《影视制作》考试系统英文翻译文档.docx

1、影视制作考试系统英文翻译文档毕业设计(论文)英文翻译学院名称： 计算机科学与技术学院 班 级： 051班 学 号： 20054440102 学生姓名： 周 怡 2009年05月12日An Online Testing and Analysis Systemfor Creative Problem-Solving Ability in SciencesaChun-Chieh Huang, bHao-Chuan Wang, aTsai-Yen Li and cChun-Yen ChangaNational Chengchi University, Taipei, TaiwanbCarnegie M

2、ellon University, Pittsburgh, PA, USAcNational Taiwan Normal University, Taipei, TaiwanEmail: g9415cs.nccu.edu.tw, haochuancs.cmu.edu,linccu.edu.tw, changcyntnu.edu.twAbstract: This paper proposes an online testing and analysis system for studying students Creative Problem-Solving (CPS) ability in s

3、ciences. Using an open-ended essay-question-type test, students are asked to express their idea and imagine how to solve problems better. Based on previous works, we utilize an automated scorer for evaluating students CPS ability. This system serves as a real-time(self-)assessment for online learner

4、s and a useful research tool to gather data for CPS studies. We have re-examined the inter-rater reliability of our automated scorer with new samples, and conducted a questionnaire survey for usability inspection. The preliminary results show the system is reliable for automated scoring and satisfac

5、tory for system usability. A roadmap for future development is also proposed.Keywords: Online Testing, Creative Problem Solving, Automated Scoring, Web-based Learning1 IntroductionThe assessment of Creative Problem-Solving (CPS) ability is taken as a useful index to investigate students creativity a

6、nd problem solving abilities in psychological and educational research (Chang & Weng, 2002). We generally believe that using assessment of an open-ended essay-question-type format is a useful approach to evaluate students CPS ability. Students are required to express their self-generated thoughts in

7、 natural languages (e.g., Chinese) in such a test, and teachers or researchers may analyze students essays, one of the most comprehensive and authentic representations of studentsthinking and reasoning, to study their comprehension, competence or creativity regarding the domain subjects. In the cont

8、ext of web-based education (WBE), it is considered essential and beneficial to incorporate this type of assessments as a valuable component of online learning environments for better instructional decision-making and more advanced educational studies.Nevertheless, although online courses have been d

9、eveloped and deployed extensively more than ever, online testing, especially open-ended essay-question-type tests, is still underemphasized. One possible factor is that technologies required for online testing were not systematically developed and incorporated by practitioners of WBE. Though the dev

10、elopment of certain types of testing, such as dichotomous-, single- and multiple-choices questions, may be intuitive and mature, the situation is very different from the case of open-ended essay-question-type testing, in which technologies for language processing and learner modeling can be more com

11、plicated and were less familiar to the Appear in Proceedings of the 10th Global Chinese Conference of Computers in Education, June 2006. WBE community. In sum, we believe that it is crucial to conduct systematic investigation on the topic of online testing for open-ended problem solving tasks in ord

12、er to further the effect of online learning.Grading is a great burden to teachers, especially for open-ended testing. With limited educational resources and given a large number of students, a prompt and valid way to evaluate students would be very useful to teachers. An approach to solve this probl

13、em is by adopting an automated scorer, made possible by Artificial Intelligence technologies, to grade a large volume of open-ended testing.In this research, we aim to develop an online testing system suitable for science education to assess students CPS abilities through open-ended essay-question-t

14、ype testing. The system is designed to support three major functions: (1) students online answering to the test questions, (2) reliably automated processing and grading upon students responses, and (3) on-line inspection of statistical results with graphical visualization tools for teachers and rese

15、archers. More specifically, this work consists of three parts: (1) developing a fully integrated system that supports the aforementioned functionalities, (2) conducting a usability study of the user interface and interaction design of this system, and (3) re-testing the reliability of our automated

16、scoring module, which was preliminarily shown to have a satisfactory performance for a small sample of students (Wang et al., 2005a). In this work, we have conducted a new empirical exploration with 70 students involved in the studies of system usability and grading reliability. In addition, we will

17、 also introduce some techniques to implement such a system, such as controlling students online resource access and collecting statistics about students scores.2 Previous WorksThe system described in this paper is based on our previous works in this series, including theoretical and empirical invest

18、igation of students CPS in sciences (Chang & Weng, 2002), user modeling for CPS (Wang et al., 2005a), and automated grading technologies designed for CPS studies (Wang et al., 2005b). The work pertaining to the research in this direction is described as follows.2.1 CPS: Creative Problem-SolvingProbl

19、em-solving ability is viewed as an ability to think critically, reason analytically, and create productively, which all involves quantitative, communication and critical-response skills (American Association for the Advancement of Science, 1993). The system we have built is based on the theories of

20、several measurements of creativity proposed in the literature (Hocevar, 1981; Yu et al., 2005). In our previous work on designing the paper-based assessment for CPS (Wu & Chang, 2002), we have taken Problem-Finding and Problem-Solving as different testing phases, and each phase is further separated

21、as two parts: ideation (i.e., divergent thinking) and explanation (i.e., convergent thinking). The separation of such reasoning phases was originated from some theories of CPS (Basadur, 1995; Osborn, 1963). The paper-based answer sheet was designed with a two-column table for students to express the

22、ir ideas and explanations, as showed in Figure 1.Figure 1. A snapshot of the answer sheet showing the pair-wise relation between ideas and reasons.In each row of the table, students may fill in two consecutive text fields with an idea and the reason(s) explaining that idea for a specific problem-sol

23、ving scenario described in the testing question. More details and discussions of the design for this assessment can be found in Wu & Chang (2002)and Wang et al.(2005a, 2005b).2.2 UPSAM: User Problem-Solving Ability ModelerOur prior work has also proposed a user modeling system, User Problem-Solving

24、Ability Modeler(UPSAM), which aims at abstracting and modeling students responses in scientific problem-solving tasks (Wang, 2005a). By representing experts and students thoughts of problem-solving, including both ideas and reasons, in the formalism of bipartite graph, ideation and explanation are m

25、odeled as two disjoint sets connected with undirected edges. UPSAM enables us to compare any two bipartite-graph-based user models to address research questions such as how well are this students answers by using the experts answers as the criteria (i.e., essay grading) and what concepts are known/u

26、nknown by the student (i.e., learning diagnosis).An important feature of the UPSAM is that it maintains an expert model representing the domain experts knowledge and vocabulary against the problem-solving tasks, and it serves as a useful module to process students open-ended answers and perform auto

27、mated grading.2.3 Automated ScorerBased on UPSAM, we have developed an automated scorer for grading students responses toward the CPS test. The core concept is to use Information Retrieval (IR) technologies to match students responses against experts standard answers and then represent each student

28、as a subset of the bipartite-graph-based expert model (Wang et al., 2005b). We have collected many experts ideations and explanations from different aspects, and organize them into an expert model, which is aimed to be objective and equitable. Because of the structured answers format from UPSAM, the

29、 automated scorer can grasp students key concepts in order to evaluate their CPS ability based on each text entry. In addition, we also invite human graders to grade manually for comparisons.In this prior work, we have conducted an evaluation to corroborate the viability of the automated scoring sch

30、eme empirically. We found the Pearsons correlation statistics r, which serves as an index of inter-rater reliability between human graders and our automated scorer, range from .67 to .82, which are positive, high, and statistically significant. The preliminary experiments showed that automated scori

31、ng is reliable to be used here, and it is also suggested that an integrated online system should be developed for testing, grading and further analyses (Wang, 2005a).3 Online Testing System: WOTASThe main objective of this paper in this line of research is to build an online system integrating our p

32、reviously developed components with the newly designed modules (e.g., UI for testing and statistics reporting.) We have designed and implemented the Web-based Online Testing and Analysis System (WOTAS) such that a great number of students can access the system online concurrently with great stability and usability and teachers or researchers can easily diagnose the learning condition from those learners. The architecture of WOTAS is described below.3.1 Design ConsiderationsFigure 2. System architecture for WOTAS.The system that we have design, as depicted in Figu