skip to main content


Title: An Automated Writing Evaluation System for Supporting Self-monitored Revising
This paper presents the design and evaluation of an automated writing evaluation system that integrates natural language processing (NLP) and user interface design to support students in an important writing skill, namely, self-monitored revising. Results from a classroom deployment suggest that NLP can accurately analyze where and what kind of revisions students make across paper drafts, that students engage in self-monitored revising, and that the interfaces for visualizing the NLP results are perceived by students to be useful.  more » « less
Award ID(s):
1735752
NSF-PAR ID:
10379514
Author(s) / Creator(s):
; ; ; ; ;
Editor(s):
Rodrigo, M.M.; Matsuda, N.; Cristea, A.I.; Dimitrova, V.
Date Published:
Journal Name:
International Conference on Artificial Intelligence in Education
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Rodrigo, M.M. ; Matsuda, N. ; Cristea, A.I. ; Dimitrova, V. (Ed.)
    This paper presents the design and evaluation of an automated writing evaluation system that integrates natural language processing (NLP) and user interface design to support students in an important writing skill, namely, self-monitored revising. Results from a classroom deployment suggest that NLP can accurately analyze where and what kind of revisions students make across paper drafts, that students engage in self-monitored revising, and that the interfaces for visualizing the NLP results are perceived by students to be useful. 
    more » « less
  2. The use of writing-based exercises in a circuit analysis course has shown promise in aiding students likely to struggle in the course by enhancing their conceptual understanding of topics related to DC circuit analysis [1]. As grading of writing samples and providing personalized feedback can be time-intensive, automating the evaluation and feedback processes through use of emerging techniques in natural language processing (NLP) could open the door for more widespread use of such writing exercises across STEM courses, thus benefiting students in most need of assistance. In this paper, the development and initial testing of two web-based writing activities that leverage a basic NLP technique to probe student writing related to DC circuits are described. The first writing exercise has students describe what happens to the power of various elements in a resistive circuit as the value of one of the resistors decreases. The second exercise has students consider situations in which the ideal independent voltage and current source models might fail. Both writing exercises are built from a template that includes several metacognitive prompts to spur self-reflection on the part of the user. A rule-based approach was taken to detect evidence of common misconceptions [2] and errors in student responses, as well as to identify sentences that revealed the student was correctly addressing the problems. Based on identified misconceptions or correct concepts in a student’s writing, the web-based application selects appropriate directed line of reasoning (DLR) feedback paths to attempt to lead the writer to an accurate understanding of the behavior of the circuits in question. Key features of the web-based application template as well as details regarding misconception detection and personalized feedback are described. Student impressions of the value of the DLR feedback is assessed using comments provided by the student within the applications. Planned modifications of the web-based writing exercise template based on this formative assessment will be given and address a broader goal of this work – to develop a web-based template that instructors across STEM disciplines, even those without a background in coding, could use to implement their own conceptual writing exercises. 
    more » « less
  3. Writing scientific explanations is a core practice in science. However, students find it difficult to write coherent scientific explanations. Additionally, teachers find it challenging to provide real-time feedback on students’ essays. In this study, we discuss how PyrEval, an NLP technology, was used to automatically assess students’ essays and provide feedback. We found that students explained more key ideas in their essays after the automated assessment and feedback. However, there were issues with the automated assessments as well as students’ understanding of the feedback and revising their essays. 
    more » « less
  4. ABET lists the ability to communicate in writing to both technical and non-technical audiences as a required outcome for baccalaureate engineering students [1]. From emails and memos to formal reports, the ability to communicate is vital to the engineering profession. This Work in Progress paper describes research being done as part of an NSF-funded project, Writing Assignment Tutor Training in STEM (WATTS). The method is designed to improve feedback writing tutors without technical backgrounds give to engineering students on technical reports. Students in engineering programs have few opportunities to develop their writing skills. Usually, composition courses are part of the general education curriculum. Students often see these courses as unrelated to their majors and careers [2]. Ideally, writing support should be integrated throughout a program. Since WATTs capitalizes on existing resources and requires only a modest amount of faculty time, it could enable engineering programs to provide additional writing support to students in multiple courses and provide a bridge for them to see the connection between writing concepts learned in composition courses and their technical reports. WATTS was developed in a junior-level circuit analysis course, where students were completing the same lab and writing individual reports. This paper focuses on a senior capstone course that utilizes concepts taught in previous courses to prepare students to complete an independent team research or design project. Projects are unique, usually based on the needs of an industrial sponsor, and are completed over three consecutive semesters. Each semester, teams write a report based on their activities during that semester, with a comprehensive report in the final semester. The multi-semester nature of the senior design project provides an opportunity for the researchers to chart longitudinal changes from the first to the students’ third semester interactions with the writing tutors, assessing the value of an integrated approach. The program’s impact on students’ attitudes toward revision and the value of tutoring, as well as the impact on tutors, are part of the assessment plan. The program hopes to change the students’ focus from simply presenting their results to communicating them. The goals of the project are to demonstrate to students that revision is essential to the writing process and that feedback can improve their written communication abilities. The expectation is that after graduation they will continue to seek critical feedback as part of their career growth. Surveys given to both students and tutors revealed that the sessions were taken seriously by the students and that meaningful collaboration was achieved between them. An evaluation of the writing in pre-tutored to final submitted report shows statistically significant improvement. Preliminary and current results will be included within the paper. [1] Criteria for Accrediting Engineering Technology Programs, ABET, Baltimore, MD., 2020, p.5, ETAC Criteria (abet.org) [2] Bergmann, L. S. and Zepernick, J., “Disciplinarity and Transfer: Students’ Perceptions of Learning to Write,” Writing Program Administration, 31, Fall/Winter 2007. 
    more » « less
  5. No skill is more important for a student of mechanics than the ability to draw a complete and accurate free-body diagram (FBD). A good FBD facilitates proper accounting of forces when writing the balances that lead to governing equations in statics, solid mechanics, and dynamics. Because this skill is essential, educational approaches that improve the ability of students to draw correct FBDs are critical for maximizing the potential of the next generation of engineers. Traditionally, learning to draw FBDs involves classroom instruction followed by homework practice consisting of problems drawn from a textbook. Homework as practice does not serve all students well, because it does not scaffold the process of drawing FBDs in terms of distinct tasks (e.g., isolating the body, considering support reactions) nor does it offer immediate feedback, which students often need to avoid falling into the same error repeatedly. To address these shortcomings, we embarked on the design, implementation, and testing of a mobile application (app) that offers an alternative venue for FBD practice. The app provides students with asynchronous opportunities for training, varied tasks that target specific FBD issues, and several levels of immediate feedback. We hypothesize that the gamified environment and puzzle-based gameplay will improve student skill and self-efficacy in drawing FBDs, particularly for women, who may feel less confident in their spatial skills. Data collected to describe student experiences may also provide additional insight into how to improve FBD instruction generally. In this paper, we detail the process for designing and implementing the app and provide initial data regarding student impressions and use. The app was piloted in Fall 2022 in a large Introduction to Statics course as a non-graded study activity; all students except one (n=97) participated in an evaluation of its design features and user experiences. Approximately half (54%) of students indicated they had played half or more of the available games. When commenting about how the FBD app did, or did not, help their learning, 49% of respondents appreciated that the app allowed additional opportunities for practice. Students used these opportunities to further develop several skills, such as visualizing the system and setting up accurate diagrams, which strengthened their confidence and reviewed key concepts. While describing the value of practicing through the app, 21% of students called out how the app provided feedback. They specifically mentioned the positive experiences of receiving feedback that is immediate, that explains boundary connections, and that deepens learning after mistakes are made. These and other findings from the pilot study are discussed with corresponding next steps for development. 
    more » « less