Writing Assignment Tutor Training in STEM (WATTS) is part of a three-year NSF IUSE grant with participants at three institutions. This research project seeks to determine to what extent students in the WATTS project show greater writing improvement than students using writing tutors not trained in WATTS. The team collected baseline, control, and experimental data. Baseline data included reports written by engineering and engineering technology students with no intervention to determine if there were variations in written communication related to student demographics and institutions. Control data included reports written by students who visited tutors with no WATTS training, and experimental data included reports written by students who visited tutors who were WATTS-trained. Reports were evaluated by the research team using a slightly modified version of the American Association of Colleges and Universities (AAC&U) Written Communication VALUE Rubric. Baseline data assessment also provided an opportunity to test the effectiveness of the rubric. This paper presents findings from the analysis of the control and experimental data to determine the impact of WATTS on student writing in lab reports. An aggregate score for each lab report was determined by averaging the reviewer scores. An analysis was run to determine if there was a statistical difference between pre-tutoring lab report scores from the baseline, control, and experimental rubric scores for each criterion and total scores; there was not a statistically significant difference. The research team ran a Wilcoxon signed-rank test to assess the relationship between control and experimental aggregate rubric scores for each criterion. The preliminary analysis of the control and experimental data shows that the WATTS intervention has a positive, statistically significant impact on written communication skills regardless of the campus student demographics. Since WATTS has been shown to be a low-cost, effective intervention to improve engineering and engineering technology students’ written communication skills at these participating campuses, it has potential use for other institutions to positively impact their students’ written communication. This material is based upon work supported by the National Science Foundation under Grant Nos. 2013467, 2013496, and 2013541.
more »
« less
This content will become publicly available on July 15, 2025
VerAs: Verify then assess STEM lab reports
With an increasing focus in STEM education on critical thinking skills, science writing plays an ever more important role. A recently published dataset of two sets of college level lab reports from an inquiry-based physics curriculum relies on analytic assessment rubrics that utilize multiple dimensions, specifying subject matter knowledge and general components of good explanations. Each analytic dimension is assessed on a 6-point scale, to provide detailed feedback to students that can help them improve their science writing skills. Manual assessment can be
slow, and difficult to calibrate for consistency across all students in large enrollment courses with many sections. While much work exists on automated assessment of open-ended questions in STEM subjects, there has been far less work on long-form writing such as lab reports. We present an end-to-end neural architecture that has separate verifier and assessment modules, inspired by approaches to Open Domain Question Answering (OpenQA). VerAs first verifies whether a report contains any content relevant to a given rubric dimension, and if so, assesses the relevant sentences. On the lab reports, VerAs outperforms multiple baselines based on OpenQA systems or Automated Essay Scoring (AES). VerAs also performs well on an analytic rubric for middle school physics essays.
more »
« less
- Award ID(s):
- 2010483
- PAR ID:
- 10515205
- Publisher / Repository:
- International Conference on Artificial Intelligence in Education 2024
- Date Published:
- Journal Name:
- International Conference on Artificial Intelligence in Education 2024
- Subject(s) / Keyword(s):
- Automated Assessment · Lab Reports · Analytic Rubrics
- Format(s):
- Medium: X
- Location:
- Recife, Brazil
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Writing Assignment Tutor Training in STEM (WATTS) is part of a three-year NSF IUSE grant with participants at three institutions. This research project seeks to determine to what extent students in the WATTS project show greater writing improvement than students using writing tutors not trained in WATTS. The team collected baseline, control, and experimental data. Baseline data included reports written by engineering and engineering technology students with no intervention to determine if there were variations in written communication related to student demographics and institutions. Control data included reports written by students who visited tutors with no WATTS training, and experimental data included reports written by students who visited tutors who were WATTS-trained. Reports were evaluated by the research team using a slightly modified version of the American Association of Colleges and Universities (AAC&U) Written Communication VALUE Rubric. Baseline data assessment also provided an opportunity to test the effectiveness of the rubric. This paper presents findings from the analysis of the control and experimental data to determine the impact of WATTS on student writing in lab reports. An aggregate score for each lab report was determined by averaging the reviewer scores. An analysis was run to determine if there was a statistical difference between pre-tutoring lab report scores from the baseline, control, and experimental rubric scores for each criterion and total scores; there was not a statistically significant difference. The research team ran a Wilcoxon signed-rank test to assess the relationship between control and experimental aggregate rubric scores for each criterion. The preliminary analysis of the control and experimental data shows that the WATTS intervention has a positive, statistically significant impact on written communication skills regardless of the campus student demographics. Since WATTS has been shown to be a low-cost, effective intervention to improve engineering and engineering technology students’ written communication skills at these participating campuses, it has potential use for other institutions to positively impact their students’ written communication.more » « less
-
Writing Assignment Tutor Training in STEM (WATTS) is part of a three-year NSF IUSE grant with participants at three institutions. This research project seeks to determine to what extent students in the WATTS project show greater writing improvement than students using writing tutors not trained in WATTS. The team collected baseline, control, and experimental data. Baseline data included reports written by engineering and engineering technology students with no intervention to determine if there were variations in written communication related to student demographics and institutions. Control data included reports written by students who visited tutors with no WATTS training, and experimental data included reports written by students who visited tutors who were WATTS-trained. Reports were evaluated by the research team using a slightly modified version of the American Association of Colleges and Universities (AAC&U) Written Communication VALUE Rubric. Baseline data assessment also provided an opportunity to test the effectiveness of the rubric. This paper presents findings from the analysis of the control and experimental data to determine the impact of WATTS on student writing in lab reports. An aggregate score for each lab report was determined by averaging the reviewer scores. An analysis was run to determine if there was a statistical difference between pre-tutoring lab report scores from the baseline, control, and experimental rubric scores for each criterion and total scores; there was not a statistically significant difference. The research team ran a Wilcoxon signed-rank test to assess the relationship between control and experimental aggregate rubric scores for each criterion. The preliminary analysis of the control and experimental data shows that the WATTS intervention has a positive, statistically significant impact on written communication skills regardless of the campus student demographics. Since WATTS has been shown to be a low-cost, effective intervention to improve engineering and engineering technology students’ written communication skills at these participating campuses, it has potential use for other institutions to positively impact their students’ written communication.more » « less
-
James, C (Ed.)Effective writing is important for communicating science ideas, and for writing-to-learn in science. This paper investigates lab reports from a large-enrollment college physics course that integrates scientific reasoning and science writing. While analytic rubrics have been shown to define expectations more clearly for students, and to improve reliability of assessment, there has been little investigation of how well analytic rubrics serve students and instructors in large-enrollment science classes. Unsurprisingly, we found that grades administered by teaching assistants (TAs) do not correlate with reliable post-hoc assessments from trained raters. More important, we identified lost learning opportunities for students, and misinformation for instructors about students’ progress. We believe our methodology to achieve post-hoc reliability is straightforward enough to be used in classrooms. A key element is the development of finer-grained rubrics for grading that are aligned with the rubrics provided to students to define expectations, but which reduce subjectivity of judgements and grading time. We conclude that the use of dual rubrics, one to elicit independent reasoning from students and one to clarify grading criteria, could improve reliability and accountability of lab report assessment, which could in turn elevate the role of lab reports in the instruction of scientific inquiry.more » « less
-
Despite the well-established importance of written communication skills for students in STEM disciplines, the quantitative assessment of STEM writing remains an evolving field. The present work seeks to measure the effectiveness of “generic” writing center tutors on the technical writing skills of senior-level Mechanical Engineering Technology students. A set of nineteen student analysis reports selected from a capstone design course were used as the source of the data. The reports were assessed both before and after a tutoring session using a version of the AAC&U VALUE rubric and a voice-development-style-diction method developed by the authors. By both methods, the improvements in student writing from before the tutoring session to afterwards were marginal at best, with some measures even showing a decrease in performance. The sole exception was that a significant increase in hedging, boosting, and attitude words appeared in the students’ work, indicative of a change in diction. It is concluded that an intervention by a “generically” trained writing center tutor has little effect on the quality of student writing outside of that due to the inclusion of additional adjectives. An intervention by tutors specifically trained using the WATTS methodology is proposed as a means to address this. Such an intervention will be investigated as an extension to the current work.more » « less
