An official website of the United States government
The purpose of this work was to test the inter-rater reliability (IRR) of a rubric used to grade technical reports in a senior-level chemical engineering laboratory course that has multiple instructors that grade deliverables. The rubric consisted of fifteen constructs that provided students detailed guidance on instructor expectations with respect to the report sections, formatting and technical writing aspects such as audience, context and purpose. Four student reports from previous years were scored using the rubric, and IRR was assessed using a two-way mixed, consistency, average-measures intraclass correlation (ICC) for each construct. Then, the instructors met as a group to discuss their scoring and reasoning. Multiple revisions were made to the rubric based on instructor feedback and constructs rated by ICC as poor. When fair or poor constructs were combined, the ICCs improved. In addition, the overall score construct continued to be rated as excellent, indicating that while different instructors may have variation at the individual construct level, they evaluate the overall quality of the report consistently. A key learning from this process was the importance of the instructor discussion around their reasoning for the scores and the importance of an ‘instructor orientation’ involving discussion and practice using the rubrics in the case of multiple instructors or a change in instructors. The developed rubric has the potential for broad applicability to engineering laboratory courses with technical writing components and could be adapted for alternative styles of technical writing genre.
more »
« less
Validation of a Senior-Level Chemical Engineering Laboratory Course Technical Report Rubric that Aligns with Industry Expectations
A challenge instructors face is developing and accurately assessing technical communication skills to ensure students can apply and transfer the skills from the academic context into the context of engineering practice. By intentionally balancing teaching transferrable communication skills relevant to engineering practice and evaluating student understanding, engineering educators can foster competence and prepare students for the expectations of their professional careers. This study addresses two questions: (1) how can chemical engineering instructors reliably and consistently assess student communication skills, and (2) are instructor expectations aligned with those of practicing engineers? The use of well-designed rubrics is important for setting clear expectations for students, providing constructive feedback, and in team taught courses, grading consistently. This study discusses how a rubric for assessing technical communication skills in senior-level chemical engineering laboratory reports was validated and demonstrated reliability across five chemical engineering instructors. Additionally, five industry partners evaluated student reports for comparison to instructor rubric scores. Expectations and perceptions of the quality of student work align between instructors and practicing engineers, but practicing engineers prioritized safety and abstract clarity, while instructors prioritized the students’ abilities to interpret results and draw conclusions.
more »
« less
- Award ID(s):
- 2120775
- PAR ID:
- 10554942
- Publisher / Repository:
- International Journal of Engineering Education
- Date Published:
- Journal Name:
- International journal of engineering education
- Volume:
- 40
- Issue:
- 1
- ISSN:
- 0949-149X
- Page Range / eLocation ID:
- 108-115
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Undergraduate writing skills in STEM fields, especially engineering, need improvement. Yet students in engineering fields often do not value them and underestimate the amount of writing they will do in their careers. University writing centers can be a helpful resource, but the peer writing tutors that often staff them need to be prepared for the differences in writing between humanities and STEM fields. The Writing Assignment Tutor Training in STEM (WATTS) model was designed to improve tutor confidence and student writing. In this innovative training, the writing center supervisor and STEM instructor collaboratively create a one-hour training for tutors about the assignment content, technical terminology, genre conventions, and instructor expectations. A research study on this multidisciplinary collaborative project is being conducted to determine the impact of WATTS on students, tutors, and faculty and to identify its mitigating and moderating effects, assessing the elements of the model that have the most impact. Data from all WATTS stakeholders—students, tutors, faculty and writing center staff—have been collected. Both quantitative and qualitative instruments were used, including pre- and post-surveys, interviews and focus groups. WATTS’ effects on student writing have been assessed by the comparison of pre- and post-tutoring reports using a normed rubric and have demonstrated statistically significantly improvement in student writing. The results are being used to develop a replicable, sustainable model for dissemination to other institutions and application within other STEM fields. Increasing collaboration between engineering instructors and writing centers is a desirable outcome and essential for WATTS dissemination to a broad audience. NSF funding of this project has enabled the investigators to expand WATTS to additional engineering courses, test key factors with more instructors, and refine the process. It is anticipated that the study will contribute valuable knowledge to facilitate the improvement of student writing in STEM fields. As the cost of higher education increases, institutions are pressured to graduate students in four years while engineering curricula are becoming more complex. WATTS presents an economical, effective method to improve student writing in the discipline. Several factors indicate that it has the potential for broad dissemination and impact and will provide a foundation for a sustainable model for future work as instructors become trainers for their colleagues, allowing additional ongoing expansion and implementation. WATTS serves as a model for institutions (large or small) to capitalize on existing infrastructure and resources to achieve large-scale improvements to undergraduate STEM writing while increasing interdisciplinary collaboration and institutional support.more » « less
-
Writing Assignment Tutor Training in STEM (WATTS) is part of a three-year NSF IUSE grant with participants at three institutions. This research project seeks to determine to what extent students in the WATTS project show greater writing improvement than students using writing tutors not trained in WATTS. The team collected baseline, control, and experimental data. Baseline data included reports written by engineering and engineering technology students with no intervention to determine if there were variations in written communication related to student demographics and institutions. Control data included reports written by students who visited tutors with no WATTS training, and experimental data included reports written by students who visited tutors who were WATTS-trained. Reports were evaluated by the research team using a slightly modified version of the American Association of Colleges and Universities (AAC&U) Written Communication VALUE Rubric. Baseline data assessment also provided an opportunity to test the effectiveness of the rubric. This paper presents findings from the analysis of the control and experimental data to determine the impact of WATTS on student writing in lab reports. An aggregate score for each lab report was determined by averaging the reviewer scores. An analysis was run to determine if there was a statistical difference between pre-tutoring lab report scores from the baseline, control, and experimental rubric scores for each criterion and total scores; there was not a statistically significant difference. The research team ran a Wilcoxon signed-rank test to assess the relationship between control and experimental aggregate rubric scores for each criterion. The preliminary analysis of the control and experimental data shows that the WATTS intervention has a positive, statistically significant impact on written communication skills regardless of the campus student demographics. Since WATTS has been shown to be a low-cost, effective intervention to improve engineering and engineering technology students’ written communication skills at these participating campuses, it has potential use for other institutions to positively impact their students’ written communication.more » « less
-
Writing Assignment Tutor Training in STEM (WATTS) is part of a three-year NSF IUSE grant with participants at three institutions. This research project seeks to determine to what extent students in the WATTS project show greater writing improvement than students using writing tutors not trained in WATTS. The team collected baseline, control, and experimental data. Baseline data included reports written by engineering and engineering technology students with no intervention to determine if there were variations in written communication related to student demographics and institutions. Control data included reports written by students who visited tutors with no WATTS training, and experimental data included reports written by students who visited tutors who were WATTS-trained. Reports were evaluated by the research team using a slightly modified version of the American Association of Colleges and Universities (AAC&U) Written Communication VALUE Rubric. Baseline data assessment also provided an opportunity to test the effectiveness of the rubric. This paper presents findings from the analysis of the control and experimental data to determine the impact of WATTS on student writing in lab reports. An aggregate score for each lab report was determined by averaging the reviewer scores. An analysis was run to determine if there was a statistical difference between pre-tutoring lab report scores from the baseline, control, and experimental rubric scores for each criterion and total scores; there was not a statistically significant difference. The research team ran a Wilcoxon signed-rank test to assess the relationship between control and experimental aggregate rubric scores for each criterion. The preliminary analysis of the control and experimental data shows that the WATTS intervention has a positive, statistically significant impact on written communication skills regardless of the campus student demographics. Since WATTS has been shown to be a low-cost, effective intervention to improve engineering and engineering technology students’ written communication skills at these participating campuses, it has potential use for other institutions to positively impact their students’ written communication.more » « less
-
Engineering undergraduates are exposed to a variety of writing curricula, such as first-year-composition courses, in their early program of study; however, they have difficulties meeting the expectations of writing in early engineering courses. On the other hand, instructors in entry-level engineering lab courses struggle to instruct lab report writing due to a wide range of student writing backgrounds and pressure to focus on technical content. When using the lens of learning transfer theories, which describe the processes and the effective extent to which past experiences affect learning and performance in a new situation, we can classify engineering students in three writing transfer modes: 1) concurrent transfer, which occurs when a rhetorically-focused technical writing class is taken concurrently or prior to engineering labs in the major; 2) vertical transfer, which occurs when a rhetorically-focused general education writing class is taken prior to engineering labs in the major; and 3) absent transfer, which occurs when no rhetorically-focused writing class exists (rather literature-focused) or writing-intensive courses are not required in the general education curriculum. This study aims to investigate how the engineering sophomore’s past writing experience, specifically in collegiate writing or writing-across-the-curriculum courses, affects their engineering lab report writing. Lab reports from four sophomore engineering courses (1 civil, 2 electrical, 1 general engineering) across three institutions collected for analysis consisted of two sets: the sample sets in early labs (for example, Lab 1) and in later labs (for example, the last lab) of the courses. A total of 46 reports (22 early and 24 later) were collected from 22 engineering sophomores during AY2019-2020. Four engineering faculty (1 civil, 1 electrical, and 2 mechanical engineering) developed a rubric based on lab report writing student outcomes, which are aligned with the existing outcomes such as ABET outcomes and the student outcomes from the Council of Writing Program Administrators (WPA). The data suggest that the greatest writing gains in a first lab course are made by vertical transfer students, while concurrent transfer students enter with skills developed in prior writing coursework. The largest improvements among the three transfer modes were found in the student outcomes related to lab data presentation, analysis, and interpretation. In these outcomes, the concurrent transfer students had relatively high scores for both early and later reports, while the vertical transfer students improved their scores from relatively low in early reports to meet expectations in later reports. Absent transfer students demonstrated inconsistent outcomes and deserve greater study with more data than was available for this study.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
