More Like this

1. What Do Peer Evaluations Represent? A Study of Rater Consensus and Target Personality

   https://doi.org/10.3389/feduc.2022.746457  

   Martin, Chris C.; Locke, Kenneth D. (March 2022, Frontiers in Education)

   When professors assign group work, they assume that peer ratings are a valid source of information, but few studies have evaluated rater consensus in such ratings. We analyzed peer ratings from project teams in a second-year university course to examine consensus. Our first goal was to examine whether members of a team generally agreed on the competence of each team member. Our second goal was to test if a target’s personality traits predicted how well they were rated. Our third goal was to evaluate whether the self-rating of each student correlated with their peer rating. Data were analyzed from 130 students distributed across 21 teams (mean team size = 6.2). The sample was diverse in gender and ethnicity. Social relations model analyses showed that on average 32% of variance in peer-ratings was due to “consensus,” meaning some targets consistently received higher skill ratings than other targets did. Another 20% of the variance was due to “assimilation,” meaning some raters consistently gave higher ratings than other raters did. Thus, peer ratings reflected consensus (target effects), but also assimilation (rater effects) and noise. Among the six HEXACO traits that we examined, only conscientiousness predicted higher peer ratings, suggesting it may be beneficial to assign one highly conscientious person to every team. Lastly, there was an average correlation of.35 between target effects and self-ratings, indicating moderate self-other agreement, which suggests that students were only weakly biased in their self-ratings. 

   more » « less
2. How Does Working on an Interdisciplinary Service-Learning Project vs. a Disciplinary Design Project Affect Peer Evaluators' Teamwork Skills?

   Kumi, I. K. (June 2023, ASEE New England annual conference)

   Over the course of several semesters, two different project-based learning approaches were used in two undergraduate engineering courses–a 100-level introductory course that covered a general education requirement on information literacy and a 300-level fluid mechanics course. One project (treatment) was an interdisciplinary service-learning project, implemented with undergraduate engineering and education students who collaborated to develop and deliver engineering lessons to fourth and fifth-grade students in a field trip model. The other projects (comparison) involved a team-based design project contained within each class. In the 100-level course, students selected their project based on personal interests and followed the engineering design process to develop, test, and redesign a prototype. In the fluid mechanics class, students designed a pumped pipeline system for a hypothetical plant. This study aimed to determine whether participating in the interdisciplinary project affected students’ evaluation of their own and their teammates’ teamwork effectiveness skills, measured using the Behaviorally Anchored Rating Scale (BARS) version of the Comprehensive Assessment of Team Member Effectiveness (CATME). The five dimensions of CATME measured in this study are (1) contribution to the team’s work, (2) interacting with teammates, (3) keeping the team on track, (4) expecting quality, and (5) having relevant knowledge, skills, and abilities (KSAs). The quantitative data from CATME were analyzed using ANCOVA. Furthermore, since data were collected over three semesters and coincided with the pre, during, and post-phases of the COVID-19 pandemic, it was possible to examine the effects of the evolving classroom constraints over the course of the pandemic on the teamwork effectiveness skills of both the treatment and comparison classes. 

   more » « less
3. Explicit Instruction of Scientific Uncertainty in an Undergraduate Geoscience Field-Based Course

   https://doi.org/10.1007/s11191-022-00345-z  

   Bateman, Kathryn M.; Wilson, Cristina G.; Williams, Randolph T.; Tikoff, Basil; Shipley, Thomas F. (May 2022, Science & Education)

   Abstract Understanding and communicating uncertainty is a key skill needed in the practice of science. However, there has been little research on the instruction of uncertainty in undergraduate science education. Our team designed a module within an online geoscience field course which focused on explicit instruction around uncertainty and provided students with an uncertainty rating scale to record and communicate their uncertainty with a common language. Students then explored a complex, real-world geological problem about which expert scientists had previously made competing claims through geologic maps. Provided with data, expert uncertainty ratings, and the previous claims, students made new geologic maps of their own and presented arguments about their claims in written form. We analyzed these reports along with assessments of uncertainty. Most students explicitly requested geologists’ uncertainty judgments in a post-course assessment when asked why scientists might differ in their conclusions and/or utilized the rating scale unprompted in their written arguments. Through the examination of both pre- and post-course assessments of uncertainty and students’ course-based assessments, we argue that explicit instruction around uncertainty can be introduced during undergraduate coursework and could facilitate geoscience novices developing into practicing geoscientists. 

   more » « less
4. Effects of an Intervention on Student Self-Efficacy and Integration in Chemical Engineering Sophomores

   Cicciarelli, Bradley A.; Sherer, Eric A.; Reeves, Timothy C.; Belk, Catherine H.; Orr, Marisa K. (January 2023, ASEE Annual Conference proceedings)

   We are assessing the impact of an intervention (a two-day voluntary workshop) on the specific factors of self-efficacy and student integration. This workshop, called the “ChemE Camp”, takes place just before the start of fall classes and includes team-building exercises, presentations from faculty about upcoming classes, a hands-on project, a lab tour, presentations from upper-level students and alumni about their experiences in the curriculum and in industry, information about academic advising and the career fair, and some recreational games. Students attending the camp learn more about chemical engineering courses and the profession and also have the opportunity to meet peers and interact with faculty and upper-level students. We hypothesized that the activities included in the camp would positively impact on students’ self-efficacy and social integration, factors which have been shown in other studies to significantly influence student experience and student success. The effects of the intervention were assessed using surveys administered to students at the start of the camp. These surveys included published subscales used in the study of self-efficacy and social and academic integration. The same surveys were administered to all second-year chemical engineering students at the beginning of the academic year (three days after the beginning of the camp) and the end of the academic year (approximately eight months later). Data collected from the previous three academic years indicate a statistically significant increase in the chemical engineering self-efficacy, coping self-efficacy, and social and academic integration ratings for students who attend the camp and these effects appear to largely be maintained throughout the sophomore year. Non-attendees enter the sophomore year with lower average ratings in these factors and show little change over the course of the year. Students’ intent to persist in the chemical engineering major was also assessed by these surveys, and while the camp attendees’ ratings showed almost no change from just prior to the camp to just after, their ratings exhibited a substantial, statistically-significant increase during the sophomore year. Non-attendees entered the year with a similar average rating to the attendees but showed a much more modest increase over the course of the year. We continue to collect data from both student survey responses and academic records, with the goal of eventually using path analysis to establish the relationships between the factors of self-efficacy and student integration and the outcomes of academic performance and persistence for second-year chemical engineering students. We currently have insufficient data to power such an analysis, but our available data suggest that the intervention is having a positive impact on these factors. 

   more » « less
5. Initial evidence that peer instruction is less effective for physics students with lower quantitative preparation in a highly heterogeneous class

   https://doi.org/10.1119/5.0070669  

   Smith, Eric N; Helwig, Jacob; Hendren, Kathryn; Huk, Jane M; Gordon, Vernita (July 2025, American Journal of Physics)

   Historically, physics education primarily consisted of lectures in which students have a largely passive role. Proponents of educational reform have rallied around active learning to increase engagement and retention in STEM fields, particularly advocating peer interactions to build a foundation of deep understanding. However, little is known about how students' prior preparation for introductory courses impacts their mastery of course material when instructors incorporate active learning. In the present study, we examine learning outcomes in two sections of an introductory mechanics course at an institution with a wide range of students' prior mathematics preparation as assessed by quantitative SAT scores. For each of three years, one section was taught using peer instruction in which much of the class time was spent in small-group discussions between students. The other section was taught by the same instructor using interactive lectures in which discussions primarily took place between volunteers from the class and the instructor. We find that students enrolled in the peer instruction sections earned lower grades in the course than did students in the interactive sections. We also find students in the peer instruction sections with lower quantitative SAT scores showed lower gains in understanding foundational concepts as assessed by the Force Concept Inventory and were less likely to earn an A in the course than comparable students in the interactive sections. While further research is needed to confirm these results, this study suggests that peer instruction might not be the optimal pedagogy for heterogeneous populations. 

   more » « less