More Like this

1. Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments

   Reid, Clodagh; Sorby, Sheryl; Raju, Gibin; Buckley, Jeff; Seery, Niall (June 2023, Review directory American Society for Engineering Education)

   This Complete Research paper investigates the holistic assessment of creativity in design solutions in engineering education. Design is a key element in contemporary engineering education, given the emphasis on its development through the ABET criteria. As such, design projects play a central role in many first-year engineering courses. Creativity is a vital component of design capability which can influence design performance; however, it is difficult to measure through traditional assessment rubrics and holistic assessment approaches may be more suitable to assess creativity of design solutions. One such holistic assessment approach is Adaptive Comparative Judgement (ACJ). In this system, student designs are presented to judges in pairs, and they are asked to select the item of work that they deem to have demonstrated the greatest level of a specific criterion or set of criteria. Each judge is asked to make multiple judgements where the work they are presented with is adaptively paired in order to create a ranked order of all items in the sample. The use of this assessment approach in technology education has demonstrated high levels of reliability among judges (~0.9) irrespective of whether the judges are students or faculty. This research aimed to investigate the use of ACJ to holistically assess the creativity of first-year engineering students design solutions. The research also sought to explore the differences, if any, that would exist between the rank order produced by first-year engineering students and the faculty who regularly teach first-year students. Forty-six first-year engineering students and 23 faculty participated in this research. A separate ACJ session was carried out with each of these groups; however, both groups were asked to assess the same items of work. Participants were instructed to assess the creativity of 101 solutions to a design task, a “Ping Pong problem,” where undergraduate engineering students had been asked to design a ping pong ball launcher to meet specific criteria. In both ACJ sessions each item of work was included in at least 11 pairwise comparisons, with the maximum number of comparisons for a single item being 29 in the faculty ACJ session and 50 in the student ACJ session. The data from the ACJ sessions were analyzed to determine the reliability of using ACJ to assess creativity of design solutions in first-year engineering education, and to explore whether the rankings produced from the first-year engineering students ACJ session differed significantly from those of the faculty. The results indicate a reasonably high level of reliability in both sessions as measured by the Scale Separation Reliability (SSR) coefficient, SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02. Further a strong correlation was observed between the ACJ ranks produced by the students and faculty both when considered in terms of the relative differences between items of work, r = .533, p < .001, and their absolute rank position, σ = .553, p < .001. These findings indicate that ACJ is a promising tool for holistically assessing design solutions in engineering education. Additionally, given the strong correlation between ranks of students and faculty, ACJ could be used to include students in their own assessment to reduce the faculty grading burden or to develop a shared construct of capability which could increase the alignment of teaching and learning. 

   more » « less
2. Work in Progress: Formation of an engineering identity in first-year students through an intervention centered on senior design projects

   Richards, Abigail M; Anderson, Ryan; Myers, Carrie B (July 2020, ASEE annual conference exposition)

   Abstract This “work in progress” paper describes a multiyear project to study the development of engineering identity in a chemical and biological engineering program at Montana State University. The project focuses on how engineering identity may be impacted by a series of interventions utilizing subject material in a senior-level capstone design course and has the senior capstone design students serve as peer-mentors to first- and second-year students. A more rapid development of an engineering identity by first- and second-year students is suspected to increase retention and persistence in this engineering program. Through a series of timed interventions scheduled to take place in the first and second year, which includes cohorts that will serve as negative controls (no intervention), we hope to ascertain the following: (1) the extent to which, relative to a control group, exposure to a peer mentor increases a students’ engineering identity development over time compared to those who do not receive peer mentoring and (2) if the quantity and/or timing of the peer interactions impact engineering identity development. While the project includes interventions for both first- and second-year students, this work in progress paper focuses on the experiences of first year freshman as a result of the interventions and their development of an engineering identity over the course of the semester. Early in the fall semester, freshman chemical engineering students enrolled in an introductory chemical engineering course and senior students in a capstone design course were administered a survey which contained a validated instrument to assess engineering identity. The first-year course has 107 students and the senior-level course has 92 students and approximately 50% of the students in both cohorts completed the survey. Mid-semester, after the first-year students were introduced to the concepts of process flow diagrams and material balances in their course, senior design student teams gave presentations about their capstone design projects in the introductory course. The presentations focused on the project goals, design process and highlighted the process flow diagrams. After the presentations, freshman and senior students attended small group dinners as part of a homework assignment wherein the senior students were directed to communicate information about their design projects as well as share their experiences in the chemical engineering program. Dinners occurred overall several days, with up to ten freshman and five seniors attending each event. Freshman students were encouraged to use this time to discover more about the major, inquire about future course work, and learn about ways to enrich their educational experience through extracurricular and co-curricular activities. Several weeks after the dinner experience, senior students returned to give additional presentations to the freshman students to focus on the environmental and societal impacts of their design projects. We report baseline engineering identity in this paper. 

   more » « less
3. Design Educators’ Conceptions of Prototyping in Engineering Design Courses

   Ali, Hadi; Lande, Micah (June 2019, ASEE Annual Conference proceedings)

   This is a research study that investigates the range of conceptions of prototyping in engineering design courses through exploring the conceptions and implementations from the instructors’ perspective. Prototyping is certainly an activity central to engineering design. The context of prototyping to support engineering education and practice has a range of implementations in an undergraduate engineering curriculum, from first-year engineering to capstone engineering design experiences. Understanding faculty conceptions’ of the reason, purpose, and place of prototyping can help illustrate how teaching and learning of the engineering design process is realistically implemented across a curriculum and how students are prepared for work practice. We seek to understand, and consequently improve, engineering design teaching and learning, through transformations of practice that are based on engineering education research. In this exploratory study, we interviewed three faculty members who teach engineering design in project-based learning courses across the curriculum of an undergraduate engineering program. This builds on related work done by the authors that previously investigated undergraduate engineering students’ conceptions of prototyping activities and process. With our instructor participants, a similar interview protocol was followed through semi-structured qualitative interviews. Data analysis has been undertaken through an emerging thematic analysis of these interview transcripts. Early findings characterize the focus on teaching the design process; the kind of feedback that the educators provide on students’ prototypes; students’ behavior while working on design projects; and educators’ perspectives on the design course. Understanding faculty conceptions with students’ conceptions of prototyping can shed light on the efficacy of using prototyping as an authentic experience in design teaching and learning. In project-based learning courses, particular issues of authenticity and assessment are under consideration, especially across the curriculum. More specifically, “proportions of problems” inform “problem solving” as one of the key characteristics in design thinking, teaching and learning. More attention to prototyping as part of the study of problem-solving processes can be useful to enhance understanding of the impact of instructional design. Challenges for teaching engineering design exist, and may be due to difficulties in framing design problems, recognizing what expertise students possess, and assessing their expertise to help them reach their goals, all at an appropriate place and ambiguity with student learning goals. Initial findings show that prototyping activities can help students become more reflective on their design. Scaffolded activities in prototyping can support self-regulated learning by students. The range of support and facilities, such as campus makerspaces, may also help students and instructors alike develop industry-ready engineering students. 

   more » « less
4. Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course

   https://doi.org/10.3389/feduc.2021.772832  

   Mentzer, Nathan; Lee, Wonki; Bartholomew, Scott Ronald (December 2021, Frontiers in Education)

   Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students ( n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity. 

   more » « less
5. Work in Progress: Cultivating Reflective Engineers: Does providing a reflective ePortfolio experience in a first-year design course lead students to be more reflective in later courses?

   R. Thomas, S. A. (July 2023, ASEE Annual Conference proceedings)

   This work in progress paper assesses whether a first-year ePortfolio experience promotes better reflection in subsequent engineering courses. While reflection is vital to promote learning, historically, reflection receives less attention in engineering education when compared to other fields [1]. Yet, cultivating more reflective engineers yields several important benefits including building self-efficacy and empowering student agency. Through continued practice, engineering students can develop a habit of reflective thinking which increases students’ ability to transfer knowledge across contexts. The adoption of ePortfolios is becoming an increasingly popular strategy to improve student learning and establish a culture of reflection. The Department of Electrical and Computer Engineering at a small liberal arts college in the northeastern United States is beginning to incorporate ePortfolios into courses. Professors of a first-year design course developed an ePortfolio assignment that gives students a space to reflect on their potential career paths and envision themselves as future engineers. We were curious about the impact this experience might have on students’ reflective thinking as they continue through the program. This work was guided by the research question: Do student ePortfolios in a first-year design course promote better reflection in subsequent technical courses? In this paper, we investigate this question by coding instances of reflection in student lab reports from a second-year design course. As a control group, lab reports from students the previous year who had not completed the ePortfolio activity were compared. We provide a quantitative summary of our analysis which concludes students that were provided with a reflective ePortfolio experience in their first-year are more reflective thinkers in their second-year. 

   more » « less