More Like this

1. Defining Assessment: Foundation Knowledge Toward Exploring Engineering Faculty’s Assessment Mental Models

   Chew, Kai Jun ; Ross, Amanda ; Katz, Andrew ( October 2022 , Frontiers in Education)

   This full research paper documents assessment definitions from engineering faculty members, mainly from Research 1 universities. Assessments are essential components of the engineering learning environment, and how engineering faculty make decisions about assessments in their classroom is a relatively understudied topic in engineering education research. Exploring how engineering faculty think and implement assessments through the mental model framework can help address this research gap. The research documented in this paper focuses on analyzing data from an informational questionnaire that is part of a larger study to understand how the participants define assessments through methods inspired by mixed method strategies. These strategies include descriptive statistics on demographic findings and Natural Language Processing (NLP) and coding on the open-ended response question asking the participants to define assessments, which yielded cluster themes that characterize the definitions. Findings show that while many participants defined assessments in relation to measuring student learning, other substantial aspects include benchmarking, assessing student ability and competence, and formal evaluation for quality. These findings serve as foundational knowledge toward deeper exploration and understanding of assessment mental models of engineering faculty that can begin to address the aforementioned research gap on faculty assessment decisions in classrooms. 

   more » « less
2. WIP: Faculty Use of Metaphors When Discussing Assessment

   Ross, Amanda ; Chew, Kai Jun ; Katz, Andrew ; Matusovich, Holly M ( June 2023 , Proceedings of the American Society for Engineering Education)

   This Work-in-Progress paper studies the mental models of engineering faculty regarding assessment, focusing on their use of metaphors. Assessments are crucial components in courses as they serve various purposes in the learning and teaching process, such as gauging student learning, evaluating instructors and course design, and documenting learning for accountability. Thus, when it comes to faculty development on teaching, assessments should consistently be considered while discussing pedagogical improvements. To contribute to faculty development research, our study illuminates several metaphors engineering faculty use to discuss assessment concepts and knowledge. This paper helps to answer the research question: which metaphors do faculty use when talking about assessment in their classrooms? Through interviews grounded in mental model theory, six metaphors emerged: (1) cooking, (2) playing golf, (3) driving a car, (4) coaching football, (5) blood tests, (6) and generically playing a sport or an instrument. Two important takeaways stemmed from the analysis. First, these metaphors were experiences commonly portrayed in the culture in which the study took place. This is important to note for someone working in faculty development as these metaphors may create communication challenges. Second, the mental model approach showed potential in eliciting ways engineering faculty describe and discuss assessments, offering opportunities for future research and practice in faculty development. The lightning talk will present further details on the findings. 

   more » « less
3. Towards an Adaptive Learning Module for Materials Science: Comparing Expert Predictions to Student Performance

   Nutnicha Nigon ; Dana C Simionescu ; Thomas W Ekstedt ; Julie D Tucker ; Milo D Koretsky ( June 2022 , ASEE Annual Conference proceedings)

   The emphasis on conceptual learning and the development of adaptive instructional design are both emerging areas in science and engineering education. Instructors are writing their own conceptual questions to promote active learning during class and utilizing pools of these questions in assessments. For adaptive assessment strategies, these questions need to be rated based on difficulty level (DL). Historically DL has been determined from the performance of a suitable number of students. The research study reported here investigates whether instructors can save time by predicting DL of newly made conceptual questions without the need for student data. In this paper, we report on the development of one component in an adaptive learning module for materials science – specifically on the topic of crystallography. The summative assessment element consists of five DL scales and 15 conceptual questions This adaptive assessment directs students based on their previous performances and the DL of the questions. Our five expert participants are faculty members who have taught the introductory Materials Science course multiple times. They provided predictions for how many students would answer each question correctly during a two-step process. First, predictions were made individually without an answer key. Second, experts had the opportunity to revise their predictions after being provided an answer key in a group discussion. We compared expert predictions with actual student performance using results from over 400 students spanning multiple courses and terms. We found no clear correlation between expert predictions of the DL and the measured DL from students. Some evidence shows that discussion during the second step made expert predictions closer to student performance. We suggest that, in determining the DL for conceptual questions, using predictions of the DL by experts who have taught the course is not a valid route. The findings in this paper can be applied to assessments in both in-person, hybrid, and online settings and is applicable to subject matter beyond materials science. 

   more » « less
4. Insights from the First Year of Project # 2044472 “Improving the Conceptual Mastery of Engineering Students in High Enrollment Engineering Courses through Oral Exams

   Qi, H. ; Lubarda M. ; Schurgers C. ; Sandoval C. ; Klement, L. ; Ghazinejad M. ; Relaford-Doyle, J. ; Kim, M. ; Minnes, M. ; Baghdadchi S. ; et al ( June 2022 , ASEE Annual Conference proceedings)

   This project aims to enhance students’ learning in foundational engineering courses through oral exams based on the research conducted at the University of California San Diego. The adaptive dialogic nature of oral exams provides instructors an opportunity to better understand students’ thought processes, thus holding promise for improving both assessments of conceptual mastery and students’ learning attitudes and strategies. However, the issues of oral exam reliability, validity, and scalability have not been fully addressed. As with any assessment format, careful design is needed to maximize the benefits of oral exams to student learning and minimize the potential concerns. Compared to traditional written exams, oral exams have a unique design space, which involves a large range of parameters, including the type of oral assessment questions, grading criteria, how oral exams are administered, how questions are communicated and presented to the students, how feedback were provided, and other logistical perspectives such as weight of oral exam in overall course grade, frequency of oral assessment, etc. In order to address the scalability for high enrollment classes, key elements of the project are the involvement of the entire instructional team (instructors and teaching assistants). Thus the project will create a new training program to prepare faculty and teaching assistants to administer oral exams that include considerations of issues such as bias and students with disabilities. The purpose of this study is to create a framework to integrate oral exams in core undergraduate engineering courses, complementing existing assessment strategies by (1) creating a guideline to optimize the oral exam design parameters for the best students learning outcomes; and (2) Create a new training program to prepare faculty and teaching assistants to administer oral exams. The project will implement an iterative design strategy using an evidence-based approach of evaluation. The effectiveness of the oral exams will be evaluated by tracking student improvements on conceptual questions across consecutive oral exams in a single course, as well as across other courses. Since its start in January 2021, the project is well underway. In this poster, we will present a summary of the results from year 1: (1) exploration of the oral exam design parameters, and its impact in students’ engagement and perception of oral exams towards learning; (2) the effectiveness of the newly developed instructor and teaching assistants training programs (3) The development of the evaluation instruments to gauge the project success; (4) instructors and teaching assistants experience and perceptions. 

   more » « less
5. How Do Engineering Faculty, Staff, and Administrators Define Engineering Thriving? An Investigation of Key Themes

   Gesun, Julianna ; Rizzo, Julia ( October 2022 , IEEE Frontiers in Education Conference Proceedings)

   In this work-in-progress research paper, we explore key themes and patterns prevalent in definitions of thriving for undergraduate engineering students. Although there is growing research and acknowledgment of the breadth and complexity of thriving in engineering, the field has limited conceptual clarity regarding its range of definitions. Data for this research was collected from 47 engineering faculty and staff who are considered long-term members of the engineering education system who also play a vital role in creating environments conducive to thriving and forming relationships with students that facilitate thriving. Participants were asked to define engineering student thriving in an open-ended survey, where their responses were analyzed using thematic analysis. 29 codes emerged from the data, 18 of which align with prior research. 10 new codes emerged, relating to positive emotions (such as happy, excited, and passionate), learning (love of learning, growing, understanding and deep learning), wellbeing, identity, belongingness, and professional experiences. These findings highlight positive aspects of engineering thriving beyond the absence or reduction of suffering and hardship. Implications of these findings include developing measures with multi-dimensional focus and emphasizing the role of emotional support and identity development in engineering students. 

   more » « less