An official website of the United States government
This Work-in-Progress paper studies the mental models of engineering faculty regarding assessment, focusing on their use of metaphors. Assessments are crucial components in courses as they serve various purposes in the learning and teaching process, such as gauging student learning, evaluating instructors and course design, and documenting learning for accountability. Thus, when it comes to faculty development on teaching, assessments should consistently be considered while discussing pedagogical improvements. To contribute to faculty development research, our study illuminates several metaphors engineering faculty use to discuss assessment concepts and knowledge. This paper helps to answer the research question: which metaphors do faculty use when talking about assessment in their classrooms? Through interviews grounded in mental model theory, six metaphors emerged: (1) cooking, (2) playing golf, (3) driving a car, (4) coaching football, (5) blood tests, (6) and generically playing a sport or an instrument. Two important takeaways stemmed from the analysis. First, these metaphors were experiences commonly portrayed in the culture in which the study took place. This is important to note for someone working in faculty development as these metaphors may create communication challenges. Second, the mental model approach showed potential in eliciting ways engineering faculty describe and discuss assessments, offering opportunities for future research and practice in faculty development. The lightning talk will present further details on the findings.
more »
« less
Defining Assessment: Foundation Knowledge Toward Exploring Engineering Faculty’s Assessment Mental Models
This full research paper documents assessment definitions from engineering faculty members, mainly from Research 1 universities. Assessments are essential components of the engineering learning environment, and how engineering faculty make decisions about assessments in their classroom is a relatively understudied topic in engineering education research. Exploring how engineering faculty think and implement assessments through the mental model framework can help address this research gap. The research documented in this paper focuses on analyzing data from an informational questionnaire that is part of a larger study to understand how the participants define assessments through methods inspired by mixed method strategies. These strategies include descriptive statistics on demographic findings and Natural Language Processing (NLP) and coding on the open-ended response question asking the participants to define assessments, which yielded cluster themes that characterize the definitions. Findings show that while many participants defined assessments in relation to measuring student learning, other substantial aspects include benchmarking, assessing student ability and competence, and formal evaluation for quality. These findings serve as foundational knowledge toward deeper exploration and understanding of assessment mental models of engineering faculty that can begin to address the aforementioned research gap on faculty assessment decisions in classrooms.
more »
« less
- Award ID(s):
- 2107008
- PAR ID:
- 10400304
- Date Published:
- Journal Name:
- IEEE Frontiers in Education Conference
- Page Range / eLocation ID:
- 1 to 8
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In this full research paper, we discuss the benefits and challenges of using GPT-4 to perform qualitative analysis to identify faculty’s mental models of assessment. Assessments play an important role in engineering education. They are used to evaluate student learning, measure progress, and identify areas for improvement. However, how faculty members approach assessments can vary based on several factors, including their own mental models of assessment. To understand the variation in these mental models, we conducted interviews with faculty members in various engineering disciplines at universities across the United States. Data was collected from 28 participants from 18 different universities. The interviews consisted of questions designed to elicit information related to the pieces of mental models (state, form, function, and purpose) of assessments of students in their classrooms. For this paper, we analyzed interviews to identify the entities and entity relationships in participant statements using natural language processing and GPT-4 as our language model. We then created a graphical representation to characterize and compare individuals’ mental models of assessment using GraphViz. We asked the model to extract entities and their relationships from interview excerpts, using GPT-4 and instructional prompts. We then compared the results of GPT-4 from a small portion of our data to entities and relationships that were extracted manually by one of our researchers. We found that both methods identified overlapping entity relationships but also discovered entities and relationships not identified by the other model. The GPT-4 model tended to identify more basic relationships, while manual analysis identified more nuanced relationships. Our results do not currently support using GPT-4 to automatically generate graphical representations of faculty’s mental models of assessments. However, using a human-in-the-loop process could help offset GPT-4’s limitations. In this paper, we will discuss plans for our future work to improve upon GPT-4’s current performance.more » « less
-
In this full research paper, we discuss the benefits and challenges of using GPT-4 to perform qualitative analysis to identify faculty’s mental models of assessment. Assessments play an important role in engineering education. They are used to evaluate student learning, measure progress, and identify areas for improvement. However, how faculty members approach assessments can vary based on several factors, including their own mental models of assessment. To understand the variation in these mental models, we conducted interviews with faculty members in various engineering disciplines at universities across the United States. Data was collected from 28 participants from 18 different universities. The interviews consisted of questions designed to elicit information related to the pieces of mental models (state, form, function, and purpose) of assessments of students in their classrooms. For this paper, we analyzed interviews to identify the entities and entity relationships in participant statements using natural language processing and GPT-4 as our language model. We then created a graphical representation to characterize and compare individuals’ mental models of assessment using GraphViz. We asked the model to extract entities and their relationships from interview excerpts, using GPT-4 and instructional prompts. We then compared the results of GPT-4 from a small portion of our data to entities and relationships that were extracted manually by one of our researchers. We found that both methods identified overlapping entity relationships but also discovered entities and relationships not identified by the other model. The GPT-4 model tended to identify more basic relationships, while manual analysis identified more nuanced relationships. Our results do not currently support using GPT-4 to automatically generate graphical representations of faculty’s mental models of assessments. However, using a human-in-the-loop process could help offset GPT-4’s limitations. In this paper, we will discuss plans for our future work to improve upon GPT-4’s current performance.more » « less
-
In this work-in-progress research paper, we explore key themes and patterns prevalent in definitions of thriving for undergraduate engineering students. Although there is growing research and acknowledgment of the breadth and complexity of thriving in engineering, the field has limited conceptual clarity regarding its range of definitions. Data for this research was collected from 47 engineering faculty and staff who are considered long-term members of the engineering education system who also play a vital role in creating environments conducive to thriving and forming relationships with students that facilitate thriving. Participants were asked to define engineering student thriving in an open-ended survey, where their responses were analyzed using thematic analysis. 29 codes emerged from the data, 18 of which align with prior research. 10 new codes emerged, relating to positive emotions (such as happy, excited, and passionate), learning (love of learning, growing, understanding and deep learning), wellbeing, identity, belongingness, and professional experiences. These findings highlight positive aspects of engineering thriving beyond the absence or reduction of suffering and hardship. Implications of these findings include developing measures with multi-dimensional focus and emphasizing the role of emotional support and identity development in engineering students.more » « less
-
Research demonstrates a growing mental health crisis in graduate education, which can contribute to productivity, departure, and well-being issues. To address this crisis and advocate for systemic change, this project explored faculty perceptions about graduate student mental health and how these perceptions intersect with direct action when student mental health challenges arise. We were guided by phenomenological inquiry to explore how faculty attitudes (n = 3) about mental health shape programmatic and individual decisions around supporting mental health. We thematically analyzed interviews discussing stress and mental health focused on faculty experiences. Faculty interviews demonstrated varying attitudes toward graduate student stress and mental health. Faculty desires to engage in discussions about stress or mental health were on a wide spectrum, often with work productivity guiding these discussions. Further, faculty highlighted levels of discomfort with engaging in discussions about mental health, especially with the students they work closest with. Findings indicate a need to foster faculty skill and comfort with engaging with students about their mental health while also providing clear institutional policies that support these actions to address the mental health crisis.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/FIE56618.2022.9962667
