An official website of the United States government
Abstract BackgroundStudents' identification with engineering is intertwined culturally with being smart. Broadly, engineering students are often considered to be smart by others and by themselves, and these beliefs about smartness—what it is and who has enough of it to be an engineer—are a fundamental and limiting aspect of students' experiences. PurposeThe purpose of this study was to explore how undergraduate engineering students describe themselves as smart enough to be engineers. We aimed to develop rich descriptions of the complex ways they articulate their identities as smart before coming to college and during the first two years of their undergraduate degrees. Design/MethodWe collected data through a series of interviews with 25 participants. We iteratively and collaboratively analyzed the data to determine the predominant ways the participants articulated their identities as smart enough to be engineers. We generated a qualitative data display to check for patterns related to pathways into engineering programs and privileged social identities. ResultsWe found that engineering students have three different ways to articulate that they are smart enough to be engineers: (1) they have innate abilities, (2) they are hardworking and dedicated to learning, and (3) they have skills and experience related to engineering. Additionally, we provide qualitative evidence that the innate abilities articulation relates to privilege. Discussion/ConclusionThe study participants engaged in identity work that produced the three articulations. As engineering educators, we need to take responsibility for the ways in which our participation in the cultural practice of smartness reproduces inequity.
more »
« less
Affordances and limitations of using large language models to generate qualitative data about mental health perceptions in engineering
Abstract BackgroundGenerative artificial intelligence (AI) large‐language models (LLMs) have significant potential as research tools. However, the broader implications of using these tools are still emerging. Few studies have explored using LLMs to generate data for qualitative engineering education research. Purpose/HypothesisWe explore the following questions: (i) What are the affordances and limitations of using LLMs to generate qualitative data in engineering education, and (ii) in what ways might these data reproduce and reinforce dominant cultural narratives in engineering education, including narratives of high stress? Design/MethodsWe analyzed similarities and differences between LLM‐generated conversational data (ChatGPT) and qualitative interviews with engineering faculty and undergraduate engineering students from multiple institutions. We identified patterns, affordances, limitations, and underlying biases in generated data. ResultsLLM‐generated content contained similar responses to interview content. Varying the prompt persona (e.g., demographic information) increased the response variety. When prompted for ways to decrease stress in engineering education, LLM responses more readily described opportunities for structural change, while participants' responses more often described personal changes. LLM data more frequently stereotyped a response than participants did, meaning that LLM responses lacked the nuance and variation that naturally occurs in interviews. ConclusionsLLMs may be a useful tool in brainstorming, for example, during protocol development and refinement. However, the bias present in the data indicates that care must be taken when engaging with LLMs to generate data. Specially trained LLMs that are based only on data from engineering education hold promise for future research.
more »
« less
- Award ID(s):
- 2342384
- PAR ID:
- 10650930
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Journal of Engineering Education
- Volume:
- 115
- Issue:
- 1
- ISSN:
- 1069-4730
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Purpose The goal of this study is to explore an immediate step in understanding the lived experiences of under-represented students through metaphor construction and possibly collect more in-depth data through photograph-based interviews. Design/Methodology/Approach This article introduced photo-elicitation based narrative interviews as a qualitative methodology while interviewing fourteen undergraduate community college students mostly from underrepresented groups (URGs). At the beginning of each interview, the authors probed the participants with 8 photographs chosen by the research team to represent a diverse set of experiences in engineering. The authors conducted a thematic analysis of the interview data. Findings The findings suggested that the inclusion of photo-elicitation often catalyzed consumption of representations, images, metaphors, and voice to stories passed unnoticed; and finally produces more detailed descriptions and complements semi-structured narrative interviews. Research Limitations/Implications This study advances the scholarship that extends photograph driven interviews/photo elicitation methodology while interviewing marginalized population and offers a roadmap for what a multi-modal, arts-based analysis process might look like for in-depth interviews. Practical Implications The use of photo-elicitation in our research enabled a deeper, more poignant exploration of the URG students' experience of navigating engineering. The participants were able to relate to the photographs and shared their life narratives through them; hence, use of photographs can be adapted in future research. Social Implications Our research revealed that PEI has excellent potential to capture marginalized narratives of URGs, which is not well explored in educational research, specially, in higher education. In our research, PEI promoted more culturally inclusive approaches positioning the participants as experts of their own narratives. Originality/Value The study presented in this paper serves as an example of qualitative research that expands methodological boundaries and centers the role of power, marginalization, and creativity in research. This work serves as a unique and important contribution to the photo-elicitation literature, offering a critical roadmap for researchers who are drawn to photo elicitation/photograph driven interviews as a method to explore their inquiry.more » « less
-
Large Language Models (LLMs) have become increasingly incorporated into everyday life for many internet users, taking on significant roles as advice givers in the domains of medicine, personal relationships, and even legal matters. The importance of these roles raise questions about how and what responses LLMs make in difficult political and moral domains, especially questions about possible biases. To quantify the nature of potential biases in LLMs, various works have applied Moral Foundations Theory (MFT), a framework that categorizes human moral reasoning into five dimensions: Harm, Fairness, Ingroup Loyalty, Authority, and Purity. Previous research has used the MFT to measure differences in human participants along political, national, and cultural lines. While there has been some analysis of the responses of LLM with respect to political stance in role-playing scenarios, no work so far has directly assessed the moral leanings in the LLM responses, nor have they connected LLM outputs with robust human data. In this paper we analyze the distinctions between LLM MFT responses and existing human research directly, investigating whether commonly available LLM responses demonstrate ideological leanings — either through their inherent responses, straightforward representations of political ideologies, or when responding from the perspectives of constructed human personas. We assess whether LLMs inherently generate responses that align more closely with one political ideology over another, and additionally examine how accurately LLMs can represent ideological perspectives through both explicit prompting and demographic-based role-playing. By systematically analyzing LLM behavior across these conditions and experiments, our study provides insight into the extent of political and demographic dependency in AI-generated responses.more » « less
-
In this methods paper, the development and utility of composite narratives will be explored. Composite narratives, which involve combining aspects of multiple interviews into a single narrative, are a relatively modern methodology used in the qualitative research literature for several purposes: to do justice to complex accounts while maintaining participant anonymity, summarize data in a more engaging personal form and retain the human face of the data, illustrate specific aspects of the research findings, enhance the transferability of research findings by invoking empathy, illuminate collective experiences, and enhance research impact by providing findings in a manner more accessible to those outside of academia. Composite narratives leverage the power of storytelling, which has shown to be effective in studies of neurology and psychology; i.e., since humans often think and process information in narrative structures, the information conveyed in story form can be imprinted more easily on readers’ minds or existing schema. Engineering education researchers have increasingly begun using narrative research methods. Recently, researchers have begun exploring composite narratives as an approach to enable more complex and nuanced understandings of engineering education while mitigating potential issues around the confidentiality of participants. Because this is a relatively new methodology in higher education more broadly and in engineering education specifically, more examples of how to construct and utilize composite narratives in both research and practice are needed. This paper will share how we created a composite narrative from interviews we collected for our work so that others can adapt this methodology for their research projects. The paper will also discuss ways we modified and enhanced these narratives to connect research to practice and impact engineering students. This approach involved developing probing questions to stimulate thinking, learning, and discussion in academic and industrial educational settings. We developed the composite narratives featured in this paper from fifteen semi-structured critical incident interviews with engineering managers about their perceptions of adaptability. The critical incidents shared were combined to develop seven composite narratives reflecting real-life situations to which engineers must adapt in the workplace. These scenarios, grounded in the data, were taken directly to the engineering classroom for discussion with students on how they would respond and adapt to the presented story. In this paper, we detail our process of creating one composite narrative from the broader study and its associated probing questions for research dissemination in educational settings. We present this detailed account of how one composite narrative was constructed to demonstrate the quality and trustworthiness of the composite narrative methodology and assist in its replication by other scholars. Further, we discuss the benefits and limitations of this methodology, highlighting the parts of the data brought into focus using this method and how that contrasts with an inductive-deductive approach to qualitative coding also taken in this research project.more » « less
-
The increasing use of machine learning and Large Language Models (LLMs) opens up opportunities to use these artificially intelligent algorithms in novel ways. This article proposes a methodology using LLMs to support traditional deductive coding in qualitative research. We began our analysis with three different sample texts taken from existing interviews. Next, we created a codebook and inputted the sample text and codebook into an LLM. We asked the LLM to determine if the codes were present in a sample text provided and requested evidence to support the coding. The sample texts were inputted 160 times to record changes between iterations of the LLM response. Each iteration was analogous to a new coder deductively analyzing the text with the codebook information. In our results, we present the outputs for these recursive analyses, along with a comparison of the LLM coding to evaluations made by human coders using traditional coding methods. We argue that LLM analysis can aid qualitative researchers by deductively coding transcripts, providing a systematic and reliable platform for code identification, and offering a means of avoiding analysis misalignment. Implications of using LLM in research praxis are discussed, along with current limitations.more » « less
