An official website of the United States government
PurposeThe purpose of this study was to examine the experiences of multiple campus teams as they engaged in the assessment of their science, technology, engineering and mathematics (STEM) mentoring ecosystems within a peer assessment dialogue exercise. Design/methodology/approachThis project utilized a qualitative multicase study method involving six campus teams, drawing upon completed inventory and visual mapping artefacts, session observations and debriefing interviews. The campuses included research universities, small colleges and minority-serving institutions (MSIs) across the United States of America. The authors analysed which features of the peer assessment dialogue exercise scaffolded participants' learning about ecosystem synergies and threats. FindingsThe results illustrated the benefit of instructor modelling, intra-team process time and multiple rounds of peer assessment. Participants gained new insights into their own campuses and an increased sense of possibility by dialoguing with peer campuses. Research limitations/implicationsThis project involved teams from a small set of institutions, relying on observational and self-reported debriefing data. Future research could centre perspectives of institutional leaders. Practical implicationsThe authors recommend dedicating time to the institutional assessment of mentoring ecosystems. Investing in a campus-wide mentoring infrastructure could align with campus equity goals. Originality/valueIn contrast to studies that have focussed solely on programmatic outcomes of mentoring, this study explored strategies to strengthen institutional mentoring ecosystems in higher education, with a focus on peer assessment, dialogue and learning exercises.
more »
« less
Affordances and limitations of using large language models to generate qualitative data about mental health perceptions in engineering
Abstract BackgroundGenerative artificial intelligence (AI) large‐language models (LLMs) have significant potential as research tools. However, the broader implications of using these tools are still emerging. Few studies have explored using LLMs to generate data for qualitative engineering education research. Purpose/HypothesisWe explore the following questions: (i) What are the affordances and limitations of using LLMs to generate qualitative data in engineering education, and (ii) in what ways might these data reproduce and reinforce dominant cultural narratives in engineering education, including narratives of high stress? Design/MethodsWe analyzed similarities and differences between LLM‐generated conversational data (ChatGPT) and qualitative interviews with engineering faculty and undergraduate engineering students from multiple institutions. We identified patterns, affordances, limitations, and underlying biases in generated data. ResultsLLM‐generated content contained similar responses to interview content. Varying the prompt persona (e.g., demographic information) increased the response variety. When prompted for ways to decrease stress in engineering education, LLM responses more readily described opportunities for structural change, while participants' responses more often described personal changes. LLM data more frequently stereotyped a response than participants did, meaning that LLM responses lacked the nuance and variation that naturally occurs in interviews. ConclusionsLLMs may be a useful tool in brainstorming, for example, during protocol development and refinement. However, the bias present in the data indicates that care must be taken when engaging with LLMs to generate data. Specially trained LLMs that are based only on data from engineering education hold promise for future research.
more »
« less
- Award ID(s):
- 2342384
- PAR ID:
- 10659449
- Publisher / Repository:
- Journal of Engineering Education
- Date Published:
- Journal Name:
- Journal of Engineering Education
- Volume:
- 115
- Issue:
- 1
- ISSN:
- 1069-4730
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Large Language Models (LLMs) have become increasingly incorporated into everyday life for many internet users, taking on significant roles as advice givers in the domains of medicine, personal relationships, and even legal matters. The importance of these roles raise questions about how and what responses LLMs make in difficult political and moral domains, especially questions about possible biases. To quantify the nature of potential biases in LLMs, various works have applied Moral Foundations Theory (MFT), a framework that categorizes human moral reasoning into five dimensions: Harm, Fairness, Ingroup Loyalty, Authority, and Purity. Previous research has used the MFT to measure differences in human participants along political, national, and cultural lines. While there has been some analysis of the responses of LLM with respect to political stance in role-playing scenarios, no work so far has directly assessed the moral leanings in the LLM responses, nor have they connected LLM outputs with robust human data. In this paper we analyze the distinctions between LLM MFT responses and existing human research directly, investigating whether commonly available LLM responses demonstrate ideological leanings — either through their inherent responses, straightforward representations of political ideologies, or when responding from the perspectives of constructed human personas. We assess whether LLMs inherently generate responses that align more closely with one political ideology over another, and additionally examine how accurately LLMs can represent ideological perspectives through both explicit prompting and demographic-based role-playing. By systematically analyzing LLM behavior across these conditions and experiments, our study provides insight into the extent of political and demographic dependency in AI-generated responses.more » « less
-
Purpose The goal of this study is to explore an immediate step in understanding the lived experiences of under-represented students through metaphor construction and possibly collect more in-depth data through photograph-based interviews. Design/Methodology/Approach This article introduced photo-elicitation based narrative interviews as a qualitative methodology while interviewing fourteen undergraduate community college students mostly from underrepresented groups (URGs). At the beginning of each interview, the authors probed the participants with 8 photographs chosen by the research team to represent a diverse set of experiences in engineering. The authors conducted a thematic analysis of the interview data. Findings The findings suggested that the inclusion of photo-elicitation often catalyzed consumption of representations, images, metaphors, and voice to stories passed unnoticed; and finally produces more detailed descriptions and complements semi-structured narrative interviews. Research Limitations/Implications This study advances the scholarship that extends photograph driven interviews/photo elicitation methodology while interviewing marginalized population and offers a roadmap for what a multi-modal, arts-based analysis process might look like for in-depth interviews. Practical Implications The use of photo-elicitation in our research enabled a deeper, more poignant exploration of the URG students' experience of navigating engineering. The participants were able to relate to the photographs and shared their life narratives through them; hence, use of photographs can be adapted in future research. Social Implications Our research revealed that PEI has excellent potential to capture marginalized narratives of URGs, which is not well explored in educational research, specially, in higher education. In our research, PEI promoted more culturally inclusive approaches positioning the participants as experts of their own narratives. Originality/Value The study presented in this paper serves as an example of qualitative research that expands methodological boundaries and centers the role of power, marginalization, and creativity in research. This work serves as a unique and important contribution to the photo-elicitation literature, offering a critical roadmap for researchers who are drawn to photo elicitation/photograph driven interviews as a method to explore their inquiry.more » « less
-
In this methods paper, the development and utility of composite narratives will be explored. Composite narratives, which involve combining aspects of multiple interviews into a single narrative, are a relatively modern methodology used in the qualitative research literature for several purposes: to do justice to complex accounts while maintaining participant anonymity, summarize data in a more engaging personal form and retain the human face of the data, illustrate specific aspects of the research findings, enhance the transferability of research findings by invoking empathy, illuminate collective experiences, and enhance research impact by providing findings in a manner more accessible to those outside of academia. Composite narratives leverage the power of storytelling, which has shown to be effective in studies of neurology and psychology; i.e., since humans often think and process information in narrative structures, the information conveyed in story form can be imprinted more easily on readers’ minds or existing schema. Engineering education researchers have increasingly begun using narrative research methods. Recently, researchers have begun exploring composite narratives as an approach to enable more complex and nuanced understandings of engineering education while mitigating potential issues around the confidentiality of participants. Because this is a relatively new methodology in higher education more broadly and in engineering education specifically, more examples of how to construct and utilize composite narratives in both research and practice are needed. This paper will share how we created a composite narrative from interviews we collected for our work so that others can adapt this methodology for their research projects. The paper will also discuss ways we modified and enhanced these narratives to connect research to practice and impact engineering students. This approach involved developing probing questions to stimulate thinking, learning, and discussion in academic and industrial educational settings. We developed the composite narratives featured in this paper from fifteen semi-structured critical incident interviews with engineering managers about their perceptions of adaptability. The critical incidents shared were combined to develop seven composite narratives reflecting real-life situations to which engineers must adapt in the workplace. These scenarios, grounded in the data, were taken directly to the engineering classroom for discussion with students on how they would respond and adapt to the presented story. In this paper, we detail our process of creating one composite narrative from the broader study and its associated probing questions for research dissemination in educational settings. We present this detailed account of how one composite narrative was constructed to demonstrate the quality and trustworthiness of the composite narrative methodology and assist in its replication by other scholars. Further, we discuss the benefits and limitations of this methodology, highlighting the parts of the data brought into focus using this method and how that contrasts with an inductive-deductive approach to qualitative coding also taken in this research project.more » « less
-
Large Language Models (LLMs) have gained attention in research and industry, aiming to streamline processes and enhance text analysis performance. Thematic Analysis (TA), a prevalent qualitative method for analyzing interview content, often requires at least two human experts to review and analyze data. This study demonstrates the feasibility of LLM-Assisted Thematic Analysis (LATA) using GPT-4 and Gemini. Specifically, we conducted semi-structured interviews with 14 researchers to gather insights on their experiences generating and analyzing Online Social Network (OSN) communications datasets. Following Braun and Clarke's six-phase TA framework with an inductive approach, we initially analyzed our interview transcripts with human experts. Subsequently, we iteratively designed prompts to guide LLMs through a similar process. We compare and discuss the manually analyzed outcomes with responses generated by LLMs and achieve a cosine similarity score up to 0.76, demonstrating a promising prospect for LATA. Additionally, the study delves into researchers' experiences navigating the complexities of collecting and analyzing OSN data, offering recommendations for future research and application designers.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1002/jee.70037
