An official website of the United States government
Thematic Analysis (TA) is a fundamental method in healthcare research for analyzing transcript data, but it is resource-intensive and difficult to scale for large, complex datasets. This study investigates the potential of large language models (LLMs) to augment the inductive TA process in high-stakes healthcare settings. Focusing on interview transcripts from parents of children with Anomalous Aortic Origin of a Coronary Artery (AAOCA), a rare congenital heart disease, we propose an LLM-Enhanced Thematic Analysis (LLM-TA) pipeline. Our pipeline integrates an affordable state-of-the-art LLM (GPT-4o mini), LangChain, and prompt engineering with chunking techniques to analyze nine detailed transcripts following the inductive TA framework. We evaluate the LLM-generated themes against human-generated results using thematic similarity metrics, LLM-assisted assessments, and expert reviews. Results demonstrate that our pipeline outperforms existing LLM-assisted TA methods significantly. While the pipeline alone has not yet reached human-level quality in inductive TA, it shows great potential to improve scalability, efficiency, and accuracy while reducing analyst workload when working collaboratively with domain experts. We provide practical recommendations for incorporating LLMs into high-stakes TA workflows and emphasize the importance of close collaboration with domain experts to address challenges related to real-world applicability and dataset complexity.
more »
« less
This content will become publicly available on May 2, 2026
LATA: A Pilot Study on LLM-Assisted Thematic Analysis of Online Social Network Data Generation Experiences
Large Language Models (LLMs) have gained attention in research and industry, aiming to streamline processes and enhance text analysis performance. Thematic Analysis (TA), a prevalent qualitative method for analyzing interview content, often requires at least two human experts to review and analyze data. This study demonstrates the feasibility of LLM-Assisted Thematic Analysis (LATA) using GPT-4 and Gemini. Specifically, we conducted semi-structured interviews with 14 researchers to gather insights on their experiences generating and analyzing Online Social Network (OSN) communications datasets. Following Braun and Clarke's six-phase TA framework with an inductive approach, we initially analyzed our interview transcripts with human experts. Subsequently, we iteratively designed prompts to guide LLMs through a similar process. We compare and discuss the manually analyzed outcomes with responses generated by LLMs and achieve a cosine similarity score up to 0.76, demonstrating a promising prospect for LATA. Additionally, the study delves into researchers' experiences navigating the complexities of collecting and analyzing OSN data, offering recommendations for future research and application designers.
more »
« less
- Award ID(s):
- 2438144
- PAR ID:
- 10648735
- Publisher / Repository:
- Proceedings of the ACM on Human-Computer Interaction
- Date Published:
- Journal Name:
- Proceedings of the ACM on Human-Computer Interaction
- Volume:
- 9
- Issue:
- 2
- ISSN:
- 2573-0142
- Page Range / eLocation ID:
- 1 to 28
- Subject(s) / Keyword(s):
- Human-centered computing→Human computer interaction (HCI) Human computer interaction (HCI) • Computing methodologies→Artificial intelligence • Software and its engineering
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We explore the possibility of using natural language processing (NLP) and generative artificial intelligence (GAI) to streamline the process of thematic analysis (TA) for qualitative research. We followed traditional TA phases to demonstrate areas of alignment and discordance between (a) steps one might take with NLP and GAI and (b) traditional thematic analysis. Using a case study, we illustrate the application of this workflow to a real-world dataset. We start with processes involved in data analysis and translate those into analogous steps in a workflow that uses NLP and GAI. We then discuss the potential benefits and limitations of these NLP and GAI techniques, highlighting points of convergence and divergence with thematic analysis. Then, we highlight the importance of the central role of researchers during the process of NLP and GAI-assisted thematic analysis. Finally, we conclude with a discussion of the implications of this approach for qualitative research and suggestions for future work. Researchers who are interested in AI-assisted methods can benefit from the roadmap we provide in this study to understand the current landscape of NLP and GAI models for qualitative research.more » « less
-
Purpose The goal of this study is to explore an immediate step in understanding the lived experiences of under-represented students through metaphor construction and possibly collect more in-depth data through photograph-based interviews. Design/Methodology/Approach This article introduced photo-elicitation based narrative interviews as a qualitative methodology while interviewing fourteen undergraduate community college students mostly from underrepresented groups (URGs). At the beginning of each interview, the authors probed the participants with 8 photographs chosen by the research team to represent a diverse set of experiences in engineering. The authors conducted a thematic analysis of the interview data. Findings The findings suggested that the inclusion of photo-elicitation often catalyzed consumption of representations, images, metaphors, and voice to stories passed unnoticed; and finally produces more detailed descriptions and complements semi-structured narrative interviews. Research Limitations/Implications This study advances the scholarship that extends photograph driven interviews/photo elicitation methodology while interviewing marginalized population and offers a roadmap for what a multi-modal, arts-based analysis process might look like for in-depth interviews. Practical Implications The use of photo-elicitation in our research enabled a deeper, more poignant exploration of the URG students' experience of navigating engineering. The participants were able to relate to the photographs and shared their life narratives through them; hence, use of photographs can be adapted in future research. Social Implications Our research revealed that PEI has excellent potential to capture marginalized narratives of URGs, which is not well explored in educational research, specially, in higher education. In our research, PEI promoted more culturally inclusive approaches positioning the participants as experts of their own narratives. Originality/Value The study presented in this paper serves as an example of qualitative research that expands methodological boundaries and centers the role of power, marginalization, and creativity in research. This work serves as a unique and important contribution to the photo-elicitation literature, offering a critical roadmap for researchers who are drawn to photo elicitation/photograph driven interviews as a method to explore their inquiry.more » « less
-
This Work in Progress (WIP) paper will present an interview protocol development that leverages social media analysis for capturing narratives of neurodivergent (e.g., ADHD, autistic, dyslexic) engineering students. The work presented in this WIP is part of a larger mixed-methods sequential research project which aims to capture neurodivergent engineering student narratives that describe their engineering experiences in terms of strengths and challenges. Through social media analysis, we identified key language used by the neurodivergent community (e.g., neurodivergent, spoon, forget). We developed a few initial themes such as multiple pathways to recognizing one’s identity of being neurodivergent, multiple ways in which neurodivergence symptoms are experienced, and the ways in which an individual internally and outwardly interacts with their own symptoms. To capture neurodivergent narratives, we plan on conducting semi-structured interviews with neurodivergent engineering students three times over the semester (beginning, middle, end). Initial interview protocols for each interview will be developed and adjusted throughout the interview data collection process. An initial compilation of relevant interview questions was compiled from previous research and from the objectives of the research study. This initial pool of questions will then be refined based on our thematic findings and using the nuanced language identified on the social media platforms TikTok, Reddit, and Twitter. Results of this work will be presented in this paper as an interview protocol that will continue to be adapted as part of this larger research study, but can also be used as a starting point for researchers exploring similar topics in capturing the experience of neurodivergent engineering students.more » « less
-
This Work in Progress (WIP) paper will present an interview protocol development that leverages social media analysis for capturing narratives of neurodivergent (e.g., ADHD, autistic, dyslexic) engineering students. The work presented in this WIP is part of a larger mixed-methods sequential research project which aims to capture neurodivergent engineering student narratives that describe their engineering experiences in terms of strengths and challenges. Through social media analysis, we identified key language used by the neurodivergent community (e.g., neurodivergent, spoon, forget). We developed a few initial themes such as multiple pathways to recognizing one’s identity of being neurodivergent, multiple ways in which neurodivergence symptoms are experienced, and the ways in which an individual internally and outwardly interacts with their own symptoms. To capture neurodivergent narratives, we plan on conducting semi-structured interviews with neurodivergent engineering students three times over the semester (beginning, middle, end). Initial interview protocols for each interview will be developed and adjusted throughout the interview data collection process. An initial compilation of relevant interview questions was compiled from previous research and from the objectives of the research study. This initial pool of questions will then be refined based on our thematic findings and using the nuanced language identified on the social media platforms TikTok, Reddit, and Twitter. Results of this work will be presented in this paper as an interview protocol that will continue to be adapted as part of this larger research study, but can also be used as a starting point for researchers exploring similar topics in capturing the experience of neurodivergent engineering students.more » « less
