More Like this

1. Connecting dots: Coding multiple data sources to enhance qualitative analysis

   https://doi.org/10.18260/1-2--32536  

   Bir, D. D. (January 2019, ASEE Annual Conference proceedings)

   This research paper elaborates on the process used by a team of researchers to create a codebook from interviews of Civil Engineers, which included students, professors, and professionals, solving ill-structured problems. The participants solved two ill-structured problems while speaking aloud their thought process. In addition to recording the participant verbalization, the solution to their problems were also collected with the use of a smart pen. Creating a codebook from interviews is a key element of qualitative analysis forming the basis for coding. While individuals can create codebooks for analysis, a team-based approach is advantageous especially when dealing with large amounts of data. A team-based approach involves an iterative process of inter-rater reliability essential to the trustworthiness of the data obtained by coding. In addition to coding the transcripts as a team, which consisted of novice, intermediate, and experts in the engineering education field, the audio and written solution to the problems were also coded. The use of multiple data sources to obtain data, and not just the verbatim transcripts, is lesser studied in engineering education literature and provides opportunities for a more detailed qualitative analysis. Initial codes were created from existing literature, which were refined through an iterative process. This process consisted of coding data, team consensus on coded data, codebook refinement, and recoding data with the refined codes. Results show that coding verbatim transcripts might not provide an accurate representation of the problem-solving processes participants used to solve the ill-structured problem. Benefits, challenges and recommendations regarding the use of multiple sources to obtain data are discussed while considering the amount of time required to conduct such analysis. 

   more » « less
2. Examining Undergraduate Engineering Students’ Perceptions of Solving an Ill-Structured Problem in Civil Engineering

   https://doi.org/10.18260/1-2--34622  

   Akinci-Ceylan, S; Cetin, KS; Ahn, B; Cetin, B (June 2020, 2020 ASEE Virtual Annual Conference)

   Workplace engineering problems are different from the problems that undergraduate engineering students typically encounter in most classroom settings. Students are most commonly given well-structured problems which have clear solution paths along with well-defined constraints and goals. This paper reports on research that examines how undergraduate engineering students perceived solving an ill-structured problem. Eighteen undergraduate civil engineering students were asked to solve an ill-structured engineering problem, and were interviewed after they completed solving the problem. This qualitative study is guided by the following research question: What factors do students perceive to influence their solving of an ill-structured civil engineering problem? Students’ responses to seven follow-up interview questions were transcribed and reviewed by research team members, which were used to develop codes and themes associated with these responses. Students’ transcripts were then coded following the developed codes. The analysis of data revealed that students were generally aware of the main positives and negatives of their proposed solutions to the ill-structured problem and reported that their creativity influenced their solutions and problem solving processes. Student responses also indicated that specific life events such as classes that they had taken, personal experiences, and exposure to other ill-structured problems during an internship helped them develop their proposed solution. Given students’ responses and overall findings, this supports creating learning environments for engineering students where they can support increasing their creativity and be more exposed to complex engineering problems. 

   more » « less
3. An Examination of the Use of Large Language Models to Aid Analysis of Textual Data

   https://doi.org/10.1177/16094069241231168  

   Tai, Robert H; Bentley, Lillian R; Xia, Xin; Sitt, Jason M; Fankhauser, Sarah C; Chicas-Mosier, Ana M; Monteith, Barnas G (January 2024, International Journal of Qualitative Methods)

   The increasing use of machine learning and Large Language Models (LLMs) opens up opportunities to use these artificially intelligent algorithms in novel ways. This article proposes a methodology using LLMs to support traditional deductive coding in qualitative research. We began our analysis with three different sample texts taken from existing interviews. Next, we created a codebook and inputted the sample text and codebook into an LLM. We asked the LLM to determine if the codes were present in a sample text provided and requested evidence to support the coding. The sample texts were inputted 160 times to record changes between iterations of the LLM response. Each iteration was analogous to a new coder deductively analyzing the text with the codebook information. In our results, we present the outputs for these recursive analyses, along with a comparison of the LLM coding to evaluations made by human coders using traditional coding methods. We argue that LLM analysis can aid qualitative researchers by deductively coding transcripts, providing a systematic and reliable platform for code identification, and offering a means of avoiding analysis misalignment. Implications of using LLM in research praxis are discussed, along with current limitations. 

   more » « less
4. Preliminary Findings on Students’ Beliefs about Intelligence

   Adams, Allison; Betz, Amy; Dringenberg, Emily (January 2019, ASEE Annual Conference proceedings)

   The goal of this project is to better understand the beliefs that undergraduate students hold about their own intelligence and how these beliefs change during their undergraduate engineering education. The research team has used the theoretical framework established by Carol Dweck on Mindset and how different fixed and growth mindsets affect success. Fixed mindset individuals believe that their intelligence is an unchanging trait, while people with a growth mindset believe that through effort they can grow and develop greater intelligence. Prior researchers have shown that individuals with a growth mindset respond to challenges with higher levels of persistence, are more interested in improving upon past failures, and value criticism and effort more than those with a fixed mindset. The team developed an interview protocol from the theoretical framework. Then the team piloted the protocol and subsequently modified the protocol multiple times to ensure that the interviews provided rich qualitative data. Analytic memos were used to analyze and modify the piloted interview protocols. Once the final protocol was established, first-year and senior students were recruited to provide cross-sectional insight. The team also recruited using purposeful sampling to ensure that women and underrepresented minorities were included. To date, 19 interviews have been conducted with the final protocol. Of these interviews, four have been coded in detail using the “Attitudes, Values, and Beliefs” coding system. A codebook has also been started to categorize and interconnect the themes in the interview transcripts. This paper provides details of the protocol and coding process as well as preliminary findings on the themes extracted from the student interviews. 

   more » « less
5. Streamlining Field Note Analysis: Leveraging GPT for Further Insights

   López-Fierro, S; Shehzad, U; Zandi, A; Clarke-Midura, J; Recker, M (April 2025, American Educational Research Association)

   As an integral part of qualitative research inquiry, field notes provide important data from researchers embedded in research sites. However, field notes can vary significantly, influenced by the researchers' immersion in the field, prior knowledge, beliefs, interests, and perspectives. As consequence, their interpretation presents significant challenges. This study offers a preliminary investigation into the potential of using large language models to assist researchers with the analysis and interpretation of field notes data. Our methodology consisted of two phases. First, a researcher deductively coded field notes of six classroom implementations of a novel elementary-level mathematics curriculum. In the second phase, we prompted ChatGPT-4 to code the same field notes, using the codebook, definitions, examples, and deductive coding approach employed by the researcher. We also prompted Chatgpt to provide justifications of its coding decisions We then, calculated agreements and disagreements between ChatGPT and the researcher, organized the data in a contingency table, computed Cohen's Kappa, structured the data into a confusion matrix; and using the researcher’s coding as the “gold standard”, we calculated performance measures, specifically: Accuracy, Precision, Recall, and F1 Score. Our findings revealed that while the researcher and ChatGPT appeared to generally agree on the frequency in applying the different codes, overall agreement, as measured by Cohen’s Kappa was low. In contrast, using measures from information science at the code level revealed more nuanced results. Moreover, coupled with ChatGPT justifications of coding decisions, these findings provided insights than can help support the iterative improvement of codebooks. 

   more » « less