More Like this

1. Connecting dots: Coding multiple data sources to enhance qualitative analysis

   https://doi.org/10.18260/1-2--32536  

   Bir, D. D. (January 2019, ASEE Annual Conference proceedings)

   This research paper elaborates on the process used by a team of researchers to create a codebook from interviews of Civil Engineers, which included students, professors, and professionals, solving ill-structured problems. The participants solved two ill-structured problems while speaking aloud their thought process. In addition to recording the participant verbalization, the solution to their problems were also collected with the use of a smart pen. Creating a codebook from interviews is a key element of qualitative analysis forming the basis for coding. While individuals can create codebooks for analysis, a team-based approach is advantageous especially when dealing with large amounts of data. A team-based approach involves an iterative process of inter-rater reliability essential to the trustworthiness of the data obtained by coding. In addition to coding the transcripts as a team, which consisted of novice, intermediate, and experts in the engineering education field, the audio and written solution to the problems were also coded. The use of multiple data sources to obtain data, and not just the verbatim transcripts, is lesser studied in engineering education literature and provides opportunities for a more detailed qualitative analysis. Initial codes were created from existing literature, which were refined through an iterative process. This process consisted of coding data, team consensus on coded data, codebook refinement, and recoding data with the refined codes. Results show that coding verbatim transcripts might not provide an accurate representation of the problem-solving processes participants used to solve the ill-structured problem. Benefits, challenges and recommendations regarding the use of multiple sources to obtain data are discussed while considering the amount of time required to conduct such analysis. 

   more » « less
2. Examining Undergraduate Engineering Students’ Perceptions of Solving an Ill-Structured Problem in Civil Engineering

   https://doi.org/10.18260/1-2--34622  

   Akinci-Ceylan, S; Cetin, KS; Ahn, B; Cetin, B (June 2020, 2020 ASEE Virtual Annual Conference)

   Workplace engineering problems are different from the problems that undergraduate engineering students typically encounter in most classroom settings. Students are most commonly given well-structured problems which have clear solution paths along with well-defined constraints and goals. This paper reports on research that examines how undergraduate engineering students perceived solving an ill-structured problem. Eighteen undergraduate civil engineering students were asked to solve an ill-structured engineering problem, and were interviewed after they completed solving the problem. This qualitative study is guided by the following research question: What factors do students perceive to influence their solving of an ill-structured civil engineering problem? Students’ responses to seven follow-up interview questions were transcribed and reviewed by research team members, which were used to develop codes and themes associated with these responses. Students’ transcripts were then coded following the developed codes. The analysis of data revealed that students were generally aware of the main positives and negatives of their proposed solutions to the ill-structured problem and reported that their creativity influenced their solutions and problem solving processes. Student responses also indicated that specific life events such as classes that they had taken, personal experiences, and exposure to other ill-structured problems during an internship helped them develop their proposed solution. Given students’ responses and overall findings, this supports creating learning environments for engineering students where they can support increasing their creativity and be more exposed to complex engineering problems. 

   more » « less
3. Interviews with members of the HJA Community during the Lookout Fire, HJ Andrews Experimental Forest, 2023

   https://doi.org/10.6073/pasta/352aaa3a7881f8ec23569135ce4c7057  

   Rapp, Claire; Nelson, Michael P (January 2024, Environmental Data Initiative)

   This dataset records the interview instrument, analytical codebook, and summary of results for the 2023 Lookout Fire Qualitative Interviews. Data was collected in 2023 in Corvallis, Oregon, and over Zoom. Members of the H. J. Andrews Experimental Forest (HJA) community (e.g., university faculty and administrative professionals, agency scientists and personnel, students, alumni and emeritus from the aforementioned communities) were interviewed between September 26th and November 8th 2023. At the time, the fire had largely stopped growing (no significant runs occurred during the interview period), but the fire was not fully contained and the fire severity was not yet known by the community. Data collection is complete. The interview included questions about emotional reactions to the Lookout Fire, current and foreseen impacts to research at the HJA, social relationships and the fire, naturalness of the fire, and climate change, climate anxiety, and the fire. Interviews were semi-structured; while interviews were guided by the interview protocol, conversation was allowed to proceed organically. In total, 40 respondents were interviewed. Interviews were transcribed verbatim and analyzed inductively and deductively. A finalized codebook was developed iteratively; the included codebook are the final codes used to analyze the full dataset. Interview transcripts and other potentially identifying information is not available to protect respondent confidentiality and anonymity. This dataset summarizes the key interview results. 

   more » « less
4. An Examination of the Use of Large Language Models to Aid Analysis of Textual Data

   https://doi.org/10.1177/16094069241231168  

   Tai, Robert H; Bentley, Lillian R; Xia, Xin; Sitt, Jason M; Fankhauser, Sarah C; Chicas-Mosier, Ana M; Monteith, Barnas G (January 2024, International Journal of Qualitative Methods)

   The increasing use of machine learning and Large Language Models (LLMs) opens up opportunities to use these artificially intelligent algorithms in novel ways. This article proposes a methodology using LLMs to support traditional deductive coding in qualitative research. We began our analysis with three different sample texts taken from existing interviews. Next, we created a codebook and inputted the sample text and codebook into an LLM. We asked the LLM to determine if the codes were present in a sample text provided and requested evidence to support the coding. The sample texts were inputted 160 times to record changes between iterations of the LLM response. Each iteration was analogous to a new coder deductively analyzing the text with the codebook information. In our results, we present the outputs for these recursive analyses, along with a comparison of the LLM coding to evaluations made by human coders using traditional coding methods. We argue that LLM analysis can aid qualitative researchers by deductively coding transcripts, providing a systematic and reliable platform for code identification, and offering a means of avoiding analysis misalignment. Implications of using LLM in research praxis are discussed, along with current limitations. 

   more » « less
5. Preliminary Findings on Students’ Beliefs about Intelligence

   Adams, Allison; Betz, Amy; Dringenberg, Emily (January 2019, ASEE Annual Conference proceedings)

   The goal of this project is to better understand the beliefs that undergraduate students hold about their own intelligence and how these beliefs change during their undergraduate engineering education. The research team has used the theoretical framework established by Carol Dweck on Mindset and how different fixed and growth mindsets affect success. Fixed mindset individuals believe that their intelligence is an unchanging trait, while people with a growth mindset believe that through effort they can grow and develop greater intelligence. Prior researchers have shown that individuals with a growth mindset respond to challenges with higher levels of persistence, are more interested in improving upon past failures, and value criticism and effort more than those with a fixed mindset. The team developed an interview protocol from the theoretical framework. Then the team piloted the protocol and subsequently modified the protocol multiple times to ensure that the interviews provided rich qualitative data. Analytic memos were used to analyze and modify the piloted interview protocols. Once the final protocol was established, first-year and senior students were recruited to provide cross-sectional insight. The team also recruited using purposeful sampling to ensure that women and underrepresented minorities were included. To date, 19 interviews have been conducted with the final protocol. Of these interviews, four have been coded in detail using the “Attitudes, Values, and Beliefs” coding system. A codebook has also been started to categorize and interconnect the themes in the interview transcripts. This paper provides details of the protocol and coding process as well as preliminary findings on the themes extracted from the student interviews. 

   more » « less