This research paper elaborates on the process used by a team of researchers to create a codebook from interviews of Civil Engineers who included students, professors, and professionals, solving ill-structured problems. The participants solved two ill-structured problems while speaking aloud their thought process. In addition to recording the participant verbalization, the solution to their problems were also collected with the use of a smart pen. Creating a codebook from interviews is a key element of qualitative analysis forming the basis for coding. While individuals can create codebooks for analysis, a team-based approach is advantageous especially when dealing with large amounts of data. A team-based approach involves an iterative process of inter-rater reliability essential to the trustworthiness of the data obtained by coding. In addition to coding the transcripts as a team, which consisted of novice, intermediate, and experts in the engineering education field, the audio and written solution to the problems were also coded. The use of multiple data sources to obtain data, and not just the verbatim transcripts, is lesser studied in engineering education literature and provides opportunities for a more detailed qualitative analysis.
Initial codes were created from existing literature, which were refined through an iterative process. This process consisted of coding data, team consensus on coded data, codebook refinement, and recoding data with the refined codes. Results show that coding verbatim transcripts might not provide an accurate representation of the problem-solving processes participants used to solve the ill-structured problem. Benefits, challenges and recommendations regarding the use of multiple sources to obtain data are discussed while considering the amount of time required to conduct such analysis.
more »
« less
Connecting dots: Coding multiple data sources to enhance qualitative analysis
This research paper elaborates on the process used by a team of researchers to create a codebook from interviews of Civil Engineers, which included students, professors, and professionals, solving ill-structured problems. The participants solved two ill-structured problems while speaking aloud their thought process. In addition to recording the participant verbalization, the solution to their problems were also collected with the use of a smart pen. Creating a codebook from interviews is a key element of qualitative analysis forming the basis for coding. While individuals can create codebooks for analysis, a team-based approach is advantageous especially when dealing with large amounts of data. A team-based approach involves an iterative process of inter-rater reliability essential to the trustworthiness of the data obtained by coding. In addition to coding the transcripts as a team, which consisted of novice, intermediate, and experts in the engineering education field, the audio and written solution to the problems were also coded. The use of multiple data sources to obtain data, and not just the verbatim transcripts, is lesser studied in engineering education literature and provides opportunities for a more detailed qualitative analysis.
Initial codes were created from existing literature, which were refined through an iterative process. This process consisted of coding data, team consensus on coded data, codebook refinement, and recoding data with the refined codes. Results show that coding verbatim transcripts might not provide an accurate representation of the problem-solving processes participants used to solve the ill-structured problem. Benefits, challenges and recommendations regarding the use of multiple sources to obtain data are discussed while considering the amount of time required to conduct such analysis.
more »
« less
- Award ID(s):
- 2013144
- NSF-PAR ID:
- 10322863
- Date Published:
- Journal Name:
- ASEE Annual Conference proceedings
- ISSN:
- 1524-4644
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Workplace engineering problems are different from the problems that undergraduate engineering students typically encounter in most classroom settings. Students are most commonly given well-structured problems which have clear solution paths along with well-defined constraints and goals. This paper reports on research that examines how undergraduate engineering students perceived solving an ill-structured problem. Eighteen undergraduate civil engineering students were asked to solve an ill-structured engineering problem, and were interviewed after they completed solving the problem. This qualitative study is guided by the following research question: What factors do students perceive to influence their solving of an ill-structured civil engineering problem? Students’ responses to seven follow-up interview questions were transcribed and reviewed by research team members, which were used to develop codes and themes associated with these responses. Students’ transcripts were then coded following the developed codes. The analysis of data revealed that students were generally aware of the main positives and negatives of their proposed solutions to the ill-structured problem and reported that their creativity influenced their solutions and problem solving processes. Student responses also indicated that specific life events such as classes that they had taken, personal experiences, and exposure to other ill-structured problems during an internship helped them develop their proposed solution. Given students’ responses and overall findings, this supports creating learning environments for engineering students where they can support increasing their creativity and be more exposed to complex engineering problems.more » « less
-
The increasing use of machine learning and Large Language Models (LLMs) opens up opportunities to use these artificially intelligent algorithms in novel ways. This article proposes a methodology using LLMs to support traditional deductive coding in qualitative research. We began our analysis with three different sample texts taken from existing interviews. Next, we created a codebook and inputted the sample text and codebook into an LLM. We asked the LLM to determine if the codes were present in a sample text provided and requested evidence to support the coding. The sample texts were inputted 160 times to record changes between iterations of the LLM response. Each iteration was analogous to a new coder deductively analyzing the text with the codebook information. In our results, we present the outputs for these recursive analyses, along with a comparison of the LLM coding to evaluations made by human coders using traditional coding methods. We argue that LLM analysis can aid qualitative researchers by deductively coding transcripts, providing a systematic and reliable platform for code identification, and offering a means of avoiding analysis misalignment. Implications of using LLM in research praxis are discussed, along with current limitations.more » « less
-
The goal of this project is to better understand the beliefs that undergraduate students hold about their own intelligence and how these beliefs change during their undergraduate engineering education. The research team has used the theoretical framework established by Carol Dweck on Mindset and how different fixed and growth mindsets affect success. Fixed mindset individuals believe that their intelligence is an unchanging trait, while people with a growth mindset believe that through effort they can grow and develop greater intelligence. Prior researchers have shown that individuals with a growth mindset respond to challenges with higher levels of persistence, are more interested in improving upon past failures, and value criticism and effort more than those with a fixed mindset. The team developed an interview protocol from the theoretical framework. Then the team piloted the protocol and subsequently modified the protocol multiple times to ensure that the interviews provided rich qualitative data. Analytic memos were used to analyze and modify the piloted interview protocols. Once the final protocol was established, first-year and senior students were recruited to provide cross-sectional insight. The team also recruited using purposeful sampling to ensure that women and underrepresented minorities were included. To date, 19 interviews have been conducted with the final protocol. Of these interviews, four have been coded in detail using the “Attitudes, Values, and Beliefs” coding system. A codebook has also been started to categorize and interconnect the themes in the interview transcripts. This paper provides details of the protocol and coding process as well as preliminary findings on the themes extracted from the student interviews.more » « less
-
Background: Because of prior experience solving well-structured problems that have single, correct answers, students often struggle to direct their own design work and may not understand the need to frame ill-structured design problems. Purpose: Framing agency—defined as making decisions that are consequential to framing design problems and learning through this process—sheds light on students’ treatment of design problems; by framing, we mean the various actions designers take to understand, define, and bound the problem. Using the construct framing agency, we sought to characterize design team discourse to detect whether students treated design problems as ill- or well-structured and examine the consequences of this treatment. Method: Data were collected through extended participant observation of a capstone design course in a biomedical engineering program at a large research university. Data included audio and video records of design team meetings over the course of framing and solving industry-sponsored problems. For this paper, we analyzed three cases using sociolinguistic content analysis to characterize framing agency and compared the cases to illuminate the nuances of framing agency. Results: All teams faced impasses; one team navigated the impasse by framing the problem, whereas the others treated the problem as given. We identified markers of agency in students’ discourse, including tentative language, personal pronouns, and sharing ownership. Conclusions: Framing agency clarifies the kinds of learning experiences students need in order to overcome past experiences dominated by solving archetypical well-structured problems with predetermined solutions.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.18260/1-2--32536
