An official website of the United States government
This Research paper discusses the opportunities that utilizing a computer program can present in analyzing large amounts of qualitative data collected through a survey tool. When working with longitudinal qualitative data, there are many challenges that researchers face. The coding scheme may evolve over time requiring re-coding of early data. There may be long periods of time between data analysis. Typically, multiple researchers will participate in the coding, but this may introduce bias or inconsistencies. Ideally the same researchers would be analyzing the data, but often there is some turnover in the team, particularly when students assist with the coding. Computer programs can enable automated or semi-automated coding helping to reduce errors and inconsistencies in the coded data. In this study, a modeling survey was developed to assess student awareness of model types and administered in four first-year engineering courses across the three universities over the span of three years. The data collected from this survey consists of over 4,000 students’ open-ended responses to three questions about types of models in science, technology, engineering, and mathematics (STEM) fields. A coding scheme was developed to identify and categorize model types in student responses. Over two years, two undergraduate researchers analyzed a total of 1,829 students’ survey responses after ensuring intercoder reliability was greater than 80% for each model category. However, with much data remaining to be coded, the research team developed a MATLAB program to automatically implement the coding scheme and identify the types of models students discussed in their responses. MATLAB coded results were compared to human-coded results (n = 1,829) to assess reliability; results matched between 81%-99% for the different model categories. Furthermore, the reliability of the MATLAB coded results are within the range of the interrater reliability measured between the 2 undergraduate researchers (86-100% for the five model categories). With good reliability of the program, all 4,358 survey responses were coded; results showing the number and types of models identified by students are presented in the paper.
more »
« less
Mitigating Ceiling Effects in a Longitudinal Study of Doctoral Engineering Student Stress and Persistence
Aim/Purpose: The research reported here aims to demonstrate a method by which novel applications of qualitative data in quantitative research can resolve ceiling effect tensions for educational and psychological research.Background: Self-report surveys and scales are essential to graduate education and social science research. Ceiling effects reflect the clustering of responses at the highest response categories resulting in non-linearity, a lack of variability which inhibits and distorts statistical analyses. Ceiling effects in stress reported by students can negatively impact the accuracy and utility of the resulting data.Methodology: A longitudinal sample example from graduate engineering students’ stress, open-ended critical events, and their early departure from doctoral study considerations demonstrate the utility and improved accuracy of adjusted stress measures to include open-ended critical event responses. Descriptive statistics are used to describe the ceiling effects in stress data and adjusted stress data. The longitudinal stress ratings were used to predict departure considerations in multilevel modeling ANCOVA analyses and demonstrate improved model predictiveness.Contribution: Combining qualitative data from open-ended responses with quantitative survey responses provides an opportunity to reduce ceiling effects and improve model performance in predicting graduate student persistence. Here, we present a method for adjusting stress scale responses by incorporating coded critical events based on the Taxonomy of Life Events, the application of this method in the analysis of stress responses in a longitudinal data set, and potential applications.Findings: The resulting process more effectively represents the doctoral student experience within statistical analyses. Stress and major life events significantly impact engineering doctoral students’ departure considerations.Recommendations for Practitioners: Graduate educators should be aware of students’ life events and assist students in managing graduate school expectations while maintaining progress toward their degree. Recommendation for Researchers: Integrating coded open-ended qualitative data into statistical models can increase the accuracy and representation of the lived student experience. The new approach improves the accuracy and presentation of students’ lived experiences by incorporating qualitative data into longitudinal analyses. The improvement assists researchers in correcting data with ceiling effects for use in longitudinal analyses.Impact on Society: The method described here provides a framework to systematically include open-ended qualitative data in which ceiling effects are present.Future Research: Future research should validate the coding process in similar samples and in samples of doctoral students in different fields and master’s students.
more »
« less
- Award ID(s):
- 1844878
- PAR ID:
- 10464169
- Publisher / Repository:
- Informing Science Institute
- Date Published:
- Journal Name:
- IJEE International Journal of Engineering Education
- Volume:
- 39
- Issue:
- 1
- ISSN:
- 2540-9808
- Page Range / eLocation ID:
- 199 to 227
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Purpose The purpose of this study is to examine how student agency influences career decision-making for doctoral students in biological sciences. The authors address the following questions: How do biological science graduate students navigate career indecision? And how does agency relate to their experiences with career indecision? Design/methodology/approach The authors analyzed interview data collected from 84 PhD biology graduate students. Researchers used a grounded theory approach. After open codes were developed and data were coded, code reports were generated, which were used to determine themes. Findings More than half of the sample had not committed to a career path, and undecided students were bifurcated into two categories: Uncommitted and Uncertain. Uncommitted graduate students demonstrated agency in their approach and were focused on exploration and development. Uncertain students demonstrated less agency, were more fearful and perceived less control and clarity about their options and strategies to pursue career goals. Practical implications Findings suggest some forms of indecision can be productive and offer institutional leaders guidance for increasing the efficacy of career development and exploration programming. Originality/value Research on doctoral student career decision-making is often quantitative and rarely explores the role of agency. This qualitative study focuses on the relationship between student agency and career indecision, which is an understudied aspect of career development.more » « less
-
A psychologically safe environment is characterized by people who feel safe to voice ideas and concerns, willingly seek feedback, have positive intentions to one another, engage in constructive confrontation, and feel safe to take risks and experiment. Outside of academia, psychological safety has been shown to impact creativity, work performance, and work engagement. In academic research environments, faculty have a major leadership role in cultivating a psychologically safe environment amongst their academic research teams. Effective graduate student mentoring, which includes both career and psychosocial support, is critical to the development and retention of talented engineers in the US workforce. There is a need to better understand how engineering departments can cultivate more inclusive, psychologically safe environments in which graduate students feel safe to engage in interpersonal risk-taking, especially in research settings. Guided by the Conservation of Resources theory, this project aims to address the following research question: What are the relationships between faculty advisor mentoring, doctoral student psychological safety, and the subsequent positive and negative outcomes for doctoral students? This work in progress paper presents the first quantitative phase of an explanatory mixed methods research design within the overarching project. The quantitative phase will address the following research aims: 1) Identify relationships between mentorship, psychological safety, and engineering doctoral student mental health, 2) Identify mentoring competencies that are predictive of research group psychological safety, and 3) Identify how different demographics experience mentoring and psychological safety in their research groups. Researchers developed a survey consisting of five pre-existing scales, four open-ended questions, and demographics questions. The scales include dyadic and team psychological safety, mentoring competency, mental health and well-being, and job stress. The survey was reviewed by graduate students outside of the participant pool at multiple institutions as well as an external advisory board panel and revised to improve clarity and ensure the selection of appropriate subscales. The survey will be administered via Qualtrics. Graduate students who have been enrolled in their doctoral program for at least one year and currently have a doctoral research advisor will be recruited to participate in the survey at four public, research-intensive institutions. The planned target sample size is 200-300 graduate students. This paper will present the design of the survey and preliminary survey results. As the first part of a larger mixed-methods study, the survey responses provide insight into graduate level engineering education and how doctoral students can be better supported.more » « less
-
Doctoral students experience high rates of mental health distress and dropout; however, the mental health and wellness of engineering doctoral students is understudied. Studies of student persistence, wellness, and success often aggregate fields together, such as by studying all engineering students. Thus, little work has considered the experiences of biomedical engineering (BME) doctoral students, despite differences between doctoral BME research, course content, and career expectations compared with other engineering disciplines. In this qualitative interview case study, we explore stressors present in the BME graduate experience that are unique from engineering students in other disciplines. Methods We analyzed a longitudinal interview study of doctoral engineering students across four timepoints within a single academic year, consisting of a subsample (n=6) of doctoral students in a BME discipline, among a larger sample of engineering doctoral students (N=55). BME students in the sample experienced some themes generated from a larger thematic analysis differently compared with other engineering disciplines. These differences are presented and discussed, grounded in a model of workplace stress. Results BME participants working in labs with biological samples expressed a lack of control over the timing and availability of materials for their research projects. BME participants also had more industry-focused career plans and described more commonly coming to BME graduate studies from other fields (e.g., another engineering major) and struggling with the scope and content of their introductory coursework. A common throughline for the stressors was the impact of the interdisciplinary nature of BME programs, to a greater extent compared with other engineering student experiences in our sample. Conclusions We motivate changes for researchers, instructors, and policymakers which specifically target BME students and emphasize the importance of considering studies at various unit levels (university department level vs college level vs full institution) when considering interventions targeting student stress and wellness.more » « less
-
Abstract BackgroundAlthough most engineering graduate students are funded and usually complete their degrees faster than other disciplines, attrition remains a problem in engineering. Existing research has explored the psychological and sociological factors contributing to attrition but not the structural factors impacting attrition. Purpose/HypothesisUsing systems theory, this study seeks to understand nuance in how underlying structural causes affect engineering graduate students' attrition experiences in ways that may differ from their official reasons for departure. Design/MethodsData were collected through semi‐structured interviews with seven departing or already departed engineering doctoral students from R1 graduate programs across the United States. Using thematic analysis, root cause analyses were conducted to understand participants' attrition experiences to explore how structures influence causes of departure. ResultsThe ways participants discuss root causes of their departure indicate differences in formal reasons for departure and underlying causes of departure. We highlight the role of informal and formal policy as root causes of a different attrition rationale often passed off as interpersonal issues. When interpreted as evidence of structural issues, the causes of departure show ways in which action–inaction, policy–“null” policy serve as structural features governing student attrition decision processes. We also highlight a form of benign neglect toward struggling graduate students. ConclusionThis study reveals important nuances underlying face‐value reasons of attrition indicating foundational structural issues contributing to engineering graduate student attrition. Coaching faculty in team management and encouraging close revision of departmental policies could help mitigate students' negative graduate experiences and decrease unnecessary attrition.more » « less
