skip to main content


Title: Board 90: EAGER: Measuring Student Support in STEM: Insights from Year 2
The purpose of the project is to identify how to measure various types of institutional support as it pertains to underrepresented and underserved populations in colleges of engineering and science. We are grounding this investigation in the Model of Co-Curricular Support, a conceptual framework that emphasizes the breadth of assistance currently used to support undergraduate students in engineering and science. The results from our study will help prioritize the elements of institutional support that should appear somewhere in a college’s suite of support efforts to improve engineering and science learning environments and design effective programs, activities, and services. Our poster will present: 1) an overview of the instrument development process; 2) evaluation of the prototype for face and content validity from students and experts; and 3) instrument revision and data collection to determine test validity and reliability across varied institutional contexts. In evaluating the initial survey, we included multiple rounds of feedback from students and experts, receiving feedback from 46 participants (38 students, 8 administrators). We intentionally sampled for representation across engineering and science colleges; gender identity; race/ethnicity; international student status; and transfer student status. The instrument was deployed for the first time in Spring 2018 to the institutional project partners at three universities. It was completed by 722 students: 598 from University 1, 51 from University 2, and 123 from University 3. We tested the construct validity of these responses using a minimum residuals exploratory factor analysis and correlation. A preliminary data analysis shows evidence of differences in perception on types of support college of engineering and college of science students experience. The findings of this preliminary analysis were used to revise the instrument further prior to the next round of testing. Our target sample for the next instrument deployment is 2,000 students, so we will survey ~13,000 students based on a 15% anticipated response rate. Following data collection, we will use confirmatory factor analysis to continue establishing construct validity and report on the stability of constructs emerging from our piloting on a new student sample(s). We will also investigate differences across these constructs by subpopulations of students.  more » « less
Award ID(s):
1704350
NSF-PAR ID:
10146796
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
ASEE Annual Conference & Exposition
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Research shows that the LGBTQ climate in engineering, and other STEM, undergraduate degree programs is rife with heteronormativity and cissexism, leading LGBTQ students to leave STEM majors and careers at higher rates than their heterosexual, cisgender peers. In order to develop a diverse STEM workforce and adequately prepare the next generation of professionals in STEM, higher education, and especially engineering education, must address inequities such as these to ensure broad participation in STEM fields. This NSF CAREER-funded project helps meet this need by examining the participation of LGBTQ students in STEM fields. The project focuses on three primary research aims to address this purpose: test the relationships between the composition of LGBTQ students’ social networks and non-cognitive STEM outcomes, compare STEM degree completion rates between LGBTQ students and their cisgender, heterosexual peers, and explore the intersection of STEM discipline-based identity (e.g., engineering identity, science identity) with sexual and gender identity. This project stands to improve our understanding of how to broaden participation in STEM by pursuing robust research efforts that illuminate the ways sexual and gender identity shape trajectories into, through, and out of STEM. The purpose of this poster is to present preliminary outcomes from the first research aim of the project, which is to test the relationship between composition of students’ social networks and non-cognitive outcomes, and compare these relationships by sexual and gender identities. We hypothesize that homophily within students’ social networks, especially for heterosexual and cisgender students, will predict greater levels of identification with one’s STEM discipline, sense of belonging in STEM, and commitment to a STEM major. LGBTQ students whose LGBTQ connections are primarily outside STEM are hypothesized to feel more of a pull away from STEM. This poster focuses on the social network analysis phase of the project, including instrument development, data collection procedures, and preliminary analysis of the data. Data collection will commence in the spring 2022 semester. Social network analysis (SNA) is a method that measures and represents the patterns and information of contextually bound structural relationships to explain why the relationships occur and the outcomes of their existence, and SNA is only recently gaining ground in educational research. We developed a survey that incorporates generating an ego-centric social network, or the people an individual relies on most for support, with existing measures for sense of belonging, discipline-based identity, and commitment to field of study, adapted for this study’s purpose. The survey validation procedure included cognitive interviews with undergraduate students and expert reviews by engineering education and institutional research experts. Data collection will occur at five colleges and universities nation-wide, representing a range of institutional types, geographical diversity, and student body diversity. The poster will detail the theory and procedures that constitute SNA research, the survey development process for this phase of the project, and preliminary results from analysis of the data. 
    more » « less
  2. High levels of stress and anxiety are common amongst college students, particularly engineering students. Students report lack of sleep, grades, competition, change in lifestyle, and other significant stressors throughout their undergraduate education (1, 2). Stress and anxiety have been shown to negatively impact student experience (3-6), academic performance (6-8), and retention (9). Previous studies have focused on identifying factors that cause individual students stress while completing undergraduate engineering degree programs (1). However, it not well-understood how a culture of stress is perceived and is propagated in engineering programs or how this culture impacts student levels of identification with engineering. Further, the impact of student stress has not been directly considered in engineering regarding recruitment, retention, and success. Therefore, our guiding research question is: Does the engineering culture create stress for students that hinder their engineering identity development? To answer our research question, we designed a sequential mixed methods study with equal priority of quantitative survey data and qualitative individual interviews. Our study participants are undergraduate engineering students across all levels and majors at a large, public university. Our sample goal is 2000 engineering student respondents. We combined three published surveys to build our quantitative data collection instrument, including the Depression Anxiety Stress Scales (DASS), Identification with engineering subscale, and Engineering Department Inclusion Level subscale. The objective of the quantitative instrument is to illuminate individual perceptions of the existence of an engineering stress culture (ESC) and create an efficient tool to measure the impact ESC on engineering identity development. Specifically, we seek to understand the relationships among the following constructs; 1) identification with engineering, 2) stress and anxiety, and 3) feelings of inclusion within their department. The focus of this paper presents the results of the pilot of the proposed instrument with 20 participants and a detailed data collection and analysis process. In an effort to validate our instrument, we conducted a pilot study to refine our data collection process and the results will guide the data collection for the larger study. In addition to identifying relationships among construct, the survey data will be further analyzed to specify which demographics are mediating or moderating factors of these relationships. For example, does a student’s 1st generation status influence their perception of stress or engineering identity development? Our analysis may identify discipline-specific stressors and characterize culture components that promote student anxiety and stress. Our objective is to validate our survey instrument and use it to inform the protocol for the follow-up interviews to gain a deeper understanding of the responses to the survey instrument. Understanding what students view as stressful and how students identify stress as an element of program culture will support the development of interventions to mitigate student stress. References 1. Schneider L (2007) Perceived stress among engineering students. A Paper Presented at St. Lawrence Section Conference. Toronto, Canada. Retrieved from: www. asee. morrisville. edu. 2. Ross SE, Niebling BC, & Heckert TM (1999) Sources of stress among college students. Social psychology 61(5):841-846. 3. Goldman CS & Wong EH (1997) Stress and the college student. Education 117(4):604-611. 4. Hudd SS, et al. (2000) Stress at college: Effects on health habits, health status and self-esteem. College Student Journal 34(2):217-228. 5. Macgeorge EL, Samter W, & Gillihan SJ (2005) Academic Stress, Supportive Communication, and Health A version of this paper was presented at the 2005 International Communication Association convention in New York City. Communication Education 54(4):365-372. 6. Burt KB & Paysnick AA (2014) Identity, stress, and behavioral and emotional problems in undergraduates: Evidence for interaction effects. Journal of college student development 55(4):368-384. 7. Felsten G & Wilcox K (1992) Influences of stress and situation-specific mastery beliefs and satisfaction with social support on well-being and academic performance. Psychological Reports 70(1):291-303. 8. Pritchard ME & Wilson GS (2003) Using emotional and social factors to predict student success. Journal of college student development 44(1):18-28. 9. Zhang Z & RiCharde RS (1998) Prediction and Analysis of Freshman Retention. AIR 1998 Annual Forum Paper. 
    more » « less
  3. The purpose of this work-in-progress paper is to share insights from current efforts to develop and test the validity of an instrument to measure undergraduate students’ perceived support in science, technology, engineering, and mathematics (STEM). The development and refinement of our survey instrument ultimately functions to extend, operationalize, and empirically test the Model of Co-curricular Support (MCCS). The MCCS is a conceptual framework of student support that demonstrates the breadth of assistance currently used to support undergraduate students in STEM, particularly those from underrepresented groups. We are currently gathering validity evidence for an instrument that evaluates the extent to which colleges of engineering and science offer supportive environments. To date, exploratory factor analysis and correlation for construct validity have helped us develop 14 constructs for student support in STEM. Future work will focus on modeling relationships between these constructs and student outcomes, providing the explanatory power needed to explain empirically how co-curricular supports contribute to different forms of student success in STEM. We hope that operationalizing the MCCS through this survey will shift how we conceptualize and offer student support, enabling college administrators and student support practitioners to evaluate their portfolio of student support efforts. 
    more » « less
  4. Student-retention theories traditionally focus on institutional retention, even though efforts to support students in science, technology, engineering, and mathematics (STEM) occur at the college level. This study bridges this gap between research and practice by extending and empirically testing the Model of Co-Curricular Support (MCCS), which specifically focuses on supporting and retaining underrepresented groups in STEM. The MCCS is a student-retention model that demonstrates the breadth of assistance currently used to support undergraduate students in STEM, particularly those from underrepresented groups. The aim of this exploratory research is to develop and validate a survey instrument grounded in the MCCS that can be used by college administrators and student-support practitioners to assess the magnitude of institutional support received by undergraduate students in STEM. To date, such an instrument does not exist. Our poster will present a summary of the instrument development process that has occurred to date. We are developing the survey following best practices outlined in the literature. We are clearly defining the construct of interest and target population; reviewing related tests; developing the prototype of the survey instrument; evaluating the prototype for face and content validity from students and experts; revising and testing based on suggestion; and collecting data to determine test validity and reliability across four institutional contexts. Our institutional sample sites were purposefully selected because of their large size and diversity with respect to undergraduates in STEM. The results from our study will help prioritize the elements of institutional support that should appear somewhere in a college’s suite of support efforts. Our study will provide scientific evidence that STEM researchers, educators, administrators, and policy makers need to make informed decisions to improve STEM learning environments and design effective programs, activities, and services. 
    more » « less
  5. The purpose of this study is to develop an instrument to measure student perceptions about the learning experiences in their online undergraduate engineering courses. Online education continues to grow broadly in higher education, but the movement toward acceptance and comprehensive utilization of online learning has generally been slower in engineering. Recently, however, there have been indicators that this could be changing. For example, ABET has accredited online undergraduate engineering degrees at Stony Brook University and Arizona State University (ASU), and an increasing number of other undergraduate engineering programs also offer online courses. During this period of transition in engineering education, further investigation about the online modality in the context of engineering education is needed, and survey instrumentation can support such investigations. The instrument presented in this paper is grounded in a Model for Online Course-level Persistence in Engineering (MOCPE), which was developed by our research team by combining two motivational frameworks used to study student persistence: the Expectancy x Value Theory of Achievement Motivation (EVT), and the ARCS model of motivational design. The initial MOCPE instrument contained 79 items related to students’ perceptions about the characteristics of their courses (i.e., the online learning management system, instructor practices, and peer support), expectancies of course success, course task values, perceived course difficulties, and intention to persist in the course. Evidence of validity and reliability was collected using a three-step process. First, we tested face and content validity of the instrument with experts in online engineering education and online undergraduate engineering students. Next, the survey was administered to the online undergraduate engineering student population at a large, Southwestern public university, and an exploratory factor analysis (EFA) was conducted on the responses. Lastly, evidence of reliability was obtained by computing the internal consistency of each resulting scale. The final instrument has seven scales with 67 items across 10 factors. The Cronbach alpha values for these scales range from 0.85 to 0.97. The full paper will provide complete details about the development and psychometric evaluation of the instrument, including evidence of and reliability. The instrument described in this paper will ultimately be used as part of a larger, National Science Foundation-funded project investigating the factors influencing online undergraduate engineering student persistence. It is currently being used in the context of this project to conduct a longitudinal study intended to understand the relationships between the experiences of online undergraduate engineering students in their courses and their intentions to persist in the course. We anticipate that the instrument will be of interest and use to other engineering education researchers who are also interested in studying the population of online students. 
    more » « less