skip to main content

Title: Developing an Instrument to Measure Online Engineering Undergraduate Students’ Learning Experiences and Intentions to Persist
The purpose of this study is to develop an instrument to measure student perceptions about the learning experiences in their online undergraduate engineering courses. Online education continues to grow broadly in higher education, but the movement toward acceptance and comprehensive utilization of online learning has generally been slower in engineering. Recently, however, there have been indicators that this could be changing. For example, ABET has accredited online undergraduate engineering degrees at Stony Brook University and Arizona State University (ASU), and an increasing number of other undergraduate engineering programs also offer online courses. During this period of transition in engineering education, further investigation about the online modality in the context of engineering education is needed, and survey instrumentation can support such investigations. The instrument presented in this paper is grounded in a Model for Online Course-level Persistence in Engineering (MOCPE), which was developed by our research team by combining two motivational frameworks used to study student persistence: the Expectancy x Value Theory of Achievement Motivation (EVT), and the ARCS model of motivational design. The initial MOCPE instrument contained 79 items related to students’ perceptions about the characteristics of their courses (i.e., the online learning management system, instructor practices, and peer support), expectancies more » of course success, course task values, perceived course difficulties, and intention to persist in the course. Evidence of validity and reliability was collected using a three-step process. First, we tested face and content validity of the instrument with experts in online engineering education and online undergraduate engineering students. Next, the survey was administered to the online undergraduate engineering student population at a large, Southwestern public university, and an exploratory factor analysis (EFA) was conducted on the responses. Lastly, evidence of reliability was obtained by computing the internal consistency of each resulting scale. The final instrument has seven scales with 67 items across 10 factors. The Cronbach alpha values for these scales range from 0.85 to 0.97. The full paper will provide complete details about the development and psychometric evaluation of the instrument, including evidence of and reliability. The instrument described in this paper will ultimately be used as part of a larger, National Science Foundation-funded project investigating the factors influencing online undergraduate engineering student persistence. It is currently being used in the context of this project to conduct a longitudinal study intended to understand the relationships between the experiences of online undergraduate engineering students in their courses and their intentions to persist in the course. We anticipate that the instrument will be of interest and use to other engineering education researchers who are also interested in studying the population of online students. « less
Authors:
; ;
Award ID(s):
1825732
Publication Date:
NSF-PAR ID:
10188464
Journal Name:
Proceedings of the American Society for Engineering Education
Sponsoring Org:
National Science Foundation
More Like this
  1. This research paper examines the influence of interpersonal interactions on the course-level persistence intentions of online undergraduate engineering students. Online learning is increasing in enrollment and importance in engineering education. Online courses also continue to confront issues with comparatively higher course dropout levels than face-to-face courses. This study correspondingly explores relevant student perceptions of their online course experiences to better understand the factors that contribute to students’ choices to remain in or drop out of their online undergraduate engineering courses. Data presented in this study were collected during fall 2019 and spring 2020 from three ABET-accredited online undergraduate engineering courses at a large southwestern public university: electrical engineering, engineering management, and software engineering. Participants were asked to respond to surveys at 12-time points during their 7.5-week online course. Each survey measured students’ perceptions of course LMS dialog, perceptions of instructor practices, and peer support for completing the course. Participants also reported their intentions to persist in the course during each survey administration. A multi-level modeling analysis revealed that LMS dialog, perceptions of instructor practices, and peer support are related to course persistence intentions. Time was also a significant predictor of persistence intentions and indicated that the course persistence intentions decreasemore »towards the end of the course. Additionally, interactions between demographic variables and other predictors (perceptions of course LMS dialog, perceptions of instructor practices, and perceptions of peer support) were significant. With the increase in perceptions of course LMS dialog, perceptions of instructor practices, and perceptions of peer support, there was a relatively smaller increase in the persistence intentions of veterans than non-veterans. There is relatively more increase in the persistence intentions of females than males as their perceptions of instructor practices increase. Finally, increasing perceptions of peer support led to a relatively larger increase in the persistence intentions of non-transfer students than transfer students and a relatively smaller increase in persistence intentions of students working full-time than other students.« less
  2. The purpose of this study is to re-examine the validity evidence of the engineering design self-efficacy (EDSE) scale scores by Carberry et al. (2010) within the context of secondary education. Self-efficacy refers to individuals’ belief in their capabilities to perform a domain-specific task. In engineering education, significant efforts have been made to understand the role of self-efficacy for students considering its positive impact on student outcomes such as performance and persistence. These studies have investigated and developed measures for different domains of engineering self-efficacy (e.g., general academic, domain-general, and task-specific self-efficacy). The EDSE scale is a frequently cited measure that examines task-specific self-efficacy within the domain of engineering design. The original scale contains nine items that are intended to represent the engineering design process. Initial score validity evidence was collected using a sample consisting of 202 respondents with varying degrees of engineering experience including undergraduate/graduate students and faculty members. This scale has been primarily used by researchers and practitioners with engineering undergraduate students to assess changes in their engineering design self-efficacy as a result of active learning interventions, such as project-based learning. Our work has begun to experiment using the scale in a secondary education context in conjunction with anmore »increased introduction to engineering in K-12 education. Yet, there still is a need to examine score validity and reliability of this scale in non-undergraduate populations such as secondary school student populations. This study fills this important gap by testing construct validity of the original nine items of the EDSE scale, supporting proper use of the scale for researchers and practitioners. This study was conducted as part of a larger, e4usa project investigating the development and implementation of a yearlong project-based engineering design course for secondary school students. Evidence of construct validity and reliability was collected using a multi-step process. First, a survey that includes the EDSE scale was administered to the project participating students at nine associated secondary schools across the US at the beginning of Spring 2020. Analysis of collected data is in progress and includes Exploratory Factor Analysis (EFA) on the 137 responses. The evidence of score reliability will be obtained by computing the internal consistency of each resulting factor. The resulting factor structure and items will be analyzed by comparing it with the original EDSE scale. The full paper will provide details about the psychometric evaluation of the EDSE scale. The findings from this paper will provide insights on the future usage of the EDSE scale in the context of secondary engineering education.« less
  3. The United Nation’s Sustainable Development Goals state climate change could irreversibly affect future generations and is one of the most urgent issues facing society. To date, most education research on climate change examines middle and high school students’ knowledge without considering the link between understanding and interest to address such issues in their career. In research on students’ attitudes about sustainability, we found that half of first-year college engineering students, in our nationally representative sample of all college students at 4-year institutions (n = 937), do not believe climate change is caused by humans. This lack of belief in human-caused climate change is a significant problem in engineering education because our results also indicate engineering students who do not believe in human caused climate change are less likely to want to address climate change in their careers. This dismal finding highlights a need for improving student understanding and attitudes toward climate change in order to produce engineers prepared and interested in solving complex global problems in sustainability. To advance understanding about students’ understanding of climate change and their agency to address the issue, we developed the CLIMATE survey to measure senior undergraduate engineering students’ Climate change literacy, engineering identity, careermore »motivations, and agency through engineering. The survey was designed for students in their final senior design, or capstone course, just prior to entering the workforce. We developed the survey using prior national surveys and newly written questions categorized into six sections: (1) career goals and motivation, (2) college experiences, (3) agency, (4) climate literacy, (5) people and the planet, and (6) demographic information. We conducted focus groups with students to establish face and content validity of the survey. We collected pilot data with 200 engineering students in upper-level engineering courses to provide validity evidence for the use of these survey items to measure students and track changes across the undergraduate curriculum for our future work. In this paper, we narrate the development of the survey supported by literature and outline the next step for further validation and distribution on a national scale. Our intent is to receive feedback and input about the questions being asked and the CLIMATE instrument. Our objective is to share the nationally representative non-identifiable responses (the estimated goal is 4,000 responses) openly with education researchers interested in students understanding about climate change, their engineering identity, career motivations, and agency through engineering. Ultimately, we want this research to become a catalyst for teaching about topics related to climate change in engineering and its implications for sustainability.« less
  4. The United Nation’s Sustainable Development Goals state climate change could irreversibly affect future generations and is one of the most urgent issues facing society. To date, most education research on climate change examines middle and high school students’ knowledge without considering the link between understanding and interest to address such issues in their career. In research on students’ attitudes about sustainability, we found that half of first-year college engineering students, in our nationally representative sample of all college students at 4-year institutions (n = 937), do not believe climate change is caused by humans. This lack of belief in human-caused climate change is a significant problem in engineering education because our results also indicate engineering students who do not believe in human caused climate change are less likely to want to address climate change in their careers. This dismal finding highlights a need for improving student understanding and attitudes toward climate change in order to produce engineers prepared and interested in solving complex global problems in sustainability. To advance understanding about students’ understanding of climate change and their agency to address the issue, we developed the CLIMATE survey to measure senior undergraduate engineering students’ Climate change literacy, engineering identity, careermore »motivations, and agency through engineering. The survey was designed for students in their final senior design, or capstone course, just prior to entering the workforce. We developed the survey using prior national surveys and newly written questions categorized into six sections: (1) career goals and motivation, (2) college experiences, (3) agency, (4) climate literacy, (5) people and the planet, and (6) demographic information. We conducted focus groups with students to establish face and content validity of the survey. We collected pilot data with 200 engineering students in upper-level engineering courses to provide validity evidence for the use of these survey items to measure students and track changes across the undergraduate curriculum for our future work. In this paper, we narrate the development of the survey supported by literature and outline the next step for further validation and distribution on a national scale. Our intent is to receive feedback and input about the questions being asked and the CLIMATE instrument. Our objective is to share the nationally representative non-identifiable responses (the estimated goal is 4,000 responses) openly with education researchers interested in students understanding about climate change, their engineering identity, career motivations, and agency through engineering. Ultimately, we want this research to become a catalyst for teaching about topics related to climate change in engineering and its implications for sustainability.« less
  5. Giving a voice to marginalized groups and understanding the double bind is critical, especially after the Charlotte, VA protests and the white supremacist discourse that has pervaded our country. The result of the discourse, more subtle beliefs about white superiority and institutional barriers is an overrepresentation of women of color (WOC) in the leaky STEM pipeline and thus the loss of their presence and expertise. The absence of WOC hinders knowledge production and innovation that is essential for societal advancements and scientific discovery. The “chilly climate” is often cited as an explanation for the loss of WOC from STEM. However, interactions that allow the “chilly climate” to persist have yet to be characterized. This lack of understanding can inhibit the professional engineering identity construction of WOC. Additionally, engineering education research typically focuses on a single identity dimension such as gender or socio-economic status. These studies connect an identity dimension to student outcomes and few studies clarify how the identity is situated within the social context of the engineering culture. Consequently, a need exist to examine how the engineering culture impacts multiple components of identity and intersecting identities of WOC. To address this gap, our study illuminates the intersections of identitymore »of WOC and how they perceive the double bind of race and gender within the context of their engineering education. The data reported here are a part of a larger, sequential mixed-methods study (N=276) of undergraduate female engineering students at a large Midwestern research university. This project applies the framework of intersectionality with the following scales: Engineering Identity, Ethnic Identity, Womanist Identity, Microaggressions, and Depression. We use intersectionality to investigate the interaction between intersecting social identities and educational conditions. We introduce the Womanist Identity Attitude scale to engineering education research, which provides an efficient way to understand gender, racial, and intersecting identity development of WOC. We utilize the microaggressions scale, in order to develop quantitative measures of gender-racial discrimination in STEM and compare to previous research. We also included the Patient Health Questionnaire (PHQ-9), an instrument for measuring depression, to assess health outcomes of respondents’ experiences of gender-racial microaggressions. Our three emergent findings suggest instrument accuracy and provide insight into the identity and depression subscales. Factor analysis established a basis to refine our quantitative survey instruments, and indicated that 23 items could offer greater accuracy than the original 54 items instrument. Second, the majority of participants report a high level of identification with engineering. This result rebuffs the long-held stereotypes that females are less interested in engineering. Third, a significant portion of female respondents self-reported PHQ-9 scores in the 15-19 range, which corresponds with a “major depression, moderately severe” provisional diagnosis, the second-highest in severity in the PHQ-9 provisional diagnosis scale. These elevated levels of depression correlated significantly to frequent instances of microaggressions. These preliminary findings are providing never-before seen insight into the experiences of WOC in engineering. Our results suggest a path to accurately describe the experiences of WOC in engineering, while revealing options for improving inclusion efforts.« less