More Like this

1. Exploring the Validity of the Engineering Design Self-Efficacy Scale for Secondary School Students (Research To Practice)

   Lee, E.; Carberry, A. R.; Dalal, M.; Miller, M. J. (July 2021, 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference)

   null (Ed.)

   The purpose of this study is to re-examine the validity evidence of the engineering design self-efficacy (EDSE) scale scores by Carberry et al. (2010) within the context of secondary education. Self-efficacy refers to individuals’ belief in their capabilities to perform a domain-specific task. In engineering education, significant efforts have been made to understand the role of self-efficacy for students considering its positive impact on student outcomes such as performance and persistence. These studies have investigated and developed measures for different domains of engineering self-efficacy (e.g., general academic, domain-general, and task-specific self-efficacy). The EDSE scale is a frequently cited measure that examines task-specific self-efficacy within the domain of engineering design. The original scale contains nine items that are intended to represent the engineering design process. Initial score validity evidence was collected using a sample consisting of 202 respondents with varying degrees of engineering experience including undergraduate/graduate students and faculty members. This scale has been primarily used by researchers and practitioners with engineering undergraduate students to assess changes in their engineering design self-efficacy as a result of active learning interventions, such as project-based learning. Our work has begun to experiment using the scale in a secondary education context in conjunction with an increased introduction to engineering in K-12 education. Yet, there still is a need to examine score validity and reliability of this scale in non-undergraduate populations such as secondary school student populations. This study fills this important gap by testing construct validity of the original nine items of the EDSE scale, supporting proper use of the scale for researchers and practitioners. This study was conducted as part of a larger, e4usa project investigating the development and implementation of a yearlong project-based engineering design course for secondary school students. Evidence of construct validity and reliability was collected using a multi-step process. First, a survey that includes the EDSE scale was administered to the project participating students at nine associated secondary schools across the US at the beginning of Spring 2020. Analysis of collected data is in progress and includes Exploratory Factor Analysis (EFA) on the 137 responses. The evidence of score reliability will be obtained by computing the internal consistency of each resulting factor. The resulting factor structure and items will be analyzed by comparing it with the original EDSE scale. The full paper will provide details about the psychometric evaluation of the EDSE scale. The findings from this paper will provide insights on the future usage of the EDSE scale in the context of secondary engineering education. 

   more » « less
2. Towards a Survey Instrument for Use In Proactive Advising

   https://doi.org/10.18260/1-2--48167  

   West, Kenneth; Carroll, Bruce; Shin, Jinnie; Crippen, Kent (June 2024, ASEE Conferences)

   This paper introduces the pilot implementation of the Evidence Based Personas survey instrument for assessing non-cognitive attributes of relevance from undergraduate students at different stages of their engineering degree for the purpose of informing proactive advising processes. The survey instrument was developed with two key objectives: first, to assess its potential for streamlining and shortening existing instruments, and second, to explore the possibility of consolidating items from different surveys that measure the same or closely related constructs. A proactive advising system is being developed that uses the Mediation Model of Research Experiences (MMRE) as a framework. Within this framework, participation in various educational activities is linked to increased Commitment to Engineering via three mediating parameters: Self-Efficacy, Teamwork/Leadership Self-Efficacy, and Engineering Identity. The existing, validated MMRE survey instrument was used as a starting point for development of the current instrument with a goal of streamlining / shortening the number of questions. Ultimately, we envision augmenting the shortened instrument with items related to broader non-cognitive and affective constructs from the SUCCESS instrument. Noting that both the MMRE and SUCCESS instruments include measures of Self-Efficacy and Engineering Identity, selected questions from both were included and compared. Data was collected from 395 total respondents, and subsequent data analysis was based on 337 valid participants. Factor Analysis techniques, both exploratory and confirmatory, were employed to uncover underlying or latent variables within the results, particularly in the areas of Self-Efficacy where the combined items of the SUCCESS instrument and the MMRE instrument were used. Cronbach’s alpha analysis was employed to assess the internal consistency of the survey instrument. The Teamwork, Engineering Identity, and Commitment to Engineering constructs all produced a Cronbach’s alpha value in excess of 0.80. The Self-Efficacy construct fell below the 0.80 threshold at 0.77 which is considered to be respectable but is indicative of some short comings compared to that of the other constructs. The results of the EFA four-factor pattern matrix show the SUCCESS instrument items breaking out into their own components while the MMRE items merge with some of the items from the Engineering Identity construct suggesting a distinction in the underlying concepts these items may be measuring. This finding is further supported in the CFA through an assessment of the Goodness of Fit (GFI), Tucker-Lewis Index (TLI), and Root Mean Square Error of Approximation (RMSEA) of these constructs. The initial groupings of the four constructs produced a robust CFI value of 0.853, robust TLI value of 0.838, and a robust RMSEA value of 0.075. Self-Efficacy is broken out into two sub-scales one defined by the three items from the SUCCESS instrument and the other defined by the four remaining items from the MMRE instrument. Engineering Identity was also broken into two sub-scales. The robust CFI and TLI report values of 0.928 and 0.919 respectively, and the robust RMSEA is reported to be 0.053. The findings of the factor analyses indicate that a shortened form of the MMRE survey instrument will provide reliable measures of the underlying constructs. Additionally, the results suggest that the self-efficacy as measured by items from the MMRE and from the SUCCESS instruments are related to two separate aspects of self-efficacy and do not load well into a single factor. 

   more » « less
3. Developing an Instrument to Measure Online Engineering Undergraduate Students’ Learning Experiences and Intentions to Persist

   Lee, E.; Brunhaver, S.; Bekki, J. (January 2020, Proceedings of the American Society for Engineering Education)

   The purpose of this study is to develop an instrument to measure student perceptions about the learning experiences in their online undergraduate engineering courses. Online education continues to grow broadly in higher education, but the movement toward acceptance and comprehensive utilization of online learning has generally been slower in engineering. Recently, however, there have been indicators that this could be changing. For example, ABET has accredited online undergraduate engineering degrees at Stony Brook University and Arizona State University (ASU), and an increasing number of other undergraduate engineering programs also offer online courses. During this period of transition in engineering education, further investigation about the online modality in the context of engineering education is needed, and survey instrumentation can support such investigations. The instrument presented in this paper is grounded in a Model for Online Course-level Persistence in Engineering (MOCPE), which was developed by our research team by combining two motivational frameworks used to study student persistence: the Expectancy x Value Theory of Achievement Motivation (EVT), and the ARCS model of motivational design. The initial MOCPE instrument contained 79 items related to students’ perceptions about the characteristics of their courses (i.e., the online learning management system, instructor practices, and peer support), expectancies of course success, course task values, perceived course difficulties, and intention to persist in the course. Evidence of validity and reliability was collected using a three-step process. First, we tested face and content validity of the instrument with experts in online engineering education and online undergraduate engineering students. Next, the survey was administered to the online undergraduate engineering student population at a large, Southwestern public university, and an exploratory factor analysis (EFA) was conducted on the responses. Lastly, evidence of reliability was obtained by computing the internal consistency of each resulting scale. The final instrument has seven scales with 67 items across 10 factors. The Cronbach alpha values for these scales range from 0.85 to 0.97. The full paper will provide complete details about the development and psychometric evaluation of the instrument, including evidence of and reliability. The instrument described in this paper will ultimately be used as part of a larger, National Science Foundation-funded project investigating the factors influencing online undergraduate engineering student persistence. It is currently being used in the context of this project to conduct a longitudinal study intended to understand the relationships between the experiences of online undergraduate engineering students in their courses and their intentions to persist in the course. We anticipate that the instrument will be of interest and use to other engineering education researchers who are also interested in studying the population of online students. 

   more » « less
4. Development of the Chemistry Mindset Instrument (CheMI) for use with introductory undergraduate chemistry students

   https://doi.org/10.1039/D2RP00102K  

   Santos, Deborah L.; Barbera, Jack; Mooring, Suazette R. (July 2022, Chemistry Education Research and Practice)

   Chemistry education research has increasingly considered the role of affect when investigating chemistry learning environments over the past decade. Despite its popularity in educational spheres, mindset has been understudied from a chemistry-specific perspective. Mindset encompasses one's beliefs about the ability to change intelligence with effort and has been shown to be a domain-specific construct. For this reason, students’ mindset would be most relevant in chemistry if it were measured as a chemistry-specific construct. To date, no instrument has been developed for use in chemistry learning contexts. Here we present evidence supporting the development process and final product of a mindset instrument designed specifically for undergraduate chemistry students. The Chemistry Mindset Instrument (CheMI) was developed through an iterative design process requiring multiple implementations and revisions. We analyze the psychometric properties of CheMI data from a sample of introductory (general and organic) chemistry students enrolled in lecture courses. We achieved good data-model fit via confirmatory factor analysis and high reliability for the newly developed items, indicating that the instrument functions well with the target population. Significant correlations were observed for chemistry mindset with students’ self-efficacy, mastery goals, and course performance, providing external validity evidence for the construct measurement. 

   more » « less
5. Assessing Drawing Self-efficacy: A Validation Study Using Exploratory Factor Analysis (EFA) for the Drawing Self-efficacy Instrument (DSEI)

   https://doi.org/10.18260/1-2--36704  

   Jaison, Donna; Merzdorf, Hillary; Williford, Blake; White, Lance; Watson, Karan; Douglas, Kerrie; Hammond, Tracy (July 2021, 2021 ASEE Virtual Annual Conference)

   Drawing, as a skill, is closely tied to many creative fields and it is a unique practice for every individual. Drawing has been shown to improve cognitive and communicative abilities, such as visual communication, problem-solving skills, students’ academic achievement, awareness of and attention to surrounding details, and sharpened analytical skills. Drawing also stimulates both sides of the brain and improves peripheral skills of writing, 3-D spatial recognition, critical thinking, and brainstorming. People are often exposed to drawing as children, drawing their families, their houses, animals, and, most notably, their imaginative ideas. These skills develop over time naturally to some extent, however, while the base concept of drawing is a basic skill, the mastery of this skill requires extensive practice and it can often be significantly impacted by the self-efficacy of an individual. Sketchtivity is an AI tool developed by Texas A&M University to facilitate the growth of drawing skills and track their performance. Sketching skill development depends in part on students’ self-efficacy associated with their drawing abilities. Gauging the drawing self-efficacy of individuals is critical in understanding the impact that this drawing practice has had with this new novel instrument, especially in contrast to traditional practicing methods. It may also be very useful for other researchers, educators, and technologists. This study reports the development and initial validation of a new 13-item measure that assesses perceived drawing self efficacy. The13 items to measure drawing self efficacy were developed based on Bandura’s guide for constructing Self-Efficacy Scales. The participants in the study consisted of 222 high school students from engineering, art, and pre-calculus classes. Internal consistency of the 13 observed items were found to be very high (Cronbach alpha: 0.943), indicating a high reliability of the scale. Exploratory Factor Analysis was performed to further investigate the variance among the 13 observed items, to find the underlying latent factors that influenced the observed items, and to see if the items needed revision. We found that a three model was the best fit for our data, given fit statistics and model interpretability. The factors are: Factor 1: Self-efficacy with respect to drawing specific objects; Factor 2: Self-efficacy with respect to drawing practically to solve problems, communicating with others, and brainstorming ideas; Factor 3: Self-efficacy with respect to drawing to create, express ideas, and use one’s imagination. An alternative four-factor model is also discussed. The purpose of our study is to inform interventions that increase self-efficacy. We believe that this assessment will be valuable especially for education researchers who implement AI-based tools to measure drawing skills.This initial validity study shows promising results for a new measure of drawing self-efficacy. Further validation with new populations and drawing classes is needed to support its use, and further psychometric testing of item-level performance. In the future, this self-efficacy assessment could be used by teachers and researchers to guide instructional interventions meant to increase drawing self-efficacy. 

   more » « less