skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • The Continued Development and Validity Testing of an Engineering Design Value-Expectancy Scale (EDVES) for High School Students
Title: The Continued Development and Validity Testing of an Engineering Design Value-Expectancy Scale (EDVES) for High School Students
As more institutions create first year engineering programs that teach an engineering design process, there is a growing desire to prepare students for this coursework in the high school setting. When exposing such a broad population to these ideas, a primary question arises regarding student attitudes toward engineering and how these attitudes develop over time. That is, how does this exposure to engineering design influence student attitudes toward engineering? Moreover, answering this question will allow educators to better understand what motivates students to learn, how much their motivation impacts their overall mastery of these skills, and how these aspects of engineering self-efficacy and engineering design may differ between those who are on a pre-engineering track and those who are not. To begin answering this question, high school students enrolled in the Olathe City school system of Olathe, Kansas completed Engineering Problem-Framing Design Activities (EPDAs) in participating science courses (AP physics, physics, advanced biotechnology, chemistry, honors chemistry, biology, honors biology, and physical science specifically) of the traditional science and engineering academy curriculums offered by the district. Student engineering self-efficacy and motivation was also measured at the beginning and end of their coursework. This was conducted via a new instrument, the Engineering Design Value-Expectancy Scale (EDVES), which includes 38 items across three primary subscales: expectancy of success in, perceived value of, and identification with engineering and design. The development of this tool was presented and discussed in a previous study where the EDVES instrument was analyzed for validity among first-year undergraduate engineering students. In this work, the responses of high school students on the EDVES were analyzed to establish validity in this new population and to begin exploring trends in student responses based on their sub-population. Validity testing was completed via Cook’s validation evidence model with respect to scoring, generalization, and extrapolation evidence. The pre-course EDVES responses obtained were used to complete validation and trend analysis (note that post-course data was not readily available at the time of analysis).  more » « less
Award ID(s):
1812823
PAR ID:
10433044
Author(s) / Creator(s):
Date Published:
Journal Name:
ASEE Annual Conference & Exposition
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, 2021 ASEE Virtual Annual Conference)
    null (Ed.)
    As high school programs are increasingly incorporating engineering content into their curricula, a question is raised as to the impacts of those programs on student attitudes towards engineering, in particular engineering design. From a collegiate perspective, there is a related question as to how first-year engineering programs at the college level should adapt to a greater percentage of incoming students with prior conceptions about engineering design and how to efficaciously uncover what those conceptions may be. Further, there is a broader question within engineering design as to how various design experiences, especially introductory experiences, may influence student attitudes towards the subject and towards engineering more broadly. Student attitudes is a broad and well-studied area and a wide array of instruments have been shown to be valid and reliable assessments of various aspects of student motivation, self-efficacy, and interests. In terms of career interests, the STEM Career Interest Survey (STEM-CIS) has been widely used in grade school settings to gauge student intentions to pursue STEM careers, with a subscale focused on engineering. In self-efficacy and motivation, the Value-Expectancy STEM Assessment Scale (VESAS) is a STEM-focused adaptation of the broader Values, Interest, and Expectations Scale (VIES), which in turns builds upon Eccles’ Value-Expectancy model of self-efficacy. When it comes to engineering design, there have been a few attempts to develop more focused instruments, such as Carberry’s Design Self-Efficacy Instrument. For the purposes of this work, evaluating novice and beginning designer attitudes about engineering design, the available instruments were not found to assess the desired attributes. Design-focused instruments such as Carberry’s were too narrowly focused on the stages of the design process, many of which required a certain a priori knowledge to effectively evaluate. Broader instruments such as the VESAS were too focused on working and studying engineering, rather than doing or identifying with engineering. A new instrument, the Engineering Design Value-Expectancy Scale (EDVES) was developed to meet this need. In its current form the EDVES includes 38 items across several subscales covering expectancy of success in, perceived value of, and identification with engineering and design. This work presents the EDVES and discusses the development process of the instrument. It presents validity evidence following the Cook validation evidence model, including scoring, generalization, and extrapolation validity evidence. This validation study was conducted using pre- and post-course deployment with 192 first-year engineering students enrolled in a foundational engineering design course. 
    more » « less
  2. ; ; (, Proceedings of the American Society for Engineering Education)
    The purpose of this study is to develop an instrument to measure student perceptions about the learning experiences in their online undergraduate engineering courses. Online education continues to grow broadly in higher education, but the movement toward acceptance and comprehensive utilization of online learning has generally been slower in engineering. Recently, however, there have been indicators that this could be changing. For example, ABET has accredited online undergraduate engineering degrees at Stony Brook University and Arizona State University (ASU), and an increasing number of other undergraduate engineering programs also offer online courses. During this period of transition in engineering education, further investigation about the online modality in the context of engineering education is needed, and survey instrumentation can support such investigations. The instrument presented in this paper is grounded in a Model for Online Course-level Persistence in Engineering (MOCPE), which was developed by our research team by combining two motivational frameworks used to study student persistence: the Expectancy x Value Theory of Achievement Motivation (EVT), and the ARCS model of motivational design. The initial MOCPE instrument contained 79 items related to students’ perceptions about the characteristics of their courses (i.e., the online learning management system, instructor practices, and peer support), expectancies of course success, course task values, perceived course difficulties, and intention to persist in the course. Evidence of validity and reliability was collected using a three-step process. First, we tested face and content validity of the instrument with experts in online engineering education and online undergraduate engineering students. Next, the survey was administered to the online undergraduate engineering student population at a large, Southwestern public university, and an exploratory factor analysis (EFA) was conducted on the responses. Lastly, evidence of reliability was obtained by computing the internal consistency of each resulting scale. The final instrument has seven scales with 67 items across 10 factors. The Cronbach alpha values for these scales range from 0.85 to 0.97. The full paper will provide complete details about the development and psychometric evaluation of the instrument, including evidence of and reliability. The instrument described in this paper will ultimately be used as part of a larger, National Science Foundation-funded project investigating the factors influencing online undergraduate engineering student persistence. It is currently being used in the context of this project to conduct a longitudinal study intended to understand the relationships between the experiences of online undergraduate engineering students in their courses and their intentions to persist in the course. We anticipate that the instrument will be of interest and use to other engineering education researchers who are also interested in studying the population of online students. 
    more » « less
  3. ; ; ; ; (, ASEE Annual Conference & Exposition)
    The purpose of the project is to identify how to measure various types of institutional support as it pertains to underrepresented and underserved populations in colleges of engineering and science. We are grounding this investigation in the Model of Co-Curricular Support, a conceptual framework that emphasizes the breadth of assistance currently used to support undergraduate students in engineering and science. The results from our study will help prioritize the elements of institutional support that should appear somewhere in a college’s suite of support efforts to improve engineering and science learning environments and design effective programs, activities, and services. Our poster will present: 1) an overview of the instrument development process; 2) evaluation of the prototype for face and content validity from students and experts; and 3) instrument revision and data collection to determine test validity and reliability across varied institutional contexts. In evaluating the initial survey, we included multiple rounds of feedback from students and experts, receiving feedback from 46 participants (38 students, 8 administrators). We intentionally sampled for representation across engineering and science colleges; gender identity; race/ethnicity; international student status; and transfer student status. The instrument was deployed for the first time in Spring 2018 to the institutional project partners at three universities. It was completed by 722 students: 598 from University 1, 51 from University 2, and 123 from University 3. We tested the construct validity of these responses using a minimum residuals exploratory factor analysis and correlation. A preliminary data analysis shows evidence of differences in perception on types of support college of engineering and college of science students experience. The findings of this preliminary analysis were used to revise the instrument further prior to the next round of testing. Our target sample for the next instrument deployment is 2,000 students, so we will survey ~13,000 students based on a 15% anticipated response rate. Following data collection, we will use confirmatory factor analysis to continue establishing construct validity and report on the stability of constructs emerging from our piloting on a new student sample(s). We will also investigate differences across these constructs by subpopulations of students. 
    more » « less
  4. ; ; (, Journal of Research in Science Teaching)
    Abstract Demand for engineering‐interested and proficient high school graduates continues to grow across the nation. However, there remains a severe gap in college participation and employment in engineering fields for students with learning disabilities (SWLDs). One potential way to encourage SWLDs to consider engineering as a profession and promote the development of key science attitudes may be through engineering and technology career and technical education (E‐CTE) coursework. In this study, we address the following research questions: Do SWLDs take E‐CTE courses in the early years of high school at different rates compared to students without learning disabilities? What is the relationship between early E‐CTE coursetaking and science attitudes (self‐efficacy, utility, identity), and does this differ for students with and without learning disabilities? How do specific engineering career expectations change with respect to enrollment in early E‐CTE coursework, and do these differ for students with and without learning disabilities? We utilize the High School Longitudinal Study of 2009 (HSLS) to respond to the research questions through moderation models and a student fixed effects methodology. Ultimately, we found no evidence of SWLD underrepresentation in E‐CTE in high school. However, SWLDs were expected to benefit more than the general population from E‐CTE participation with respect to higher levels of science self‐efficacy and science identity. Implications from these findings include how to encourage persistence along the engineering pathway, the growth of career pathway policies at the state level, and how to incorporate E‐CTE practices in academic courses. 
    more » « less
  5. ; ; ; ; ; (, 2021 American Society for Engineering Education Annual Conference)
    null (Ed.)
    This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 246) groups, wherein the former participated in a two-to-three-week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email