Since their release in 2013, the Next Generation Science Standards advocate the implementation of numerous engineering and engineering design practices into the K-12 learning space [1]. With engineering design becoming so prevalent in this space, educators readily acknowledge it has benefited higher institutions. In particular, students pursuing engineering after high school are more acclimated to the engineering design curriculum they encounter in first-year engineering courses [2]. However, before one begins to understand how engineering design in the K-12 space affects students continuing on to collegiate engineering programs, one must understand how this exposure to engineering design in the high school setting influences their collective attitudes and beliefs towards engineering. With this recognition, educators will be better able to understand the motivations of student learning, the subsequent impact on skill mastery, and the difference between engineering self-efficacy and value-expectancy for students on a pre-engineering track versus those who are not. Building off this recognition, educators can also dynamically develop a curriculum that ensures all students understand how to apply engineering design and why it is applicable in a range of situations, even if they don't find much value in it or intend to pursue engineering.
To begin achieving this end, the Engineering Design Value Expectancy Scale (EDVES) was created and resulted from the analysis of several tools already in existence: the Value-Expectancy STEM Assessment Scale (VESAS), the Value-Expectancy Model of Motivation, Carberry's Design Self-Efficacy Instrument, and the STEM Career Interest Survey (STEM-CIS) [3][4][5][6]. This work builds upon initial presentation and validity testing of the EDVES in the first-year engineering setting by Hylton et. al. [7]. Here, the 2014 Standards for Educational and Psychological Measurement were applied as the basis for evidence gathering and Cook's evidence validation model was used for instrument validity [8]. Preliminary reliability testing depicts the EDVES having reasonable reliability in this general population based on computed inter-item correlations, item-to-scale total correlations, and Cronbach's alpha with few items being removed from analysis due to poor correlation values. With the validity of the instrument assessed by Hylton et. al. in the first-year engineering context, this work observes the validity of the EDVES in the K-12 space through the same evidence gathering process and application of Cook's evidence validity model.
As previously noted, the EDVES arose from previously documented instruments and models to meet an unmet need. Derived from the Expectancy-Value Theory proposed by Eccles and Wigfield, the VESAS observes student motivation, values, and expectations that influence their desire to stay or transfer out of university STEM programs [3] [9-10]. Building upon the original Expectancy-Value Theory proposed by Eccles and Wigfield, the Expectancy-Value-Cost Model of motivation observes the importance of expectancy of success, the perceived value for engaging in a task, and the cost of doing such for an individual [4]. Carberry's instrument focuses on measuring an individuals' beliefs towards engineering design activities and specific steps of the engineering design process [5]. Lastly, the STEM-CIS analyzes student desire to pursue a STEMrelated career [6].
While all of the noted tools juxtapose our motivation for this work, none concretely aid our pursuit and ultimately required the creation of a new tool to do so. The EDVES is comprised of 38 items across three sub-scales: expectancy of success in, perceived value of, and identification with engineering and various engineering-related tasks. With these sub-scales, one can assess student attitudes toward engineering and how they may change as they are exposed to and practice engineering ideas.
Numerous instruments were influential in the creation of the EDVES, however, the Expectancy-Value Model of Motivation by Eccles and Wigfield served as our primary basis for development [9][10]. More recently, a third factor in the model was introduced by Barron and Hulleman: the cost associated with partaking in a given task [4]. For the purpose of our work, cost was not included when creating the various EDVES items as it did not relate very well into our current context. This model of motivation was the primary basis for the EDVES as it relates closely to other motivation theories such as Self-Efficacy Theory proposed by Bandura and Self-Determination Theory proposed by Deci and Ryan [11][12].
In terms of developing the individual items that comprise the EDVES, the VESAS, Carberry's Engineering Design Self-Efficacy Scale, and the STEM-CIS were the primary contributors to item content and wording in the EDVES while Eccles' Expectancy-Value Theory grounded the attitudefocused items [3-6] [9-10]. Note that the three scales exhibited their own validity and reliability by their creators, and subsequently allowed us to ensure EDVES items were created with established, high-quality practices in mind. Upon assembling and finalizing all items, the instrument was reviewed by two engineering faculty members and a psychometrician. Additional revision of the instrument was conducted upon receiving their feedback and gave rise to the current form of the EDVES (see Appendix 1) where items measure expectancy of success, perceived value, and identity with engineering in three separate sub-scales. Within these sub-scales are sub-sections that further classify the topic of each EDVES item (see Table 1 in Appendix 1). The first sub-scale (expectancy of success) contains three sub-sections as so: expectancy of success in science, expectancy of success in engineering, and expectancy of success in problem framing. The second sub-scale (perceived value) contains three new subsections: engineering intrinsic value, engineering attainment value, and identification with engineering. The final sub-scale (identification with engineering) has three sub-sections as well: engineering extrinsic utility value, problem framing skill extrinsic utility value, and engineering career interest. Each item is assessed via 1-7 Likert type scale with 1 indicating "strongly disagree" and 7 indicating "strongly agree" with the item.
The 2014 Standards for Education and Psychological Measurement along with Cook's evidence validation model serve as the primary means for content validity of the EDVES [6] [8]. In particular, the former was used as the primary means for gathering validity evidence while the latter was used as the guideline for establishing validity. The various evidence types and the manner in which verification for the EDVES occurred is presented in Appendix 2.
At the onset of the 2021-2022 academic year, the EDVES was deployed to students (N = 569) enrolled in a public school district of a suburb of Kansas City, Olathe, Kansas. With the goal to explore the effects of engineering content interventions across the high school population, the project was made available to all science teachers in the district. This was to ensure a wide range of subjects and grade levels were exposed to the project content and as so, 12 science teachers agreed to participate and gave rise to this population size. Grade levels ranged from 9 th to 12 th grade with course subjects including Advanced Biotechnology: Cellular & Molecular I, Biology, Honors Biology, Chemistry, Honors Chemistry, AP Physics I, Physics, and Physical Science. Of these course subjects, 36 students were enrolled in Advanced Biotechnology: Cellular & Molecular I, 86 were enrolled in Biology, 52 were enrolled in Honors Biology, 13 were enrolled in Chemistry, 95 were enrolled in Honors Chemistry, 111 were enrolled in AP Physics I, 134 were enrolled in Physics, and 36 were enrolled in Physical Science. Three of these courses (Honors Biology, Honors Chemistry, and AP Physics I) are part of an engineering academy in the school system while the remaining five courses are part of the traditional science curriculum. Note that six students did not complete the EDVES in its entirety and were removed from the dataset prior to analysis. Also, the EDVES was deployed prior to the coverage of any engineering related content.
All student responses to every EDVES item were recorded in an Excel workbook for descriptive statistical analysis, correlation analyses, and the calculation of Cronbach's alpha to determine the instrument's internal reliability. Every calculation was conducted via IBM SPSS software. EDVES items completed by all participating students were first analyzed for excessive skewness and kurtosis where the former held true for values greater than 3.0 and the latter for values greater than 10.0 [13]. Any item exhibiting either trait was subsequently removed from analysis. From there, item-to-scale total correlations were calculated and correlations less than 0.30 were removed from analysis. Inter-item correlations were then calculated for each sub-section of the three sub-scales against a 0.30 threshold. Lastly, Cronbach's alpha was computed using all completed responses for each of the three sub-scales and their corresponding sub-sections.
In addition to collectively analyzing all recorded responses, responses were divided by course subject and analyzed for basic statistics, excessive skewness, kurtosis, item-to-scale total correlations, inter-item correlations, and Cronbach's alpha. The same process was carried out for responses in each course subject where items that exhibited excess skewness/kurtosis or poor itemto-scale total or inter-item correlations were removed from analysis. Analysis of such statistics was also observed between courses on the engineering track versus those that are not. This subsequent analysis allowed for exploration of whether early interest in engineering affects the responses supplied on the EDVES and serves as another basis to satisfy Cook's evidence validation model [8].
Upon collection of all EDVES responses from the participating courses, analysis over the aggregate data was conducted and while no items exhibited excessive skew or kurtosis, two items, items 4 and 10, exhibited poor item-to-scale total correlations. These items were thus removed from analysis and are highlighted red in Appendix 1. All remaining item-to-scale total correlations and inter-item correlations were above 0.30 and used for Cronbach's alpha calculation.
Descriptive statistics, the number of items analyzed by sub-scale and sub-section, and resultant alpha values are presented in Table 2. Assuming the safest range for alpha values to indicate internal reliability is 0.60-0.80 [14], all sub-scales and their corresponding sub-sections exhibited alpha values above 0.60, thereby indicating the items within each are well-related for this general population and measure the specific concepts of interest as intended. However, many of these values are above the upper threshold of 0.80, and while that does not immediately mean the items used to compute those alpha values do not measure the same concept or factor very well, they should be taken with caution and will be elaborated upon in the following section. In terms of the basic statistics presented in Table 2, the averages and corresponding standard deviations of student responses for items in each sub-scale and section indicate not that they have a lack of confidence in any one section, but rather a slight confidence in each (recall the 1-7 Likert scale applied where 1 translates to "strongly disagree" while 7 becomes "strongly agree"). This indicates that in the high school setting, students expect to have some success using their science, engineering, and problem framing abilities, generally value the offerings of engineering design, and somewhat identify with engineering and the extrinsic value it offers.
Similar to the analysis conducted over the aggregate data, EDVES responses were divided by the course subject students were enrolled in and subsequently analyzed for excessive skewness, kurtosis, low item-to-scale and inter-item correlations. All course subjects exhibited values within the set acceptable ranges for these statistics aside from the Honors Biology course. In this course, excessive kurtosis was found with EDVES item 27 and was removed from further analysis. Note that item 27 has also been highlighted red in Appendix 1 to denote this.
Tables 3 and4 This indicates the items in the corresponding sub-scales and sub-sections do not measure the ideas of interest as well for this sub-population. Also, among all three courses on the engineering track, the alpha values computed for Honors Biology were consistently the lowest and closest to the lower threshold. Therefore, the EDVES items did not measure the intended ideas for this particular sub-set of students as well as it did for those in AP Physics I or Honors Chemistry. Although this is the case among the engineering track courses, note that the EDVES still collectively measured the specific ideas for Honors Biology in a reliable manner for many items and all three sub-scales based on the overall alpha values computed. While there is one alpha value below the 0.60 threshold for the Expectancy of Success in Problem Framing sub-section, the differential from the computed value to 0.60 is very minor and encourages one to believe this sub-section measures the intended ideas well enough as is with room for future improvement. Beyond this sub-scale, the perceived value sub-scale was the sole scale where all course subjects had alpha values greater than the 0.60 threshold [14]. As such, the items consisting of this scale relate and measure the intended idea (perceived value) most reliably of all three sub-scales. However, note the expectancy of success sub-scale follows close behind with only one alpha value slightly below 0.60. Lastly, the third sub-scale (identity with engineering) contained the largest number of alpha values below 0.60 among all course subjects which is not expected considering the high alpha value found in the aggregate set.
The general agreement students have with the various EDVES items is readily tracked with the average and standard deviations computed for the sub-scales and corresponding sub-sections.
Recall the Likert scale used for this instrument where 1 corresponds to "strongly agree" while 7 corresponds to "strongly disagree". As seen in Table 3, many averages for the courses on the engineering track are at least a 5 which translates to students primarily agreeing with the items presented in the given sub-scales and sub-sections of the instrument. However, with Table 4, most averages reside around a 4, translating to the response option "neither agree nor disagree". Here, students seem less inclined in one way or the other with respect to the success and value they attach to engineering design, and overall interest in pursuing engineering. The differences seen in computed averages for the identity with engineering subscale verifies that students enrolled in courses on the engineering track are more interested in pursuing an engineering career than those who are not.
In this work, the EDVES was deployed to students ranging from grades 9-12 enrolled in several science courses: AP Physics I, Advanced Biotechnology: Cellular & Molecular I, Biology, Chemistry, Honors Biology, Honors Chemistry, Physical Science and Physics. To begin determining the validity of the instrument with this general population, responses to the EDVES were analyzed for excessive skewness or kurtosis and for low item-to-scale total correlations or inter-item correlations. Any such items were subsequently removed before calculating Cronbach's alpha for each sub-scale and sub-section of the instrument.
In the aggregate dataset, all sub-scales and sub-sections exhibited high alpha values, indicating items may closely relate to one another and measure the idea of interest for particular scale or section well. In other words, for the general population of 9 th -12 th grade students, the EDVES does appear to reliably measure their expectancy of success in, perceived value of, and overall identity with engineering and engineering design as desired. These alpha values were generally higher than those exhibited by first-year undergraduate engineering students enrolled in a Foundations of Design course that also completed the EDVES where few alpha values exceeded 0.80 [7]. Also, a total of five items were removed prior to analyzing the first-year engineering data whereas only two were removed in this work, indicating the EDVES is reliable in this aggregate group and perhaps even more than for the first-year engineering students [7]. While high alpha values typically suggest internal consistency of sub-scales in a given instrument and is desired, some of these values may be too high, particularly those above 0.80. Recently, researchers have been questioning whether alpha values can be too high and no longer convey items are measuring juxtaposing ideas under the umbrella of a particular sub-scale, but rather are redundant, eliciting answers for the same question and measuring the same single idea posited in one item [15]. To more readily conclude this as the case, exploratory factor analysis should be carried out and will allow us to determine if each EDVES item explains one factor and one factor only rather than multiple factors simultaneously. If the latter is found to be true, then this implies redundancy among EDVES items much like extraordinarily high alpha values. Preliminary work on exploratory factor analysis has been completed with the aggregate data set and found items to explain several factors simultaneously, indicating a re-wording of items may be needed to better establish a one-factor-to-one-item relationship with the instrument.
When dividing the dataset by course subject, many sub-scales and sub-sections exhibited similarly high alpha values and may indicate interrelatedness of the EDVES items in their sub-scales and sections. While this is the preferred outcome, it may also be indicative of redundancy across items in the instrument, and therefore needs extensive exploratory factor analysis to further establish if items need to be re-worded and re-deployed. As noted earlier, the perceived value scale resulted in the most alpha values above 0.60 while the identity with engineering scale resulted in the most below 0.60. This encourages us to believe the perceived value scale reliably measures the intended ideas not only in the aggregate set, but along the course subjects in this work as well. Moreover, this may be due to students generally finding importance with the ideas associated with engineering design and subsequently connect to them. However, with the identity in engineering scale, there is consistent evidence with the noted alpha values below 0.60 that these items are not measuring reliably the identity students hold with engineering, and should be re-worded to more accurately measure this concept. If these particular alpha values are below 0.60 upon re-wording of items and re-deploying the EDVES, then it's possible the students in the sub-populations of interest have a general lack of interest in engineering and do not connect well to the ideas presented in the items.
The application of an Engineering Design Value Expectancy Scale (EDVES) grounded in Expectancy-Value theory as proposed by Eccles and Wigfield and synthesized from several preexisting instruments readily allows evaluators of such an instrument to reflect on their selfefficacy, value-expectancy, and identity with engineering and engineering design [3-6] [9-10]. Such an instrument also allows educators to determine where students stand and dynamically teach relevant concepts in a manner that enhances all three of these facets for students. In particular, applying this tool in the K-12 space is vital as students are actively shaping their identity and beliefs relative to the primary subjects being taught. While the value of this tool is readily identified, reliability and validity of it must be established to ensure the results obtained from it are viable. To ground the EDVES in terms of its validity, Cook's validation evidence model was applied in this analysis.
In this work, the EDVES was deployed to students ranging from 9 th -12 th grade enrolled in one of eight science courses: Advanced Biotechnology: Cellular & Molecular I, Biology, Honors Biology, Chemistry, Honors Chemistry, AP Physics I, Physics, and Physical Science. Descriptive statistics were carried out for both the aggregate dataset and course subject-based dataset. In the validity scope, excessive skewness, kurtosis, low item-to-scale total correlations, and low interitem correlations were computed for both datasets. Any items found to exhibit such characteristics were removed from analysis. From here, Cronbach's alpha was calculated for the three sub-scales and their corresponding sub-sections of the overall instrument.
Overall, alpha values were typically above 0.60 and indicated strong inter-reliability of the subscales for both the entire population and sub-populations divided by course subject. However, many values were potentially too high (above 0.80), and may indicate redundancy of items in the instrument. Another point of observation regards the Honors Biology dataset which had the most alpha values under 0.60 and in the identity with engineering sub-scale across all course subjects. This indicates the EDVES did not measure the various ideas of interest in the instrument for Honors Biology sub-population of student well and may be remedied by re-wording items. However, the perceived value sub-scale of the EDVES resulted in all alpha values above 0.60, potentially indicating that students may find the concepts highlighted in the corresponding EDVES items of importance and generally connect to them.
While there are some preliminary conclusions to draw from this work, there is ample room for future work that can further verify or determine changes that should be made to the instrument to ensure it is viable. One way to move forward from here is to re-word the EDVES items that exhibited high alpha values. Currently, it is possible that some items measure or ask the same one idea rather than juxtaposing ideas that reside under one umbrella, and this may be remedied by rewording some items. A second way to move forward is to conduct extensive exploratory factor analysis with the data. If items are found to simultaneously explain multiple factors at once rather than solely one factor, then one can conclude the presence of redundancy in the EDVES and further verify that items must be re-worded before moving forward. Although there are several points to improve upon with the EDVES, we readily identify the value it brings to the K-12 learning space as it will support educators in understanding what their students are thinking and how that impacts their performance and motivation to thrive. The EDVES can actively shape how educators deliver the important concepts of engineering design to students and ultimately enhance their learning.