Numerous groups focused on education and workforce preparation have issued calls for increased exposure to engineering content and skills among K-12 students, and researchers and educators have documented the resultant rise in engineering instruction at the K-12 level [2,3,4,5]. Engineering as a discipline can be defined as <any engagement in a systematic practice of design to achieve solutions to particular human problems= [3] and focuses largely on problem solving and application of students' knowledge and skills. Engineering work draws significantly upon students' math and science content knowledge backgrounds and can motivate and reinforce learning of foundational math and science concepts. If students are engaged in working through applied engineering problems, and addressing those problems requires the utilization of math and science content and skills, students may feel driven to apply, practice, and refine those content and skills in ways that traditional core math and science assignments do not always elicit [5,6,7]. Early exposure to engineering can also serve to promote interest in engineering as a potential career path while students still have time to take actions needed to pursue it [8].
To ensure a reliable and impactful delivery of engineering education to K-12 students, there is a critical need for quality curricula and teacher training [1]. In 2010, GA Tech took on a large, NSF-funded AMP-IT-UP (Award#1238089) project designed to develop, implement, and test a set of three, 18-week engineering curricula for grades 6 -8. This curriculum uses applied engineering problems, Problem-Based Learning (PBL), and an engaging, single, semester-long context for each grade level. The curriculum creates an experience designed to promote student engagement in engineering work, self-efficacy for engineering skills, persistence in engineering, and enhanced academic performance in not only engineering but also science and math. This approach is grounded in the literature [5,9,10,11,12] as well as relevant teaching experiences among the curriculum designers. PBL, a cognitive-apprenticeship model with collaborative problem solving at its core, requires students to identify gaps in their current understanding, plan how they can address those gaps, conduct research, and interpret their findings to solve a given problem. Prior research has identified a host of benefits associated with the PBL approach, many of which were also observed in research on the original implementation of the curriculum [1,13,14,15]. The focus on the middle school grade band is supported by prior research identifying this time period as critical for promoting engagement with math and science, as well as developing career interest in engineering [8]. Research has suggested that middle school grades present a key period for promoting interest and awareness, as <it is during the junior high (middle) school age that a student's beliefs about competency and interests begin to solidify= [16]. The engineering course is taught as an elective in the district where we tested this curriculum, allowing for more flexibility in the pacing, content, and skills taught as compared to a core class like mathematics or language arts. Furthermore, teachers in this class are less beholden to high-stakes standardized testing and district-level requirements, as compared to their peers teaching core classes, and as a result, can present material in a more relaxed and flexible environment.
Results from our prior project presented compelling evidence that practice with interesting and engaging engineering problems embedded with math and science skills and knowledge provided students with a significant, positive improvement in both science and math achievement and engagement with STEM [1]. Furthermore, research results indicated students' successful transfer of knowledge between engineering and core math and science courses. Additionally, Cunnigham et. al's (2020) study of elementary engineering curriculum implementation supports better learning outcomes in both engineering and science when it is compared to regular curriculum [17]. The study shows the importance of curriculum design for student learning.
The current study is part of a new NSF DRK-12 project (#2101441), Measuring the effectiveness of Middle School STEM-Innovation and Engineering Design Curricula, awarded in 2021, in which we aim to further refine the curriculum, accompanying teacher resources, and teacher professional development experiences, as well as replicate the initial research results in a different educational setting. The original study was held in a small school district (4 middle schools) considered rural fringe. The demographics of students in the original study school district are 32% white, 51% African-American, 11% Asian, and 6% other. The current study school district is the largest in the state (29 middle schools) and has a student body with the following demographics: 18% white, 33% African-American, 11% Asian, 35% Hispanic/Latino and 3% other. The results presented in this paper primarily comprise students' perceptions on a variety of attitudinal measures prior to and after their semester-long experience with the course, and these perceptions are a key component towards the replication of the study. The research question being investigated by the student survey is: What is the effect of participation in the engineering course on students' social-emotional outcomes such as cognitive and behavioral engagement, science interest, and math and science anxiety?
The STEM-Innovation and Engineering Design (STEM-ID) curricula are guided by the PBL approach, as discussed above. A key engineering practice students are intended to learn and interact with throughout the curriculum is the engineering design process (EDP). While there are numerous published versions of the EDP [18], the one selected for use in the STEM-ID curriculum is presented in Figure 1. The EDP offers students a scaffolded series of steps to follow to understand and work through an engineering problem. The EDP is utilized heavily in all grade level curricula, such that students enrolled in more than one semester of STEM-ID will work with this framework extensively, with the intention that repeated exposure will solidify their ability to apply the EDP. Math and science foundational skills incorporated into the curriculum are drawn from the Standards for Mathematical Practice (SMP) [19] and Next Generation Science Standards (NGSS) [20], respectively. In this paper, we focus on investigating the extent to which exposure to the STEM-ID course is associated with the intended student level effects of increased engagement, decreased math and science anxiety, and increased science interest. The effects of course enrollment on student achievement and understanding of the EDP will be explored elsewhere and are beyond the intended scope of this paper.
The multiyear engineering course sequence consists of three 18-week courses. The courses are designed to satisfy the state-level engineering and technology course standards for 6 th , 7 th and 8 th grade, but can also be implemented as STEM Connections courses. Courses are structured into a sequence of four challenges, with the initial three focusing on the development of various skills. The fourth challenge is a multi-week design challenge that consolidates the acquired experience from the first three challenges. The challenges are named the Data Challenge, Systems Challenge, Visualization Challenge, and Design Challenge.
The engineering courses in this study generally prioritize foundational mathematics and science skills over grade-specific disciplinary content in mathematics and science. Students routinely apply skills such as measuring, computing, estimating, graphing, and employing mathematical reasoning, aligning with their grade-level mathematics and science courses. The curriculum of these courses place emphasis on specific mathematics and science practices, organized under three overarching themes related to data collection, visualization, interpretation, and communication. These three themes are (1) Experimental Design, (2) Data Visualization, and (3) Data-Driven Decision Making.
The Experimental Design theme delves into concepts aligned with NGSS [20] Practice #3 (Planning and Carrying Out Investigations), SMP #1 [19] (Making Sense of Problems, e.g., planning a solution pathway), and SMP #5 [19] (Using Appropriate Tools Strategically). Through these practices, students engage in identifying and controlling variables, crafting procedures, conducting experiments, utilizing data-collection tools, and analyzing data. The engineering classes challenge students to establish and conduct tests, pinpoint variables causing inconsistent results, establish standard procedures, rerun tests, and graph data to illustrate data convergence as procedures standardize.
The Data Visualization theme encompasses concepts drawn from NGSS [20] Practice #4 (Analyzing and Interpreting Data, e.g., creating and using graphical displays), SMP #1 [19] (Making Sense of Problems, e.g., graphing data and identifying regularities or trends), and SMP #4 [19] (Modeling with Mathematics, e.g., mapping relationships using diagrams, two-way tables, graphs). This theme underscores the idea that data can be represented in various ways, different visualizations offer diverse insights into evidence, and the most effective data visualization is the one that best conveys the intended concept. Students learn to consider what aspects of their data they wish to emphasize, and which visualization tools are best for a variety of applications.
As for the third theme, Data-Driven Decision Making, students are tasked with making decisions or designing solutions based on data in scenarios lacking simple solutions and involving potential trade-offs. The engineering courses introduce decision matrices as a tool for organizing data to derive meaning and guide decisions, with students subsequently articulating and defending their choices. This theme integrates NGSS [20] Practice #6 (Constructing Explanations and Designing Solutions) and NGSS [20] Practice #7 (Engaging in Argument from Evidence), along with the communication aspect of NGSS Practice [20] #8 (Obtaining, Evaluating, and Communicating Information). It also aligns with the mathematics standards of SMP #1 [19] (Making Sense of Problems, e.g., analyzing givens, constraints, relationships, and goals) and SMP #3 [19] (Construct Viable Arguments).
Table 1 outlines the skills associated with each challenge, using the 8th-grade course as an illustration. These skills are categorized into Engineering & Problem Solving, Science Disciplinary Content, Science Practices, Foundational Math, and Communication. The final Design Challenge requires students to showcase all the skills developed and refined in the preceding challenges. Table 1. Structure of the Engineering Course 8 th Grade Engineering Course
paper (2022-2023), only one middle school had implemented the engineering curriculum prior to that school year. In our original study [1], the results showed that students who participated in the engineering course at least twice during their middle school years showed significant changes in terms of social-emotional outcomes, such as cognitive and behavioral engagement, science interest, math anxiety, and science anxiety. In an attempt to replicate these results, this study utilizes data solely from the school which had previously implemented the engineering curriculum, as students at this school had the opportunity to participate in the engineering course prior to the focal year. In subsequent years of data collection, students at nearly all schools will have had the opportunity to participate in the engineering course two, or in some cases, three times during their middle school years.
This study utilizes 2022-2023 pre-post survey data from 58 students, all of whom had previously participated in the engineering course. The study participants were comprised of 7 th and 8 th graders, and the race/ethnicity subgroups represented in the study population include White (19%), Black (26%), Hispanic (14%), Asian (40%) and other (1%). Additionally, 65% of the students in the study identified as male, and 35% of the students identified as female. Approximately 52% of the students in the school qualify for free/reduced lunch.
The engineering teacher received professional development on Project-Based Inquiry Learning, LEGO robotics, individualized instruction on the course curriculum, and regular ongoing support through email, phone calls and texts, and classroom visits.
The student survey utilized in this study was developed and validated in 2013 with a middle school population to measure change in specific 21 st Century Skills and other socialemotional outcomes in the original study [1]. The instrument consists of 51 Likert-type selfreport items in which students are asked to describe their level of agreement. The response options range from <Strongly Disagree= (=1) to <Strongly Agree= (=4). For the purpose of this paper, only the constructs presented in Table 2 are analyzed, each of which shows very good reliability based on Cronbach's alpha (> 0.80) [21].
Engagement: Two types of engagement are captured by the student survey: Behavioral and Cognitive [22]. Behavioral Engagement is defined as positive conduct, such as paying attention and following rules. This construct was measured with three items that were adapted from the School Engagement Scale [23]. Cognitive Engagement has a broad description in the literature, encompassing students' desire to put effort into learning and school activities, as well as seeing the benefits of such activities. This construct consists of five items and was adapted from the Science Motivation Questionnaire [24].
Anxiety has two elements: the feeling of nervousness and the act of worrying. The feeling of nervousness refers to the uneasy or sick feeling that students may experience when contemplating their schoolwork, assignments, or exams. On the other hand, worrying involves the fear of not performing well in these academic tasks. In this study, anxiety was gauged through surveys focusing on both science and mathematics. Existing literature on anxiety encompasses measures that capture both negative and positive emotions related to mathematics and science, such as concerns about exam performance or finding the subject interesting [25,26]. However, this study specifically concentrates on the negative aspects of anxiety associated with mathematics and science. The items (11 items) utilized for measurement were adapted from the Science Motivation Questionnaire [24] and the Mathematics Anxiety Scale-Revised [26].
Science Interest: This construct (7 items) centers on the combination of interest and personal relevance that students perceive in the content they are studying. The items were adapted from the Science Motivation Questionnaire [24].
Open-ended item: Additionally, students were asked to respond to a single open-ended item presented on the engineering students' post-tests, <Do you think the math and science that you do in your engineering class helps in your core math and science classes? If so, how? Data Collection: A pre-post survey design was used, with the student survey administered in paper and pencil format at the start and end of the semester in which students were enrolled in STEM-ID courses (either Fall, 2022 or Spring, 2023 for this analysis).
Pretest-posttest comparisons were conducted using paired samples t-tests. The sample for these analyses included students who took the engineering course at least twice during their middle school experience. In addition to statistical significance, effect sizes were determined using Cohen's d [27]. Effect sizes that were found to be around 0.3 or less are considered small, around 0.5 are considered moderate, and around 0.8 and above are considered large. Student responses to the open-ended item were analyzed using thematic analysis, which emphasizes identifying, analyzing, and interpreting patterns of meaning within qualitative data [28].
Despite the disparity in middle school sample sizes between this study (one school) and the original (four schools), the findings demonstrated similar trends to our initial investigation. Significant statistical changes were observed across all constructs. Table 3 shows the mean change for each construct. It is important to highlight that the items related to math and science anxiety were subjected to reverse coding, such that a lower mean indicates high anxiety, while a higher mean indicates low anxiety. Results of a paired t-test show that the students' level of perceived behavioral and cognitive engagement increased (t = -6.22, p < .001, d = .73, t = -2.48, p < .05, d = .42, respectively). According to Brewster & Fager (2000), the involvement of students in school frequently diminishes as they progress from elementary to middle school, and school involvement experiences another decline during the transition to high school [29]. Our findings indicate that when students have more experiences with inquiry-based learning, which allows them to ask questions and participate in hands-on activities, their perceived engagement with school becomes more positive.
Additionally, the change in Science Interest was statistically significant (t = -3.01, p < .01, d = .55). Similar to the original study, this outcome implies that the engineering course enhances the relevance of science for students, thereby increasing their interest in the subject. The science interest construct specifically centers on personal relevance and interest. There was no statistically significant decrease in science anxiety found after participating in the engineering course; however, it was slightly lower after participating in the engineering course. Lastly, the findings show a statistically significant decrease in math anxiety at the end of the school year compared to the beginning of the year (t = -2.25, p < .05, d = .71). The curriculum specifically focuses on practicing foundational math skills, which attempts to reduce math anxiety, and increase students' academic self-efficacy.
Open-ended item results also support the survey findings. The overall responses in openended items were largely positive, with many students recognizing the value of their engineering classes in reinforcing math and science concepts. Eighty-seven percent of respondents agreed that there was a connection between the engineering class and their science and math classes. The connection between problem-solving skills developed in engineering and their application in real-world scenarios was a recurring theme. While some acknowledge the specificity of engineering content, others find a beneficial overlap, suggesting that the interdisciplinary nature of the engineering course contributes to a more holistic understanding of math and science subjects.
• Positive Impact on Problem-Solving: Many students believe that their engineering classes enhance their problem-solving skills, providing them with valuable tools to tackle challenges. As one student stated <
• Application of Math and Science in Engineering: Several students express that the math and science they learn in engineering is directly applicable to the field, citing examples of calculations, measurements, and concepts used in both subjects.
• Preparation for Future Topics: Some students feel that their engineering classes prepare them for upcoming topics in math and science, giving them a head start and background information. As one of the students highlighted: < have a bit of background information of it. That gives me a head start.=
• Overlap and Integration: There is a recognition of overlap and integration between engineering, math, and science. Students mention instances where topics introduced in one class reappear in another, creating connections between the subjects.
• Enhancement of Understanding: A number of students believe that their engineering classes enhance their understanding of math and science, making certain topics clearer and more tangible.
The findings of this study support the results of the original study. They also support our hypothesis that providing students with opportunities to apply foundational science and mathematics skills within captivating middle school engineering classes can have a substantial positive impact on their engagement and academic performance in mathematics and science. Specifically, the results indicate that these engineering courses, conducted concurrently with essential science and mathematics classes, contribute significantly to student attitudes related to school engagement, science interest, and math anxiety. These findings align with the theoretical and applied work around K-12 applied engineering curricula conducted by other researchers [5,11,13,14] , some of whose worked informed the original study design.
Given that a primary purpose of the current study is to replicate results from our previous deployment of this curriculum, these initial results are encouraging as to the potential of these curricula to impart similar student benefits in a different setting with a different population. The current study, because of the timing of this paper, only allowed for the inclusion of a small sample of students from one school. However, our overall study design allows the collection and analysis of survey data from a much larger pool of students, with more students having the opportunity to take the engineering course twice or even three times during their middle school years. We are optimistic that these initial replication results will hold in the larger sample of students in the future. K-12 curricula focusing on STEM integration and engineering have become more common offerings in recent years. However, it is critical that these curricula be subject to rigorous investigation so that specifics related to school and teacher requirements for successful implementation, as well as potential impacts on students, can be fully understood. The next step in this research is to conduct longitudinal studies across years that also include comparison groups, as well as to continue with the analyses described in this paper with larger and more varied samples of students. Our prior work [1] and the current study are intended to provide such investigation so that an evidence-based middle school engineering curriculum can be studied under varying conditions, iterated upon, optimized, and ultimately disseminated to a wide range of teachers and students. The positive impacts on both social-emotional outcomes such as engagement, science interest, and math and science anxiety, as well as academic achievement in engineering, science, and math, aim to benefit both students and teachers participating in engineering, broadly.