Despite limited success in broadening participation in engineering with rural and Appalachian youth, there remain challenges such as misunderstandings around engineering careers, misalignments with youth’s sociocultural background, and other environmental barriers. In addition, middle school science teachers may be unfamiliar with engineering or how to integrate engineering concepts into science lessons. Furthermore, teachers interested in incorporating engineering into their curriculum may not have the time or resources to do so. The result may be single interventions such as a professional development workshop for teachers or a career day for students. However, those are unlikely to cause major change or sustained interest development. To address these challenges, we have undertaken our NSF ITEST project titled, Virginia Tech Partnering with Educators and Engineers in Rural Schools (VT PEERS). Through this project, we sought to improve youth awareness of and preparation for engineering related careers and educational pathways. Utilizing regular engagement in engineering-aligned classroom activities and culturally relevant programming, we sought to spark an interest with some students. In addition, our project involves a partnership with teachers, school districts, and local industry to provide a holistic and, hopefully, sustainable influence. By engaging over time we aspired to promote sustainability beyond this NSF project via increased teacher confidence with engineering related activities, continued integration within their science curriculum, and continued relationships with local industry. From the 2017-2020 school years the project has been in seven schools across three rural counties. Each year a grade level was added; that is, the teachers and students from the first year remained for all three years. Year 1 included eight 6th grade science teachers, year 2 added eight 7th grade science teachers, and year 3 added three 8th grade science teachers and a career and technology teacher. The number of students increased from over 500 students in year 1 to over 2500 in year 3. Our three industry partners have remained active throughout the project. During the third and final year in the classrooms, we focused on the sustainable aspects of the project. In particular, on how the intervention support has evolved each year based on data, support requests from the school divisions, and in scaffolding “ownership” of the engineering activities. Qualitative data were used to support our understanding of teachers’ confidence to incorporate engineering into their lessons plans and how their confidence changed over time. Noteworthy, our student data analysis resulted in an instrument change for the third year; however due to COVID, pre and post data was limited to schools who taught on a semester basis. Throughout the project we have utilized the ITEST STEM Workforce Education Helix model to support a pragmatic approach of our research informing our practice to enable an “iterative relationship between STEM content development and STEM career development activities… within the cultural context of schools, with teachers supported by professional development, and through programs supported by effective partnerships.” For example, over the course of the project, scaffolding from the University leading interventions to teachers leading interventions occurred.
more »
« less
Gauging Informal STEM Youth Program Impact: A Conceptual Framework and a Measurement Instrument
STEM education programs are often formulated with a "hands-on activities" focus across a wide array of topics from robotics to rockets to ecology. Traditionally, the impact of these programs is based on surveys of youth on program-specific experiences or the youths’ interest and impressions of science in general. In this manuscript, we offer a new approach to analyzing science programming design and youth participant impact. The conceptual framework discussed here concentrates on the organization and analysis of common learning activities and instructional strategies. We establish instrument validity and reliability through an analysis of validity threats and pilot study results. We conclude by using this instrument in an example analysis of a STEM education program.
more »
« less
- Award ID(s):
- 1811265
- NSF-PAR ID:
- 10335662
- Date Published:
- Journal Name:
- Journal of Youth Development
- Volume:
- 16
- Issue:
- 4
- ISSN:
- 2325-4017
- Page Range / eLocation ID:
- 103 to 133
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
National Science Foundation (NSF) funded Engineering Research Centers (ERC) must complement their technical research with various education and outreach opportunities to: 1) improve and promote engineering education, both within the center and to the local community; 2) encourage and include the underrepresented populations to participate in Engineering activities; and 3) advocate communication and collaboration between industry and academia. ERCs ought to perform an adequate evaluation of their educational and outreach programs to ensure that beneficial goals are met. Each ERC has complete autonomy in conducting and reporting such evaluation. Evaluation tools used by individual ERCs are quite similar, but each ERC has designed their evaluation processes in isolation, including evaluation tools such as survey instruments, interview protocols, focus group protocols, and/or observation protocols. These isolated efforts resulted in redundant resources spent and lacking outcome comparability across ERCs. Leaders from three different ERCs led and initiated a collaborative effort to address the above issue by building a suite of common evaluation instruments that all current and future ERCs can use. This leading group consists of education directors and external evaluators from all three partners ERCs and engineering education researchers, who have worked together for two years. The project intends to address the four ERC program clusters: Broadening Participation in Engineering, Centers and Networks, Engineering Education, and Engineering Workforce Development. The instruments developed will pay attention to culture of inclusion, outreach activities, mentoring experience, and sustained interest in engineering. The project will deliver best practices in education program evaluation, which will not only support existing ERCs, but will also serve as immediate tools for brand new ERCs and similar large-scale research centers. Expanding the research beyond TEEC and sharing the developed instruments with NSF as well as other ERCs will also promote and encourage continual cross-ERC collaboration and research. Further, the joint evaluation will increase the evaluation consistency across all ERC education programs. Embedded instrumental feedback loops will lead to continual improvement to ERC education performance and support the growth of an inclusive and innovative engineering workforce. Four major deliveries are planned. First, develop a common quantitative assessment instrument, named Multi-ERC Instrument Inventory (MERCII). Second, develop a set of qualitative instruments to complement MERCII. Third, create a web-based evaluation platform for MERCII. Fourth, update the NSF ERC education program evaluation best practice manual. These deliveries together will become part of and supplemented by an ERC evaluator toolbox. This project strives to significantly impact how ERCs evaluate their educational and outreach programs. Single ERC based studies lack the sample size to truly test the validity of any evaluation instruments or measures. A common suite of instruments across ERCs would provide an opportunity for a large scale assessment study. The online platform will further provide an easy-to-use tool for all ERCs to facilitate evaluation, share data, and reporting impacts.more » « less
-
nterest in science, technology, engineering, and mathematics (STEM) begins as early as elementary and middle school. As youth enter adolescence, they begin to shape their personal identities and start making decisions about who they are and could be in the future. Students form their career aspirations and interests related to STEM in elementary school, long before they choose STEM coursework in high school or college. Much of the literature examines either science or STEM identity and career aspirations without separating out individual sub-disciplines. Therefore, the purpose of this paper is to describe the development of a survey instrument to specifically measure engineering identity and career aspirations in adolescents and preadolescents. When possible, we utilized existing measures of STEM identity and career aspirations, adapting them when necessary to the elementary school level and to fit the engineering context. The instrument was developed within the context of a multi-year, NSF-funded research project examining the dynamics between undergraduate outreach providers and elementary students to understand the impact of the program on students’ engineering identity and career aspirations. Three phases of survey development were conducted that involved 492 elementary students from diverse communities in the United States. Three sets of items were developed and/or adapted throughout the four phases. The first set of items assessed Engineering Identity. Recent research suggests that identity consists of three components: recognition, interest, and performance/competence. Items assessing each of these constructs were included in the survey. The second and third sets of items reflected Career Interests and Aspirations. Because elementary and middle school students often have a limited or nascent awareness of what engineers do or misconceptions about what a job in science or engineering entails, it is problematic to measure their engineering identity or career aspirations by directly asking them whether they want to be a scientist/engineer or by using a checklist of broad career categories. Therefore, similar to other researchers, the second set of items assessed the types of activities that students are interested in doing as part of a future career, including both non-STEM and STEM (general and engineering-specific) activities. These items were created by the research team or adapted from activity lists used in existing research. The third set of items drew from career counseling measures relying on Holland’s Career Codes. We adapted the format of these instruments by asking students to choose the activity they liked the most from a list of six activities that reflected each of the codes rather than responding to their interest about each activity. Preliminary findings for each set of items will be discussed. Results from the survey contribute to our understanding of engineering identities and career aspirations in preadolescent and adolescent youth. However, our instrument has the potential for broader application in non-engineering STEM environments (e.g., computer science) with minor wording changes to reflect the relevant science subject area. More research is needed in determining its usefulness in this capacity.more » « less
-
Engineering Explorations are curriculum modules that engage children across contexts in learning about science and engineering. We used them to leverage multiple education sectors (K–12 schools, museums, higher education, and afterschool programs) across a community to provide engineering learning experiences for youth, while increasing local teachers’ capacity to deliver high-quality engineering learning opportunities that align with school standards. Focusing on multiple partners that serve youth in the same community provides opportunities for long-term collaborations and programs developed in response to local needs. In a significant shift from earlier sets of standards, the Next Generation Science Standards include engineering design, with the goal of providing students with a foundation “to better engage in and aspire to solve the major societal and environmental challenges they will face in decades ahead” (NGSS Lead States 2013, Appendix I). Including engineering in K–12 standards is a positive step forward in introducing students to engineering; however, K–12 teachers are not prepared to facilitate high-quality engineering activities. Research has consistently shown that elementary teachers are not confident in teaching science, especially physical science, and generally have little knowledge of engineering (Trygstad 2013). K–12 teachers, therefore, will need support. Our goal was to create a program that took advantage of the varied resources across a STEM (science, technology, engineering, and math) education ecosystem to support engineering instruction for youth across multiple contexts, while building the capacity of educators and meeting the needs of each organization. Specifically, we developed mutually reinforcing classroom and field trip activities to improve student learning and a curriculum to improve teacher learning. This challenging task required expertise in school-based standards, engineering education, informal education, teacher professional development, and classroom and museum contexts.more » « less
-
Abstract Background Billions of dollars are spent annually on grant-funded STEM (science, technology, engineering, and mathematics) education programs. These programs help students stay on track toward STEM careers when standard educational practices do not adequately prepare them for these careers. It is important to know that reliable and accurate student participation and completion data are being collected about these programs. This multiple case study investigates how student data are collected and reported for a national STEM education program in the United States, the National Science Foundation (NSF) Advanced Technological Education (ATE) program. Our overall aim is to provide insights to funding agencies, STEM education faculty, and others who are interested in addressing issues related to the collection and reporting of student participation and completion data within their own contexts. Emphasis is placed on the barriers encountered in collecting participation and completion data, particularly with regard to unduplicated participation counts and marketable credential data. The ATE program was selected for this study because there is already a mechanism (known as the ATE Survey) in place for annually collecting systematic data across all projects within the program. Results A multiple case study, including interviews of primary investigators, allowed for in-depth analysis of the ATE Survey’s point-in-time data on project-level participation in various activities, and for identification of the following barriers to tracking student-level data: lack of time and help to gather these data, lack of a consistent system for tracking students across different institutions, and a perceived lack of guidance from the funding agency about what data to track. We also saw that different data are needed from different projects to determine a project’s true impact. Defining “success” the same way across all projects is inadequate. Conclusions Although, due to the limited sample size, these findings cannot be generalized to the larger ATE population, they provide specific insights into the various barriers that projects encounter in collecting participation and completion data.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.5195/jyd.2021.981
