skip to main content


Title: Gauging Informal STEM Youth Program Impact: A Conceptual Framework and a Measurement Instrument
STEM education programs are often formulated with a "hands-on activities" focus across a wide array of topics from robotics to rockets to ecology. Traditionally, the impact of these programs is based on surveys of youth on program-specific experiences or the youths’ interest and impressions of science in general. In this manuscript, we offer a new approach to analyzing science programming design and youth participant impact. The conceptual framework discussed here concentrates on the organization and analysis of common learning activities and instructional strategies. We establish instrument validity and reliability through an analysis of validity threats and pilot study results. We conclude by using this instrument in an example analysis of a STEM education program.  more » « less
Award ID(s):
1811265
NSF-PAR ID:
10335662
Author(s) / Creator(s):
; ; ; ; ; ; ;
Date Published:
Journal Name:
Journal of Youth Development
Volume:
16
Issue:
4
ISSN:
2325-4017
Page Range / eLocation ID:
103 to 133
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Engineering Explorations are curriculum modules that engage children across contexts in learning about science and engineering. We used them to leverage multiple education sectors (K–12 schools, museums, higher education, and afterschool programs) across a community to provide engineering learning experiences for youth, while increasing local teachers’ capacity to deliver high-quality engineering learning opportunities that align with school standards. Focusing on multiple partners that serve youth in the same community provides opportunities for long-term collaborations and programs developed in response to local needs. In a significant shift from earlier sets of standards, the Next Generation Science Standards include engineering design, with the goal of providing students with a foundation “to better engage in and aspire to solve the major societal and environmental challenges they will face in decades ahead” (NGSS Lead States 2013, Appendix I). Including engineering in K–12 standards is a positive step forward in introducing students to engineering; however, K–12 teachers are not prepared to facilitate high-quality engineering activities. Research has consistently shown that elementary teachers are not confident in teaching science, especially physical science, and generally have little knowledge of engineering (Trygstad 2013). K–12 teachers, therefore, will need support. Our goal was to create a program that took advantage of the varied resources across a STEM (science, technology, engineering, and math) education ecosystem to support engineering instruction for youth across multiple contexts, while building the capacity of educators and meeting the needs of each organization. Specifically, we developed mutually reinforcing classroom and field trip activities to improve student learning and a curriculum to improve teacher learning. This challenging task required expertise in school-based standards, engineering education, informal education, teacher professional development, and classroom and museum contexts. 
    more » « less
  2. Abstract

    In and out‐of‐school time (OST) experiences are viewed as complementary in contributing to students’ interest, engagement, and performance in science, technology, engineering, and mathematics (STEM). While tools exist to measure quality in general afterschool settings and others to measure structured science classroom experiences, there is a need for reliable measures of STEM program quality in OST settings such as afterschool programs, summer camps, and museum or science center programming. In this paper we present the development of the Dimensions of Success (DoS) tool, which defines twelve key components of informal, exploratory STEM programming that goes beyond the school day. Additionally, we present a validity argument that includes reliability evidence for the DoS tool based on two studies: Study 1 (n = 284 observations) and Study 2 (n = 56 observations). Our findings suggest that the coherence of the constructs and validity evidence, as well as the training and certification procedures in place for DoS, make it an important tool to understand the quality of STEM experiences for youth beyond the school day. A tool like DoS has several implications, including the ability to make national comparisons across programs, create aggregate databases, improve program quality and professional development, as well as to link program quality to student‐level outcomes.

     
    more » « less
  3. Undergraduate education in the US is racially/ethnically stratified, and there is limited mobility for Black and Latinx BS recipients in STEM majors into the PhD programs from which faculty hiring disproportionately occurs. Bridge programs are proliferating as a means of increasing minoritized students’ enrollment in STEM graduate programs, but little social science examines mechanisms of their impact or how impacts depend on the graduate programs to which students seek access. This sequential mixed methods study of the Cal-Bridge program analyzed trust networks and mechanisms of relational trust as factors in graduate school application, admissions, and enrollment decisions. First, using social network analysis, we examined patterns in the graduate programs to which seven cohorts of Cal-Bridge scholars applied, were admitted, and chose to enroll. Then, we conducted an in-depth case study of the organization in the Cal-Bridge network with the highest centrality: University of California, Irvine’s physics and astronomy PhD program. We find the positive admission and enrollment outcomes at UC Irvine were due to intentional, institutional change at multiple organizational levels. Change efforts complemented the activities of the Cal-Bridge program, creating conditions that cultivated lived experiences of mutual, relational trust between bridge scholars and their faculty advisors and mentors. Findings illustrate mechanisms and antecedents of trust in the transition to graduate education. We use these findings to propose a framework that may inform the design of future research and practical efforts to account for the role of trust in inequities and creating more equitable cultures in STEM. 
    more » « less
  4. Abstract Background Billions of dollars are spent annually on grant-funded STEM (science, technology, engineering, and mathematics) education programs. These programs help students stay on track toward STEM careers when standard educational practices do not adequately prepare them for these careers. It is important to know that reliable and accurate student participation and completion data are being collected about these programs. This multiple case study investigates how student data are collected and reported for a national STEM education program in the United States, the National Science Foundation (NSF) Advanced Technological Education (ATE) program. Our overall aim is to provide insights to funding agencies, STEM education faculty, and others who are interested in addressing issues related to the collection and reporting of student participation and completion data within their own contexts. Emphasis is placed on the barriers encountered in collecting participation and completion data, particularly with regard to unduplicated participation counts and marketable credential data. The ATE program was selected for this study because there is already a mechanism (known as the ATE Survey) in place for annually collecting systematic data across all projects within the program. Results A multiple case study, including interviews of primary investigators, allowed for in-depth analysis of the ATE Survey’s point-in-time data on project-level participation in various activities, and for identification of the following barriers to tracking student-level data: lack of time and help to gather these data, lack of a consistent system for tracking students across different institutions, and a perceived lack of guidance from the funding agency about what data to track. We also saw that different data are needed from different projects to determine a project’s true impact. Defining “success” the same way across all projects is inadequate. Conclusions Although, due to the limited sample size, these findings cannot be generalized to the larger ATE population, they provide specific insights into the various barriers that projects encounter in collecting participation and completion data. 
    more » « less
  5. The National Science Foundation (2019) points to Black, Hispanic/Latino, American Indian, Alaskan Native, Native Hawaiian and other Pacific Island peoples as underrepresented in science, technology, engineering and math (STEM) college majors and professional pathways. This underrepresentation results from an interplay of representation and process, meaning that it in part results from STEM opportunities and programming that do not reflect the experiences of racially and ethnically diverse people, offer insight into the needs of diverse communities, nor address barriers that prevent participation (McGee 2020). One way that institutions of higher education (IHEs) and community organizations try to address inequities in STEM is through pre-college programs aimed at supporting racially and ethnically minoritized (REM) youth. These pre-college STEM programs (PCSPs) work to foster increased STEM awareness and support students in achieving academic milestones that make college pathways more viable. Out-of-school learning (OSL) and informal STEM programming have the potential to fill gaps in STEM K–12 education, as well as complement and support connections with K–12 STEM by offering REM youth opportunities to connect STEM with their lives and influence both their capabilities and dispositions toward STEM (Kitchen et al. 2018). Studies have pointed to positive connections between OSL STEM participation and outcomes such as high school graduation, sustained interest in STEM, and matriculation to university (Penuel, Clark, and Bevan 2016). PCSPs may also support IHE’s development of infrastructure to increase admissions of REM students into college STEM programs. For PCSPs to be an effective element of reducing inequities in STEM, they must successfully engage with and recruit REM youth to participate. In this article, we focus on the recruitment of REM youth into STEM OSL to better understand what is effective for REM youth and their families, as well as highlight connections between OSL and in-school STEM opportunities. Our goals in this work are to (1) identify program practices with the aim of broadening STEM educational and career pathways for REM youth and (2) support strengthened pathways between STEM in K–12 schools and IHEs. 
    more » « less