skip to main content


Title: Measuring the long-term effects of informal science education experiences: challenges and potential solutions
Abstract Despite the fact that most science learning takes place outside of school, little is known about how engagement in informal science learning (ISL) experiences affects learners’ knowledge, skill development, interest, or identities over long periods of time. Although substantial ISL research has documented short-term outcomes such as the learning that takes place during a science center visit, research suggests that the genuine benefits of informal experiences are long-term transformations in learners as they pursue a “cascade” of experiences subsequent to the initial educational event. However, a number of major methodological challenges have limited longitudinal research projects investigating the long-term effects of ISL experiences. In this paper we identify and address four key issues surrounding the critical but challenging area of how to study and measure the long-term effects or impacts of ISL experiences: attribution, attrition, data collection, and analytic approaches. Our objective is to provide guidance to ISL researchers wishing to engage in long-term investigations of learner outcomes and to begin a dialogue about how best to address the numerous challenges involved in this work.  more » « less
Award ID(s):
1514593
NSF-PAR ID:
10285659
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Disciplinary and Interdisciplinary Science Education Research
Volume:
3
Issue:
1
ISSN:
2662-2300
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Aims and objectives/purpose/research questions: Language brokering (LB) is an informal translation experience where bilinguals serve as linguistic and cultural intermediaries for family members. LB may have long-term socio-emotional and cognitive outcomes, yet little is known about its effects on executive functions (EFs). This study examines how first language (L1) proficiency and negative emotions tied to language brokering experiences affect EF performance on a Simon task (ST). Design/methodology/approach: Fifty-three Mexican American Spanish–English bilinguals with LB experience performed a ST, and reported their feelings towards LB for their mother. Data and analysis: Mean reaction times (RTs) and accuracy rates for correct ST trials were analyzed using linear mixed effects modeling, with trial type, proficiency and negative emotions tied to LB experience as factors and their interactions as additional predictors. Findings/conclusions: The L1 proficiency and negative emotions tied to brokering experiences have divergent, but combined effects on EF. Contrary to our hypotheses, low L1 proficiency predicted better performance and the smallest Simon effect was found for brokers with low L1 proficiency and low negative emotional brokering experiences. However, high L1 proficiency predicted better performance (smallest RTs) regardless of negative emotions tied to brokering experiences. Originality: This study takes a different perspective on the examination of individual differences among bilinguals, in which we examine how negative emotions tied to brokering experiences coupled with L1 proficiency relates to EF performance. Significance/implications: Our results provide support for the need to understand how individual differences in bilingual language experiences, such as L1 proficiency and negative emotions tied to LB, interact with performance on the ST. 
    more » « less
  2. null (Ed.)
    To meet the rising demand for computer science (CS) courses, K-12 educators need to be prepared to teach introductory concepts and skills in courses such as Computer Science Principles (CSP), which takes a breadth-first approach to CS and includes topics beyond programming such as data, impacts of computing, and networks. Educators are now also being asked to teach more advanced concepts in courses such as the College Board's Advanced Placement Computer Science A (CSA) course, which focuses on advanced programming using Java and includes topics such as objects, inheritance, arrays, and recursion. Traditional CSA curricula have not used content or pedagogy designed to engage a broad range of learners and support their success. Unlike CSP, which is attracting more underrepresented students to computing as it was designed, CSA continues to enroll mostly male, white, and Asian students [College Board 2019, Ericson 2020, Sax 2020]. In order to expand CS education opportunities, it is crucial that students have an engaging experience in CSA similar to CSP. Well-designed differentiated professional development (PD) that focuses on content and pedagogy is necessary to meet individual teacher needs, to successfully build teacher skills and confidence to teach CSA, and to improve engagement with students [Darling-Hammond 2017]. It is critical that as more CS opportunities and courses are developed, teachers remain engaged with their own learning in order to build their content knowledge and refine their teaching practice [CSTA 2020]. CSAwesome, developed and piloted in 2019, offers a College Board endorsed AP CSA curriculum and PD focused on supporting the transition of teachers and students from CSP to CSA. This poster presents preliminary findings aimed at exploring the supports and challenges new-to-CSA high school level educators face when transitioning from teaching an introductory, breadth-first course such as CSP to teaching the more challenging, programming-focused CSA course. Five teachers who completed the online CSAwesome summer 2020 PD completed interviews in spring 2021. The project employed an inductive coding scheme to analyze interview transcriptions and qualitative notes from teachers about their experiences learning, teaching, and implementing CSP and CSA curricula. Initial findings suggest that teachers’ experience in the CSAwesome PD may improve their confidence in teaching CSA, ability to effectively use inclusive teaching practices, ability to empathize with their students, problem-solving skills, and motivation to persist when faced with challenges and difficulties. Teachers noted how the CSAwesome PD provided them with a student perspective and increased feelings of empathy. Participants spoke about the implications of the COVID-19 pandemic on their own learning, student learning, and teaching style. Teachers enter the PD with many different backgrounds, CS experience levels, and strengths, however, new-to-CSA teachers require further PD on content and pedagogy to transition between CSP and CSA. Initial results suggest that the CSAwesome PD may have an impact on long-term teacher development as new-to-CSA teachers who participated indicated a positive impact on their teaching practices, ideologies, and pedagogies. 
    more » « less
  3. null (Ed.)
    Abstract Sense of place describes both affective and cognitive — emotional and intellectual — connections to place. Affective outcomes, tied to arts and humanities education, can facilitate these connections. But little research explores environmental science, arts and humanities (eSAH) curricula on place relationships. Additionally, most research on the sense of place focuses on repeated visits to a place over time, rather than short-term experiences like a field trip. Finally, digital technology is a growing trend across science education, but little research investigates its use in field-based contexts. Our research begins to address these gaps. This article describes an eSAH field trip for middle and high school learners. Using a conventional content analysis, we present pilot data from two high school field trips. Our findings illuminate a framework for understanding active and passive place relationships in the context of short-term interdisciplinary field learning experiences. 
    more » « less
  4. Need/Motivation (e.g., goals, gaps in knowledge) The ESTEEM implemented a STEM building capacity project through students’ early access to a sustainable and innovative STEM Stepping Stones, called Micro-Internships (MI). The goal is to reap key benefits of a full-length internship and undergraduate research experiences in an abbreviated format, including access, success, degree completion, transfer, and recruiting and retaining more Latinx and underrepresented students into the STEM workforce. The MIs are designed with the goals to provide opportunities for students at a community college and HSI, with authentic STEM research and applied learning experiences (ALE), support for appropriate STEM pathway/career, preparation and confidence to succeed in STEM and engage in summer long REUs, and with improved outcomes. The MI projects are accessible early to more students and build momentum to better overcome critical obstacles to success. The MIs are shorter, flexibly scheduled throughout the year, easily accessible, and participation in multiple MI is encouraged. ESTEEM also establishes a sustainable and collaborative model, working with partners from BSCS Science Education, for MI’s mentor, training, compliance, and building capacity, with shared values and practices to maximize the improvement of student outcomes. New Knowledge (e.g., hypothesis, research questions) Research indicates that REU/internship experiences can be particularly powerful for students from Latinx and underrepresented groups in STEM. However, those experiences are difficult to access for many HSI-community college students (85% of our students hold off-campus jobs), and lack of confidence is a barrier for a majority of our students. The gap between those who can and those who cannot is the “internship access gap.” This project is at a central California Community College (CCC) and HSI, the only affordable post-secondary option in a region serving a historically underrepresented population in STEM, including 75% Hispanic, and 87% have not completed college. MI is designed to reduce inequalities inherent in the internship paradigm by providing access to professional and research skills for those underserved students. The MI has been designed to reduce barriers by offering: shorter duration (25 contact hours); flexible timing (one week to once a week over many weeks); open access/large group; and proximal location (on-campus). MI mentors participate in week-long summer workshops and ongoing monthly community of practice with the goal of co-constructing a shared vision, engaging in conversations about pedagogy and learning, and sustaining the MI program going forward. Approach (e.g., objectives/specific aims, research methodologies, and analysis) Research Question and Methodology: We want to know: How does participation in a micro-internship affect students’ interest and confidence to pursue STEM? We used a mixed-methods design triangulating quantitative Likert-style survey data with interpretive coding of open-responses to reveal themes in students’ motivations, attitudes toward STEM, and confidence. Participants: The study sampled students enrolled either part-time or full-time at the community college. Although each MI was classified within STEM, they were open to any interested student in any major. Demographically, participants self-identified as 70% Hispanic/Latinx, 13% Mixed-Race, and 42 female. Instrument: Student surveys were developed from two previously validated instruments that examine the impact of the MI intervention on student interest in STEM careers and pursuing internships/REUs. Also, the pre- and post (every e months to assess longitudinal outcomes) -surveys included relevant open response prompts. The surveys collected students’ demographics; interest, confidence, and motivation in pursuing a career in STEM; perceived obstacles; and past experiences with internships and MIs. 171 students responded to the pre-survey at the time of submission. Outcomes (e.g., preliminary findings, accomplishments to date) Because we just finished year 1, we lack at this time longitudinal data to reveal if student confidence is maintained over time and whether or not students are more likely to (i) enroll in more internships, (ii) transfer to a four-year university, or (iii) shorten the time it takes for degree attainment. For short term outcomes, students significantly Increased their confidence to continue pursuing opportunities to develop within the STEM pipeline, including full-length internships, completing STEM degrees, and applying for jobs in STEM. For example, using a 2-tailed t-test we compared means before and after the MI experience. 15 out of 16 questions that showed improvement in scores were related to student confidence to pursue STEM or perceived enjoyment of a STEM career. Finding from the free-response questions, showed that the majority of students reported enrolling in the MI to gain knowledge and experience. After the MI, 66% of students reported having gained valuable knowledge and experience, and 35% of students spoke about gaining confidence and/or momentum to pursue STEM as a career. Broader Impacts (e.g., the participation of underrepresented minorities in STEM; development of a diverse STEM workforce, enhanced infrastructure for research and education) The ESTEEM project has the potential for a transformational impact on STEM undergraduate education’s access and success for underrepresented and Latinx community college students, as well as for STEM capacity building at Hartnell College, a CCC and HSI, for students, faculty, professionals, and processes that foster research in STEM and education. Through sharing and transfer abilities of the ESTEEM model to similar institutions, the project has the potential to change the way students are served at an early and critical stage of their higher education experience at CCC, where one in every five community college student in the nation attends a CCC, over 67% of CCC students identify themselves with ethnic backgrounds that are not White, and 40 to 50% of University of California and California State University graduates in STEM started at a CCC, thus making it a key leverage point for recruiting and retaining a more diverse STEM workforce. 
    more » « less
  5. Obeid, Iyad ; Picone, Joseph ; Selesnick, Ivan (Ed.)
    The Neural Engineering Data Consortium (NEDC) is developing a large open source database of high-resolution digital pathology images known as the Temple University Digital Pathology Corpus (TUDP) [1]. Our long-term goal is to release one million images. We expect to release the first 100,000 image corpus by December 2020. The data is being acquired at the Department of Pathology at Temple University Hospital (TUH) using a Leica Biosystems Aperio AT2 scanner [2] and consists entirely of clinical pathology images. More information about the data and the project can be found in Shawki et al. [3]. We currently have a National Science Foundation (NSF) planning grant [4] to explore how best the community can leverage this resource. One goal of this poster presentation is to stimulate community-wide discussions about this project and determine how this valuable resource can best meet the needs of the public. The computing infrastructure required to support this database is extensive [5] and includes two HIPAA-secure computer networks, dual petabyte file servers, and Aperio’s eSlide Manager (eSM) software [6]. We currently have digitized over 50,000 slides from 2,846 patients and 2,942 clinical cases. There is an average of 12.4 slides per patient and 10.5 slides per case with one report per case. The data is organized by tissue type as shown below: Filenames: tudp/v1.0.0/svs/gastro/000001/00123456/2015_03_05/0s15_12345/0s15_12345_0a001_00123456_lvl0001_s000.svs tudp/v1.0.0/svs/gastro/000001/00123456/2015_03_05/0s15_12345/0s15_12345_00123456.docx Explanation: tudp: root directory of the corpus v1.0.0: version number of the release svs: the image data type gastro: the type of tissue 000001: six-digit sequence number used to control directory complexity 00123456: 8-digit patient MRN 2015_03_05: the date the specimen was captured 0s15_12345: the clinical case name 0s15_12345_0a001_00123456_lvl0001_s000.svs: the actual image filename consisting of a repeat of the case name, a site code (e.g., 0a001), the type and depth of the cut (e.g., lvl0001) and a token number (e.g., s000) 0s15_12345_00123456.docx: the filename for the corresponding case report We currently recognize fifteen tissue types in the first installment of the corpus. The raw image data is stored in Aperio’s “.svs” format, which is a multi-layered compressed JPEG format [3,7]. Pathology reports containing a summary of how a pathologist interpreted the slide are also provided in a flat text file format. A more complete summary of the demographics of this pilot corpus will be presented at the conference. Another goal of this poster presentation is to share our experiences with the larger community since many of these details have not been adequately documented in scientific publications. There are quite a few obstacles in collecting this data that have slowed down the process and need to be discussed publicly. Our backlog of slides dates back to 1997, meaning there are a lot that need to be sifted through and discarded for peeling or cracking. Additionally, during scanning a slide can get stuck, stalling a scan session for hours, resulting in a significant loss of productivity. Over the past two years, we have accumulated significant experience with how to scan a diverse inventory of slides using the Aperio AT2 high-volume scanner. We have been working closely with the vendor to resolve many problems associated with the use of this scanner for research purposes. This scanning project began in January of 2018 when the scanner was first installed. The scanning process was slow at first since there was a learning curve with how the scanner worked and how to obtain samples from the hospital. From its start date until May of 2019 ~20,000 slides we scanned. In the past 6 months from May to November we have tripled that number and how hold ~60,000 slides in our database. This dramatic increase in productivity was due to additional undergraduate staff members and an emphasis on efficient workflow. The Aperio AT2 scans 400 slides a day, requiring at least eight hours of scan time. The efficiency of these scans can vary greatly. When our team first started, approximately 5% of slides failed the scanning process due to focal point errors. We have been able to reduce that to 1% through a variety of means: (1) best practices regarding daily and monthly recalibrations, (2) tweaking the software such as the tissue finder parameter settings, and (3) experience with how to clean and prep slides so they scan properly. Nevertheless, this is not a completely automated process, making it very difficult to reach our production targets. With a staff of three undergraduate workers spending a total of 30 hours per week, we find it difficult to scan more than 2,000 slides per week using a single scanner (400 slides per night x 5 nights per week). The main limitation in achieving this level of production is the lack of a completely automated scanning process, it takes a couple of hours to sort, clean and load slides. We have streamlined all other aspects of the workflow required to database the scanned slides so that there are no additional bottlenecks. To bridge the gap between hospital operations and research, we are using Aperio’s eSM software. Our goal is to provide pathologists access to high quality digital images of their patients’ slides. eSM is a secure website that holds the images with their metadata labels, patient report, and path to where the image is located on our file server. Although eSM includes significant infrastructure to import slides into the database using barcodes, TUH does not currently support barcode use. Therefore, we manage the data using a mixture of Python scripts and manual import functions available in eSM. The database and associated tools are based on proprietary formats developed by Aperio, making this another important point of community-wide discussion on how best to disseminate such information. Our near-term goal for the TUDP Corpus is to release 100,000 slides by December 2020. We hope to continue data collection over the next decade until we reach one million slides. We are creating two pilot corpora using the first 50,000 slides we have collected. The first corpus consists of 500 slides with a marker stain and another 500 without it. This set was designed to let people debug their basic deep learning processing flow on these high-resolution images. We discuss our preliminary experiments on this corpus and the challenges in processing these high-resolution images using deep learning in [3]. We are able to achieve a mean sensitivity of 99.0% for slides with pen marks, and 98.9% for slides without marks, using a multistage deep learning algorithm. While this dataset was very useful in initial debugging, we are in the midst of creating a new, more challenging pilot corpus using actual tissue samples annotated by experts. The task will be to detect ductal carcinoma (DCIS) or invasive breast cancer tissue. There will be approximately 1,000 images per class in this corpus. Based on the number of features annotated, we can train on a two class problem of DCIS or benign, or increase the difficulty by increasing the classes to include DCIS, benign, stroma, pink tissue, non-neoplastic etc. Those interested in the corpus or in participating in community-wide discussions should join our listserv, nedc_tuh_dpath@googlegroups.com, to be kept informed of the latest developments in this project. You can learn more from our project website: https://www.isip.piconepress.com/projects/nsf_dpath. 
    more » « less