skip to main content

Explore Research Products in the NSF-PAR

Title: Scaling up behavioral science interventions in online education
Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom.  more » « less
Award ID(s):
1646976
NSF-PAR ID:
10164956
Author(s) / Creator(s):
; ; ; ; ; ; ; ;
Date Published:
Journal Name:
Proceedings of the National Academy of Sciences
ISSN:
0027-8424
Page Range / eLocation ID:
201921417
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ( , CBE—Life Sciences Education)
    Shortlidge, Erin (Ed.)
    The ability to navigate scientific obstacles is widely recognized as a hallmark of a scientific disposition and is one predictor of science, technology, engineering, and mathematics persistence for early-career scientists. However, the development of this competency in undergraduate research has been largely underexplored. This study addresses this gap by examining introductory students’ emotional and behavioral responses to research-related challenges and failures that occur in two sequential research-based courses. We describe commonly reported emotions, coping responses, and perceived outcomes and examine relationships between these themes, student demographics, and course enrollment. Students commonly experience frustration, confusion, and disappointment when coping with challenges and failures. Yet the predominance of students report coping responses likely to be adaptive in academic contexts despite experiencing negative emotions. Being enrolled in the second course of a research-based course sequence was related to several shifts in response to challenges during data collection, including less reporting of confusion and fewer reports of learning to be cautious from students. Overall, students in both the first and second courses reported many positive outcomes indicating improvements in their ability to cope with challenge and failure. We assert that educators can improve research-based educational courses by scaffolding students’ research trials, failures, and iterations to support students’ perseverance. 
    more » « less
  2. ; ; ; ; ( , Journal of Microbiology & Biology Education)
    null (Ed.)
    ABSTRACT The global COVID-19 pandemic left universities with few options but to turn to remote learning. With much effort, STEM courses made this change in modality; however, many laboratory skills, such as measurement and handling equipment, are more difficult to teach in an online learning environment. A cohort of instructors who are part of the NSF RCN-UBE funded Sustainable, Transformative Engagement across a Multi-Institution/Multidisciplinary STEM (STEM 2 ) Network (a working group of faculty from two community colleges and three 4-year universities) analyzed introductory biology and chemistry courses to identify essential laboratory skills that students will need in advanced courses. Seven essential laboratory proficiencies were derived from reviewing disciplinary guiding documents such as AAAS’s Vision and Change in Undergraduate Biology Education, the American Society for Microbiology’s Recommended Curriculum Guidelines for Undergraduate Microbiology Education , and the American Chemical Society’s Guidelines for Chemistry : data analysis, scientific writing, proper handling and disposal of laboratory materials, discipline-specific techniques, measurement, lab safety and personal protective equipment, and interpersonal and collaborative skills. Our analysis has determined that some of these skills are difficult to develop in a remote online setting but could be recovered with appropriate interventions. Skill recovery procedures suggested are a skills “boot camp,” department and college coordinated club events, and a triage course. The authors recommend that one of these three recovery mechanisms be offered to bridge this skill gap and better prepare STEM students for upper-level science courses and the real world. 
    more » « less
  3. ; ; ( , ASEE Annual Conference Proceedings 2020)
    There is a critical need for more students with engineering and computer science majors to enter into, persist in, and graduate from four-year postsecondary institutions. Increasing the diversity of the workforce by inclusive practices in engineering and science is also a profound identified need. According to national statistics, the largest groups of underrepresented minority students in engineering and science attend U.S. public higher education institutions. Most often, a large proportion of these students come to colleges and universities with unique challenges and needs, and are more likely to be first in their family to attend college. In response to these needs, engineering education researchers and practitioners have developed, implemented and assessed interventions to provide support and help students succeed in college, particularly in their first year. These interventions typically target relatively small cohorts of students and can be managed by a small number of faculty and staff. In this paper, we report on “work in progress” research in a large-scale, first-year engineering and computer science intervention program at a public, comprehensive university using multivariate comparative statistical approaches. Large-scale intervention programs are especially relevant to minority serving institutions that prepare growing numbers of students who are first in their family to attend college and who are also under-resourced, financially. These students most often encounter academic difficulties and come to higher education with challenging experiences and backgrounds. Our studied first-year intervention program, first piloted in 2015, is now in its 5th year of implementation. Its intervention components include: (a) first-year block schedules, (b) project-based introductory engineering and computer science courses, (c) an introduction to mechanics course, which provides students with the foundation needed to succeed in a traditional physics sequence, and (d) peer-led supplemental instruction workshops for calculus, physics and chemistry courses. This intervention study responds to three research questions: (1) What role does the first-year intervention’s components play in students’ persistence in engineering and computer science majors across undergraduate program years? (2) What role do particular pedagogical and cocurricular support structures play in students’ successes? And (3) What role do various student socio-demographic and experiential factors play in the effectiveness of first-year interventions? To address these research questions and therefore determine the formative impact of the firstyear engineering and computer science program on which we are conducting research, we have collected diverse student data including grade point averages, concept inventory scores, and data from a multi-dimensional questionnaire that measures students’ use of support practices across their four to five years in their degree program, and diverse background information necessary to determine the impact of such factors on students’ persistence to degree. Background data includes students’ experiences prior to enrolling in college, their socio-demographic characteristics, and their college social capital throughout their higher education experience. For this research, we compared students who were enrolled in the first-year intervention program to those who were not enrolled in the first-year intervention. We have engaged in cross-sectional 2 data collection from students’ freshman through senior years and employed multivariate statistical analytical techniques on the collected student data. Results of these analyses were interesting and diverse. Generally, in terms of backgrounds, our research indicates that students’ parental education is positively related to their success in engineering and computer science across program years. Likewise, longitudinally (across program years), students’ college social capital predicted their academic success and persistence to degree. With regard to the study’s comparative research of the first-year intervention, our results indicate that students who were enrolled in the first-year intervention program as freshmen continued to use more support practices to assist them in academic success across their degree matriculation compared to students who were not in the first-year program. This suggests that the students continued to recognize the value of such supports as a consequence of having supports required as first-year students. In terms of students’ understanding of scientific or engineering-focused concepts, we found significant impact resulting from student support practices that were academically focused. We also found that enrolling in the first-year intervention was a significant predictor of the time that students spent preparing for classes and ultimately their grade point average, especially in STEM subjects across students’ years in college. In summary, we found that the studied first-year intervention program has longitudinal, positive impacts on students’ success as they navigate through their undergraduate experiences toward engineering and computer science degrees. 
    more » « less
  4. ; ( , Journal of Research in Science Teaching)
    Abstract

    In response to Li, Reigh, He, and Miller's commentary,Can we and should we use artificial intelligence for formative assessment in science, we argue that artificial intelligence (AI) is already being widely employed in formative assessment across various educational contexts. While agreeing with Li et al.'s call for further studies on equity issues related to AI, we emphasize the need for science educators to adapt to the AI revolution that has outpaced the research community. We challenge the somewhat restrictive view of formative assessment presented by Li et al., highlighting the significant contributions of AI in providing formative feedback to students, assisting teachers in assessment practices, and aiding in instructional decisions. We contend that AI‐generated scores should not be equated with the entirety of formative assessment practice; no single assessment tool can capture all aspects of student thinking and backgrounds. We address concerns raised by Li et al. regarding AI bias and emphasize the importance of empirical testing and evidence‐based arguments in referring to bias. We assert that AI‐based formative assessment does not necessarily lead to inequity and can, in fact, contribute to more equitable educational experiences. Furthermore, we discuss how AI can facilitate the diversification of representational modalities in assessment practices and highlight the potential benefits of AI in saving teachers’ time and providing them with valuable assessment information. We call for a shift in perspective, from viewing AI as a problem to be solved to recognizing its potential as a collaborative tool in education. We emphasize the need for future research to focus on the effective integration of AI in classrooms, teacher education, and the development of AI systems that can adapt to diverse teaching and learning contexts. We conclude by underlining the importance of addressing AI bias, understanding its implications, and developing guidelines for best practices in AI‐based formative assessment.

     
    more » « less
  5. ; ; ( , Frontiers in Education)
    Asynchronous online courses are popular because they offer benefits to both students and instructors. Students benefit from the convenience, flexibility, affordability, freedom of geography, and access to information. Instructors and institutions benefit by having a broad geographical reach, scalability, and cost-savings of no physical classroom. A challenge with asynchronous online courses is providing students with engaging, collaborative and interactive experiences. Here, we describe how an online poster symposium can be used as a unique educational experience and assessment tool in a large-enrollment (e.g., 500 students), asynchronous, natural science, general education (GE) course. The course, Introduction to Environmental Science (ENR2100), was delivered using distance education (DE) technology over a 15-week semester. In ENR2100 students learn a variety of topics including freshwater resources, surface water, aquifers, groundwater hydrology, ecohydrology, coastal and ocean circulation, drinking water, water purification, wastewater treatment, irrigation, urban and agricultural runoff, sediment and contaminant transport, water cycle, water policy, water pollution, and water quality. Here we present a is a long-term study that takes place from 2017 to 2022 (before and after COVID-19) and involved 5,625 students over 8 semesters. Scaffolding was used to break up the poster project into smaller, more manageable assignments, which students completed throughout the semester. Instructions, examples, how-to videos, book chapters and rubrics were used to accommodate Students’ different levels of knowledge. Poster assignments were designed to teach students how to find and critically evaluate sources of information, recognize the changing nature of scientific knowledge, methods, models and tools, understand the application of scientific data and technological developments, and evaluate the social and ethical implications of natural science discoveries. At the end of the semester students participated in an asynchronous online poster symposium. Each student delivered a 5-min poster presentation using an online learning management system and completed peer reviews of their classmates’ posters using a rubric. This poster project met the learning objectives of our natural science, general education course and taught students important written, visual and verbal communication skills. Students were surveyed to determine, which parts of the course were most effective for instruction and learning. Students ranked poster assignments first, followed closely by lectures videos. Approximately 87% of students were confident that they could produce a scientific poster in the future and 80% of students recommended virtual poster symposiums for online courses. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email