skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Scaling up behavioral science interventions in online education
Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom.  more » « less
Award ID(s):
1646976
PAR ID:
10164956
Author(s) / Creator(s):
; ; ; ; ; ; ; ;
Date Published:
Journal Name:
Proceedings of the National Academy of Sciences
ISSN:
0027-8424
Page Range / eLocation ID:
201921417
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Climate change and biodiversity loss require us to engage the next generation of scientists in addressing global ecological issues. Introducing undergraduate students to citizen science allows them to learn scientific processes and content while contributing to real‐world applications. We conducted a systematic review of literature to (1) identify what types of undergraduate courses and institutions use citizen science, (2) list the projects and platforms that have been implemented in online courses in undergraduate education, (3) examine how students participated in the projects through online courses, and (4) summarize learning objectives and reported benefits of student participation. In all, 44 studies about the use of citizen science in undergraduate online courses were found in 25 papers in the published literature. The most common projects consisted of classification of species or natural history (e.g., iNaturalist), which could be done mainly online but with data collection completed at a location available to the student. Citizen science projects were incorporated into multiple course formats (e.g., lecture, lab) and class sizes, and students were most frequently asked to collect and submit data. The most frequently reported learning outcomes included increased student interest/engagement, improved appreciation for the relevance of science to the “real world,” and practice using the scientific process, but rigorous assessment data were lacking in papers. The use of citizen science in online courses and institutions appears to be increasing, and we encourage faculty using these approaches with students to publish on their efforts, providing details about their implementation, assessment, and course context. 
    more » « less
  2. Shortlidge, Erin (Ed.)
    The ability to navigate scientific obstacles is widely recognized as a hallmark of a scientific disposition and is one predictor of science, technology, engineering, and mathematics persistence for early-career scientists. However, the development of this competency in undergraduate research has been largely underexplored. This study addresses this gap by examining introductory students’ emotional and behavioral responses to research-related challenges and failures that occur in two sequential research-based courses. We describe commonly reported emotions, coping responses, and perceived outcomes and examine relationships between these themes, student demographics, and course enrollment. Students commonly experience frustration, confusion, and disappointment when coping with challenges and failures. Yet the predominance of students report coping responses likely to be adaptive in academic contexts despite experiencing negative emotions. Being enrolled in the second course of a research-based course sequence was related to several shifts in response to challenges during data collection, including less reporting of confusion and fewer reports of learning to be cautious from students. Overall, students in both the first and second courses reported many positive outcomes indicating improvements in their ability to cope with challenge and failure. We assert that educators can improve research-based educational courses by scaffolding students’ research trials, failures, and iterations to support students’ perseverance. 
    more » « less
  3. null (Ed.)
    ABSTRACT The global COVID-19 pandemic left universities with few options but to turn to remote learning. With much effort, STEM courses made this change in modality; however, many laboratory skills, such as measurement and handling equipment, are more difficult to teach in an online learning environment. A cohort of instructors who are part of the NSF RCN-UBE funded Sustainable, Transformative Engagement across a Multi-Institution/Multidisciplinary STEM (STEM 2 ) Network (a working group of faculty from two community colleges and three 4-year universities) analyzed introductory biology and chemistry courses to identify essential laboratory skills that students will need in advanced courses. Seven essential laboratory proficiencies were derived from reviewing disciplinary guiding documents such as AAAS’s Vision and Change in Undergraduate Biology Education, the American Society for Microbiology’s Recommended Curriculum Guidelines for Undergraduate Microbiology Education , and the American Chemical Society’s Guidelines for Chemistry : data analysis, scientific writing, proper handling and disposal of laboratory materials, discipline-specific techniques, measurement, lab safety and personal protective equipment, and interpersonal and collaborative skills. Our analysis has determined that some of these skills are difficult to develop in a remote online setting but could be recovered with appropriate interventions. Skill recovery procedures suggested are a skills “boot camp,” department and college coordinated club events, and a triage course. The authors recommend that one of these three recovery mechanisms be offered to bridge this skill gap and better prepare STEM students for upper-level science courses and the real world. 
    more » « less
  4. Even prior to the COVID-19 pandemic, online learning had become a fundamental part of post-secondary education. At the same time, empirical evidence from the last decade documents higher dropout online in comparison to face-to-face courses for some students. Thus, while online learning may provide students access to post-secondary education, concerns about academic momentum and degree attainment dominate the higher education online learning landscape. Because course completion is often used as a measure of effectiveness, there is a strong need for institutions to be able to predict the potential persistence of online students to direct efforts towards ameliorating dropout. Yet currently, a widely tested and validated archetypical predictive model of retention and success does not exist for undergraduate online learning. This integrative review of the literature examines evidence gathered over the last decade, organizing and summarizing major findings, to help identify potential undergraduate student characteristics for inclusion in such a model. The body of literature collected in this review suggests ten factors for consideration. 
    more » « less
  5. The purpose of this study is to develop an instrument to measure student perceptions about the learning experiences in their online undergraduate engineering courses. Online education continues to grow broadly in higher education, but the movement toward acceptance and comprehensive utilization of online learning has generally been slower in engineering. Recently, however, there have been indicators that this could be changing. For example, ABET has accredited online undergraduate engineering degrees at Stony Brook University and Arizona State University (ASU), and an increasing number of other undergraduate engineering programs also offer online courses. During this period of transition in engineering education, further investigation about the online modality in the context of engineering education is needed, and survey instrumentation can support such investigations. The instrument presented in this paper is grounded in a Model for Online Course-level Persistence in Engineering (MOCPE), which was developed by our research team by combining two motivational frameworks used to study student persistence: the Expectancy x Value Theory of Achievement Motivation (EVT), and the ARCS model of motivational design. The initial MOCPE instrument contained 79 items related to students’ perceptions about the characteristics of their courses (i.e., the online learning management system, instructor practices, and peer support), expectancies of course success, course task values, perceived course difficulties, and intention to persist in the course. Evidence of validity and reliability was collected using a three-step process. First, we tested face and content validity of the instrument with experts in online engineering education and online undergraduate engineering students. Next, the survey was administered to the online undergraduate engineering student population at a large, Southwestern public university, and an exploratory factor analysis (EFA) was conducted on the responses. Lastly, evidence of reliability was obtained by computing the internal consistency of each resulting scale. The final instrument has seven scales with 67 items across 10 factors. The Cronbach alpha values for these scales range from 0.85 to 0.97. The full paper will provide complete details about the development and psychometric evaluation of the instrument, including evidence of and reliability. The instrument described in this paper will ultimately be used as part of a larger, National Science Foundation-funded project investigating the factors influencing online undergraduate engineering student persistence. It is currently being used in the context of this project to conduct a longitudinal study intended to understand the relationships between the experiences of online undergraduate engineering students in their courses and their intentions to persist in the course. We anticipate that the instrument will be of interest and use to other engineering education researchers who are also interested in studying the population of online students. 
    more » « less