skip to main content

Title: Scaling up behavioral science interventions in online education
Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention more » policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom. « less
; ; ; ; ; ; ; ;
Award ID(s):
Publication Date:
Journal Name:
Proceedings of the National Academy of Sciences
Page Range or eLocation-ID:
Sponsoring Org:
National Science Foundation
More Like this
  1. Shortlidge, Erin (Ed.)
    The ability to navigate scientific obstacles is widely recognized as a hallmark of a scientific disposition and is one predictor of science, technology, engineering, and mathematics persistence for early-career scientists. However, the development of this competency in undergraduate research has been largely underexplored. This study addresses this gap by examining introductory students’ emotional and behavioral responses to research-related challenges and failures that occur in two sequential research-based courses. We describe commonly reported emotions, coping responses, and perceived outcomes and examine relationships between these themes, student demographics, and course enrollment. Students commonly experience frustration, confusion, and disappointment when coping with challengesmore »and failures. Yet the predominance of students report coping responses likely to be adaptive in academic contexts despite experiencing negative emotions. Being enrolled in the second course of a research-based course sequence was related to several shifts in response to challenges during data collection, including less reporting of confusion and fewer reports of learning to be cautious from students. Overall, students in both the first and second courses reported many positive outcomes indicating improvements in their ability to cope with challenge and failure. We assert that educators can improve research-based educational courses by scaffolding students’ research trials, failures, and iterations to support students’ perseverance.« less
  2. ABSTRACT The global COVID-19 pandemic left universities with few options but to turn to remote learning. With much effort, STEM courses made this change in modality; however, many laboratory skills, such as measurement and handling equipment, are more difficult to teach in an online learning environment. A cohort of instructors who are part of the NSF RCN-UBE funded Sustainable, Transformative Engagement across a Multi-Institution/Multidisciplinary STEM (STEM 2 ) Network (a working group of faculty from two community colleges and three 4-year universities) analyzed introductory biology and chemistry courses to identify essential laboratory skills that students will need in advanced courses.more »Seven essential laboratory proficiencies were derived from reviewing disciplinary guiding documents such as AAAS’s Vision and Change in Undergraduate Biology Education, the American Society for Microbiology’s Recommended Curriculum Guidelines for Undergraduate Microbiology Education , and the American Chemical Society’s Guidelines for Chemistry : data analysis, scientific writing, proper handling and disposal of laboratory materials, discipline-specific techniques, measurement, lab safety and personal protective equipment, and interpersonal and collaborative skills. Our analysis has determined that some of these skills are difficult to develop in a remote online setting but could be recovered with appropriate interventions. Skill recovery procedures suggested are a skills “boot camp,” department and college coordinated club events, and a triage course. The authors recommend that one of these three recovery mechanisms be offered to bridge this skill gap and better prepare STEM students for upper-level science courses and the real world.« less
  3. There is a critical need for more students with engineering and computer science majors to enter into, persist in, and graduate from four-year postsecondary institutions. Increasing the diversity of the workforce by inclusive practices in engineering and science is also a profound identified need. According to national statistics, the largest groups of underrepresented minority students in engineering and science attend U.S. public higher education institutions. Most often, a large proportion of these students come to colleges and universities with unique challenges and needs, and are more likely to be first in their family to attend college. In response to thesemore »needs, engineering education researchers and practitioners have developed, implemented and assessed interventions to provide support and help students succeed in college, particularly in their first year. These interventions typically target relatively small cohorts of students and can be managed by a small number of faculty and staff. In this paper, we report on “work in progress” research in a large-scale, first-year engineering and computer science intervention program at a public, comprehensive university using multivariate comparative statistical approaches. Large-scale intervention programs are especially relevant to minority serving institutions that prepare growing numbers of students who are first in their family to attend college and who are also under-resourced, financially. These students most often encounter academic difficulties and come to higher education with challenging experiences and backgrounds. Our studied first-year intervention program, first piloted in 2015, is now in its 5th year of implementation. Its intervention components include: (a) first-year block schedules, (b) project-based introductory engineering and computer science courses, (c) an introduction to mechanics course, which provides students with the foundation needed to succeed in a traditional physics sequence, and (d) peer-led supplemental instruction workshops for calculus, physics and chemistry courses. This intervention study responds to three research questions: (1) What role does the first-year intervention’s components play in students’ persistence in engineering and computer science majors across undergraduate program years? (2) What role do particular pedagogical and cocurricular support structures play in students’ successes? And (3) What role do various student socio-demographic and experiential factors play in the effectiveness of first-year interventions? To address these research questions and therefore determine the formative impact of the firstyear engineering and computer science program on which we are conducting research, we have collected diverse student data including grade point averages, concept inventory scores, and data from a multi-dimensional questionnaire that measures students’ use of support practices across their four to five years in their degree program, and diverse background information necessary to determine the impact of such factors on students’ persistence to degree. Background data includes students’ experiences prior to enrolling in college, their socio-demographic characteristics, and their college social capital throughout their higher education experience. For this research, we compared students who were enrolled in the first-year intervention program to those who were not enrolled in the first-year intervention. We have engaged in cross-sectional 2 data collection from students’ freshman through senior years and employed multivariate statistical analytical techniques on the collected student data. Results of these analyses were interesting and diverse. Generally, in terms of backgrounds, our research indicates that students’ parental education is positively related to their success in engineering and computer science across program years. Likewise, longitudinally (across program years), students’ college social capital predicted their academic success and persistence to degree. With regard to the study’s comparative research of the first-year intervention, our results indicate that students who were enrolled in the first-year intervention program as freshmen continued to use more support practices to assist them in academic success across their degree matriculation compared to students who were not in the first-year program. This suggests that the students continued to recognize the value of such supports as a consequence of having supports required as first-year students. In terms of students’ understanding of scientific or engineering-focused concepts, we found significant impact resulting from student support practices that were academically focused. We also found that enrolling in the first-year intervention was a significant predictor of the time that students spent preparing for classes and ultimately their grade point average, especially in STEM subjects across students’ years in college. In summary, we found that the studied first-year intervention program has longitudinal, positive impacts on students’ success as they navigate through their undergraduate experiences toward engineering and computer science degrees.« less
  4. Asynchronous online courses are popular because they offer benefits to both students and instructors. Students benefit from the convenience, flexibility, affordability, freedom of geography, and access to information. Instructors and institutions benefit by having a broad geographical reach, scalability, and cost-savings of no physical classroom. A challenge with asynchronous online courses is providing students with engaging, collaborative and interactive experiences. Here, we describe how an online poster symposium can be used as a unique educational experience and assessment tool in a large-enrollment (e.g., 500 students), asynchronous, natural science, general education (GE) course. The course, Introduction to Environmental Science (ENR2100), wasmore »delivered using distance education (DE) technology over a 15-week semester. In ENR2100 students learn a variety of topics including freshwater resources, surface water, aquifers, groundwater hydrology, ecohydrology, coastal and ocean circulation, drinking water, water purification, wastewater treatment, irrigation, urban and agricultural runoff, sediment and contaminant transport, water cycle, water policy, water pollution, and water quality. Here we present a is a long-term study that takes place from 2017 to 2022 (before and after COVID-19) and involved 5,625 students over 8 semesters. Scaffolding was used to break up the poster project into smaller, more manageable assignments, which students completed throughout the semester. Instructions, examples, how-to videos, book chapters and rubrics were used to accommodate Students’ different levels of knowledge. Poster assignments were designed to teach students how to find and critically evaluate sources of information, recognize the changing nature of scientific knowledge, methods, models and tools, understand the application of scientific data and technological developments, and evaluate the social and ethical implications of natural science discoveries. At the end of the semester students participated in an asynchronous online poster symposium. Each student delivered a 5-min poster presentation using an online learning management system and completed peer reviews of their classmates’ posters using a rubric. This poster project met the learning objectives of our natural science, general education course and taught students important written, visual and verbal communication skills. Students were surveyed to determine, which parts of the course were most effective for instruction and learning. Students ranked poster assignments first, followed closely by lectures videos. Approximately 87% of students were confident that they could produce a scientific poster in the future and 80% of students recommended virtual poster symposiums for online courses.« less
  5. Use of geospatial technology in higher education facilitates student engagement, promotes deeper understanding of material, and supports inquiry-based learning. However, technology must be applied strategically to generate optimal results. While use of web-based interactive modules and short video are constructive in curriculum, it is beneficial to combine this with exposure to hands-on, experimental, field-based technologies. Experiential learning with technology in the physical environment allows students to understand both the challenges and achievements of scientific investigation. This creates a more comprehensive understanding of science as an iterative process of experimentation and investigation and enrichens course material. This paper explores the uniquelymore »advantageous opportunity Geography educators have to combine classroom-based technology with field-based educational experiences. Classroom use of Geographic Information Systems (GIS) and Remotely Sensed data is increasingly accessible with abundant free educational resources. In addition, field-based use of technology can promote location awareness and spatial critical thinking with the use of GPS-based activities. GPS-based educational units also highlight the growing field of citizen science and can be designed as service-based learning opportunities. Use of highly affordable micro unmanned aerial vehicles (UAV) demonstrates data collection procedures. In addition, exposure to Surveying techniques and the field of Geomatics highlights real-world applications of geographic technology. We discuss the use of geospatial technologies in introductory and advanced higher education courses and examine how technology can encourage access to scientific inquiry throughout the student population.« less