skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: No Benefit for High-Dosage Time Management Interventions in Online Courses
In past work, time management interventions involving prompts, alerts, and planning tools have successfully nudged students in online courses, leading to higher engagement and improved performance. However, few studies have investigated the effectiveness of these interventions over time, understanding if the effectiveness maintains or changes based on dosage (i.e., how often an intervention is provided). In the current study, we conducted a randomized controlled trial to test if the effect of a time management intervention changes over repeated use. Students at an online computer science course were randomly assigned to receive interventions based on two schedules (i.e., high-dosage vs. low-dosage). We ran a two-way mixed ANOVA, comparing students' assignment start time and performance across several weeks. Unexpectedly, we did not find a significant main effect from the use of the intervention, nor was there an interaction effect between the use of the intervention and week of the course.  more » « less
Award ID(s):
1931419
PAR ID:
10448495
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
L@S '23: Proceedings of the Tenth ACM Conference on Learning @ Scale
Page Range / eLocation ID:
302 to 305
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Inferring the relative strength (i.e. the ratio of reproduction numbers) and relative speed (i.e. the difference between growth rates) of new SARS-CoV-2 variants is critical to predicting and controlling the course of the current pandemic. Analyses of new variants have primarily focused on characterizing changes in the proportion of new variants, implicitly or explicitly assuming that the relative speed remains fixed over the course of an invasion. We use a generation-interval-based framework to challenge this assumption and illustrate how relative strength and speed change over time under two idealized interventions: a constant-strength intervention like idealized vaccination or social distancing, which reduces transmission rates by a constant proportion, and a constant-speed intervention like idealized contact tracing, which isolates infected individuals at a constant rate. In general, constant-strength interventions change the relative speed of a new variant, while constant-speed interventions change its relative strength. Differences in the generation-interval distributions between variants can exaggerate these changes and modify the effectiveness of interventions. Finally, neglecting differences in generation-interval distributions can bias estimates of relative strength. 
    more » « less
  2. In this paper, we present a framework for quantifying the impact of interventions on the full trajectories of students' experiences. The interventions are given periodically based on student performance forecasting from an artificial intelligence (AI) model. We performed a small-scale randomized controlled trial for evaluating the impact of the AI-based intervention system on the undergraduate students of a science, technology, engineering, and mathematics (STEM) course. Intervention messaging content was based on machine learning forecasting models trained on data collected from the students in the same course over the preceding 3 years. Trial results show that the intervention produced a statistically significant increase in the proportion of students that achieved a passing grade. By applying the trajectory-analysis framework we find that the intervention impacts the stories of some types of students more than others, and use this to define new ways of identifying students who are most likely to benefit. Together these outcomes point to the potential and promise of just-in-time interventions for STEM learning and the need for larger fully-powered randomized controlled trials. 
    more » « less
  3. Retrieval practice (i.e., recalling information from memory) and elaboration (i.e., generating meaningful explanations and examples) promote learning, but students underutilize these strategies when studying. We developed a strategy-training intervention addressing prominent barriers to students’ strategy use: lack of knowledge, lack of motivation, and poor management of study time. Undergraduates in an Introductory Biology course were randomly assigned to receive the strategy-training intervention or to a healthy life habits control group. No significant differences were found between the two groups on measures of learning behavior or achievement collected across the semester, emphasizing the challenge of changing students’ learning habits. Future research should investigate strategy training with lower performing students integrated into a course. 
    more » « less
  4. Abstract BackgroundSingle‐session interventions have the potential to address young people's mental health needs at scale, but their effects are heterogeneous. We tested whether themindset + supportive contexthypothesis could help explain when intervention effects persist or fade over time. The hypothesis posits that interventions are more effective in environments that support the intervention message. We tested this hypothesis using the synergistic mindsets intervention, a preventative treatment for stress‐related mental health symptoms that helps students appraise stress as a potential asset in the classroom (e.g., increasing oxygenated blood flow) rather than debilitating. In an introductory college course, we examined whether intervention‐consistent messages from instructors sustained changes in appraisals over time, as well as impacts on students' predisposition to try demanding academic tasks that could enhance learning. MethodsWe randomly assigned 1675 students in the course to receive the synergistic mindsets intervention (or a control activity) at the beginning of the semester, and subsequently, to receive intervention‐supportive messages from their instructor (or neutral messages) four times throughout the term. We collected weekly measures of students' appraisals of stress in the course and their predisposition to take on academic challenges. Trial‐registration: OSF.io; DOI: 10.17605/osf.io/fchyn. ResultsA conservative Bayesian analysis indicated that receiving both the intervention and supportive messages led to the greatest increases in positive stress appraisals (0.35SD; 1.00 posterior probability) and challenge‐seeking predisposition (2.33 percentage points; 0.94 posterior probability), averaged over the course of the semester. In addition, intervention effects grew larger throughout the semester when complemented by supportive instructor messages, whereas without these messages, intervention effects shrank somewhat over time. ConclusionsThis study shows, for the first time, that supportive cues in local contexts can be the difference in whether a single‐session intervention's effects fade over time or persist and even amplify. 
    more » « less
  5. Team building activities are popular interventions during early stages of team development. At RIT, in the multidisciplinary capstone course with an average cohort size of around 350, the students on a particular capstone project team may not be mutually acquainted and thus may benefit from such team building activities. Prior literature has studied the effectiveness of various instructor-directed team building activities on student teams. However, our students are generally eager to spend class time working on their projects and often see in-class activities as a distraction rather than an important part of their growth. Instead, the student teams are now allowed to choose an intervention based on team consensus. In this paper, the relationship between attributes of the chosen intervention and student performance, as measured using a series of AACU VALUE rubrics, was studied using statistical measures. The analysis revealed a statistically significant effect of type of team building activity on teamwork, oral communication, and design & problem solving scores of individual students on the team. Also, a statistically significant effect of location of team building activity (on or off campus) on design & problem solving score was observed. 
    more » « less