An official website of the United States government
Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom.
more »
« less
Exploring Cross-Country Prediction Model Generalizability in MOOCs
Massive Open Online Courses (MOOCs) have increased the accessibility of quality educational content to a broader audience across a global network. They provide access for students to material that would be difficult to obtain locally, and an abundance of data for educational researchers. Despite the international reach of MOOCs, however, the majority of MOOC research does not account for demographic differences relating to the learners' country of origin or cultural background, which have been shown to have implications on the robustness of predictive models and interventions. This paper presents an exploration into the role of nation-level metrics of culture, happiness, wealth, and size on the generalizability of completion prediction models across countries. The findings indicate that various dimensions of culture are predictive of cross-country model generalizability. Specifically, learners from indulgent, collectivist, uncertainty-accepting, or short-term oriented, countries produce more generalizable predictive models of learner completion.
more »
« less
- Award ID(s):
- 1931419
- PAR ID:
- 10448499
- Date Published:
- Journal Name:
- L@S '23: Proceedings of the Tenth ACM Conference on Learning @ Scale
- Page Range / eLocation ID:
- 183 to 194
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Students who take an online course, such as a MOOC, use the course's discussion forum to ask questions or reach out to instructors when encountering an issue. However, reading and responding to students' questions is difficult to scale because of the time needed to consider each message. As a result, critical issues may be left unresolved, and students may lose the motivation to continue in the course. To help address this problem, we build predictive models that automatically determine the urgency of each forum post, so that these posts can be brought to instructors' attention. This paper goes beyond previous work by predicting not just a binary decision cut-off but a post's level of urgency on a 7-point scale. First, we train and cross-validate several models on an original data set of 3,503 posts from MOOCs at University of Pennsylvania. Second, to determine the generalizability of our models, we test their performance on a separate, previously published data set of 29,604 posts from MOOCs at Stanford University. While the previous work on post urgency used only one data set, we evaluated the prediction across different data sets and courses. The best-performing model was a support vector regressor trained on the Universal Sentence Encoder embeddings of the posts, achieving an RMSE of 1.1 on the training set and 1.4 on the test set. Understanding the urgency of forum posts enables instructors to focus their time more effectively and, as a result, better support student learning.more » « less
-
Abstract This study investigated the predictive use of dative verb constraints in Mandarin among home-country-raised native speakers and classroom learners (including both sequential L2 learners and heritage speakers). In a visual world eye-tracking experiment, participants made anticipatory looks to the upcoming argument (recipient versus theme) following categorical restrictions of non-alternating verbs and gradient bias of alternating verbs before the acoustic onset of the disambiguating noun. Crucially, no delay or reduction in the prediction effects was observed among L2 learners and heritage speakers in comparison with home-country-raised native speakers. Mandarin proficiency and dominant language (English versus other) did not modulate prediction effects among classroom learners. These findings provide direct support for the assumption of error-driven learning accounts of the dative alternation, that is, language users actively predict upcoming arguments based on verb information during real-time sentence processing.more » « less
-
Captions play a major role in making educational videos accessible to all and are known to benefit a wide range of learners. However, many educational videos either do not have captions or have inaccurate captions. Prior work has shown the benefits of using crowdsourcing to obtain accurate captions in a cost-efficient way, though there is a lack of understanding of how learners edit captions of educational videos either individually or collaboratively. In this work, we conducted a user study where 58 learners (in a course of 387 learners) participated in the editing of captions in 89 lecture videos that were generated by Automatic Speech Recognition (ASR) technologies. For each video, different learners conducted two rounds of editing. Based on editing logs, we created a taxonomy of errors in educational video captions (e.g., Discipline-Specific, General, Equations). From the interviews, we identified individual and collaborative error editing strategies. We then further demonstrated the feasibility of applying machine learning models to assist learners in editing. Our work provides practical implications for advancing video-based learning and for educational video caption editing.more » « less
-
The ongoing highly contagious coronavirus disease 2019 (COVID-19) pandemic, which started in Wuhan, China, in December 2019, has now become a global public health problem. Using publicly available data from the COVID-19 data repository of Our World in Data, we aimed to investigate the influences of spatial socio-economic vulnerabilities and neighbourliness on the COVID-19 burden in African countries. We analyzed the first wave (January–September 2020) and second wave (October 2020 to May 2021) of the COVID-19 pandemic using spatial statistics regression models. As of 31 May 2021, there was a total of 4,748,948 confirmed COVID-19 cases, with an average, median, and range per country of 101,041, 26,963, and 2191 to 1,665,617, respectively. We found that COVID-19 prevalence in an Africa country was highly dependent on those of neighbouring Africa countries as well as its economic wealth, transparency, and proportion of the population aged 65 or older (p-value < 0.05). Our finding regarding the high COVID-19 burden in countries with better transparency and higher economic wealth is surprising and counterintuitive. We believe this is a reflection on the differences in COVID-19 testing capacity, which is mostly higher in more developed countries, or data modification by less transparent governments. Country-wide integrated COVID suppression strategies such as limiting human mobility from more urbanized to less urbanized countries, as well as an understanding of a county’s social-economic characteristics, could prepare a country to promptly and effectively respond to future outbreaks of highly contagious viral infections such as COVID-19.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3573051.3593380
