There is a growing need to empirically evaluate the quality
of online instructional interventions at scale. In response,
some online learning platforms have begun to implement
rapid A/B testing of instructional interventions. In these
scenarios, students participate in series of randomized ex-
periments that evaluate problem-level interventions in quick
succession, which makes it difficult to discern the effect of
any particular intervention on their learning. Therefore, dis-
tal measures of learning such as posttests may not provide
a clear understanding of which interventions are effective,
which can lead to slow adoption of new instructional meth-
ods. To help discern the effectiveness of instructional in-
terventions, this work uses data from 26,060 clickstream se-
quences of students across 31 different online educational
experiments exploring 51 different research questions and
the students’ posttest scores to create and analyze different
proximal surrogate measures of learning that can be used
at the problem level. Through feature engineering and deep
learning approaches, next problem correctness was deter-
mined to be the best surrogate measure. As more data from
online educational experiments are collected, model based
surrogate measures can be improved, but for now, next prob-
lem correctness is an empirically effective proximal surrogate
measure of learning for analyzing rapid problem-level exper-
iments.
more »
« less
Effective Evaluation of Online Learning Interventions with Surrogate Measures
There is a growing need to empirically evaluate the quality
of online instructional interventions at scale. In response,
some online learning platforms have begun to implement
rapid A/B testing of instructional interventions. In these
scenarios, students participate in series of randomized ex-
periments that evaluate problem-level interventions in quick
succession, which makes it difficult to discern the effect of
any particular intervention on their learning. Therefore, dis-
tal measures of learning such as posttests may not provide
a clear understanding of which interventions are effective,
which can lead to slow adoption of new instructional meth-
ods. To help discern the effectiveness of instructional in-
terventions, this work uses data from 26,060 clickstream se-
quences of students across 31 different online educational
experiments exploring 51 different research questions and
the students’ posttest scores to create and analyze different
proximal surrogate measures of learning that can be used at
the problem level. Through feature engineering and deep
learning approaches, next-problem correctness was deter-
mined to be the best surrogate measure. As more data
from online educational experiments are collected, model
based surrogate measures can be improved, but for now,
next-problem correctness is an empirically effective proximal
surrogate measure of learning for analyzing rapid problem-
level experiments. The data and code
more »
« less
- Award ID(s):
- 1931523
- NSF-PAR ID:
- 10443574
- Date Published:
- Journal Name:
- In The Proceedings of the 16th International Conference on Educational Data Mining
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students’ posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problemlevel experiments. The data and code used in this work can be found at https://osf.io/uj48v/.more » « less
-
There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students’ posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problemlevel experiments. The data and code used in this work can be found at https://osf.io/uj48v/.more » « less
-
Many online learning platforms and MOOCs incorporate some amount of video-based content into their platform, but there are few randomized controlled experiments that evaluate the effectiveness of the different methods of video integration. Given the large amount of publicly available educational videos, an investigation into this content's impact on students could help lead to more effective and accessible video integration within learning platforms. In this work, a new feature was added into an existing online learning platform that allowed students to request skill-related videos while completing their online middle-school mathematics assignments. A total of 18,535 students participated in two large-scale randomized controlled experiments related to providing students with publicly available educational videos. The first experiment investigated the effect of providing students with the opportunity to request these videos, and the second experiment investigated the effect of using a multi-armed bandit algorithm to recommend relevant videos. Additionally, this work investigated which features of the videos were significantly predictive of students' performance and which features could be used to personalize students' learning. Ultimately, students were mostly disinterested in the skill-related videos, preferring instead to use the platforms existing problem-specific support, and there was no statistically significant findings in either experiment. Additionally, while no video features were significantly predictive of students' performance, two video features had significant qualitative interactions with students' prior knowledge, which showed that different content creators were more effective for different groups of students. These findings can be used to inform the design of future video-based features within online learning platforms and the creation of different educational videos specifically targeting higher or lower knowledge students. The data and code used in this work can be found at https://osf.io/cxkzf/.more » « less
-
Many online learning platforms and MOOCs incorporate some amount of video-based content into their platform, but there are few randomized controlled experiments that evaluate the effectiveness of the different methods of video integration. Given the large amount of publicly available educational videos, an investigation into this content’s impact on students could help lead to more effective and accessible video integration within learning platforms. In this work, a new feature was added into an existing online learning platform that allowed students to request skill-related videos while completing their online middle-school mathematics assignments. A total of 18,535 students participated in two large-scale randomized controlled experiments related to providing students with publicly available educational videos. The first experiment investigated the effect of providing students with the opportunity to request these videos, and the second experiment investigated the effect of using a multi-armed bandit algorithm to recommend relevant videos. Additionally, this work investigated which features of the videos were significantly predictive of students’ performance and which features could be used to personalize students’ learning. Ultimately, students were mostly disinterested in the skill-related videos, preferring instead to use the platforms existing problem specific support, and there was no statistically significant findings in either experiment. Additionally, while no video features were significantly predictive of students’ performance, two video features had significant qualitative interactions with students’ prior knowledge, which showed that different content creators were more effective for different groups of students. These findings can be used to inform the design of future video-based features within online learning platforms and the creation of different educational videos specifically targeting higher or lower knowledge students. The data and code used in this work can be found at https://osf.io/cxkzf/.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
