The use of Bayesian Knowledge Tracing (BKT) models in predicting student learning and mastery, especially in mathematics, is a well-established and proven approach in learning analytics. In this work, we report on our analysis examining the generalizability of BKT models across academic years attributed to ”detector rot.” We compare the generalizability of Knowledge Training (KT) models by comparing model performance in predicting student knowledge within the academic year and across academic years. Models were trained on data from two popular open-source curricula available through Open Educational Resources. We observed that the models generally were highly performant in predicting student learning within an academic year, whereas certain academic years were more generalizable than other academic years. We posit that the Knowledge Tracing models are relatively stable in terms of performance across academic years yet can still be susceptible to systemic changes and underlying learner behavior. As indicated by the evidence in this paper, we posit that learning platforms leveraging KT models need to be mindful of systemic changes or drastic changes in certain user demographics.
more »
« less
Knowledge Tracing Over Time: A Longitudinal Analysis.
The use of Bayesian Knowledge Tracing (BKT) models in
predicting student learning and mastery, especially in math-
ematics, is a well-established and proven approach in learn-
ing analytics. In this work, we report on our analysis exam-
ining the generalizability of BKT models across academic
years attributed to ”detector rot.” We compare the gen-
eralizability of Knowledge Training (KT) models by com-
paring model performance in predicting student knowledge
within the academic year and across academic years. Models
were trained on data from two popular open-source curric-
ula available through Open Educational Resources. We ob-
served that the models generally were highly performant in
predicting student learning within an academic year, whereas
certain academic years were more generalizable than other
academic years. We posit that the Knowledge Tracing mod-
els are relatively stable in terms of performance across aca-
demic years yet can still be susceptible to systemic changes
and underlying learner behavior. As indicated by the evi-
dence in this paper, we posit that learning platforms lever-
aging KT models need to be mindful of systemic changes or
drastic changes in certain user demographics.
more »
« less
- Award ID(s):
- 1917808
- PAR ID:
- 10425012
- Date Published:
- Journal Name:
- The Proceedings of the 16th International Conference on Educational Data Mining.
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The use of Bayesian Knowledge Tracing (BKT) models in predicting student learning and mastery, especially in mathematics, is a well-established and proven approach in learning analytics. In this work, we report on our analysis examining the generalizability of BKT models across academic years attributed to ”detector rot.” We compare the generalizability of Knowledge Training (KT) models by comparing model performance in predicting student knowledge within the academic year and across academic years. Models were trained on data from two popular open-source curricula available through Open Educational Resources. We observed that the models generally were highly performant in predicting student learning within an academic year, whereas certain academic years were more generalizable than other academic years. We posit that the Knowledge Tracing models are relatively stable in terms of performance across academic years yet can still be susceptible to systemic changes and underlying learner behavior. As indicated by the evidence in this paper, we posit that learning platforms leveraging KT models need to be mindful of systemic changes or drastic changes in certain user demographics.more » « less
-
The use of Bayesian Knowledge Tracing (BKT) models in predicting student learning and mastery, especially in mathematics, is a well-established and proven approach in learning analytics. In this work, we report on our analysis examining the generalizability of BKT models across academic years attributed to "detector rot." We compare the generalizability of Knowledge Training (KT) models by comparing model performance in predicting student knowledge within the academic year and across academic years. Models were trained on data from two popular open-source curricula available through Open Educational Resources. We observed that the models generally were highly performant in predicting student learning within an academic year, whereas certain academic years were more generalizable than other academic years. We posit that the Knowledge Tracing models are relatively stable in terms of performance across academic years yet can still be susceptible to systemic changes and underlying learner behavior. As indicated by the evidence in this paper, we posit that learning platforms leveraging KT models need to be mindful of systemic changes or drastic changes in certain user demographics.more » « less
-
The use of Bayesian Knowledge Tracing (BKT) models in predicting student learning and mastery, especially in math- ematics, is a well-established and proven approach in learn- ing analytics. In this work, we report on our analysis exam- ining the generalizability of BKT models across academic years attributed to ”detector rot.” We compare the gen- eralizability of Knowledge Training (KT) models by com- paring model performance in predicting student knowledge within the academic year and across academic years. Models were trained on data from two popular open-source curric- ula available through Open Educational Resources. We ob- served that the models generally were highly performant in predicting student learning within an academic year, whereas certain academic years were more generalizable than other academic years. We posit that the Knowledge Tracing mod- els are relatively stable in terms of performance across aca- demic years yet can still be susceptible to systemic changes and underlying learner behavior. As indicated by the evi- dence in this paper, we posit that learning platforms lever- aging KT models need to be mindful of systemic changes or drastic changes in certain user demographics.more » « less
-
Accurate modeling of student knowledge is essential for large-scale online learning systems that are increasingly used for student training. Knowledge tracing aims to model student knowledge state given the student's sequence of learning activities. Modern Knowledge tracing (KT) is usually formulated as a supervised sequence learning problem to predict students' future practice performance according to their past observed practice scores by summarizing student knowledge state as a set of evolving hidden variables. Because of this formulation, many current KT solutions are not fit for modeling student learning from non-assessed learning activities with no explicit feedback or score observation (e.g., watching video lectures that are not graded). Additionally, these models cannot explicitly represent the dynamics of knowledge transfer among different learning activities, particularly between the assessed (e.g., quizzes) and non-assessed (e.g., video lectures) learning activities. In this paper, we propose Transition-Aware Multi-activity Knowledge Tracing (TAMKOT), which models knowledge transfer between learning materials, in addition to student knowledge, when students transition between and within assessed and non-assessed learning materials. TAMKOT is formulated as a deep recurrent multi-activity learning model that explicitly learns knowledge transfer by activating and learning a set of knowledge transfer matrices, one for each transition type between student activities. Accordingly, our model allows for representing each material type in a different yet transferrable latent space while maintaining student knowledge in a shared space. We evaluate our model on three real-world publicly available datasets and demonstrate TAMKOT's capability in predicting student performance and modeling knowledge transfer.more » « less