Introduction

Mathematics education continues to be an important focus for national improvement. Achievement gaps between demographic groups continue to be an important national and state-based concern. Due to the promise of technology as a tool for improving mathematics education and closing the achievement gap, the use of educational technology in K-12 education has expanded dramatically in recent years, accelerated by the COVID-19 pandemic. The AIED and intelligent tutoring systems researchers and developers have built numerous technology-based learning platforms and programs, but few of these products have been implemented at a large scale in authentic school classroom settings. ASSISTments [1] is one of the few digital platforms that have been used widely in the U.S. Over the past two years, ASSISTments use in schools increased significantly, going from supporting 800 teachers to supporting 20,000 teachers and their 500,000 students. At the beginning of the COVID pandemic, the US Department funded a rapid review to synthesize existing evidence in online programs that promoted learning, and ASSISTments was one of the few digital learning programs recommended for use in response to the COVID pandemic [2]. The ASSISTments platform is a technology-based, formative assessment platform for improving teacher practices and student math learning outcomes. As students work through problems and enter their answers into ASSISTments, the system provides immediate feedback on the correctness of answers and offers additional assistance in the form of hints or scaffolds. ASSISTments provides teachers with real-time, easily accessible reports that summarize student work for a particular assignment, which teachers can use to target their homework review in class and tailor instruction to their students' needs. ASSISTments was identified as effective at improving 7 th grade student's learning and changing teacher's homework review practices during an efficacy study in Maine ( [3,4], meeting What Works Clearinghouse standards without reservation, g=.22, p<.01). Escueta et al. [5] indicated that out of 29 studies they reviewed that met rigorous standards of randomization, ASSISTments was one of only "Two interventions in the United States [that] stand out as being particularly promising" (page 88).

Supported by the U.S. Department of Education, we conducted a large scale randomized controlled trialfoot_0 to replicate the Maine study and see whether the found effects replicate in a heterogeneous population that more closely matches national demographics. In addition, we designed a follow-up study to measure the long-term effects of ASSISTments on student performance one year after the intervention was over. This late-breaking results paper reports on the findings from the recently completed preliminary analysis of the follow-up study. The follow-up study addresses two research questions: 1) What is the impact of ASSISTments on student math outcomes at the end of Grade 8? 2) Do the effects of ASSISTments vary for students of different demographic characteristics?

Background on ASSISTments

ASSISTments uses technology to give teachers new capabilities for assigning and reviewing homework and to give students additional support for learning as they do homework. Content in ASSISTments consists of mathematics problems with answers and hint messages. These mathematics problems are bundled into problem sets which teachers can use ASSISTments to assign to students in class or as homework. Students first do their assigned problems on paper and then enter their answers into ASSISTments to receive immediate feedback about the correctness of their answers, and/or hints on how to improve their answers or help separate multi-step problems into parts. One type of problem set is mastery-oriented "Skill Builders". Each skill builder provides opportunities for students to practice solving problems that focus on a targeted skill, until they reach a teacher-defined "mastery" threshold of proficiency (e.g., a streak of three correct answers on similar math problems). ASSISTments also automatically re-assessing students on skills and concepts that were "mastered" earlier at regular intervals and providing further opportunities for students to hone those skills.

ASSISTments provides teachers with real-time, easily accessible reports that summarize student work for a particular assignment in a grid format (e.g. Fig 1). These reports inform teachers about the average percent correct on each question, skills covered in the assignment, common wrong answers, and each student's answer to every question. Teachers use information about individual students to form instructional groups and address common difficulties. The design and development of ASSISTments is built upon the theoretical foundations and empirical research. Feedback has been identified as a powerful way to increase student learning [6]. An extensive set of studies (e.g., [7,8,9]) has found significant learning gains in response to feedback within computer-based instruction. Formative assessment [10,11] informs teachers of student learning and progress and students of their own performance in relation to learning goals. Several decades of research have shown formative assessment to be an effective way of improving both teaching and student learning [12,13]. Mastery learning programs have been shown to lead to higher student achievement than more traditional forms of teaching [14,15]. An IES Practice Guide [16] recommends spacing practice over time. Research has demonstrated that retention increases when learners are repeatedly exposed to content (see [17]).

Research Design

The replication study used a school-level, clustered randomized experimental design. Schools within each district were paired based on their demographic characteristics and student prior performance on state math and English language arts (ELA) tests and then randomly assigned to a treatment or business-as-usual control condition. All 7th math teachers in a school had the same assignment of condition. Teachers in the treatment schools used ASSISTments to support math homework. In the control condition, teachers continued their existing homework practices, including any use of online tools, but not had access to ASSISTments. The intervention was implemented by all Grade 7 teachers in treatment schools over two consecutive years-teachers learned how to use ASSISTments for a year (2018-19) and then we measured the immediate impact for students in teachers' second year of experience with the system (2019-20). Students who were in Grade 7 during the 2019-20 school year comprised the analytic sample for the research questions. The students maintained their conditions and were followed longitudinally for another year to Grade 8 when their Grade 8 performance (long-term impact) was measured at the end of Grade 8 (see Fig. 2 for an overview). During the follow-up year in Grade 8, no interventions were be provided by the team students, 3,030 control students). We examined student baseline equivalence on their 6th grade EoG test scores, gender, and ethnicity and found no significant difference between the two conditions.

To evaluate the efficacy of ASSISTments for improving students' long-term mathematics achievement in 8th grade (Research Question 1), we conduct an intent-to-treat (ITT) analysis using two-level hierarchical linear regression models (HLM). The HLM was used to account for the clustering effect of the data (students were nested within schools). We modeled the mean differences in Grade 8 EoG scores (the long-term outcome) between students in treatment and control schools, controlling for the student's Grade 6 scores and other student-and school-level covariates to improve the precision of the impact estimate. To address Research Question 2, we added the analytical model with a cross-level interaction term of the school-level treatment variable and indicator of student subgroup and conducted a series of moderator analysis examining whether the ASSISTments had a differential impact on minority (e.g., Hispanic, Black) versus nonminority (White) students. Similar models were also used to estimate the effects of the intervention on the students with other policy-relevant background variables.

The results showed that there was a statistically significant difference between students in the treatment and control conditions in their 8 th grade EoG test scores (p = 0.011, effect size Hedges' g = 0.10) with students from the treatment schools performing significantly better than those in the control group. The moderator analyses showed that the intervention benefited minority (non-White) students, who started with significantly lower 6 th grade EoG scores, significantly more than majority (white) students (p = 0.003, Hedges' g = 0.14) and that the impact was stronger for Hispanic students than non-Hispanic students (p = 0.014, Hedges' g = 0.13). Overall, the results suggested that ASSISTments had a sustained long-term impact on students' math learning, even after one year of intervention implementation. The program also helped close the achievement gap among students of different ethnicity.

Conclusion and Future Work

Enhancing mathematics education is an imperative challenge, especially given the significant learning loss that has transpired during the pandemic in the past three years. In this paper, we presented the promising long-term effects from a rigorous efficacy study of 63 schools that evaluated the impact of the ASSISTments homework support program in diverse settings and implementation circumstances. Historically, when the policy relevant state standardized tests were used as an outcome measure, few studies demonstrated impact on student learning [5] and long-term effects were rarely detected. Therefore, the findings have strong implications for math interventions. The findings demonstrate the potency of the ASSISTments program and its value in promoting math learning in middle school, as well as the promise of leveraging ASSISTments, and similar AI-supported programs, to help overcome the academic setbacks caused by the pandemic. The team is continuing with benchmarking the effect sizes and further analysis of the data to explore the variation of effects for different populations and school settings and relationships between ASSISTments usage in Grade 7 and performance in Grade 8.