To improve student learning outcomes within online learning
platforms, struggling students are often provided with
on-demand supplemental instructional content. Recently,
services like Yup (yup.com) and UPcheive (upchieve.org)
have begun to offer on-demand live tutoring sessions with
qualified educators, but the availability of tutors and the
cost associated with hiring them prevents many students
from having access to live support. To help struggling students
and offset the inequities intrinsic to high-cost services,
we are attempting to develop a process that uses large language
representation models to algorithmically identify relevant
support messages from these chat logs, and distribute
them to all students struggling with the same content. In
an empirical evaluation of our methodology we were able
to identify messages from tutors to students struggling with
middle school mathematics problems that qualified as explanations
of the content. However, when we distributed
these explanations to students outside of the tutoring sessions,
they had an overall negative effect on the students’
learning. Moving forward, we want to be able to identify
messages that will promote equity and have a positive impact
on students.
more »
« less
Identifying Explanations Within Student-Tutor Chat Logs
To improve student learning outcomes within online learning platforms, struggling students are often provided with
on-demand supplemental instructional content. Recently,
services like Yup (yup.com) and UPcheive (upchieve.org)
have begun to offer on-demand live tutoring sessions with
qualified educators, but the availability of tutors and the
cost associated with hiring them prevents many students
from having access to live support. To help struggling students and offset the inequities intrinsic to high-cost services,
we are attempting to develop a process that uses large language representation models to algorithmically identify relevant support messages from these chat logs, and distribute
them to all students struggling with the same content. In
an empirical evaluation of our methodology we were able
to identify messages from tutors to students struggling with
middle school mathematics problems that qualified as explanations of the content. However, when we distributed
these explanations to students outside of the tutoring sessions, they had an overall negative effect on the students’
learning. Moving forward, we want to be able to identify
messages that will promote equity and have a positive impact on students.
more »
« less
- Award ID(s):
- 1840771
- NSF-PAR ID:
- 10374333
- Date Published:
- Journal Name:
- Educational data Mining Conference
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possibility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually using this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to integrate new curricula into online learning platforms. To generate explanations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations using few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teachers’ process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/.more » « less
-
Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possibility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually using this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to integrate new curricula into online learning platforms. To generate explanations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations using few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teachers’ process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/.more » « less
-
Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possi- bility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually us- ing this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to inte- grate new curricula into online learning platforms. To generate expla- nations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations us- ing few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teach- ers’ process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/.more » « less
-
Large language models have recently been able to perform well in a wide variety of circumstances. In this work, we explore the possi- bility of large language models, specifically GPT-3, to write explanations for middle-school mathematics problems, with the goal of eventually us- ing this process to rapidly generate explanations for the mathematics problems of new curricula as they emerge, shortening the time to inte- grate new curricula into online learning platforms. To generate expla- nations, two approaches were taken. The first approach attempted to summarize the salient advice in tutoring chat logs between students and live tutors. The second approach attempted to generate explanations us- ing few-shot learning from explanations written by teachers for similar mathematics problems. After explanations were generated, a survey was used to compare their quality to that of explanations written by teachers. We test our methodology using the GPT-3 language model. Ultimately, the synthetic explanations were unable to outperform teacher written explanations. In the future more powerful large language models may be employed, and GPT-3 may still be effective as a tool to augment teach- ers’ process for writing explanations, rather than as a tool to replace them. The explanations, survey results, analysis code, and a dataset of tutoring chat logs are all available at https://osf.io/wh5n9/.more » « less