skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Classifying Tutor Discursive Moves at Scale in Mathematics Classrooms with Large Language Models
In mathematics tutoring, using appropriate instructional discursive strategies, called "talk moves'', is critical to support student learning. Training tutors in the appropriate use of talk moves is a key component of tutor development programs. However, tutor development at scale is a challenge. Recent research has shown that automatic talk moves classification of tutorial discourse can facilitate large-scale delivery of personalized talk moves feedback. In this paper, we build on this work and share our current progress using large language models to classify talk moves in transcripts of tutoring sessions. We report classification results from fine-tuned models, prompt optimization, and supervised embedding vectors classification. The fine-tuned strategy performed best, yielding better performance (.87 macro and .93 weighted f1 score in predicting expert labels) than the current state-of-the-art RoBERTa model. We discuss trade-offs across methods and models.  more » « less
Award ID(s):
2019805
PAR ID:
10586888
Author(s) / Creator(s):
; ; ; ; ; ;
Publisher / Repository:
ACM
Date Published:
ISBN:
9798400706332
Page Range / eLocation ID:
361 to 365
Format(s):
Medium: X
Location:
Atlanta GA USA
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; (, Proceedings of the International Conference on Artificial Intelligence in Education)
    Olney, AM; Chounta, IA; Liu, Z; Santos, OC; Bittencourt, II (Ed.)
    This work investigates how tutoring discourse interacts with students’ proximal knowledge to explain and predict students’ learning outcomes. Our work is conducted in the context of high-dosage human tutoring where 9th-grade students attended small group tutorials and individually practiced problems on an Intelligent Tutoring System (ITS). We analyzed whether tutors’ talk moves and students’ performance on the ITS predicted scores on math learning assessments. We trained Random Forest Classifiers (RFCs) to distinguish high and low assessment scores based on tutor talk moves, student’s ITS performance metrics, and their combination. A decision tree was extracted from each RFC to yield an interpretable model. We found AUCs of 0.63 for talk moves, 0.66 for ITS, and 0.77 for their combination, suggesting interactivity among the two feature sources. Specifically, the best decision tree emerged from combining the tutor talk moves that encouraged rigorous thinking and students’ ITS mastery. In essence, tutor talk that encouraged mathematical reasoning predicted achievement for students who demonstrated high mastery on the ITS, whereas tutors’ revoicing of students’ mathematical ideas and contributions was predictive for students with low ITS mastery. Implications for practice are discussed. 
    more » « less
  2. ; ; (, Artificial Intelligence in Education. AIED 2024. Lecture Notes in Computer Science)
    Olney, AM; Chounta, IA; Liu, Z; Santos; OC; Bittencourt, II (Ed.)
    This work investigates how tutoring discourse interacts with students’ proximal knowledge to explain and predict students’ learning outcomes. Our work is conducted in the context of high-dosage human tutoring where 9th-grade students (N = 1080) attended small group tutorials and individually practiced problems on an Intelligent Tutoring System (ITS). We analyzed whether tutors’ talk moves and students’ performance on the ITS predicted scores on math learning assessments. We trained Random Forest Classifiers (RFCs) to distinguish high and low assessment scores based on tutor talk moves, student’s ITS performance metrics, and their combination. A decision tree was extracted from each RFC to yield an interpretable model. We found AUCs of 0.63 for talk moves, 0.66 for ITS, and 0.77 for their combination, suggesting interactivity among the two feature sources. Specifically, the best decision tree emerged from combining the tutor talk moves that encouraged rigorous thinking and students’ ITS mastery. In essence, tutor talk that encouraged mathematical reasoning predicted achievement for students who demonstrated high mastery on the ITS, whereas tutors’ revoicing of students’ mathematical ideas and contributions was predictive for students with low ITS mastery. Implications for practice are discussed. 
    more » « less
  3. ; ; (, Springer Cham)
    Olney, A M; Chounta, I A; Liu, Z; Santos, O C; Bittencourt, I I (Ed.)
    This work investigates how tutoring discourse interacts with students’ proximal knowledge to explain and predict students’ learning outcomes. Our work is conducted in the context of high-dosage human tutoring where 9th-grade students attended small group tutorials and individually practiced problems on an Intelligent Tutoring System (ITS). We analyzed whether tutors’ talk moves and students’ performance on the ITS predicted scores on math learning assessments. We trained Random Forest Classifiers (RFCs) to distinguish high and low assessment scores based on tutor talk moves, student’s ITS performance metrics, and their combination. A decision tree was extracted from each RFC to yield an interpretable model. We found AUCs of 0.63 for talk moves, 0.66 for ITS, and 0.77 for their combination, suggesting interactivity among the two feature sources. Specifically, the best decision tree emerged from combining the tutor talk moves that encouraged rigorous thinking and students’ ITS mastery. In essence, tutor talk that encouraged mathematical reasoning predicted achievement for students who demonstrated high mastery on the ITS, whereas tutors’ revoicing of students’ mathematical ideas and contributions was predictive for students with low ITS mastery. Implications for practice are discussed. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ; et al (, Association for Computational Linguistics)
    Rambow, Owen; Wanner, Leo; Apidianaki, Marianna; Al-Khalifa, Hend; Di_Eugenio, Barbara; Schockaert, Steven (Ed.)
    Human tutoring interventions play a crucial role in supporting student learning, improving academic performance, and promoting personal growth. This paper focuses on analyzing mathematics tutoring discourse using talk moves—a framework of dialogue acts grounded in Accountable Talk theory. However, scaling the collection, annotation, and analysis of extensive tutoring dialogues to develop machine learning models is a challenging and resource-intensive task. To address this, we present SAGA22, a compact dataset, and explore various modeling strategies, including dialogue context, speaker information, pretraining datasets, and further fine-tuning. By leveraging existing datasets and models designed for classroom teaching, our results demonstrate that supplementary pretraining on classroom data enhances model performance in tutoring settings, particularly when incorporating longer context and speaker information. Additionally, we conduct extensive ablation studies to underscore the challenges in talk move modeling. 
    more » « less
  5. ; ; ; ; ; ; ; ; ; ; et al (, Association for Computational Linguistics)
    Rambow, Owen; Wanner, Owen; Apidianaki, Marianna; Al-Khalifa, Hend; Di_Eugenio, Barbara; Schockaert, Steven (Ed.)
    Human tutoring interventions play a crucial role in supporting student learning, improving academic performance, and promoting personal growth. This paper focuses on analyzing mathematics tutoring discourse using talk moves—a framework of dialogue acts grounded in Accountable Talk theory. However, scaling the collection, annotation, and analysis of extensive tutoring dialogues to develop machine learning models is a challenging and resource-intensive task. To address this, we present SAGA22, a compact dataset, and explore various modeling strategies, including dialogue context, speaker information, pretraining datasets, and further fine-tuning. By leveraging existing datasets and models designed for classroom teaching, our results demonstrate that supplementary pretraining on classroom data enhances model performance in tutoring settings, particularly when incorporating longer context and speaker information. Additionally, we conduct extensive ablation studies to underscore the challenges in talk move modeling. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email