skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms
“Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers’ ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similar transformer-based classification models.  more » « less
Award ID(s):
1837986
PAR ID:
10387286
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications
Page Range / eLocation ID:
71 to 81
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; (, International Conference on Artificial Intelligence in Education)
    null (Ed.)
    Inclusion in mathematics education is strongly tied to discourse rich classrooms, where students ideas play a central role. Talk moves are specific discursive practices that promote inclusive and equitable participation in classroom discussions. This paper describes the development of the TalkMoves application, which provides teachers with detailed feedback on their usage of talk moves based on accountable talk theory. Building on our recent work to automate the classification of teacher talk moves, we have expanded the application to also include feedback on a set of student talk moves. We present results from several deep learning models trained to classify student sentences into student talk moves with performance up to 73% F1. The classroom data used for training these models were collected from multiple sources that were pre-processed and annotated by highly reliable experts. We validated the performance of the model on an out-of-sample dataset which included 166 classroom transcripts collected from teachers piloting the application. 
    more » « less
  2. ; ; ; ; ; ; (, ACM)
    In mathematics tutoring, using appropriate instructional discursive strategies, called "talk moves'', is critical to support student learning. Training tutors in the appropriate use of talk moves is a key component of tutor development programs. However, tutor development at scale is a challenge. Recent research has shown that automatic talk moves classification of tutorial discourse can facilitate large-scale delivery of personalized talk moves feedback. In this paper, we build on this work and share our current progress using large language models to classify talk moves in transcripts of tutoring sessions. We report classification results from fine-tuned models, prompt optimization, and supervised embedding vectors classification. The fine-tuned strategy performed best, yielding better performance (.87 macro and .93 weighted f1 score in predicting expert labels) than the current state-of-the-art RoBERTa model. We discuss trade-offs across methods and models. 
    more » « less
  3. ; ; ; ; ; ; (, International Educational Data Mining Society)
    Mills, Caitlin; Alexandron, Giora; Taibi, Davide; Lo_Bosco, Giosuè; Paquette, Luc (Ed.)
    Effective feedback is essential for refining instructional practices in mathematics education, and researchers often turn to advanced natural language processing (NLP) models to analyze classroom dialogues from multiple perspectives. However, utterance-level discourse analysis encounters two primary challenges: (1) multi-functionality, where a single utterance may serve multiple purposes that a single tag cannot capture, and (2) the exclusion of many utterances from domain-specific discourse move classifications, leading to their omission in feedback. To address these challenges, we proposed a multi-perspective discourse analysis that integrates domain-specific talk moves with dialogue act (using the flattened multi-functional SWBD-MASL schema with 43 tags) and discourse relation (applying Segmented Discourse Representation Theory with 16 relations). Our top-down analysis framework enables a comprehensive understanding of utterances that contain talk moves, as well as utterances that do not contain talk moves. This is applied to two mathematics education datasets: TalkMoves (teaching) and SAGA22 (tutoring). Through distributional unigram analysis, sequential talk move analysis, and multi-view deep dive, we discovered meaningful discourse patterns, and revealed the vital role of utterances without talk moves, demonstrating that these utterances, far from being mere fillers, serve crucial functions in guiding, acknowledging, and structuring classroom discourse. These insights underscore the importance of incorporating discourse relations and dialogue acts into AI-assisted education systems to enhance feedback and create more responsive learning environments. Our framework may prove helpful for providing human educator feedback, but also aiding in the development of AI agents that can effectively emulate the roles of both educators and students. 
    more » « less
  4. ; ; ; ; ; ; (, International Educational Data Mining Society)
    Mills, Caitlin; Alexandron, Giora; Taibi, Davide; Lo_Bosco, Giosue; Paquette, Luc (Ed.)
    in mathematics education, and researchers often turn to advanced natural language processing (NLP) models to analyze classroom dialogues from multiple perspectives. However, utterance-level discourse analysis encounters two primary challenges: (1) multifunctionality, where a single utterance may serve multiple purposes that a single tag cannot capture, and (2) the exclusion of many utterances from domain-specific discourse move classifications, leading to their omission in feedback. To address these challenges, we proposed a multi-perspective discourse analysis that integrates domain-specific talk moves with dialogue act (using the flattened multi-functional SWBD-MASL schema with 43 tags) and discourse relation (applying Segmented Discourse Representation Theory with 16 relations). Our top-down analysis framework enables a comprehensive understanding of utterances that contain talk moves, as well as utterances that do not contain talk moves. This is applied to two mathematics education datasets: TalkMoves (teaching) and SAGA22 (tutoring). Through distributional unigram analysis, sequential talk move analysis, and multi-view deep dive, we discovered meaningful discourse patterns, and revealed the vital role of utterances without talk moves, demonstrating that these utterances, far from being mere fillers, serve crucial functions in guiding, acknowledging, and structuring classroom discourse. These insights underscore the importance of incorporating discourse relations and dialogue acts into AI-assisted education systems to enhance feedback and create more responsive learning environments. Our framework may prove helpful for providing human educator feedback, but also aiding in the development of AI agents that can effectively emulate the roles of both educators and students. 
    more » « less
  5. ; ; ; ; ; ; (, Association for the Advancement of Artificial Intelligence)
    null (Ed.)
    TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application combines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education community and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastructure for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss several technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email