More Like this

1. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application

   Suresh, A.; Jacobs, J.; Lai, V.; Tan, C.; Ward, W.; Martin, J.H.; Sumner, T. (January 2021, Association for the Advancement of Artificial Intelligence)

   null (Ed.)

   TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application combines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education community and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastructure for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss several technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms. 

   more » « less
2. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application

   Suresh, A.; Jacobs, J.; Lai, V.; Tan, C.; Ward, W.; Martin, J.H.; Sumner, T. (January 2021, Association for the Advancement of Artificial Intelligence in Education)

   null (Ed.)

   TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application com- bines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education com- munity and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastruc- ture for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss sev- eral technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms. 

   more » « less
3. Elementary preservice teachers' use of prompts to encourage student‐to‐student talk during simulated argumentation discussions

   https://doi.org/10.1111/ssm.12685  

   Masters, Heidi L; Lottero‐Perdue, Pamela S; Placa, Nicora; Galindo, Enrique; Mikeska, Jamie N; Howell, Heather (June 2024, School Science and Mathematics)

   Providing opportunities for students to talk directly with their peers is a critical dimension to facilitating discussions in mathematics and science, including argumentation‐focused discussions in which students construct arguments and critique others' arguments. Research suggests that supporting student‐to‐student talk and facilitating argumentation discussions are complex and challenging practices for preservice teachers (PSTs). Elementary PSTs from two mathematics and two science methods courses practiced facilitating student‐to‐student talk within the context of an argumentation‐focused discussion. This study's main purpose was to explore the prompts that 29 PSTs used to encourage student‐to‐student talk in a simulated classroom. Findings show the PSTs were able to use direct prompts that encourage student‐to‐student talk but were just as likely to use prompts that may discourage students from talking to each other. Most direct prompts PSTs used to encourage student‐to‐student talk were for the purpose of argumentation construction and/or critique. PSTs were more likely to use indirect prompts, much like Talk Moves, that encourage students to consider others' ideas rather than requesting that students talk with each other. These findings have important implications for future research, as well as for teacher educators and professional learning facilitators who support teachers learning to encourage student‐to‐student talk during argumentation‐focused discussions. 

   more » « less
4. Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms

   https://doi.org/10.18653/v1/2022.bea-1.11  

   Suresh, Abhijit; Jacobs, Jennifer; Perkoff, Margaret; Martin, James H.; Sumner, Tamara (July 2022, Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications)

   “Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers’ ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similar transformer-based classification models. 

   more » « less
5. Classifying Tutor Discursive Moves at Scale in Mathematics Classrooms with Large Language Models

   https://doi.org/10.1145/3657604.3664664  

   Moreau-Pernet, Baptiste; Tian, Yu; Sawaya, Sandra; Foltz, Peter; Cao, Jie; Milne, Brent; Christie, Thomas (July 2024, ACM)

   In mathematics tutoring, using appropriate instructional discursive strategies, called "talk moves'', is critical to support student learning. Training tutors in the appropriate use of talk moves is a key component of tutor development programs. However, tutor development at scale is a challenge. Recent research has shown that automatic talk moves classification of tutorial discourse can facilitate large-scale delivery of personalized talk moves feedback. In this paper, we build on this work and share our current progress using large language models to classify talk moves in transcripts of tutoring sessions. We report classification results from fine-tuned models, prompt optimization, and supervised embedding vectors classification. The fine-tuned strategy performed best, yielding better performance (.87 macro and .93 weighted f1 score in predicting expert labels) than the current state-of-the-art RoBERTa model. We discuss trade-offs across methods and models. 

   more » « less