More Like this

1. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application

   Suresh, A. ; Jacobs, J. ; Lai, V. ; Tan, C. ; Ward, W. ; Martin, J.H. ; Sumner, T. ( January 2021 , Association for the Advancement of Artificial Intelligence)

   TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application combines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education community and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastructure for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss several technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms.
2. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application

   Suresh, A. ; Jacobs, J. ; Lai, V. ; Tan, C. ; Ward, W. ; Martin, J.H. ; Sumner, T. ( January 2021 , Association for the Advancement of Artificial Intelligence in Education)

   TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application com- bines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education com- munity and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastruc- ture for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss sev- eral technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms.
3. The TalkMoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves.

   Suresh, A. ; Jacobs, J. ; Clevenger, C. ; Perkoff, M. ; Martin, J. ; Sumner, T. ( July 2022 , Proceedings of the 13th Conference on Language Resources and Evaluation)

   Transcripts of teaching episodes can be effective tools to understand discourse patterns in classroom instruction. According to most educational experts, sustained classroom discourse is a critical component of equitable, engaging, and rich learning environments for students. This paper describes the TalkMoves dataset, composed of 567 human annotated K-12 mathematics lesson transcripts (including entire lessons or portions of lessons) derived from video recordings. The set of transcripts primarily includes in-person lessons with whole-class discussions and/or small group work, as well as some online lessons. All of the transcripts are human-transcribed, segmented by the speaker (teacher or student), and annotated at the sentence level for ten discursive moves based on accountable talk theory. In addition, the transcripts include utterance-level information in the form of dialogue act labels based on the Switchboard Dialog Act Corpus. The dataset can be used by educators, policymakers, and researchers to understand the nature of teacher and student discourse in K-12 math classrooms. Portions of this dataset have been used to develop the TalkMoves application, which provides teachers with automated, immediate, and actionable feedback about their mathematics instruction.
4. Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms

   https://doi.org/10.18653/v1/2022.bea-1.11  

   Suresh, Abhijit ; Jacobs, Jennifer ; Perkoff, Margaret ; Martin, James H. ; Sumner, Tamara ( July 2022 , Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications)

   “Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers’ ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similarmore »transformer-based classification models.« less
5. A balancing act: Elementary teachers and their students balancing tradeoffs in engineering design projects

   Johnson, Matthew M. ; Gil, Minyoung ( August 2022 , ASEE annual conference proceedings)

   This fundamental research in pre-college education engineering study investigates the ways in which elementary school students and their teacher balance the tradeoffs in engineering design. STEM education reforms promote the engagement of K-12 students in the epistemic practices of disciplinary experts to teach content.1,2,3 This emphasis on practices is a paradigm shift that requires both extensive professional development and research to learn about the ways in which students and teacher learn about and participate in these practices. Balancing tradeoffs is an important practice in engineering but most often in classroom curricula it is embedded in the concept of iteration1,4; however, improving a design is not always the same as balancing trade-offs.1 Optimizing a multivariate problem requires students to engage in a number of engineering practices, like considering multiple solution, making tradeoffs between criteria and constraints, applying math and science knowledge to problem solving, constructing models, making evidence-based decisions, and assessing the implications of solutions5. The ways in which teachers and students collectively balance these tradeoffs in a design has been understudied1. Our primary research questions are, “How do teachers and students make decisions about making tradeoffs between criteria and constraints” and “How do experiences in teacher workshops affect the waysmore »they implement engineering projects in their classes.” We take an ethnographic perspective to investigate these phenomena, and collected video data, field notes, student journals, and semi-structured interviews of eight elementary teachers in a workshop and similar data from two of the workshop teachers’ classes as they implemented the curriculum they learned in the workshop. Our analyses focus on the disciplinary practices teachers and students use to make decisions for balancing tradeoffs, how they are supported (or impeded) by teachers, and how they justify these decisions. Similarly, we compared two of the teachers wearing their “student hat” in the workshop as well as their “teacher hat” in the classroom5. Our analyses suggest three significant findings. First, teachers and students tended to focus on one criterion (e.g. cost, performance) and had few discussions about trying to minimize cost and maximize performance. Second, curriculum design significantly impacts the choices students make. Using two examples, we will show the impact of weighting criteria differently on the design strategies teachers and students make. Last, we noted most of the feedback given was related to managing classroom activity rather than supporting students’ designs. Implications of this study are relevant to both engineering educators and engineering curriculum developers.« less