More Like this

1. Using AI to Promote Equitable Classroom Discussions: The TalkMoves Application

   Suresh, A. ; Jacobs, J. ; Clevenger, C. ; Lai, V. ; Tan, C. ; Martin, J. H. ; Sumner, T. ( July 2021 , International Conference on Artificial Intelligence in Education)

   Inclusion in mathematics education is strongly tied to discourse rich classrooms, where students ideas play a central role. Talk moves are specific discursive practices that promote inclusive and equitable participation in classroom discussions. This paper describes the development of the TalkMoves application, which provides teachers with detailed feedback on their usage of talk moves based on accountable talk theory. Building on our recent work to automate the classification of teacher talk moves, we have expanded the application to also include feedback on a set of student talk moves. We present results from several deep learning models trained to classify student sentences into student talk moves with performance up to 73% F1. The classroom data used for training these models were collected from multiple sources that were pre-processed and annotated by highly reliable experts. We validated the performance of the model on an out-of-sample dataset which included 166 classroom transcripts collected from teachers piloting the application.
2. Examining the Literacy Practices of Engineers to Develop a Model of Disciplinary Literacy Instruction for K-12 Engineering (Work in Progress)

   Green, Theresa ; Minichiello, Angela ; Wilson-Lopez, Amy ( June 2018 , ASEE Annual Conference proceedings)

   Despite efforts to diversify the science, technology, engineering, and mathematics (STEM) workforce, engineering remains a White, male-dominated profession. Often, women and underrepresented students do not identify with STEM careers and many opt out of STEM pathways prior to entering high school or college. In order to broaden participation in engineering, new methods of engaging and retaining those who are traditionally underrepresented in engineering are needed. This work is based on a promising approach for encouraging and supporting diverse participation in engineering: disciplinary literacy instruction (DLI). Generally, teachers use DLI to provide K-12 students with a framework for interpreting, evaluating, and generating discipline-specific texts. This instruction provides students with an understanding of how experts in the discipline read, engage, and generate texts used to solve problems or communicate information. While models of disciplinary literacy have been developed and disseminated in several humanities and science fields, there is a lack of empirical and theoretical research that examines the use of DLI within the engineering domain. It is thought that DLI can be used to foster diverse student interest in engineering from a young age by removing literacy-based barriers that often discourage underrepresented students from entering and pursuing careers in STEM fields. Thismore »work-in-progress paper describes a new study underway to develop and disseminate a model of disciplinary literacy in engineering. During this project, researchers will observe, interview, and collect written artifacts from engineers working across four sub-disciplines of engineering: aerospace/mechanical, biological, civil/environmental, and electrical/computer. Data that will be collected include interview transcripts, observation field notes, engineer logs of literacy practices, and photographs of texts that the engineers read and write. Data will be analyzed using constant comparative analytic (CCA) methods. CCA will be used to generate theoretical codes from the data that will form the basis for a model of disciplinary literacy in engineering. As a primary outcome of this research, the engineering DLI model will promote the use of DLI practices within K-12 engineering instruction in order to assist and encourage diverse, underrepresented students to engage in engineering courses of study and pursue STEM careers. Thus far, the research team has begun collecting and analyzing data from two electrical engineers. This work in progress paper will report on preliminary findings, as well as implications for K-12 classroom instruction. For instance, this study has shed insights on how engineers use texts as part of the process of conducting failure analysis, and the research team has begun to conceptualize how these types of texts might be used with K-12 students to help them conduct failure analyses during design testing. Ultimately, this project will result in a list of grade-appropriate texts, evaluative frameworks, and activities (e.g., failure analysis in testing) that K-12 engineering teachers can use to prepare their diverse students to think, act, read, and write like engineers.« less
3. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application

   Suresh, A. ; Jacobs, J. ; Lai, V. ; Tan, C. ; Ward, W. ; Martin, J.H. ; Sumner, T. ( January 2021 , Association for the Advancement of Artificial Intelligence)

   TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application combines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education community and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastructure for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss several technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms.
4. Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application

   Suresh, A. ; Jacobs, J. ; Lai, V. ; Tan, C. ; Ward, W. ; Martin, J.H. ; Sumner, T. ( January 2021 , Association for the Advancement of Artificial Intelligence in Education)

   TalkMoves is an innovative application designed to support K-12 mathematics teachers to reflect on, and continuously improve their instructional practices. This application com- bines state-of-the-art natural language processing capabilities with automated speech recognition to automatically analyze classroom recordings and provide teachers with personalized feedback on their use of specific types of discourse aimed at broadening and deepening classroom conversations about mathematics. These specific discourse strategies are referred to as “talk moves” within the mathematics education com- munity and prior research has documented the ways in which systematic use of these discourse strategies can positively impact student engagement and learning. In this article, we describe the TalkMoves application’s cloud-based infrastruc- ture for managing and processing classroom recordings, and its interface for providing teachers with feedback on their use of talk moves during individual teaching episodes. We present the series of model architectures we developed, and the studies we conducted, to develop our best-performing, transformer-based model (F1 = 79.3%). We also discuss sev- eral technical challenges that need to be addressed when working with real-world speech and language data from noisy K-12 classrooms.
5. Fine-tuning Transformers with Additional Context to Classify Discursive Moves in Mathematics Classrooms

   https://doi.org/10.18653/v1/2022.bea-1.11  

   Suresh, Abhijit ; Jacobs, Jennifer ; Perkoff, Margaret ; Martin, James H. ; Sumner, Tamara ( July 2022 , Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications)

   “Talk moves” are specific discursive strategies used by teachers and students to facilitate conversations in which students share their thinking, and actively consider the ideas of others, and engage in rich discussions. Experts in instructional practices often rely on cues to identify and document these strategies, for example by annotating classroom transcripts. Prior efforts to develop automated systems to classify teacher talk moves using transformers achieved a performance of 76.32% F1. In this paper, we investigate the feasibility of using enriched contextual cues to improve model performance. We applied state-of-the-art deep learning approaches for Natural Language Processing (NLP), including Robustly optimized bidirectional encoder representations from transformers (Roberta) with a special input representation that supports previous and subsequent utterances as context for talk moves classification. We worked with the publically available TalkMoves dataset, which contains utterances sourced from real-world classroom sessions (human- transcribed and annotated). Through a series of experimentations, we found that a combination of previous and subsequent utterances improved the transformers’ ability to differentiate talk moves (by 2.6% F1). These results constitute a new state of the art over previously published results and provide actionable insights to those in the broader NLP community who are working to develop similarmore »transformer-based classification models.« less