skip to main content

Title: Automatically Measuring Question Authenticity in Real-World Classrooms
Analyzing the quality of classroom talk is central to educational research and improvement efforts. In particular, the presence of authentic teacher questions, where answers are not predetermined by the teacher, helps constitute and serves as a marker of productive classroom discourse. Further, authentic questions can be cultivated to improve teaching effectiveness and consequently student achievement. Unfortunately, current methods to measure question authenticity do not scale because they rely on human observations or coding of teacher discourse. To address this challenge, we set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically. Our methods were iteratively refined using classroom audio and human coded observational data from two sources: (a) a large archival database of text transcripts of 451 observations from 112 classrooms; and (b) a newly collected sample of 132 high-quality audio recordings from 27 classrooms, obtained under technical constraints that anticipate large-scale automated data collection and analysis. Correlations between human coded and computer-coded authenticity at the classroom level were sufficiently high (r = .602 for archival transcripts and .687 for audio recordings) to provide a valuable complement to human coding in research efforts.  more » « less
Award ID(s):
Author(s) / Creator(s):
Date Published:
Journal Name:
Educational researcher
Page Range / eLocation ID:
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Transcripts of teaching episodes can be effective tools to understand discourse patterns in classroom instruction. According to most educational experts, sustained classroom discourse is a critical component of equitable, engaging, and rich learning environments for students. This paper describes the TalkMoves dataset, composed of 567 human annotated K-12 mathematics lesson transcripts (including entire lessons or portions of lessons) derived from video recordings. The set of transcripts primarily includes in-person lessons with whole-class discussions and/or small group work, as well as some online lessons. All of the transcripts are human-transcribed, segmented by the speaker (teacher or student), and annotated at the sentence level for ten discursive moves based on accountable talk theory. In addition, the transcripts include utterance-level information in the form of dialogue act labels based on the Switchboard Dialog Act Corpus. The dataset can be used by educators, policymakers, and researchers to understand the nature of teacher and student discourse in K-12 math classrooms. Portions of this dataset have been used to develop the TalkMoves application, which provides teachers with automated, immediate, and actionable feedback about their mathematics instruction. 
    more » « less
  2. Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool—with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts.

    more » « less
  3. Gardner, Stephanie (Ed.)
    Stronger metacognition, or awareness and regulation of thinking, is related to higher academic achievement. Most metacognition research has focused at the level of the individual learner. However, a few studies have shown that students working in small groups can stimulate metacognition in one another, leading to improved learning. Given the increased adoption of interactive group work in life science classrooms, there is a need to study the role of social metacognition, or the awareness and regulation of the thinking of others, in this context. Guided by the frameworks of social metacognition and evidence-based reasoning, we asked: 1) What metacognitive utterances (words, phrases, statements, or questions) do students use during small-group problem solving in an upper-division biology course? 2) Which metacognitive utterances are associated with small groups sharing higher-quality reasoning in an upper-division biology classroom? We used discourse analysis to examine transcripts from two groups of three students during breakout sessions. By coding for metacognition, we identified seven types of metacognitive utterances. By coding for reasoning, we uncovered four categories of metacognitive utterances associated with higher-quality reasoning. We offer suggestions for life science educators interested in promoting social metacognition during small-group problem solving. 
    more » « less
  4. Bringing Research into the Classroom (BRIC) engaged rural K-12 science teachers in sustained, mentored science research. BRIC’s goal was to equip teachers with the knowledge, skills, and dispositions to provide high-quality biomedical research opportunities for K-12 students and teachers. Programmatic elements included authentic, place-based, microbiology outreach in K-12 classrooms, summer teacher research academies focused on content knowledge and research, and a capstone symposium. Over 9,000 Montana students collected and tested environmental samples to isolate new-toscience bacteriophages (viruses that infect bacteria). University scientists, faculty, and students mentored K-12 teachers and students during classroom outreach visits and teacher research academies. BRIC aimed to increase teacher and student bacteriophage content knowledge and research skills through meaningful, mentored research projects. BRIC researchers hypothesized greater program impacts from intensive teacher professional development combined with classroom outreach, compared to classroom outreach visits alone. Program evaluation compared two cohorts of teachers, which each received all programmatic elements through a four-year, staggered rollout. Teachers and students were assessed for gains in knowledge, skills, and science attitudes. A subset of our evaluation instruments and outcomes, program dissemination, lessons learned, and recommendations for replicating the BRIC model are discussed. 
    more » « less
  5. null (Ed.)
    This Research Work in Progress paper presents a case study that demonstrates how a secondary school teacher with a non-STEM background identifies parallels between the engineering design process and music creation to embrace teaching an engineering course for the first time. Multiple interviews and classroom observations were open coded using a two-cycle coding approach to reveal four themes: overcoming imposter syndrome, connections between engineering and music, challenges encountered, and changes in practice. These themes highlight the processes involved in transferring a pedagogical philosophy that can inform future efforts to explore the necessary preconditions for bridging seemingly disparate and unconnected content areas. Further exploration building on these findings will inform efforts to broaden the pool of teachers capable of teaching pre-college engineering classes. 
    more » « less