skip to main content


Title: Measures of mathematics teachers’ behavior and affect: An examination of the assessment landscape
Although the paradigm wars between quantitative and qualitative research methods and the associated epistemologies may have settled down in recent years within the mathematics education research community, the high value placed on quantitative methods and randomized control trials remain as the gold standard at the policy-making level (USDOE, 2008). Although diverse methods are valued in the mathematics education community, if mathematics educators hope to influence policy to cultivate more equitable education systems, then we must engage in rigorous quantitative research. However, quantitative research is limited in what it can measure by the quantitative tools that exist. In mathematics education, it seems as though the development of quantitative tools and studying their associated validity and reliability evidence has lagged behind the important constructs that rich qualitative research has uncovered. The purpose of this study is to describe quantitative instruments related to mathematics teacher behavior and affect in order to better understand what currently exists in the field, what validity and reliability evidence has been published for such instruments, and what constructs each measure. 1. How many and what types of instruments of mathematics teacher behavior and affect exist? 2. What types of validity and reliability evidence are published for these instruments? 3. What constructs do these instruments measure? 4. To what extent have issues of equity been the focus of the instruments found?  more » « less
Award ID(s):
1920621
NSF-PAR ID:
10332442
Author(s) / Creator(s):
; ; ; ; ; ; ;
Date Published:
Journal Name:
Annual meeting program American Educational Research Association
ISSN:
0163-9676
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. The purpose of this working group is to continue to bring together scholars with an interest in examining the use of and access to large-scale quantitative tools used to measure student- and teacher-related outcomes in mathematics education. The working group session will focus on (1) updating the workgroup on the progress made since the first working group at PME-NA in Tucson, Arizona, specifically focusing on the outcomes of the Validity Evidence for Measurement in Mathematics Education conference that took place in April, 2017, in San Antonio, (2) continued development of a document of available tools and their associated validity evidence, and (3) identification of potential follow-up activities to continue this work. The efforts of the group will be summarized and extended through both social media tools and online collaboration tools to further promote this work. 
    more » « less
  2. In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes. 
    more » « less
  3. Recent calls to action focus on using educational tools that promote mathematics learning through evidence-based and equity-forward practices (NCTM, 2018). These practices may be derived from scholarship that examines factors related to mathematics teaching and learning using quantitative measures. A purpose of this presentation is to highlight areas of strength and opportunity related to the use of quantitative measures in scholarship examining K-12 mathematics settings. One outcome from this research-in progress is that scholars may become more aware of quantitative assessments for use in their research. A second outcome from this research is to foster conversations among colleagues around collaborative scholarship as well as areas for growth within mathematics education assessment. As a result, scholars may be better equipped to engage in quantitative research within mathematics contexts. Recognizing what is available and relevant to a desired area of study has potential to address contexts connected to topics described in Catalyzing Change (NCTM, 2018, 2020, 2020). That is, scholars cannot quantitatively measure constructs described in Catalyzing Change until it is known what measures are available and what they assess. This research-in progress aims to engage researchers in ongoing research and promote discussions across attendees. 
    more » « less
  4. The purpose of this working group is to bring together scholars with an interest in examining the research on quantitative tools and measures for gathering meaningful data, and to spark conversations and collaboration across individuals and groups with an interest in synthesizing the literature on large-scale tools used to measure student- and teacher-related outcomes. While syntheses of measures for use in mathematics education can be found in the literature, few can be described as a comprehensive analysis. The working group session will focus on (1) defining terms identified as critical (e.g., large-scale, quantitative, and validity evidence) for bounding the focus of the group, (2) initial development of a document of available tools and their associated validity evidence, and (3) identification of potential follow-up activities to continue the work to identify tools and developed related synthesis documents (e.g., the formation of sub-groups around potential topics of interest). The efforts of the group will be summarized and extended through both social media tools (e.g., creating a Facebook group) and online collaboration tools (e.g., Google hangouts and documents) to further promote this work. 
    more » « less
  5. Teachers’ beliefs can have powerful consequences on instructional decisions and student learning. However, little research focuses on how teachers’ beliefs about the role of race and gender in mathematics teaching and learning influence educational equity within classrooms. This gap is partly due to the lack of studies focused on variation within classrooms, which in turn is hampered by the lack of instruments designed to measure mathematics-specific equity beliefs. In this study of 313 preservice and practicing elementary teachers, we report evidence of construct validity for the Attributions of Mathematical Excellence Scale. Factor analyses provide support for a four-factor structure, including genetic, social, personal, and educational attributions. The findings suggest that the same system of attribution beliefs underlies both racial and gender prejudice among elementary mathematics teachers. The Attributions of Mathematical Excellence Scale has the potential to provide a useful outcome measure for equity-focused interventions in teacher education and professional development. 
    more » « less