More Like this

1. Practices in instrument use and development in chemistry education research and practice 2010–2021

   https://doi.org/10.1039/D2RP00275B  

   Lazenby, Katherine; Tenney, Kristin; Marcroft, Tina A; Komperda, Regis (July 2023, Chemistry Education Research and Practice)

   Assessment instruments that generate quantitative data on attributes (cognitive, affective, behavioral,etc.) of participants are commonly used in the chemistry education community to draw conclusions in research studies or inform practice. Recently, articles and editorials have stressed the importance of providing evidence for the validity and reliability of data collected with these instruments following guidance from the Standards for Educational and Psychological Testing. This study examines how quantitative instruments have been used in the journalChemistry Education Research and Practice(CERP) from 2010–2021. Of the 369 unique researcher-developed instruments used during this time frame, the majority only appeared in a single publication (89.7%) and were rarely reused. Cognitive topics were the most common target of the instruments (56.6%). Validity and/or reliability evidence was provided in 64.4% of instances where instruments were used inCERPpublications. The most frequently reported evidence was single administration reliability (e.g., coefficient alpha), appearing in 47.9% of instances. Only 37.2% of instances reported evidence of both validity and reliability. These results indicate that, as a field, opportunities exist to increase the amount of validity and reliability evidence available for data collected with instruments and that reusing instruments may be one method of increasing this type of data quality evidence for instruments used by the chemistry education community. 

   more » « less
2. EXPLORING AND EXAMINING QUANTITATIVE MEASURES

   Shih, J.; Bostic, J.; Carney, M.; Krupa, E. (October 2017, Psychology of Mathematics Education North America)

   The purpose of this working group is to continue to bring together scholars with an interest in examining the use of and access to large-scale quantitative tools used to measure student- and teacher-related outcomes in mathematics education. The working group session will focus on (1) updating the workgroup on the progress made since the first working group at PME-NA in Tucson, Arizona, specifically focusing on the outcomes of the Validity Evidence for Measurement in Mathematics Education conference that took place in April, 2017, in San Antonio, (2) continued development of a document of available tools and their associated validity evidence, and (3) identification of potential follow-up activities to continue this work. The efforts of the group will be summarized and extended through both social media tools and online collaboration tools to further promote this work. 

   more » « less
3. An Inquiry into the Use of Intercoder Reliability Measures in Qualitative Research

   Wilson-Lopez, A.; Minichiello, A.; Green, T. (July 2019, ASEE Annual Conference proceedings)

   In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes. 

   more » « less
4. K-12 Student and Teacher Math Measures: What’s out there? What do we need?

   Bostic, J; Krupa, E; Jong, C (October 2023, NCTM)

   Recent calls to action focus on using educational tools that promote mathematics learning through evidence-based and equity-forward practices (NCTM, 2018). These practices may be derived from scholarship that examines factors related to mathematics teaching and learning using quantitative measures. A purpose of this presentation is to highlight areas of strength and opportunity related to the use of quantitative measures in scholarship examining K-12 mathematics settings. One outcome from this research-in progress is that scholars may become more aware of quantitative assessments for use in their research. A second outcome from this research is to foster conversations among colleagues around collaborative scholarship as well as areas for growth within mathematics education assessment. As a result, scholars may be better equipped to engage in quantitative research within mathematics contexts. Recognizing what is available and relevant to a desired area of study has potential to address contexts connected to topics described in Catalyzing Change (NCTM, 2018, 2020, 2020). That is, scholars cannot quantitatively measure constructs described in Catalyzing Change until it is known what measures are available and what they assess. This research-in progress aims to engage researchers in ongoing research and promote discussions across attendees. 

   more » « less
5. Exploring and examining quantitative measures

   Bostic, J. (October 2016, Proceedings for the 38th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education)

   The purpose of this working group is to bring together scholars with an interest in examining the research on quantitative tools and measures for gathering meaningful data, and to spark conversations and collaboration across individuals and groups with an interest in synthesizing the literature on large-scale tools used to measure student- and teacher-related outcomes. While syntheses of measures for use in mathematics education can be found in the literature, few can be described as a comprehensive analysis. The working group session will focus on (1) defining terms identified as critical (e.g., large-scale, quantitative, and validity evidence) for bounding the focus of the group, (2) initial development of a document of available tools and their associated validity evidence, and (3) identification of potential follow-up activities to continue the work to identify tools and developed related synthesis documents (e.g., the formation of sub-groups around potential topics of interest). The efforts of the group will be summarized and extended through both social media tools (e.g., creating a Facebook group) and online collaboration tools (e.g., Google hangouts and documents) to further promote this work. 

   more » « less