More Like this

1. Exploring and examining quantitative measures

   Bostic, J. ( October 2016 , Proceedings for the 38th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education)

   The purpose of this working group is to bring together scholars with an interest in examining the research on quantitative tools and measures for gathering meaningful data, and to spark conversations and collaboration across individuals and groups with an interest in synthesizing the literature on large-scale tools used to measure student- and teacher-related outcomes. While syntheses of measures for use in mathematics education can be found in the literature, few can be described as a comprehensive analysis. The working group session will focus on (1) defining terms identified as critical (e.g., large-scale, quantitative, and validity evidence) for bounding the focus of the group, (2) initial development of a document of available tools and their associated validity evidence, and (3) identification of potential follow-up activities to continue the work to identify tools and developed related synthesis documents (e.g., the formation of sub-groups around potential topics of interest). The efforts of the group will be summarized and extended through both social media tools (e.g., creating a Facebook group) and online collaboration tools (e.g., Google hangouts and documents) to further promote this work. 

   more » « less
2. Developing a measure to capture middle school students’ interpretive understanding of engineering design

   Piña, Jeremiah ; Ellis, Glenn W ; Rudnitsky, Al ; Mazur, Rebecca ; McGinnis-Cavanaugh, Beth ; Huff, Isabel ( July 2021 , 2021 American Society for Engineering Education Annual Conference)

   null (Ed.)

   This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 246) groups, wherein the former participated in a two-to-three-week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 

   more » « less
3. Developing a measure to capture middle school students’ interpretive understanding of engineering design

   Piña, Jeremiah ; Ellis, Glenn W ; Rudnitsky, Al ; Mazur, Rebecca ; McGinnis-Cavanaugh, Beth ; Huff, Isabel ( July 2021 , 2021 American Society for Engineering Education Annual Conference)

   null (Ed.)

   This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non- treatment (n = 246) groups, wherein the former participated in a two-to-three- week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann- Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 

   more » « less
4. Measures of mathematics teachers’ behavior and affect: An examination of the assessment landscape

   Gallagher, M. ; Bardelli, E. ; Folger, T. ; Neely, A. ; Bostic, J. ; Walkowiak, T. ; Wilhelm, A. ; Zelkowski, J. ( April 2022 , Annual meeting program American Educational Research Association)

   Although the paradigm wars between quantitative and qualitative research methods and the associated epistemologies may have settled down in recent years within the mathematics education research community, the high value placed on quantitative methods and randomized control trials remain as the gold standard at the policy-making level (USDOE, 2008). Although diverse methods are valued in the mathematics education community, if mathematics educators hope to influence policy to cultivate more equitable education systems, then we must engage in rigorous quantitative research. However, quantitative research is limited in what it can measure by the quantitative tools that exist. In mathematics education, it seems as though the development of quantitative tools and studying their associated validity and reliability evidence has lagged behind the important constructs that rich qualitative research has uncovered. The purpose of this study is to describe quantitative instruments related to mathematics teacher behavior and affect in order to better understand what currently exists in the field, what validity and reliability evidence has been published for such instruments, and what constructs each measure. 1. How many and what types of instruments of mathematics teacher behavior and affect exist? 2. What types of validity and reliability evidence are published for these instruments? 3. What constructs do these instruments measure? 4. To what extent have issues of equity been the focus of the instruments found? 

   more » « less
5. Gender differences in mathematics and its cognitive and non-cognitive predictors in community college students

   Boada, C. ; Cirino, P.T. ( February 2023 , International Neuropsychological Society)

   Objective Historically, numerous studies have supported a male advantage in math. While more recent literature has shown that the gender gap is either decreasing or non-significant, a gender difference remains for higher level math (high school and college) (Hyde et. al. 1990; Casey et. al. 1995). It is known that both cognitive and non-cognitive factors influence math performance. There is little evidence for gender differences in working memory (Miller & Bichsel, 2004), which is a key predictor for mathematics. There is, however, evidence for gender differences in the non-cognitive domain, including math anxiety, with females having higher levels (Miller & Bichsel, 2004; Goetz, et. al. 2013). This study evaluates gender differences in both standardized and everyday math performances, and the way that cognitive and non-cognitive factors impact math. The study is focused on a very understudied group with high levels of math difficulty, namely community college students. We expected to find gender differences in math, and expect these to be in part accounted for by gender differences in strong mathematical predictors, particularly non-cognitive factors. Participants and Methods Participants included 94 community college students enrolled in their first math class (60 female; 34 male). Participants were administered the Kaufman Test of Educational Achievement – 3rd edition (KTEA3): Math Computation (MC) and Math Concepts Application (MCA) subtests, as well as an original Everyday Math (EM) measure which assessed their math ability in the context of common uses for math (e.g., financial and health numeracy). Additional measures included math anxiety, self-efficacy, and confidence. Finally, measures of complex span working memory tasks were administered to assess verbal and spatial working memory. Analyses were performed using correlation and regression to examine relationships between the cognitive and non-cognitive variables and standardized and everyday math measures. Results Correlations showed that all cognitive and non-cognitive variables are significantly correlated with all three math measures (all p < .05). There were no significant gender differences for any of the math measures, nor the working memory, or non-cognitive measures. Regression showed that across all three math outcomes, math anxiety and verbal working memory are significantly predictive of math performance. Overall R2 values were significant (range 27% to 37%, all p < .001). Working memory and math anxiety were unique predictors in all three regressions (all p < .05), but other non-cognitive variables such as self-efficacy did not show unique prediction (all p > .05). Conclusions There was no evidence for gender differences on any studied variable. This stands in contrast to prior studies, although few studies have included community college students. On the other hand, both cognitive and non-cognitive factors were complimentary in the prediction of math outcomes, which is consistent with prior work. Among non-cognitive predictors, math anxiety was particularly prominent. This study clarifies prior conflicting work regarding gender differences, and highlights the role of both math anxiety and working memory as relevant for multiple math outcomes. 

   more » « less