More Like this

1. Creating a coupled multiple response assessment for modeling in lab courses

   https://doi.org/10.1119/perc.2020.pr.pollard  

   Pollard, Benjamin; Fox, Michael F.; Ríos, Laura; Lewandowski, H. J. (September 2020, 2020 Physics Education Research Conference Proceedings)

   null (Ed.)

   Research-based assessment instruments (RBAIs) are essential tools to measure aspects of student learning and improve pedagogical practice. RBAIs are designed to measure constructs related to a well-defined learning goal. However, relatively few RBAIs exist that are suitable for the specific learning goals of upper-division physics lab courses. One such learning goal is modeling, the process of constructing, testing, and refining models of physical and measurement systems. Here, we describe the creation of one component of an RBAI to measure proficiency with modeling. The RBAI is called the Modeling Assessment for Physics Laboratory Experiments (MAPLE). For use with large numbers of students, MAPLE must be scalable, which includes not requiring impractical amounts of labor to analyze its data as is often the case with large free-response assessments. We, therefore, use the coupled multiple response (CMR) format, from which data can be analyzed by a computer, to create items for measuring student reasoning in this component of MAPLE.We describe the process we used to create a set of CMR items for MAPLE, provide an example of this process for an item, and lay out an argument for construct validity of the resulting items based on our process. 

   more » « less
2. Item Response Theory – A Statistical Framework for Educational and Psychological Measurement

   Yunxiao Chen, Xiaoou Li (March 2023, Statistical science)

   Item response theory (IRT) has become one of the most popular statistical models for psychometrics, a field of study concerned with the theory and techniques of psychological measurement. The IRT models are latent factor models tailored to the analysis, interpretation, and prediction of individuals’ behaviors in answering a set of measurement items that typically involve categorical response data. Many important questions of measurement are directly or indirectly answered through the use of IRT models, including scoring individuals’ test performances, validating a test scale, linking two tests, among others. This paper provides a review of item response theory, including its statistical framework and psychometric applications. We establish connections between item response theory and related topics in statistics, including empirical Bayes, nonparametric methods, matrix completion, regularized estimation, and sequential analysis. Possible future directions of IRT are discussed from the perspective of statistical learning. 

   more » « less
3. Evaluating the differential impact of dichotomous and partial credit scoring models on student problem-solving assessment outcomes.

   May, T.; Koskey, K.; Bostic, J.; Stone, G.; Kruse, L.; Matney, G. (February 2023, School science and mathematics)

   Determining the most appropriate method of scoring an assessment is based on multiple factors, including the intended use of results, the assessment's purpose, and time constraints. Both the dichotomous and partial credit models have their advantages, yet direct comparisons of assessment outcomes from each method are not typical with constructed response items. The present study compared the impact of both scoring methods on the internal structure and consequential validity of a middle-grades problem-solving assessment called the problem solving measure for grade six (PSM6). After being scored both ways, Rasch dichotomous and partial credit analyses indicated similarly strong psychometric findings across models. Student outcome measures on the PSM6, scored both dichotomously and with partial credit, demonstrated strong, positive, significant correlation. Similar demographic patterns were noted regardless of scoring method. Both scoring methods produced similar results, suggesting that either would be appropriate to use with the PSM6. 

   more » « less
4. Examining how using dichotomous and partial credit scoring models influence sixth‐grade mathematical problem‐solving assessment outcomes

   https://doi.org/10.1111/ssm.12570  

   May, Toni A.; Koskey, Kristin L. K.; Bostic, Jonathan D.; Stone, Gregory E.; Kruse, Lance M.; Matney, Gabriel (February 2023, School Science and Mathematics)

   Abstract Determining the most appropriate method of scoring an assessment is based on multiple factors, including the intended use of results, the assessment's purpose, and time constraints. Both the dichotomous and partial credit models have their advantages, yet direct comparisons of assessment outcomes from each method are not typical with constructed response items. The present study compared the impact of both scoring methods on the internal structure and consequential validity of a middle‐grades problem‐solving assessment called the problem solving measure for grade six (PSM6). After being scored both ways, Rasch dichotomous and partial credit analyses indicated similarly strong psychometric findings across models. Student outcome measures on the PSM6, scored both dichotomously and with partial credit, demonstrated strong, positive, significant correlation. Similar demographic patterns were noted regardless of scoring method. Both scoring methods produced similar results, suggesting that either would be appropriate to use with the PSM6. 

   more » « less
5. Examining how using dichotomous and partial credit scoring models influence sixth‐grade mathematical problem‐solving assessment outcomes

   May, T. (January 2023, School science and mathematics)

   Determining the most appropriate method of scoring an assessment is basedon multiple factors, including the intended use of results, the assessment's pur-pose, and time constraints. Both the dichotomous and partial credit modelshave their advantages, yet direct comparisons of assessment outcomes fromeach method are not typical with constructed response items. The presentstudy compared the impact of both scoring methods on the internal structureand consequential validity of a middle-grades problem-solving assessmentcalled the problem solving measure for grade six (PSM6). After being scoredboth ways, Rasch dichotomous and partial credit analyses indicated similarlystrong psychometric findings across models. Student outcome measures on thePSM6, scored both dichotomously and with partial credit, demonstratedstrong, positive, significant correlation. Similar demographic patterns werenoted regardless of scoring method. Both scoring methods produced similarresults, suggesting that either would be appropriate to use with the PSM6. 

   more » « less