skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: A Diagnostic Facet Status Model (DFSM) for Extracting Instructionally Useful Information from Diagnostic Assessment
Modern assessment demands, resulting from educational reform efforts, call for strengthening diagnostic testing capabilities to identify not only the understanding of expected learning goals but also related intermediate understandings that are steppingstones on pathways to learning goals. An accurate and nuanced way of interpreting assessment results will allow subsequent instructional actions to be targeted. An appropriate psychometric model is indispensable in this regard. In this study, we developed a new psychometric model, namely, the diagnostic facet status model (DFSM), which belongs to the general class of cognitive diagnostic models (CDM), but with two notable features: (1) it simultaneously models students’ target understanding (i.e., goal facet) and intermediate understanding (i.e., intermediate facet); and (2) it models every response option, rather than merely right or wrong responses, so that each incorrect response uniquely contributes to discovering students’ facet status. Given that some combination of goal and intermediate facets may be impossible due to facet hierarchical relationships, a regularized expectation–maximization algorithm (REM) was developed for model estimation. A log-penalty was imposed on the mixing proportions to encourage sparsity. As a result, those impermissible latent classes had estimated mixing proportions equal to 0. A heuristic algorithm was proposed to infer a facet map from the estimated permissible classes. A simulation study was conducted to evaluate the performance of REM to recover facet model parameters and to identify permissible latent classes. A real data analysis was provided to show the feasibility of the model.  more » « less
Award ID(s):
2300382
PAR ID:
10529470
Author(s) / Creator(s):
Publisher / Repository:
Springer
Date Published:
Journal Name:
Psychometrika
ISSN:
0033-3123
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; (, Frontiers in Psychology)
    null (Ed.)
    Selected response items and constructed response (CR) items are often found in the same test. Conventional psychometric models for these two types of items typically focus on using the scores for correctness of the responses. Recent research suggests, however, that more information may be available from the CR items than just scores for correctness. In this study, we describe an approach in which a statistical topic model along with a diagnostic classification model (DCM) was applied to a mixed item format formative test of English and Language Arts. The DCM was used to estimate students’ mastery status of reading skills. These mastery statuses were then included in a topic model as covariates to predict students’ use of each of the latent topics in their written answers to a CR item. This approach enabled investigation of the effects of mastery status of reading skills on writing patterns. Results indicated that one of the skills, Integration of Knowledge and Ideas, helped detect and explain students’ writing patterns with respect to students’ use of individual topics. 
    more » « less
  2. ; ; ; ; (, American Society for Engineering Education)
    Team-based learning is commonly used in engineering introductory courses. As students of a team may be from vastly different backgrounds, academically and non-academically, it is important for faculty members to know what aid or hinder team success. The dataset that is used in this paper includes student personality inputs, self-and-peer-assessments of teamwork, and perceptions of teamwork outcomes. Using this information, we developed several bayesian models that are able to predict if a team is working well. We also constructed and estimated Q-matrices which are crucial in explaining the relationship between latent traits and students’ characteristics in cognitive diagnostic models. The prediction and diagnostic models are able to help faculty members and instructors to gain insights into finding ways to separate students into teams more effectively so that students have a positive team-based learning experience. 
    more » « less
  3. ; ; ; ; (, Psychometrika)
    Sinharay, Sandip (Ed.)
    Diagnostic classification models (DCMs) have seen wide applications in educational and psychological measurement, especially in formative assessment. DCMs in the presence of testlets have been studied in recent literature. A key ingredient in the statistical modeling and analysis of testlet-based DCMs is the superposition of two latent structures, the attribute profile and the testlet effect. This paper extends the standard testlet DINA (T-DINA) model to accommodate the potential correlation between the two latent structures. Model identifiability is studied and a set of sufficient conditions are proposed. As a byproduct, the identifiability of the standard T-DINA is also established. The proposed model is applied to a dataset from the 2015 Programme for International Student Assessment. Comparisons are made with DINA and T-DINA, showing that there is substantial improvement in terms of the goodness of fit. Simulations are conducted to assess the performance of the new method under various settings. 
    more » « less
  4. ; ; ; ; ; ; ; (, Physical Review Physics Education Research)
    In physics education research, instructors and researchers often use research-based assessments (RBAs) to assess students’ skills and knowledge. In this paper, we support the development of a mechanics cognitive diagnostic to test and implement effective and equitable pedagogies for physics instruction. Adaptive assessments using cognitive diagnostic models provide significant advantages over fixed-length RBAs commonly used in physics education research. As part of a broader project to develop a cognitive diagnostic assessment for introductory mechanics within an evidence-centered design framework, we identified and tested the student models of four skills that cross content areas in introductory physics: apply vectors, conceptual relationships, algebra, and visualizations. We developed the student models in three steps. First, we based the model on learning objectives from instructors. Second, we coded the items on RBAs using the student models. Finally, we then tested and refined this coding using a common cognitive diagnostic model, the deterministic inputs, noisy “and” gate model. The data included 19 889 students who completed either the Force Concept Inventory, Force and Motion Conceptual Evaluation, or Energy and Momentum Conceptual Survey on the LASSO platform. The results indicated a good to adequate fit for the student models with high accuracies for classifying students with many of the skills. The items from these three RBAs do not cover all of the skills in enough detail, however, they will form a useful initial item bank for the development of the mechanics cognitive diagnostic. 
    more » « less
  5. ; ; ; (, 2021 IEEE Frontiers in Education Conference (FIE))
    This research-track work-in-progress paper contributes to engineering education by documenting progress in developing a new standard Engineering Computational Thinking Diagnostic to measure engineering student success in five factors of computational thinking. Over the past year, results from an initial validation attempt were used to refine diagnostic questions. A second statistical validation attempt was then completed in Spring 2021 with 191 student participants at three universities. Statistics show that all diagnostic questions had statistically significant factor loadings onto one general computational thinking factor that incorporates the five original factors of (a) Abstraction, (b) Algorithmic Thinking, (c) Decomposition, (d) Data Representation and Organization, and (e) Impact of Computing. This result was unexpected as our goal was a diagnostic that could discriminate among the five factors. A small population size caused by the virtual delivery of courses during the COVID-19 pandemic may be the explanation and a third round of validation in Fall 2021 is expected to result in a larger population given the return to face-to-face instruction. When statistical validation is completed, the diagnostic will help institutions identify students with strong entry level skills in computational thinking as well as students that require academic support. The diagnostic will inform curriculum design by demonstrating which factors are more accessible to engineering students and which factors need more time and focus in the classroom. The long-term impact of a successfully validated computational thinking diagnostic will be introductory engineering courses that better serve engineering students coming from many backgrounds. This can increase student self- efficacy, improve student retention, and improve student enculturation into the engineering profession. Currently, the diagnostic identifies general computational thinking skill 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email