More Like this

1. Sample Size Requirements for Applying Diagnostic Classification Models

   https://doi.org/10.3389/fpsyg.2020.621251  

   Sen, Sedat; Cohen, Allan S. (January 2021, Frontiers in Psychology)

   null (Ed.)

   Results of a comprehensive simulation study are reported investigating the effects of sample size, test length, number of attributes and base rate of mastery on item parameter recovery and classification accuracy of four DCMs (i.e., C-RUM, DINA, DINO, and LCDMREDUCED). Effects were evaluated using bias and RMSE computed between true (i.e., generating) parameters and estimated parameters. Effects of simulated factors on attribute assignment were also evaluated using the percentage of classification accuracy. More precise estimates of item parameters were obtained with larger sample size and longer test length. Recovery of item parameters decreased as the number of attributes increased from three to five but base rate of mastery had a varying effect on the item recovery. Item parameter and classification accuracy were higher for DINA and DINO models. 

   more » « less
2. Bayesian Model Assessment for Jointly Modeling Multidimensional Response Data with Application to Computerized Testing

   https://doi.org/10.1007/s11336-022-09845-x  

   Liu, Fang; Wang, Xiaojing; Hancock, Roeland; Chen, Ming-Hui (March 2022, Psychometrika)

   von Davier, Matthias (Ed.)

   Computerized assessment provides rich multidimensional data including trial-by-trial accuracy and response time (RT) measures. A key question in modeling this type of data is how to incorporate RT data, for example, in aid of ability estimation in item response theory (IRT) models. To address this, we propose a joint model consisting of a two-parameter IRT model for the dichotomous item response data, a log-normal model for the continuous RT data, and a normal model for corresponding paper-and-pencil scores. Then, we reformulate and reparameterize the model to capture the relationship between the model parameters, to facilitate the prior specification, and to make the Bayesian computation more efficient. Further, we propose several new model assessment criteria based on the decomposition of deviance information criterion (DIC) the logarithm of the pseudo-marginal likelihood (LPML). The proposed criteria can quantify the improvement in the fit of one part of the multidimensional data given the other parts. Finally, we have conducted several simulation studies to examine the empirical performance of the proposed model assessment criteria and have illustrated the application of these criteria using a real dataset from a computerized educational assessment program. 

   more » « less
3. A unified EM framework for estimation and inference of normal ogive item response models

   https://doi.org/10.1111/bmsp.12356  

   Meng, Xiangbin; Xu, Gongjun (October 2024, British Journal of Mathematical and Statistical Psychology)

   Abstract Normal ogive (NO) models have contributed substantially to the advancement of item response theory (IRT) and have become popular educational and psychological measurement models. However, estimating NO models remains computationally challenging. The purpose of this paper is to propose an efficient and reliable computational method for fitting NO models. Specifically, we introduce a novel and unified expectation‐maximization (EM) algorithm for estimating NO models, including two‐parameter, three‐parameter, and four‐parameter NO models. A key improvement in our EM algorithm lies in augmenting the NO model to be a complete data model within the exponential family, thereby substantially streamlining the implementation of the EM iteration and avoiding the numerical optimization computation in the M‐step. Additionally, we propose a two‐step expectation procedure for implementing the E‐step, which reduces the dimensionality of the integration and effectively enables numerical integration. Moreover, we develop a computing procedure for estimating the standard errors (SEs) of the estimated parameters. Simulation results demonstrate the superior performance of our algorithm in terms of its recovery accuracy, robustness, and computational efficiency. To further validate our methods, we apply them to real data from the Programme for International Student Assessment (PISA). The results affirm the reliability of the parameter estimates obtained using our method. 

   more » « less
4. Prior choice and data requirements of Bayesian multivariate hierarchical models fit to tag‐recovery data: The need for power analyses

   https://doi.org/10.1002/ece3.9847  

   Deane, Cody E; Carlson, Lindsay G; Cunningham, Curry J; Doak, Pat; Kielland, Knut; Breed, Greg A (March 2023, Ecology and Evolution)

   Abstract Recent empirical studies have quantified correlation between survival and recovery by estimating these parameters as correlated random effects with hierarchical Bayesian multivariate models fit to tag‐recovery data. In these applications, increasingly negative correlation between survival and recovery has been interpreted as evidence for increasingly additive harvest mortality. The power of these hierarchal models to detect nonzero correlations has rarely been evaluated, and these few studies have not focused on tag‐recovery data, which is a common data type. We assessed the power of multivariate hierarchical models to detect negative correlation between annual survival and recovery. Using three priors for multivariate normal distributions, we fit hierarchical effects models to a mallard (Anas platyrhychos) tag‐recovery data set and to simulated data with sample sizes corresponding to different levels of monitoring intensity. We also demonstrate more robust summary statistics for tag‐recovery data sets than total individuals tagged. Different priors led to substantially different estimates of correlation from the mallard data. Our power analysis of simulated data indicated most prior distribution and sample size combinations could not estimate strongly negative correlation with useful precision or accuracy. Many correlation estimates spanned the available parameter space (−1,1) and underestimated the magnitude of negative correlation. Only one prior combined with our most intensive monitoring scenario provided reliable results. Underestimating the magnitude of correlation coincided with overestimating the variability of annual survival, but not annual recovery. The inadequacy of prior distributions and sample size combinations previously assumed adequate for obtaining robust inference from tag‐recovery data represents a concern in the application of Bayesian hierarchical models to tag‐recovery data. Our analysis approach provides a means for examining prior influence and sample size on hierarchical models fit to capture–recapture data while emphasizing transferability of results between empirical and simulation studies. 

   more » « less
5. Theoretical Insights Into Multiclass Classification: A high-dimensional Asymptotic View

   Thrampoulidis, Christos (January 2020, Neural Information Processing Systems (NeuRIPS 2020))

   null (Ed.)

   Contemporary machine learning applications often involve classification tasks with many classes. Despite their extensive use, a precise understanding of the statistical properties and behavior of classification algorithms is still missing, especially in modern regimes where the number of classes is rather large. In this paper, we take a step in this direction by providing the first asymptotically precise analysis of linear multiclass classification. Our theoretical analysis allows us to precisely character- ize how the test error varies over different training algorithms, data distributions, problem dimensions as well as number of classes, inter/intra class correlations and class priors. Specifically, our analysis reveals that the classification accuracy is highly distribution-dependent with different algorithms achieving optimal per- formance for different data distributions and/or training/features sizes. Unlike linear regression/binary classification, the test error in multiclass classification relies on intricate functions of the trained model (e.g., correlation between some of the trained weights) whose asymptotic behavior is difficult to characterize. This challenge is already present in simple classifiers, such as those minimizing a square loss. Our novel theoretical techniques allow us to overcome some of these chal- lenges. The insights gained may pave the way for a precise understanding of other classification algorithms beyond those studied in this paper. 

   more » « less