More Like this

1. DIF Statistical Inference Without Knowing Anchoring Items

   https://doi.org/10.1007/s11336-023-09930-9  

   Chen, Yunxiao; Li, Chengcheng; Ouyang, Jing; Xu, Gongjun (December 2023, Psychometrika)

   Abstract Establishing the invariance property of an instrument (e.g., a questionnaire or test) is a key step for establishing its measurement validity. Measurement invariance is typically assessed by differential item functioning (DIF) analysis, i.e., detecting DIF items whose response distribution depends not only on the latent trait measured by the instrument but also on the group membership. DIF analysis is confounded by the group difference in the latent trait distributions. Many DIF analyses require knowing several anchor items that are DIF-free in order to draw inferences on whether each of the rest is a DIF item, where the anchor items are used to identify the latent trait distributions. When no prior information on anchor items is available, or some anchor items are misspecified, item purification methods and regularized estimation methods can be used. The former iteratively purifies the anchor set by a stepwise model selection procedure, and the latter selects the DIF-free items by a LASSO-type regularization approach. Unfortunately, unlike the methods based on a correctly specified anchor set, these methods are not guaranteed to provide valid statistical inference (e.g., confidence intervals andp-values). In this paper, we propose a new method for DIF analysis under a multiple indicators and multiple causes (MIMIC) model for DIF. This method adopts a minimal$$L_1$$ $L_1$norm condition for identifying the latent trait distributions. Without requiring prior knowledge about an anchor set, it can accurately estimate the DIF effects of individual items and further draw valid statistical inferences for quantifying the uncertainty. Specifically, the inference results allow us to control the type-I error for DIF detection, which may not be possible with item purification and regularized estimation methods. We conduct simulation studies to evaluate the performance of the proposed method and compare it with the anchor-set-based likelihood ratio test approach and the LASSO approach. The proposed method is applied to analysing the three personality scales of the Eysenck personality questionnaire-revised (EPQ-R). 

   more » « less
2. Item Response Theory – A Statistical Framework for Educational and Psychological Measurement

   Yunxiao Chen, Xiaoou Li (March 2023, Statistical science)

   Item response theory (IRT) has become one of the most popular statistical models for psychometrics, a field of study concerned with the theory and techniques of psychological measurement. The IRT models are latent factor models tailored to the analysis, interpretation, and prediction of individuals’ behaviors in answering a set of measurement items that typically involve categorical response data. Many important questions of measurement are directly or indirectly answered through the use of IRT models, including scoring individuals’ test performances, validating a test scale, linking two tests, among others. This paper provides a review of item response theory, including its statistical framework and psychometric applications. We establish connections between item response theory and related topics in statistics, including empirical Bayes, nonparametric methods, matrix completion, regularized estimation, and sequential analysis. Possible future directions of IRT are discussed from the perspective of statistical learning. 

   more » « less
3. Adaptation of the Delphi technique in the development of assessments of problem-solving in computer adaptive testing environments (DEAP-CAT).

   https://doi.org/10.21125/iceri.2021.2142  

   Koskey, K. L.; Bright, D.; Struloeff, K.; Sondergeld, T.; Stone, G.; Bostic, J.; Matney, G. (November 2021, Proceedings of the International Conference of Education, Research and Innovation)

   The Standards for educational and psychological assessment were developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (AERA et al., 2014). The Standards specify assessment developers establish five types of validity evidence: test content, response processes, internal structure, relationship to other variables, and consequential/bias. Relevant to this proposal is consequential validity evidence that identifies the potential negative impact of testing or bias. Standard 3.1 of The Standards (2014) on fairness in testing states that “those responsible for test development, revision, and administration should design all steps of the testing process to promote valid score interpretations for intended score uses for the widest possible range of individuals and relevant sub-groups in the intended populations” (p. 63). Three types of bias include construct, method, and item bias (Boer et al., 2018). Testing for differential item functioning (DIF) is a standard analysis adopted to detect item bias against a subgroup (Boer et al., 2018). Example subgroups include gender, race/ethnic group, socioeconomic status, native language, or disability. DIF is when “equally able test takers differ in their probabilities answering a test item correctly as a function of group membership” (AERA et al., 2005, p. 51). DIF indicates systematic error as compared to real mean group differences (Camilli & Shepard, 1994). Items exhibiting significant DIF are removed or reviewed for sources leading to bias to determine modifications to retain and further test an item. The Delphi technique is an emergent systematic research method whereby expert panel members review item content through an iterative process (Yildirim & Büyüköztürk, 2018). Experts independently evaluate each item for potential sources leading to DIF, researchers group their responses, and experts then independently complete a survey to rate their level of agreement with the anonymously grouped responses. This process continues until saturation and consensus are reached among experts as established through some criterion (e.g., median agreement rating, item quartile range, and percent agreement). The technique allows researchers to “identify, learn, and share the ideas of experts by searching for agreement among experts” (Yildirim & Büyüköztürk, 2018, p. 451). Research has illustrated this technique applied after DIF is detected, but not before administering items in the field. The current research is a methodological illustration of the Delphi technique applied in the item construction phase of assessment development as part of a five-year study to develop and test new problem-solving measures (PSM; Bostic et al., 2015, 2017) for U.S.A. grades 6-8 in a computer adaptive testing environment. As part of an iterative design-science-based methodology (Middleton et al., 2008), we illustrate the integration of the Delphi technique into the item writing process. Results from two three-person panels each reviewing a set of 45 PSM items are utilized to illustrate the technique. Advantages and limitations identified through a survey by participating experts and researchers are outlined to advance the method. 

   more » « less
4. Refinement of an instrument measuring science teachers’ knowledge of language through mixed method

   https://doi.org/10.1186/s43031-023-00080-7  

   Ding, Chenchen; Lammert, Catherine; Fulmer, Gavin_W; Hand, Brian; Suh, Jee_K (August 2023, Disciplinary and Interdisciplinary Science Education Research)

   Abstract Teachers must know how to use language to support students in knowledge generation environments that align to the Next Generation Science Standards. To measure this knowledge, this study refines a survey on teachers’ knowledge of language as an epistemic tool. Rasch modelling was used to examine 15 items’ fit statistics and the functioning of a previously-designed questionnaire’s response categories. Cronbach’s alpha reliability was also examined. Additionally, interviews were used to investigate teachers’ interpretations of each item to identify ambiguous items. The results indicated that three ambiguous items were deleted based on qualitative data and three more items were deleted because of negative correlation and mismatched fit statistics. Finally, we present a revised language questionnaire with nine items and acceptable correlation and good fit statistics, with utility for science education researchers and teacher educators. This research contributes a revised questionnaire to measure teachers’ knowledge of language that could inform professional development efforts. This research also describes instrument refinement processes that could be applied elsewhere. 

   more » « less
5. Peer preferences and characteristics of same-group and cross-group social interactions among autistic and non-autistic adolescents

   https://doi.org/10.1177/13623613211005918  

   Chen, Yu-Lun; Senande, Laura L; Thorsen, Michael; Patten, Kristie (October 2021, Autism)

   Social interaction is inherently bidirectional, but research on autistic peer interactions often frames communication as unidirectional and in isolation from the peer context. This study investigated natural peer interactions among six autistic and six non-autistic adolescents in an inclusive school club over 5 months (14 45-min sessions in total) to examine the students’ peer preferences in real-world social interactions and how the preferences changed over time. We further examined whether social behavior characteristics differ between student and peer neurotype combinations. Findings showed that autistic students were more likely to interact with autistic peers then non-autistic peers. In both autistic and non-autistic students, the likelihood of interacting with a same-neurotype peer increased over time. Autistic and non-autistic students’ within-neurotype social interactions were more likely to reflect relational than functional purposes, be characterized as sharing thoughts and experiences rather than requesting help or objects, and be highly reciprocal, as compared with cross-neurotype interactions. These peer preferences and patterns of social interactions were not found among student-peer dyads with the same genders. These findings suggest that peer interaction is determined by more than just a student’s autism diagnosis, but by a combination of student and peer neurotypes. Lay abstract Autistic students often experience challenges in peer interactions, especially for young adolescents who are navigating the increased social expectations in secondary education. Previous research on the peer interactions of autistic adolescents mainly compared the social behaviors of autistic and non-autistic students and overlooked the peers in the social context. However, recent research has shown that the social challenges faced by autistic may not be solely contributed by their social differences, but a mismatch in the social communication styles between autistic and non-autistic people. As such, this study aimed to investigate the student-and-peer match in real-world peer interactions between six autistic and six non-autistic adolescents in an inclusive school club. We examined the odds of autistic and non-autistic students interacting with either an autistic peer, a non-autistic peer, or multiple peers, and the results showed that autistic students were more likely to interact with autistic peers then non-autistic peers. This preference for same-group peer interactions strengthened over the 5-month school club in both autistic and non-autistic students. We further found that same-group peer interactions, in both autistic and non-autistic students, were more likely to convey a social interest rather than a functional purpose or need, be sharing thoughts, experiences, or items rather than requesting help or objects, and be highly reciprocal than cross-group social behaviors. Collectively, our findings support that peer interaction outcomes may be determined by the match between the group memberships of the student and their peers, either autistic or non-autistic, rather than the student’s autism diagnosis. 

   more » « less