More Like this

1. Serial order depends on item-dependent and item-independent contexts.

   https://doi.org/10.1037/rev0000422  

   Logan, Gordon D. ; Cox, Gregory E. ( March 2023 , Psychological Review)
2. Flip it: An exploratory (versus explanatory) sequential mixed methods design using Delphi and differential item functioning to evaluate item bias

   https://doi.org/10.1016/j.metip.2023.100117  

   Koskey, Kristin L.K. ; May, Toni A. ; Fan, Yiyun “Kate” ; Bright, Dara ; Stone, Gregory ; Matney, Gabriel ; Bostic, Jonathan D. ( November 2023 , Methods in Psychology)
3. Multi-Item Mechanisms without Item-Independence: Learnability via Robustness

   https://doi.org/10.1145/3391403.3399541  

   Brustle, Johannes ; Cai, Yang ; Daskalakis, Constantinos ( July 2020 , 21st ACM Conference on Economics and Computation)

   We study the sample complexity of learning revenue-optimal multi-item auctions. We obtain the first set of positive results that go beyond the standard but unrealistic setting of item-independence. In particular, we consider settings where bidders' valuations are drawn from correlated distributions that can be captured by Markov Random Fields or Bayesian Networks -- two of the most prominent graphical models. We establish parametrized sample complexity bounds for learning an up-to-ε optimal mechanism in both models, which scale polynomially in the size of the model, i.e. the number of items and bidders, and only exponential in the natural complexity measure of the model, namely either the largest in-degree (for Bayesian Networks) or the size of the largest hyper-edge (for Markov Random Fields). We obtain our learnability results through a novel and modular framework that involves first proving a robustness theorem. We show that, given only "approximate distributions" for bidder valuations, we can learn a mechanism whose revenue is nearly optimal simultaneously for all "true distributions" that are close to the ones we were given in Prokhorov distance. Thus, to learn a good mechanism, it suffices to learn approximate distributions. When item values are independent, learning in Prokhorov distance is immediate, hence our framework directly implies the main result of Gonczarowski and Weinberg. When item values are sampled from more general graphical models, we combine our robustness theorem with novel sample complexity results for learning Markov Random Fields or Bayesian Networks in Prokhorov distance, which may be of independent interest. Finally, in the single-item case, our robustness result can be strengthened to hold under an even weaker distribution distance, the Levy distance. 

   more » « less
4. Mapping Unobserved Item–Respondent Interactions: A Latent Space Item Response Model with Interaction Map

   https://doi.org/10.1007/s11336-021-09762-5  

   Jeon, Minjeong ; Jin, Ick Hoon ; Schweinberger, Michael ; Baugh, Samuel ( January 2021 , Psychometrika)

   null (Ed.)

   Classic item response models assume that all items with the same difficulty have the same response probability among all respondents with the same ability. These assumptions, however, may very well be violated in practice, and it is not straightforward to assess whether these assumptions are violated, because neither the abilities of respondents nor the difficulties of items are observed. An example is an educational assessment where unobserved heterogeneity is present, arising from unobserved variables such as cultural background and upbringing of students, the quality of mentorship and other forms of emotional and professional support received by students, and other unobserved variables that may affect response probabilities. To address such violations of assumptions, we introduce a novel latent space model which assumes that both items and respondents are embedded in an unobserved metric space, with the probability of a correct response decreasing as a function of the distance between the respondent’s and the item’s position in the latent space. The resulting latent space approach provides an interaction map that represents interactions of respondents and items, and helps derive insightful diagnostic information on items as well as respondents. In practice, such interaction maps enable teachers to detect students from underrepresented groups who need more support than other students. We provide empirical evidence to demonstrate the usefulness of the proposed latent space approach, along with simulation results. 

   more » « less
5. Flip it: An exploratory (versus explanatory) sequential mixed methods design using Delphi and differential item functioning to evaluate item bias.

   Koskey, K. ; May, T. ; Fan, Y. ; Bright, D. ; Stone, G. ; Matney, G. ; Bostic, J. ( March 2023 , Methods in psychology)

   The Delphi method has been adapted to inform item refinements in educational and psychological assessment development. An explanatory sequential mixed methods design using Delphi is a common approach to gain experts' insight into why items might have exhibited differential item functioning (DIF) for a sub-group, indicating potential item bias. Use of Delphi before quantitative field testing to screen for potential sources leading to item bias is lacking in the literature. An exploratory sequential design is illustrated as an additional approach using a Delphi technique in Phase I and Rasch DIF analyses in Phase II. We introduce the 2 × 2 Concordance Integration Typology as a systematic way to examine agreement and disagreement across the qualitative and quantitative findings using a concordance joint display table. A worked example from the development of the Problem-Solving Measures Grades 6–8 Computer Adaptive Tests supported using an exploratory sequential design to inform item refinement. The 2 × 2 Concordance Integration Typology (a) crystallized instances where additional refinements were potentially needed and (b) provided for evaluating the distribution of bias across the set of items as a whole. Implications are discussed for advancing data integration techniques and using mixed methods to improve instrument development. 

   more » « less