More Like this

1. Debiasing Evaluations That Are Biased by Evaluations

   Wang, Jingyan; Stelmakh, Ivan; Wei, Yuting; Shah, Nihar (May 2021, Proceedings of the AAAI Conference on Artificial Intelligence)
2. Debiasing Evaluations That Are Biased by Evaluations

   Wang, Jingyan; Stelmakh, Ivan; Wei, Yuting; Shah, Nihar (February 2024, Journal of machine learning research)

   It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews. However, in these applications, students often give a higher rating to a course if they receive higher grades in a course, and authors often give a higher rating to the reviews if their papers are accepted to the conference. In this work, we call these external factors the" outcome" experienced by people, and consider the problem of mitigating these outcome-induced biases in the given ratings when some information about the outcome is available. We formulate the information about the outcome as a known partial ordering on the bias. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint, and also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization. We provide theoretical guarantees on the performance of our algorithm, as well as experimental evaluations. 

   more » « less
3. Debiasing Evaluations that are Biased by Evaluations

   Jingyan Wang, Ivan Stelmakh (January 2021, AAAI)
4. Coherence shifts in attribute evaluations.

   https://doi.org/10.1037/dec0000151  

   Lee, Douglas G.; Holyoak, Keith J. (October 2021, Decision)
5. Students’ Scientific Evaluations of Astronomical Origins

   https://doi.org/10.32374/AEJ.2022.2.1.032ra  

   Dobaria, Archana; Bailey, Janelle M.; Klavon, Timothy G.; Lombardi, Doug (September 2022, Astronomy Education Journal)

   Students often encounter alternative explanations about astronomical phenomena. However, inconsistent with astronomers’ practices, students may not be scientific, critical, and evaluative when comparing alternatives. Instructional scaffolds, such as the Model-Evidence Link (MEL) diagram, where students weigh connections between lines of evidence and alternative explanations, may help facilitate students’ scientific evaluation and deepen their learning about astronomy. Our research team has developed two forms of the MEL: (a) the preconstructed MEL (pcMEL), where students are given four lines of evidence and two alternative explanatory models about the formation of Earth’s Moon and (b) the build-a-MEL (baMEL), where students construct their own diagrams by choosing four lines scientific evidence out of eight choices and two alternative explanatory model out of three choices, about the origins of the Universe. The present study compared the more autonomy-supportive baMEL to the less autonomy-supportive pcMEL and found that both scaffolds shifted high school student and preservice teacher participants’ plausibility judgments toward a more scientific stance and increased their knowledge about the topics. Additional analyses revealed that the baMEL resulted in deeper evaluations and had stronger relations between levels of evaluation and post-instructional plausibility judgements and knowledge compared to the pcMEL. This present study, focused on astronomical topics, supports our team’s earlier research that scaffolds such as the MELs in combination with more autonomy-supportive classrooms may be one way to deepen students’ scientific thinking and increase their knowledge of complex scientific phenomena. 

   more » « less