More Like this

1. Assessing Educational Quality: Comparative Analysis of Crowdsourced, Expert, and AI-Driven Rubric Applications

   https://doi.org/10.1609/hcomp.v12i1.31606  

   Moore, Steven; Bier, Norman; Stamper, John (October 2024, Proceedings of the AAAI Conference on Human Computation and Crowdsourcing)

   Exposing students to low-quality assessments such as multiple-choice questions (MCQs) and short answer questions (SAQs) is detrimental to their learning, making it essential to accurately evaluate these assessments. Existing evaluation methods are often challenging to scale and fail to consider their pedagogical value within course materials. Online crowds offer a scalable and cost-effective source of intelligence, but often lack necessary domain expertise. Advancements in Large Language Models (LLMs) offer automation and scalability, but may also lack precise domain knowledge. To explore these trade-offs, we compare the effectiveness and reliability of crowdsourced and LLM-based methods for assessing the quality of 30 MCQs and SAQs across six educational domains using two standardized evaluation rubrics. We analyzed the performance of 84 crowdworkers from Amazon's Mechanical Turk and Prolific, comparing their quality evaluations to those made by the three LLMs: GPT-4, Gemini 1.5 Pro, and Claude 3 Opus. We found that crowdworkers on Prolific consistently delivered the highest-quality assessments, and GPT-4 emerged as the most effective LLM for this task. Our study reveals that while traditional crowdsourced methods often yield more accurate assessments, LLMs can match this accuracy in specific evaluative criteria. These results provide evidence for a hybrid approach to educational content evaluation, integrating the scalability of AI with the nuanced judgment of humans. We offer feasibility considerations in using AI to supplement human judgment in educational assessment. 

   more » « less
2. 4th Crowd Science Workshop - CANDLE: Collaboration of Humans and Learning Algorithms for Data Labeling

   https://doi.org/10.1145/3539597.3572703  

   Ustalov, Dmitry; Savage, Saiph; van Berkel, Niels; Liu, Yang (February 2023, WSDM: ACM International Web Seach and Data Mining Conference 2023.)

   rowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNET, SuperGLUE, etc. Since the rise of crowdsourcing in early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles. We welcome the studies on developing and enhancing of crowdworker-centric tools, that offer task matching, requester assessment, instruction validation, among other topics. We are also interested in exploring methods that leverage the integration of crowdworkers to improve the recognition and performance of the machine learning models. Thus, we invite studies that focus on shipping active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines. Moreover, we invite works on designing and applying such techniques in various domains, including e-commerce and medicine. 

   more » « less
3. Estimating crowd-worker's reliability with interval-valued labels to improve the quality of crowdsourced work

   https://doi.org/10.1109/SSCI50451.2021.9660043  

   Spurling, Makenzie; Hu, Chenyi; Zhan, Huixin; Sheng, Victor S. (December 2021, 2021 IEEE Symposium Series on Computational Intelligence (SSCI))

   With inputs from human crowds, usually through the Internet, crowdsourcing has become a promising methodology in AI and machine learning for applications that require human knowledge. Researchers have recently proposed interval-valued labels (IVLs), instead of commonly used binary-valued ones, to manage uncertainty in crowdsourcing [19]. However, that work has not yet taken the crowd worker’s reliability into consideration. Crowd workers usually come with various social and economic backgrounds, and have different levels of reliability. To further improve the overall quality of crowdsourcing with IVLs, this work presents practical methods that quantitatively estimate worker’s reliability in terms of his/her correctness, confidence, stability, and predictability from his/her IVLs. With worker’s reliability, this paper proposes two learning schemes: weighted interval majority voting (WIMV) and weighted preferred matching probability (WPMP). Computational experiments on sample datasets demonstrate that both WIMV and WPMP can significantly improve learning results in terms of higher precision, accuracy, and F1-score than other methods. 

   more » « less
4. Can Crowds Customize Instructional Materials with Minimal Expert Guidance?: Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor

   https://doi.org/10.1145/3449193  

   Yang, Kexin_Bella; Nagashima, Tomohiro; Yao, Junhui; Williams, Joseph_Jay; Holstein, Kenneth; Aleven, Vincent (April 2021, Proceedings of the ACM on Human-Computer Interaction)

   AI-based educational technologies may be most welcome in classrooms when they align with teachers' goals, preferences, and instructional practices. Teachers, however, have scarce time to make such customizations themselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelines have traditionally focused on content generation. It is an open question how a pipeline might be designed so the crowd can succeed in a revision/customization task. In this paper, we explore an initial version of a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-based tutoring system so they fit teachers' preferences, while requiring minimal expert guidance. In two experiments involving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipeline could save experts' time and produce better crowd-revised hints (in terms of teacher satisfaction) than two comparison conditions. The revised hints however, did not improve on the existing hints in the AI tutor, which were carefully-written but still have room for improvement and customization. Further analysis revealed that the main challenge for crowdworkers may lie in understanding teachers' brief written comments and implementing them in the form of effective edits, without introducing new problems. We also found that teachers preferred their own revisions over other sources of hints, and exhibited varying preferences for hints. Overall, the results confirm that there is a clear need for customizing hints to individual teachers' preferences. They also highlight the need for more elaborate scaffolds so the crowd can have specific knowledge of the requirements that teachers have for hints. The study represents a first exploration in the literature of how to support crowds with minimal expert guidance in revising and customizing instructional materials. 

   more » « less
5. A Novel Crowd-Resilient Visual Localization Algorithm Via Robust Pca Background Extraction

   https://doi.org/10.1109/ICASSP.2018.8461286  

   Yang, Zhuorui; Duarte, Marco F.; Ganz, Aura (April 2018, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP))

   We present a novel egocentric visual localization algorithm for an indoor navigation system, called PERCEPT-V, which is designed to assist the blind and visually impaired users traveling independently in an unfamiliar indoor space. Through the integration of a background extraction module based on Robust Principle Component Analysis (RPCA) into the localization algorithm, we successfully improve the resilience of camera localization to the presence of crowds in the observed scene. Experiments using datasets of videos containing various levels of crowd activity show that the proposed algorithm can increase prominently the reliability of localization performance. 

   more » « less