More Like this

1. La Independiente: Designing Ubiquitous Systems for Latin American and Caribbean Women Crowdworkers

   https://doi.org/10.1145/3594739.3610728  

   De Los Santos, Maya; Chávez, Norma Elva; Navarrete, Alberto; Martínez Pinto, Cristina; González, Luz Elena; Telles-Calderon, Tatiana; Savage, Saiph (October 2023, ACM Ubicomp for All Symposium)

   Since 2018, Venezuelans have contributed to 75% of leading AI crowd work platforms’ total workforce, and it is very likely other Latin American and Caribbean (LAC) countries will follow in the context of the post covid-19 economic recovery. While crowd work presents new opportunities for employment in regions of the world where local economies have stagnated, few initiatives have investigated the impact of such work in the Global South through the lens of feminist theory. To address this knowledge gap, we surveyed 55 LAC women on the crowd work platform Toloka to understand their personal goals, professional values, and hardships faced in their work. Our results revealed that most participants shared a desire to hear the experiences of other women crowdworkers, mainly to help them navigate tasks, develop technical and soft skills, and manage their finances more efficiently. Additionally, 75% of the women reported that they completed crowd work tasks on top of caring for their families, while over 50% confirmed they needed to negotiate their family responsibilities to pursue crowd work in the first place. These findings demonstrated a vital component lacking from the experiences of these women was a sense of connection with one another. Based on these observations, we propose a system designed to foster community between LAC women in crowd work to improve their personal and professional advancement. 

   more » « less
2. The Challenges of Crowd Workers in Rural and Urban America

   Saviaga, C; Li, Y; Hanrahan, B; Bigham, J; Savage, S. (January 2020, Human computation)

   null (Ed.)

   Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic opportunities, e.g., rural areas. However, we currently have limited information about the challenges facing crowd workers from rural and super rural areas as they struggle to make a living through crowd work sites. This paper examines the challenges and advantages of rural and super rural Amazon Mechanical Turk (MTurk) crowd workers and contrasts them with those of workers from urban areas. Based on a survey of 421 crowd workers from differing geographic regions in the U.S., we identified how across regions, people struggled with being onboarded into crowd work. We uncovered that despite the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We also identified cultural traits, relating to time dimension and individualism, that offer us an insight into crowd workers and the necessary qualities for them to succeed on gig platforms. We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools. 

   more » « less
3. The Challenges of Crowd Workers in Rural and Urban America

   Saviaga, C; Li, Y; Hanrahan, B; Bigham, J; Savage, S. (January 2020, Human computation)

   null (Ed.)

   Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic opportunities, e.g., rural areas. However, we currently have limited information about the challenges facing crowd workers from rural and super rural areas as they struggle to make a living through crowd work sites. This paper examines the challenges and advantages of rural and super rural Amazon Mechanical Turk (MTurk) crowd workers and contrasts them with those of workers from urban areas. Based on a survey of 421 crowd workers from differing geographic regions in the U.S., we identified how across regions, people struggled with being onboarded into crowd work. We uncovered that despite the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We also identified cultural traits, relating to time dimension and individualism, that offer us an insight into crowd workers and the necessary qualities for them to succeed on gig platforms. We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools. 

   more » « less
4. Comparing Experts and Novices for AI Data Work: Insights on Allocating Human Intelligence to Design a Conversational Agent

   https://doi.org/10.1609/hcomp.v10i1.21999  

   Sun, Lu; Liu, Yuhan; Joseph, Grace; Yu, Zhou; Zhu, Haiyi; Dow, Steven P. (October 2022, Proceedings of the AAAI Conference on Human Computation and Crowdsourcing)

   Many AI system designers grapple with how best to collect human input for different types of training data. Online crowds provide a cheap on-demand source of intelligence, but they often lack the expertise required in many domains. Experts offer tacit knowledge and more nuanced input, but they are harder to recruit. To explore this trade off, we compared novices and experts in terms of performance and perceptions on human intelligence tasks in the context of designing a text-based conversational agent. We developed a preliminary chatbot that simulates conversations with someone seeking mental health advice to help educate volunteer listeners at 7cups.com. We then recruited experienced listeners (domain experts) and MTurk novice workers (crowd workers) to conduct tasks to improve the chatbot with different levels of complexity. Novice crowds perform comparably to experts on tasks that only require natural language understanding, such as correcting how the system classifies a user statement. For more generative tasks, like creating new lines of chatbot dialogue, the experts demonstrated higher quality, novelty, and emotion. We also uncovered a motivational gap: crowd workers enjoyed the interactive tasks, while experts found the work to be tedious and repetitive. We offer design considerations for allocating crowd workers and experts on input tasks for AI systems, and for better motivating experts to participate in low-level data work for AI. 

   more » « less
5. 4th Crowd Science Workshop - CANDLE: Collaboration of Humans and Learning Algorithms for Data Labeling

   https://doi.org/10.1145/3539597.3572703  

   Ustalov, Dmitry; Savage, Saiph; van Berkel, Niels; Liu, Yang (February 2023, WSDM: ACM International Web Seach and Data Mining Conference 2023.)

   rowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNET, SuperGLUE, etc. Since the rise of crowdsourcing in early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles. We welcome the studies on developing and enhancing of crowdworker-centric tools, that offer task matching, requester assessment, instruction validation, among other topics. We are also interested in exploring methods that leverage the integration of crowdworkers to improve the recognition and performance of the machine learning models. Thus, we invite studies that focus on shipping active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines. Moreover, we invite works on designing and applying such techniques in various domains, including e-commerce and medicine. 

   more » « less