Crowdsourcing markets, and their APIs, can help absorb some of the costs associated with crowd work [51,82]. From the requesters' perspective, these platforms provide an alwaysavailable pool of workers and an easy-to-use payment API to contract workers and start getting work done [5,63,75]. From the workers' perspective, these markets provide a central place to find work and offer them the flexibility of working from wherever they desire [1,87].
However, recent research has identified that some of these costs do not actually get absorbed by the crowdsourcing platform, but rather, they are passed onto the workers in the form of invisible labor [15,51]. Invisible labor is defined as "unpaid activities that occur within the context of paid employment that workers perform in response to requirements (either implicit or explicit) from employers and that are crucial for workers to generate income, to obtain or retain their jobs, and to further their careers, yet are often overlooked, ignored, and/or devalued by employers, consumers, workers, and ultimately the legal system itself [15]."
Invisible labor is also present in crowd work and it includes activities such as: the unpaid time workers have to invest in finding work, figuring out on their own how to complete the job at hand, or managing their payments [51,67]. The problem is that crowd workers are forced to engage in these unpaid activities just to be able to complete the labor for which they are paid [91,107]. If we are aiming to create a future where crowd work is fair and equitable to workers, we need to ensure that workers receive a fair wage for all of the labor they do, whether it is the actual tasks for which they get paid, or the unpaid invisible work they do above and beyond that work.
The central question this work addresses is how much time do workers actually spend on invisible work, and how does this affect their overall hourly wages? This is an important question not only to ensure that workers receive a fair wage now but also to ensure that workers receive a fair wage in the future. Notice that our research is addressing a critical problem because a common use case for crowd work is to train machine learning algorithms, or to provide a human-in-the-loop approach when A.I. fails [13,44,100,106] . Since we are in the midst of an "A.I. revolution," it is plausible that we will see dramatic growth in the use of crowd labor [10,45,64,65,99]. In addition, post-COVID-19, there will likely be a large increase in people who need to work from home, whether that is for safety reasons or because of the massive number of worldwide layoffs [29,31,104].
Measuring invisible labor in crowd work will only grow in importance going forward.
To start to quantify the invisible labor in crowd work, we develop a web plugin 1 that allows us to detect when a worker is performing invisible labor and quantitatively measure the amount of time the worker spends on such efforts on Amazon Mechanical Turk (MTurk), one of the most popular crowdsourcing markets [73]. We conduct a field study with our plugin to measure in the wild the amount of time crowd workers invest in invisible labor, and contrast with the amount of time workers spent on paid labor. Armed with our plugin, we had 100 crowd workers use our plugin for one week. Workers completed 40,903 human intelligence tasks (HITs). Through our plugin, we uncovered that crowd workers spent 33% of their time on MTurk doing unpaid work. Relatively similar to prior work, we found that workers' median hourly wage considering only paid labor was $3.76 [57]. But, if we consider the time workers spent on invisible labor, we calculated that workers' median hourly wage dropped to $2.83. We also found that the amount of time that workers dedicated to invisible labor varied across workers' skill level and demographics. We found that master workers spent 23% less time on invisible work than regular workers. We also observed that the time spent in invisible work appears to be heavily correlated with demographic factors.
The invisible labor in which crowd workers spent the greatest portion of their time revolved around payments. In particular, this most time consuming activity involved doing tasks for which workers were not paid because they experienced a "time out" (and hence they did not receive any payment for any of the labor they conducted for the task). Workers spent a median of 4.5 minutes daily on this activity. Overall, invisible labor around "payments" was the most time-consuming for workers; it was also among the most common. In fact, 97% of the workers in our study practiced invisible labor around visiting the earnings section on their workers' dashboard (perhaps to ensure they had gotten paid fairly [112]). The second most time-consuming category of invisible labor involved hyper-vigilance where workers were "on-call" vigilantly watching over requesters' profiles ready to do, at all hours of the day, the labor that certain requesters posted, as well as vigilantly searching for work on Amazon Mechanical Turk [24,51,114]. Understanding invisible labor is key to creating positive change in crowd work [15]; however, it has remained so far understudied. Bettering our understanding of invisible labor will allow us to design fairer crowdsourcing markets.
Our research builds on two main pieces of literature: (1) research on invisible labor, especially within digital labor markets [15]; and (2) methodologies to quantitatively measure the time crowd workers spend completing paid labor on MTurk [96,98], i.e., HITs.
The literature has traditionally characterized invisible labor as work that is "economically devalued through cultural, legal, and/or spatial dynamics" [59]. Under this definition, invisible labor is usually done in private rather than public [18,50]. Usually, housework is one of the most commonly cited examples of invisible labor [14,30], and it involves both psychical labor and mental labor (e.g., planning what types of household chores should be done and in what manner.)
In their book Invisible Labor [15], Crain et al. describe the concept of invisible work as the "activities that occur within the context of paid employment that workers perform in response to requirements (either implicit or explicit) from employers." They explain how this concept has existed in different offline settings before, but nowadays, technology has enabled a large part of all invisible labor. In particular, a number of technology companies are passing several aspects of digitization labor to consumers and workers, e.g., consumers are expected to install all the required Internet infrastructure at their homes. This labor is typically presented as something that is mundane, flexible, and part of the "do-it-yourself" culture [48]. However, this dynamic also reduces the meaning of this type of labor, making it invisible, and something for which people are not paid.
Gray and Suri [51] explored these concepts of invisible labor with a particular angle toward crowd workers. Through in-depth interviews with crowd workers, their book "Ghost Work" unveils the current conditions to which crowd workers are exposed and explains how companies have placed on the shoulders of workers a great portion of the invisible labor that companies themselves would traditionally do. The book also went a step further and started to describe the different types of invisible labor present in crowd work. Within this setting of describing invisible labor in crowd work, it is important to consider that crowd work does not emerge only from the requesters' side of the market; it is also something that crowdsourcing platforms, such as Amazon Mechanical Turk facilitate [51] and could, with different design choices, help alleviate. For instance, crowdsourcing platforms could help match workers to tasks to reduce search time. Also, they could potentially pay workers for the time they spend searching for tasks or reading messages from requesters, which is something that companies have traditionally covered [20,51].
Furthermore, when thinking about invisible labor in crowd work, we have to consider that much of the labor that crowd workers complete is fed into machine learning models that power the A.I. industry [72,106]. For instance, crowd workers might label content so that Facebook's News Feed algorithm will not recommend posts that are filled with hate speech or pedophilia [17,46]. Crowd workers might also transcribe audio to help Amazon's Alexa better understand the user [6]. Given that most end-users are unaware that there are humans helping to power the A.I. services they access [51], the work done by workers and their possible unfair labor conditions, are hidden from sight. Notice that here the invisibility of crowd work is again not just due to requesters and their HIT design choices, but rather it is an issue within the A.I. industry as a whole. In this particular research, we focus on measuring the different categories of invisible work that the book of Ghost Work identified that exist within the context of crowd work [51]. We believe that by quantifying the different costs that invisible work has in this setting, we can design better solutions to improve crowd workers' conditions. Notice that invisible labor in crowd work includes activities that go unnoticed while doing paid work, such as finding HITs and communicating with requesters to resolve conflicts [42,55,57,97]. Invisible labor in crowd work has recently gained more attention because it has become clearer that the independent nature of crowd work has led workers to now have to assume invisible labor that was traditionally taken by companies and employers [20,95]. In this work, we present computational mechanisms for quantifying for the first time the invisible labor that exists in crowd work and bring much needed light to a critical topic.
Saito et al. [96] studied different ways to measure the time crowd workers spent completing HITs using their system called TurkScanner. They found that through web plugins, they could quantify how much time workers spent completing HITs. We built upon their methodology to properly measure the working time on HITs and expand their method to now also measure time spent in unpaid work. Hara et al. [57] also used plugins to measure the wages of MTurk workers. Ignoring unpaid work, they estimated an average hourly wage of $3.18, which roughly coincides with the average hourly estimate of $3.76 that our study uncovered when we also ignore unpaid work. This shows that our measurement instrument is roughly calibrated to theirs (we likely had higher hourly wages because we considered more conservative measurements). We build on their work and provide a more detailed accounting and classification of the unpaid work that workers endure.
The goal of our IRB-approved field study is to measure and contrast the time that crowd workers spent on invisible labor and the time they spent on regular paid labor (i.e., completing HITs.) Since this data is not part of the official MTurk API, and prior work has not been able to measure invisible labor at the level of detail in which we were interested, we build computational mechanisms to measure these variables. Armed with these computational mechanisms, we conduct a field study to investigate in the wild how much time workers on MTurk dedicate to invisible labor. In the following, we describe how we measured these activities through the computational mechanisms that we designed and detail how we conducted our field study. It is important to highlight that our computational methods for measuring invisible labor focus on measuring invisible labor in a conservative manner. We consider it is best to err in underestimating the amount of time that workers spend in invisible labor than to overestimate. We make this design decision because quantifying invisible labor can potentially call attention to the structural issues surrounding crowdsourcing markets and the conditions they provide workers. Operating in a conservative manner helps us to avoid being labeled as "exaggerated activists" and allows us to present the study in a scientific, objective way. This approach helps us to bring much-needed attention to understanding invisible labor in crowd work. As we will see, even with erring on the side of underestimating invisible labor, it is still a sizeable overhead for the workers.
3.1 Computational Mechanisms to Measure Invisible & Paid Labor For our study, we need computational mechanisms for: (1) detecting when a worker is doing invisible labor or when she is doing paid work; and (2) measuring how much time a worker invests in each of these two activities. To address these two points, we created a Chrome Extension (plugin).
3.1.1 Methods for Quantifying Paid Labor. Our plugin builds on prior research that was able to detect and measure with plugins when a crowd worker was completing a HIT, the amount of time the worker invested in completing the HIT, and the daily earnings that workers made from the HITs (notice that this value is important as it can help us to quantify the monetary costs of invisible labor) [96]. In particular, building on prior work, we developed a plugin that can: 1) automatically record the exact times when a worker accepts a HIT and when she finishes and submits the HIT; 2) track when a tab about a HIT is in focus and automatically record the time period in which the worker is active on the HIT page tab by checking whether there were any type of interactions from the worker (e.g., mouse movements, typing) under a given time window; and also 3) measure the daily income that each worker makes from these HITs by querying the information from their workers' dashboard on MTurk. In summary, as a starting point, we developed our own plugin that mimics prior work and quantifies the amount of time that a given worker dedicates to completing HITs and the earnings that the worker is making.
Labor. Next, we expand the plugin to now provide new functionality through which we can also track and measure the time workers spend on invisible labor. Notice that we focus on quantifying invisible labor in a conservative manner, which means that we prefer to err on the side of under measuring the invisible labor (we took this methodological decision based on the reasoning stated above). Our conservative approach to the measurement of invisible labor comes in especially when we consider cases where there is disagreement in the literature on whether an activity is invisible work or not [47,94]. In such cases, we prefer not to label the activity as invisible labor. We prefer to underreport so that the invisible labor we measure will be at least as large as we quantify here, if not larger. Some of the discussions around what is and what is not invisible labor especially arise for the activities of "reading instructions of the HITs", and "taking breaks" [25,36,43,47,51,76]. Gray and Suri [51] label "reading instructions" and "taking breaks" as examples of invisible labor activities. However, we decided not to categorize these activities as invisible labor because there is research that considers these two activities as part of paid work [25,36,43,47,76]. Now, given that workers are not actually paid for either of these two activities, we designed our computational methods to detect when workers take breaks or read instructions; but, we do not count these activities as either paid nor invisible labor. It is important to highlight that because workers are not paid any wages for reading instructions, it is incorrect to categorize the work as being paid.
Our plugin, therefore, in addition to what prior work had already developed, provides now the novelty of being able to detect and quantify all other activities that workers do aside from completing HITs. For this purpose, we developed new computational mechanisms to detect when a worker is visiting other parts of the MTurk platform that are different from the HIT page tabfoot_3 (e.g., perhaps the worker entered the MTurk page to search for HITs 3 or the worker entered the MTurk page for sending messages to requesters 4 ). Our plugin tracks the exact time when a worker enters one of these other MTurk domain pages and then scrapes and parses the HTML of the page to understand how the worker interacted with the page and identifies the intervals of time in which the worker is active on these other pages. We consider a worker to be active on a page when the worker has the page in focus and does any type of user interaction on that page, e.g., mouse movements, scrolls, clicks, keyboard typing. Notice that we do not track what a worker does on these pages (e.g., we do not track what they type). We simply detect that they are active on a particular MTurk page. To accomplish all of this, we developed two new components into our plugin: a page crawler and a time-driven background process that detects the different browser events that happen on MTurk (e.g., that the worker visited another page on MTurk, or that she started typing, or began a new HIT). The page crawler detects the current MTurk domain page that the worker is on, as well as the status of the page (e.g., that the page is loaded, active, inactive, or closed). The background process focuses on detecting the HITs that the worker is currently doing and identifying which she has finished. In order to accomplish this, the background process polls workers' task queues on MTurk every 30 seconds. From the task queue, the background process obtains the metadata and status of all the HITs the worker has accepted to do. Notice that the page crawler is the primary element that we use to detect whether the worker is completing paid labor or invisible labor. The background process helps our plugin to be able to better detect when the worker is completing HITs (some of them reside outside the MTurk platform) and also when the worker is multi-tasking (doing multiple HITs at the same time.) Through this, we create a plugin that automatically detects when a worker is doing invisible or paid labor and the amount of time the worker invested in each of these two activities. Our plugin is available here: https://github.com/anonym-research/invisible-labor.
Labor. We were not only interested in detecting whether or not a worker was doing invisible work; we also wanted to understand what type of invisible labor was the most taxing and contextualize our results with prior interview work that started to document the invisible labor that workers perceived by conducting interview studies with them [51]. In the following, we present the different types of invisible labor we consider (i.e., broad categories) and how we detected their related individual activities. The categories and activities we study are based on prior interview research that studied invisible labor [51,113]. Note that for most cases, we detect that a worker started a new activity when they loaded, focused, or changed their browser tab to a page on MTurk related to that particular type of invisible labor (below, we mention which pages relate to specific invisible labor activities). Similarly, our plugin considers that a worker paused or finished an activity when the worker changed to another tab, unloaded, blurred, or closed the MTurk page related to that particular activity. The categories and activities we consider are:
(a) Category: Hypervigilance. This category involves workers spending time in: (1) identifying good work, e.g., "wading and sorting through spam or suspicious offers for athome-work." [22]; and (2) being "on-call," ready to do HITs for requesters at any time. Invisible activities include:
• Watching over requesters' profile: Notice that this activity relates to Hypervigilance because workers are visiting requesters' profiles to be ready to do any HIT that requesters post. In other words, workers are "on-call." To detect this activity, our page crawler detects when a worker is on a requester's profile page.
• Searching for general HITs (unfiltered): To detect this activity, our page crawler identifies that a worker is on the main page where HITs are posted. • Searching for filtered HITs: Our page crawler detects when the search URL for the main page of HITs has a query in it to filter HIT results. This activity relates to hypervigilance as it involves "wading and sorting" through HITs.
• Managing their queued HITs: this activity relates to Hypervigilance as it involves workers filtering out fraudulent HITs and focusing on HITs from specific requesters (i.e., being "on call"). To detect this, our crawler identifies when a worker is visiting her tasks queue.
• Checking their own qualifications: This activity relates to Hypervigilance as prior work has identified that workers watch over their own qualifications to vigilantly identify whether they could now access certain HITs and thus more effectively find and access quality labor [51]. In this case, our crawler detects when the worker is viewing her earned qualifications.
(b) Category: Lack of Guidance. Crowd workers are generally left on their own to figure out how to do jobs as fast and accurately as possible [86]. Activities related involve: • Starting HITs but then returning them: This activity relates to "Lack of Guidance" as it usually occurs because workers believe the HITs will be different than what they actually end up being [49,88] (e.g., less complex or of another nature.) The lack of guidance leads workers to have to return HITs they already started. In this case, our crawler detects when workers click the return HIT button on MTurk.
• Sending messages: Workers send messages to requesters to ask them questions about a HIT and better understand what the requester wants. To detect this type of invisible labor, our crawler detects when a worker opens MTurk's messaging form to send a message. • Reading HIT information: Page crawler detects when a worker clicks the "More Info" option while previewing or working on a HIT. Notice that this activity is different from reading HIT instructions, as reading HIT information helps workers get a preview of what a HIT is about. It is an activity that workers have to do in order to obtain guidance. • Previewing HITs: Page crawler detects when the page of a HIT is open in preview mode.
Notice that here we could potentially say that workers are previewing HITs in order to "vigilantly" find tasks from certain requesters (and hence this activity could be labeled as being from the category of Hypervigilance). However, the search filtering option allows workers to do that more effectively, and that is also not the main purpose of the preview [70]. We, therefore, decided to categorize this activity as Lack of Guidance.
Additionally, prior work has labeled this activity as related to guidance [83,116]. • Reading platform help: Page crawler detects when workers are in MTurk support sections. Notice that within this category, we could have considered the activity of reading instructions as part of the invisible labor that a worker has to do related to the lack of guidance. However, as mentioned before, we opted to just detect the activity but not label it as invisible nor paid labor. To detect the activity of "reading instructions," the page crawler detects the time that passes from when a worker accepted a HIT until the worker has her first interaction with the HIT (e.g., she presses a key, or she opens another tab related to the HIT, etc.) We assume that this time-lapse corresponds to when the worker is reading instructions.
(c) Category: Payments. In crowd work, even after workers have vigilantly identified legitimate labor and they have been able to figure out how to complete the work, they still run the risk that they will not get paid for their efforts. The broad category of "Payments" encompasses the invisible labor that workers do to ensure payment and also instances where they worked on HITs but were not paid. This category of invisible labor includes:
• Visiting their worker's dashboard: workers visit their dashboard to oversee if requesters have paid them and ensure they made a certain amount of daily income. To detect this activity, our crawler identifies when workers are visiting their general MTurk dashboard. • Doing HITs that eventually timeout: Some HITs have an expiration time on them. If workers take longer to complete the HIT than the allowed expiration time, the HIT times out. In these cases, workers are not paid for any of the labor they have done on the HIT, and thus we consider this activity within the broader category of Payments. To detect these instances, the background process of our plugin identifies when a HIT has an end time equal to or higher to the HIT expiration time. Our plugin also checks in the worker's dashboard that the worker was never paid for those HITs.
Multi-Tasking. When measuring the time workers spent in complet ing HITs, it is important to properly detect when workers are multi-tasking and properly measure and account for the time they spent doing so [74]. In our study, we refer to multi-tasking as when a worker accepts multiple HITs or batches of HITs around the same time and then starts completing these multiple HITs. The background process of our plugin checks the workers' tasks queue to detect workers completing HITs via multi-tasking. To account for this time, we adopt an approach similar to prior work [96]. A common feature of working in this manner is that the HITs are chained in succession. This means that the start and end times may overlap with one or more HITs in the batch. Also, similar to prior work [ 96], our study does not consider batch HITs that take more than one day to be completed (0.6% of our sample). We filtered out all the multi-day batches and HITs since these imply computing the effective working schedule of each worker.
The purpose of our field study was to have workers use our plugin and through it measure in the wild the amount of time workers dedicate daily to invisible labor. Armed with our plugin that could detect and measure the amount of time workers dedicated to different types of invisible labor, as well as the time they dedicated to paid labor, we conducted a field study that lasted a week. Note that we included weekends in our analysis as MTurk presents itself as a platform that offers workers the flexibility to work whenever workers want (weekends included). Similar to prior work [58,98], we did not see changes in the days workers completed tasks.
We recruited workers from MTurk by posting a HIT inviting workers to our study. We also used mailing lists of Turkers (workers on MTurk) who had participated previously in studies with us. For our study, we first surveyed participants on their perceptions of how much time they estimated that they spent on invisible labor. We asked workers to report how much time they felt they invested on MTurk: searching for work; looking over their worker dashboard; sending messages to requesters; and doing HITs that eventually timeout. This helped us understand workers' prior beliefs and awareness of invisible labor and how much time they believed they spent on it. We also asked workers about how COVID-19 had affected them (none of our participants expressed any work disruptions). Our initial survey also asked workers about their basic demographic information, such as current location, gender, disabilities, etc. Overall, we based our survey on prior work [37,58].
After the initial survey, we asked participants to: (1) install and use our plugin for a week; (2) work on MTurk as normal; (3) visit the plugin dashboard, which showed to each worker graphs of how much time the plugin detected that they invested in different MTurk activities for a given day. At the end of the field study, workers completed a short survey evaluating the accuracy of the plugin in detecting and measuring the amount of time they spent on different activities on MTurk. In general, workers in our study stated that they felt that our plugin was able to adequately track the time they spent daily on MTurk completing HITs and doing different invisible labor activities (the median score for the plugin's accuracy was 4 on a 5 point Likert scale). We paid each participant $10 USD for taking part in our study. Notice that this accounts for the US federal minimum wage ($7.25/hour) as our initial survey took 5-8 minutes to complete, the installation of our plugin took less than 4 minutes, and the end survey we gave participants took 5-8 minutes.
We had 100 MTurk workers install and use our plugin for a week. We allowed all types of workers to participate in our study. This resulted in us recruiting 21 "master workers" and 79 "non-master" workers. Note that we considered that a worker was a master worker if we detected that they had Maximum number of HITs a worker did in a batch 689 Table 1. Summary statistics of the workers in our study with regard to: HITs workers did, the time they invested in working, workers' earnings, and their multi-tasking information.
completed at least one HIT with master qualifications. Table 1 presents the statistics of the workers in our study and their general labor patterns. We had 73 men and 27 women, who had a median age of 30 years old. 41 participants were from the United States, 45 from India, five from Brazil, three from Italy, and the remaining six from Venezuela, Spain, Mexico, United Kingdom, United States Virgin Islands, and Thailand.
Through our plugin, we identified that workers did a median of 30 HITs each day. The median daily earnings of each worker were $8.07 US dollars. Figure 1 workers invested daily in completing HITs was 1 hour 7 minutes, and the median time that workers invested in invisible labor was an additional 33 minutes, with some workers spending a maximum of 5 hours and 31 minutes daily. Notice that to calculate this value, we summed up all of the time that workers invested in the different invisible labor activities that our plugin detected. Workers spent a median of 33% of their daily time on MTurk doing invisible labor.
We also graphed a histogram of the amount of time that workers dedicated to completing HITs (see Fig. 2). This graph helps to calibrate whether our plugin is measuring paid labor adequately as we can compare our findings to prior work [98]. Note that we used a log scale on the y-axis so that the distribution was easier to visualize. From here, we observe that similar to prior work [96], the distribution of the time that workers invested in completing HITs had a long tail that was heavily weighted towards shorter tasks, meaning workers usually did HITs that took under a minute.
Next, we were interested in studying whether there was a significant correlation between the time workers spent working and the time they spent conducting invisible labor (as this can help us to better understand the phenomena of invisible labor). For this purpose, we computed the Spearman's correlation and obtained 0.283 (p-value 0.004) for the time workers spent working and time doing invisible labor, and 0.517 (p-value 0.000) for the percentage of time working and time in invisible labor. Given these values, for both cases, we reject the null hypothesis that the samples are uncorrelated, i.e., we identified that there is correlation between the time workers' spent working and the time they spent completing invisible labor. Future work could thus study the type of paid labor that might minimize the amount of time a worker has to dedicate to invisible labor.
We aimed here to understand the eco nomic costs that invisible labor has on workers' wages. For this purpose, we first visualized in greater depth the median time workers spent daily in invisible labor and in completing HITs (see Fig. 3). We also aimed to understand the distribution of payments of the completed HITs (see Fig. 4). Armed with this information, we calculated the median hourly wage of workers. We used an approach similar to prior work [57,98]. We first calculated the total hours a worker spent completing HITs on a given day D. We call this the worker's �� ����������������������, and it is the sum of all the time series (Time �� ), measured in hours, that the worker dedicated to doing HITs on day D within the time period ��: (1) After this, we obtain the total ������������ �� the worker made on day ��. We take this value from the rewards and bonuses logged on the worker's "Daily Income" on her MTurk dashboard. For worker ��, her overall hourly wage for day �� is:
With this, we calculate for each worker her hourly wage for each day of our study. We then use that information to calculate the median hourly wage of the 100 workers participating in our study. Excluding invisible labor, we calculated that workers earned a median hourly wage of $3.76, which roughly coincides with prior work, which calculated $3.18 [57]. Notice that it is likely that we calculate a slightly higher salary because we utilize a slightly more conservative approach for our measurements, with the purpose of limiting the overreporting of invisible labor that workers do. Now, if we include invisible work into the calculation of the hourly wage, the median hourly wage of workers drops to $2.83. Next, we were interested in better understanding the dynamics around invisible labor and wages. Figure 5 presents a scatter plot where each point represents a worker. The X-axis represents the median percentage of time a worker invested in invisible labor daily, and the Y-axis the worker's median daily wage. From Fig. 5, we observe that the highest-earning workers, in general, all invested less than 50% of their time in invisible labor. Given this result, there might be value in exploring coaching systems that teach workers how to best manage their invisible labor to ensure high wages.
In this section, we provide a breakdown of the different demographics of workers in our study (segments) and study the type of invisible labor they presented in their work practices. This analysis is important as research has started to showcase how workers' different demographics can impact how they approach work on MTurk [37,58,90,102]. We were thus interested in further studying and understanding this aspect, but now for invisible labor. In Table 2, we present an overview of the amount of paid labor and invisible labor that different population segments conducted. Notice that in the table, we also calculate the "unpaid percentage ratio," which denotes the percentage of the total working time that is unpaid. We calculated the unpaid percentage rate as follows, where ������������������_����������_��������, is the median time workers in a particular segment spent on invisible labor, and ��������_����������_ �������� the median time workers in that segment spent on paid work.
= ������������������_����������_������ �� ������������������_����������_ �������� + ��������_����������_�������� (3) Armed with these measurements, we next conducted statistical analysis to study whether there were significant differences between how invisible labor impacted the different segments of workers. First, over each worker segment we performed the Shapiro-Wilk test, which allows us to identify whether our distribution is normal or not. We found that for all the segments, the p-value was less than .05, so we rejected the null hypothesis (i.e., our distribution is not normal). Given that we do not have a normal distribution, we proceeded to use a nonparametric analysis of variance. We performed a Kruskal-Wallis H Test as a non-parametric alternative to the parametric one-way between-groups analysis of variance for independent groups. We found that there was a significant difference (p-value < 0.05) in the invisible labor time between the workers who were: Masters and Non-masters (p-value 0.00), male and female (p-value 0.04), tool users and non-tool users (p-value 0.01), from English speaking countries and non-English speaking countries (pvalue 0.02). We did not find a significant difference among the following groups: workers without disabilities and workers with some disabilities (p-value 0.64); workers who do multitasking and workers who do no multi-tasking (p-value 0.32); workers in the U.S. and workers in India (p-value 0.07).
Next, we dug deeper into several of these results to better understand the dynamics behind invisible labor and workers' demographics. Table 2 shows that from the 100 Turkers who participated in our study, 21 of these were MTurk Masters, and 79 were not. We found that the median amount of time that master workers invested in completing HITs daily was 1 hour and 37 min and the median amount of time they invested in invisible work daily was just 24 minutes, as shown in Sending messages (Lack of Guidance) 2.4 1.9 0.7 51% Watching over requesters' profiles (Hypervigilance) 15.0 1.1 12.9 69% Searching for general HITs (Hypervigilance) 3.6 0.9 5.6 96% Managing queued HITs (Hypervigilance) 3.2 0.7 4.6 93% Previewing HITs (Lack of Guidance) 1.5 0.6 1.0 66%
Viewing their earnings (Payments) 0.9 0.5 0.3 85% Searching for filtered HITs (Hypervigilance) 3.9 0.5 0.6 46% Checking Worker's qualifications (Hypervigilance) 0.4 0.2 0.0 27% Login to MTurk (General Logistics) 0.3 0.1 0.1 64% Reading HIT information (Lack of Guidance) 0.1 0.0 0.0 63% Reading Platform Help (Lack of Guidance) 0.0 0.0 0.0 0% Table 3. Overview of the invisible labor activities that workers did, the amount of time they dedicated daily to each activity per day, and the percentage of workers who engaged in the activity. Doing HITs that eventually time out was the median most time consuming activity; viewing their earnings was the most common activity.
that Masters workers have more experience using tools. 86% of our Masters worker participants reported using tools, while only 57% of the non-masters workers reported tool use. Additionally, these workers might not be using these tools as effectively as the master workers. Prior work had identified that there are differences in how experts and nonexperts use tools [67,98].
However, it is important to highlight that Table 2 does show that workers who used tools spent more time doing paid work (30 minutes more) and earned substantially higher wages ($1.3 USD more daily, when measuring workers' median wages.) Notice that these results might be emerging because most tools focus on increasing the wages that workers receive for their paid labor [67]. But, given our results, we believe there is value in exploring mechanisms through which workers learn how to better navigate crowdsourcing markets to focus primarily on paid work.
Within this study of worker segments, we also studied the relationship between adopting particular strategies and invisible labor. Prior work has shown that experienced workers often use strategies to boost their performance [54,98]. This can include using different tools or multi-tasking. Our study identified that workers who completed HITs in batches did 9.6% less invisible work than workers who did not (see Table 2). The reasoning behind this finding is likely that within batches, the same type of tasks is continuously presented to workers (one after the other). Therefore, workers do not have to search for new tasks (thus reducing their invisible labor). Batch tasks are also usually similar, so workers do not have to spend time context switching [74].
4.0.3 Quantifying Categories of Invisible Labor. We were also interested in understanding the type of invisible labor that was the most taxing for workers. Table 3 presents an overview of the different invisible labor activities that our plugin detected that workers did and the percentage of workers who engaged in each activity. For each activity, we also present in parenthesis the main categories to which the activity belongs. In Table 4, we present a summary of the time workers invested in each of these main categories. From Tables 3 and4, we observe that the invisible labor category of "Payments" was the most timeconsuming category (especially when taking the median value) and was also highly common among workers. For example, Table 3 shows how 97% of all workers in our study engaged in the Payments related activity of checking their daily earnings on their worker dashboard. Similarly, the most time-consuming activity was "doing HITs that eventually timeout," which took a median of 4.5 minutes. Luckily, timeouts were not as common (only 37% of workers engaged in this activity). It is important to mention that timeouts relate to "Payments" because requesters on MTurk have to specify the amount of time that workers have for completing their tasks; if workers take longer than that time, the HIT is timed out, and workers do not get paid for any of the labor that they did for the HIT.
We calculated timeouts only if the worker was actually working on the HIT (had any current mouse or keyboard-related activity on the HIT). The timeouts we detected were, therefore, cases where the worker was actively doing labor but at the end did not get paid for it.
To understand the details of the workers who engaged the most in this type of highly taxing invisible labor, we first identified the workers who were outliers (i.e., invested the most time in this activity) and then conducted a manual inspection of their digital traces. We considered outliers to be the workers whose time invested for this particular activity was above the 95th percentile (typical method to calculate outliers [56]). We observed that in this case, the outliers tended to be workers who accepted a high number of HITs within a given time window (likely to avoid having other workers take the HIT before them). However, the problem was that it would sometimes take workers significant time to get to some of the HITs they had "reserved" for themselves, and hence they experienced timeouts. We thus believe there is value in exploring tools [81], that based on workers' log data, can automatically learn the best amount of time that should be allocated for a given task and then recommend to requesters to use a significantly higher time window than that time to avoid timeouts and also be sympathetic with the labor practices of some workers.
Table 4 also shows that the second most time-consuming category was that of Hypervigilance, taking workers' a median of 11 minutes daily. The Hypervigilance activity that took the most time was watching over requesters' profiles. It is likely that workers engaged in this activity because through this they could more easily grab the HITs that their favorite requesters posted [51]. Upon manual inspection of workers' digital traces, we identified that the workers who invested the most time in this activity (i.e., the outliers, which we calculated with a similar method as stated above), were the workers who appeared to hunt the profiles of multiple requesters ready to be "on-call". (In specific, these workers opened the profile pages of multiple requesters and then iterated through the list of profile pages, likely inspecting if the requesters had posted anything new.) Finally, the third most time-consuming category was "Lack of Guidance," which took a median of six minutes daily. The most time-consuming activity here were cases when workers started a HIT but then decided to return it. There are several reasons why workers might engage in this behavior; for example: workers realize that the HIT is more time-consuming than they expected; or the HIT involves skills that the worker lacks; or the HIT consists of activities that the worker does not enjoy. In general, these are instances where the HIT instructions likely did not correctly guide the worker on the type of labor to expect, and hence the worker had to return the HIT. Prior work has already reported how the lack of guidance can lead to these types of dynamics [43,83]. From Table 3, we note that the activities related to the Lack of Guidance were actually some of the most commonplace for workers and also some of the most time consuming (e.g., 92% engaged in starting HITs but then returning them; and this was also the second most time consuming activity.) It was surprising to see the large percentage who returned HITs. Upon manual inspection of the outliers, we observed that they appeared to primarily follow a discard-by-doing labor pattern [68].
Workers from digital labor platforms typically underestimate the actual amount of time and effort they dedicate to invisible labor [113]. However, workers' perceptions of invisible labor can play a strong role in how they feel about their work. In this section, we investigate the amount of time that crowd workers believe they invested in invisible labor and their satisfaction. For this part, we use the initial survey that we gave workers, which was inspired by prior work [4]. Through this, we found that workers in our study estimated that they spent a median of 2 hours daily on invisible labor on MTurk (with the minimum time that some workers' estimated as 0 and a maximum of 8 hours.) Figure 6 plots the actual time workers invested in invisible labor against perceived time. Notice that each point represents a worker in our study, and workers are color-coded based on whether workers are master-workers (dark gray) or non-master workers (light gray). We made this distinction given that prior work has identified that there are differences in how more experienced workers operate [54,98], and our results in the previous sections were also highlighting these differences. Notice that workers in Figure 6 who were able to accurately guess the amount of time they spent on invisible labor are located on the diagonal line, as that is when the actual time is equal to the perceived time. The cluster of points that we observe above the diagonal line close to the Y-axis showcases that the majority of crowd workers in our study overestimated the amount of time that they thought they invested in invisible labor. Notice that this overestimation occurred for both master and non-master workers. Next, we quantify the relative error of workers in estimating how much time they invested daily in invisible labor:
Through this, we identified that the median relative error was -0.14. Notice that the negative value highlights that workers are overestimating how much time they dedicate to invisible labor, but our conservative metrics used to quantify invisible labor might also contribute to the error. Next, we analyze workers' satisfaction with the time they perceived they invested in invisible labor. Table 5 presents a summary of these statistics.
Only 10% felt they invested too little time on invisible labor (8% felt they spent "too little time" and 2% "far too little"), while slightly more than half (63%) felt they invested too much in invisible labor (38% "too much time," and 25% "far too much time"). Lastly, 27% of workers considered they invested the right amount of time in invisible labor. Future work could study the type of labor dynamics that might lead workers to feel more satisfied with the amount of invisible labor that they do, and also what circumstances might lead them to feel the most dissatisfied.
The core result from our study is that crowd workers spent a median of 33 minutes of their daily time on MTurk doing invisible labor, and this labor leads workers to drop their median hourly wage from $3.76 to $2.83. Notice that because we used conservative methods to measure invisible labor, we are obtaining a lower bound of the amount of invisible labor that exists on MTurk. However, this lower bound is still highlighting and providing quantitative support to the literature's qualitative claim that invisible work makes up a substantial fraction of the work done in crowdsourcing markets and, therefore, dramatically reduces workers' hourly wages [51]. Considering that the median hourly wage of workers is just $3.76 (without considering invisible labor), it is clear that crowd workers still need a dramatic increase in their wages before we can consider this labor fair. However, this is not only something for requesters to consider, but also something for platforms, workers, and even policy makers. In this section, we discuss: the details of the most taxing categories of invisible labor that our study uncovered; design and policy solutions to mitigate invisible labor on crowdsourcing platforms. Additionally, we make an effort to connect with invisible labor in other workplaces, as well as with critical theory, to have a broader discussion on the implications of our research.
Labor. The invisible labor that the over whelming majority of workers in our study practiced was around Payments. In fact, 97% of the workers in our study visited the earnings section on their worker's dashboard at least once daily. Crowd workers are likely visiting their earnings dashboard to ensure that they: (1) were paid for their labor; and (2) made a certain daily income amount [67]. For the first point, it is important to note that crowd workers typically have to deal with faceless requesters, machines that are outdated, unreliable internet connections, and have nowhere to report when things go wrong (e.g., report that a requester decided to unjustly withhold payment, or report that due to technical issues they can no longer access their MTurk account and earnings.) Pew Research reported that 30% of on-demand gig workers experienced situations where they were not paid for their labor [ 61,103]. Similarly, the US Freelancers Union found that 71% of freelancers have struggled to collect payment for their work. As we note, ensuring payment is a critical and stressful aspect of crowd work [62]. For the second point, we have to be aware that most crowd workers struggle to make a minimum wage [57]. Therefore, another likely reason why workers were visiting their earnings dashboard was to see if they had made sufficient wages. The stress of not receiving payment for their labor or not receiving enough appears to be very present and real in crowd work.
Invisible labor around payments was actually also the most time-consuming, and one of the most critical, as it relates to workers' livelihood. To address this problem, designers could explore interfaces where workers are constantly informed of their current earnings. However, seeing their earnings constantly could also create stress on workers. Future work could explore optimal settings for displaying wages in crowd work. We also believe there is value in further exploring interfaces where requesters, platforms, and workers agree to fair wages [101]. Offering workers a space where they know they will be treated fairly could reduce repeated payment checking [112].
It is important to mention that this type of invisible labor is also present in other digital workspaces [53,92]. For example, Uber had reports of drivers and passengers organizing to check how much a passenger was actually charged for a ride vs. how much the driver received. This dynamic emerged after Uber changed its pricing algorithm and did not provide transparency on how it functioned [12]. The lack of transparency not only led drivers and passengers to have to engage in this type of invisible labor, it also led them to feel cheated and betrayed by the platform [92]. Here it is important to highlight that this invisible labor does not only emerge due to the fault of requesters (passengers). But rather, platforms can play a key part in the promotion of this invisible labor. Here it can be important for platforms to see that this type of invisible labor is likely emerging out of mistrust and has the potential to alienate people from their platforms.
Our study uncovered that the invisible labor category of Hypervigilance was the second most time-consuming for crowd workers. Workers spent a median of 11 minutes daily on this category of invisible labor. Crowd work has been championed as offering people the unique flexibility of working anytime and from anywhere [21,117]. However, our work highlights how this flexibility is likely more of a myth. Crowd workers have to dedicate significant time daily to search for work and be on-call for requesters. Intuitively, this suggests that there are more workers on the site than there is work to be done. (If there were lots of requesters constantly posting lots of highpaying jobs, workers would not feel the need to be on call to get the good work.) This connects with prior work that shows that requesters have the majority of the power in this market partly due to the fact that there is an extreme concentration of a few requesters who post the majority of the tasks [28,69]. Thus, workers are forced to take whatever jobs at whatever pay these few requesters post.
A way to start addressing this problem could be to build off the different tools and computa tional methods that have been developed to achieve fair compensation [112]. Potentially these computational methods could be extended by incorporating an invisible labor component. For instance, workers could be computationally guided to cooperate with each other to ensure fairer wages and minimize the amount of invisible labor in which they engage [32], such approach could be extended to potentially lead to reduced invisible labor. Similarly, we could also consider how algorithms that facilitate automatic task assignment and recommendations [60], could be helpful in reducing invisible labor by minimizing the task search time.
When thinking about the invisible labor around Hypervigilance, it is also important to notice that this type of invisible labor is one that promises workers high returns (especially as by being vigilant, workers can potentially earn high wages). Here, it can be important to identify that other digital labor platforms have started to weaponize this type of invisible labor to manipulate workers to stay longer on their platforms [52,110]. For example, Uber sends drivers messages to motivate them to keep being vigilant of surge pricing [12]. The following is an example message that Uber sends drivers to motivate them to remain vigilant of surge pricing: "The weekend is here, and demand is on the rise in Lehigh Valley! Plan to go online tonight, and keep an eye out for surge around the area, where you can earn over 3X on fares! Stay online through midnight to take advantage of the highest fares. Uber on!" [92]. In this context, we believe there is value in providing workers with tools that can help them to visualize how digital labor platforms might be manipulating them to engage them in free labor. Related, there is likely also value in tools that can inform workers of the likelihood of achieving specific wages if they engage in hypervigilance within particular time windows.
Researchers have argued that within our "capitalist societies", there is a propensity to manage the workforce in ways that will profit the "capitalists" (who in this context could be considered to be Amazon or the requesters.) [26,33]. Such "workforce management" can include defining what labor is counted and what labor is turned invisible [18,19]. Labor visibility (what is counted) is considered to be especially important in this societal context because the cultural worth of a piece of labor is directly connected to how much the labor costs [89]. Work that is done for free (invisible) usually will fail to be valued [3,33,108]. Several labor collectives, researchers, practitioners, and individual citizens have therefore fought to empower workers to gain visibility and recognition for their work [26,41]. For instance, the International Feminist Collective has been fighting for decades to give more visibility to the housework that women perform [33]. The collective has argued that housework has been undervalued, underpaid, and its invisibility has been used as a means to empower primarily "white middleclass men to do lucrative waged jobs," e.g., office work [19,40]. This in return has profited companies and factories as they now have a more specialized and dedicated workforce [38,39,78]).
In 2013, several of these collectives had a breakthrough when labor statisticians agreed inter nationally to begin measuring in official workforce surveys both paid and unpaid labor, such as housework [9,11]. This inclusion influenced the development of new policies around invisible labor [9,11]. Historically, policymakers had overlooked unpaid labor simply because the work was not included in the official statistics that they used to define policy [9,111]. Its exclusion also meant that policymakers did not understand why the labor was problematic or the number of citizens who were impacted. But, by now counting and including the labor within the official stats that policymakers used for their decision-making, they were able to more easily pay attention to this type of labor, grasp its problematic, and design policy to address its problematics.
Inspired by the impact that the quantification of invisible labor has had in transforming policy within other industries and workplaces, our hope is that our plugin tool, study, and anonymized worker data, can in the future also be used to motivate new policies to improve the labor conditions of crowdworkers. However, given that the use of data in policymaking is usually an organic, political process [23] (which might not be obvious to outsiders, e.g., workers and their advocates), we believe there is value in designing socio-technical mechanisms that guide citizens on how they can best use the data from our plugin to drive policy innovation [16]. This could include tools that guide citizens on the time in which they should release the data on invisible labor to match the political cycle. Being in tune with the political cycle could help citizens to have a better chance at influencing policymakers [111]. Similarly, other tools could focus on helping citizens to easily visualize which policymakers might be most influenced by seeing the stats from our plugin on invisible labor.
There is likely also value in tools that can guide citizens on how to use our plugin's data to gather the public's support and create pressure on policymakers [7,105].
Future work could explore mechanisms to help workers manage the time overheads from invisible labor. Notice that here there are still numerous aspects of invisible labor that need to be further investigated. For instance, are more experienced crowd workers able to reduce the amount of time they spent in invisible labor in comparison with novices? Our results highlight that at least master and non-master workers have similar perceptions of the amount of invisible labor they do. But more analysis in this space is necessary. Especially because there might be a benefit in designing tools that help novice workers adopt some of the strategies from more experienced workers [54,98]. Other questions we are interested in exploring in this space are: How does the way that workers manage their invisible labor relate to their wages? How exactly does multi-tasking and context switching relate to invisible labor? Is a worker's invisible labor increased when workers have to switch between HITs? Are there certain HITs or requesters that magnify workers' invisible labor? Our hope is that by releasing our plugin, we will enable the scientific community to study this.
Notice that our plugin tool can be easily extrapolated to other digital labor platforms to help workers quantify the amount of invisible labor that they spend on those other workplaces (the only main piece that needs to be changed is the mapping between the websites the workers use and the work done on each platform; primarily if it is paid or unpaid labor). Our hope is that our tool will inspire cross-platform studies on invisible labor and will help the scientific community to derive principles around how invisible labor looks like across digital workplaces. As we described above, our plugin and study could also help to motivate action from policymakers. Facilitating tools for cross-platform auditing can be extremely important as digital labor platforms have traditionally been black boxes. But, to design better platforms or drive policy change, it is crucial to understand what happens inside these platforms. Our hope is that our research will be a step forward to better understand and address the dynamics existing in these online spaces.
We believe there is likely value in exploring data visualizations that could help to better showcase the different types of invisible labor that crowd workers have to do. Here, we could take inspiration from the visualizations that Github has developed to showcase the labor surrounding the writing of collaborative code [26,77]. Github has made great strides to provide visualizations that help people to rapidly understand the quantity, frequency, and duration of the contributions made by each individual to a codebase. Such visualizations in this context could help requesters to better grasp the amount and type of invisible labor that their tasks are forcing workers to do and potentially lead requesters to better compensate workers for their effort and time [112]. It is important here to consider how to design such visualizations to also not incite unhealthy competition between workers or enable abuse and surveillance from requesters [71,93].
Workplaces. An important question in CSCW is whether a new design truly engages with the root cause of a societal problem or if it is primarily dealing with the symptoms of a problem [2]. For example, a design could make a societal problem bearable. However, this might lead people to no longer have a need for addressing the root problem. In this setting, the design could provide enjoyable experiences to end-users; but it could also reinforce the structural issues that are harming end-users. Within this context, Herbert Marcuse, a theorist from the Frankfurt School of Critical Theory [66], introduced the concept of "onedimensional" people who have a conformist understanding of society that does not allow them to critique or question how society could be different [84]. Marcuse argues that the one-dimensional person has lost her ability to critique society because consumerism has tricked her into having false needs and wishes (notice that consumerism is considered to be "a social and economic order that encourages the acquisition of goods and services in ever-increasing amounts" [8]). As a result, the person focuses on fulfilling those "fake needs" instead of questioning the problematic societal structures in which she is immersed. According to Marcuse, this dynamic leads us to be imprisoned into one-dimensional thinking, and that makes it extremely challenging to critically question the structures and processes that exist in our society.
As CSCW researchers, we believe it is crucial that we question to what extent we are falling into one-dimensional thinking and possibly strengthening the structural issues that are already in place. This is especially important when designing interfaces that aim to address the problem of invisible labor in crowdwork and also within other digital labor platforms. Without this critical analysis, we might fall into designing interfaces that make the problem of invisible labor bearable; but we never address the systematic problems surrounding workers, requesters, and digital platform owners. Notice that engaging in such critical analysis is an ambitious, complex, and difficult undertaking, but as Marcuse discusses, it is very much necessary [2,84].
Marcuse argues that a way to engage in such critical analysis and challenge our onedimensional thought is by participating in artistic creativity that allows us to leave the reality that has been defined by society [2,84,85]. Artistic creativity facilitates developing new designs that are not confined by the current reality of what is possible and allows us to consider designs we might have been blind to consider otherwise. Based on this, we believe there is value in engaging with workers in "creative artistic co-design sessions." These sessions would allow workers to creatively define the type of digital labor platforms that they would like to see and how they would design to address invisible labor [109].
Similarly, we believe there is value in drawing on scholarship that has studied the link between fiction and design [27,35,79,115]. Here we envision we could engage researchers, workers, platform owners, and practitioners to use fictional narratives to design "alternative realities" to contemporary digital labor platforms and tools [34,80].
The insights from our research are limited by the methodology and population we studied. Our study also focused on breadth instead of depth to start to shed needed light on the quantification of invisible labor in crowd work. Notice that we had to develop specific tools in order to do our field study, which is not simple. However, these types of studies are important, especially given the lack of transparency that MTurk or other crowdsourcing platforms provide around invisible labor. Upon publication, we will open-source our plugin and anonymous worker data so that the scientific community can conduct longitudinal studies around invisible labor, as well as study other principles surrounding invisible labor.
We developed a new computational tool to be able to quantify and study the invisible labor of crowd workers on MTurk. We have demonstrated that the invisible labor that workers do can take a toll on their wages. Particularly, we saw that if we consider the amount of time that crowd workers invest in invisible labor, their hourly wages go down to $2.83 from $3.76. We also identified that the two most time-consuming categories of invisible labor revolved around payments and hyper-vigilance. Additionally, our study identified that workers tended to overestimate the amount of invisible labor that they believed they did. Our results also suggest there is a wide range of dynamics that influence the amount of invisible labor that a particular worker conducts. These different dynamics deserve more investigation.
Finally, we hope that our plugin tool inspires the auditing of different digital labor platforms and helps to potentially generate a range of positive policy innovations in digital work. Our paper has provided much-needed light to the invisible labor of crowd workers.
https://github.com/anonym-research/invisible-labor Proc. ACM Hum.-Comput. Interact., Vol.
2, No. CSCW, Article 37. Publication date: November 2021. Quantifying the Invisible Labor in Crowd Work 37:3
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021. Quantifying the Invisible Labor in Crowd Work 37:5
https://worker.mturk.com/
https://worker.mturk.com/?filters
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021. 37:6 Toxtli et al.
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021. Quantifying the Invisible Labor in Crowd Work 37:7
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021.
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021. Quantifying the Invisible Labor in Crowd Work 37:17
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021. 37:18 Toxtli et al.
Proc. ACM Hum.-Comput. Interact., Vol. 2, No. CSCW, Article 37. Publication date: November 2021. 37:22 Toxtli et al.