An official website of the United States government
Flexibility is essential for optimizing crowdworker performance in the digital labor market, and prior research shows that integrating diverse devices can enhance this flexibility. While studies on Amazon Mechanical Turk show the need for tailored workflows and varied device usage and preferences, it remains unclear if these insights apply to other platforms. To explore this, we conducted a survey on another major crowdsourcing platform, Prolific, involving 1,000 workers. Our findings reveal that desktops are still the primary devices for crowdwork, but Prolific workers display more diverse usage patterns and a greater interest in adopting smartwatches, smart speakers, and tablets compared to MTurk workers. While current use of these newer devices is limited, there is growing interest in employing them for future tasks. These results underscore the importance for crowdsourcing platforms to develop platform-specific strategies that promote more flexible and engaging workflows, better aligning with the diverse needs of their crowdworkers.
more »
« less
Replication: How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys
Privacy and security researchers often rely on data collected through online crowdsourcing platforms such as Amazon Mechanical Turk (MTurk) and Prolific. Prior work---which used data collected in the United States between 2013 and 2017---found that MTurk responses regarding security and privacy were generally representative for people under 50 or with some college education. However, the landscape of online crowdsourcing has changed significantly over the last five years, with the rise of Prolific as a major platform and the increasing presence of bots. This work attempts to replicate the prior results about the external validity of online privacy and security surveys. We conduct an online survey on MTurk (n=800), a gender-balanced survey on Prolific (n=800), and a representative survey on Prolific (n=800) and compare the responses to a probabilistic survey conducted by the Pew Research Center (n=4272). We find that MTurk response quality has degraded over the last five years, and our results do not replicate the earlier finding about the generalizability of MTurk responses. By contrast, we find that data collected through Prolific is generally representative for questions about user perceptions and experiences, but not for questions about security and privacy knowledge. We also evaluate the impact of Prolific settings, attention check questions, and statistical methods on the external validity of online surveys, and we develop recommendations about best practices for conducting online privacy and security surveys.
more »
« less
- Award ID(s):
- 1948344
- PAR ID:
- 10404815
- Date Published:
- Journal Name:
- Eighteenth Symposium on Usable Privacy and Security
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Internet of Things (IoT) device manufacturers provide little information to consumers about their security and data handling practices. Therefore, IoT consumers cannot make informed purchase choices around security and privacy. While prior research has found that consumers would likely consider security and privacy when purchasing IoT devices, past work lacks empirical evidence as to whether they would actually pay more to purchase devices with enhanced security and privacy. To fill this gap, we conducted a two-phase incentive compatible online study with 180 Prolific participants. We measured the impact of five security and privacy factors (e.g., access control) on participants’ purchase behaviors when presented individually or together on an IoT label. Participants were willing to pay a significant premium for devices with better security and privacy practices. The biggest price differential we found was for de-identified rather than identifiable cloud storage. Mainly due to its usability challenges, the least valuable improvement for participants was to have multi-factor authentication as opposed to passwords. Based on our findings, we provide recommendations on creating more effective IoT security and privacy labeling programs.more » « less
-
Mobile and web apps are increasingly relying on the data generated or provided by users such as from their uploaded documents and images. Unfortunately, those apps may raise significant user privacy concerns. Specifically, to train or adapt their models for accurately processing huge amounts of data continuously collected from millions of app users, app or service providers have widely adopted the approach of crowdsourcing for recruiting crowd workers to manually annotate or transcribe the sampled ever-changing user data. However, when users' data are uploaded through apps and then become widely accessible to hundreds of thousands of anonymous crowd workers, many human-in-the-loop related privacy questions arise concerning both the app user community and the crowd worker community. In this paper, we propose to investigate the privacy risks brought by this significant trend of large-scale crowd-powered processing of app users' data generated in their daily activities. We consider the representative case of receipt scanning apps that have millions of users, and focus on the corresponding receipt transcription tasks that appear popularly on crowdsourcing platforms. We design and conduct an app user survey study (n=108) to explore how app users perceive privacy in the context of using receipt scanning apps. We also design and conduct a crowd worker survey study (n=102) to explore crowd workers' experiences on receipt and other types of transcription tasks as well as their attitudes towards such tasks. Overall, we found that most app users and crowd workers expressed strong concerns about the potential privacy risks to receipt owners, and they also had a very high level of agreement with the need for protecting receipt owners' privacy. Our work provides insights on app users' potential privacy risks in crowdsourcing, and highlights the need and challenges for protecting third party users' privacy on crowdsourcing platforms. We have responsibly disclosed our findings to the related crowdsourcing platform and app providers.more » « less
-
Background Chatbots are being piloted to draft responses to patient questions, but patients’ ability to distinguish between provider and chatbot responses and patients’ trust in chatbots’ functions are not well established. Objective This study aimed to assess the feasibility of using ChatGPT (Chat Generative Pre-trained Transformer) or a similar artificial intelligence–based chatbot for patient-provider communication. Methods A survey study was conducted in January 2023. Ten representative, nonadministrative patient-provider interactions were extracted from the electronic health record. Patients’ questions were entered into ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider’s response. In the survey, each patient question was followed by a provider- or ChatGPT-generated response. Participants were informed that 5 responses were provider generated and 5 were chatbot generated. Participants were asked—and incentivized financially—to correctly identify the response source. Participants were also asked about their trust in chatbots’ functions in patient-provider communication, using a Likert scale from 1-5. Results A US-representative sample of 430 study participants aged 18 and older were recruited on Prolific, a crowdsourcing platform for academic studies. In all, 426 participants filled out the full survey. After removing participants who spent less than 3 minutes on the survey, 392 respondents remained. Overall, 53.3% (209/392) of respondents analyzed were women, and the average age was 47.1 (range 18-91) years. The correct classification of responses ranged between 49% (192/392) to 85.7% (336/392) for different questions. On average, chatbot responses were identified correctly in 65.5% (1284/1960) of the cases, and human provider responses were identified correctly in 65.1% (1276/1960) of the cases. On average, responses toward patients’ trust in chatbots’ functions were weakly positive (mean Likert score 3.4 out of 5), with lower trust as the health-related complexity of the task in the questions increased. Conclusions ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower-risk health questions. It is important to continue studying patient-chatbot interaction as chatbots move from administrative to more clinical roles in health care.more » « less
-
The United Nation’s Sustainable Development Goals state climate change could irreversibly affect future generations and is one of the most urgent issues facing society. To date, most education research on climate change examines middle and high school students’ knowledge without considering the link between understanding and interest to address such issues in their career. In research on students’ attitudes about sustainability, we found that half of first-year college engineering students, in our nationally representative sample of all college students at 4-year institutions (n = 937), do not believe climate change is caused by humans. This lack of belief in human-caused climate change is a significant problem in engineering education because our results also indicate engineering students who do not believe in human caused climate change are less likely to want to address climate change in their careers. This dismal finding highlights a need for improving student understanding and attitudes toward climate change in order to produce engineers prepared and interested in solving complex global problems in sustainability. To advance understanding about students’ understanding of climate change and their agency to address the issue, we developed the CLIMATE survey to measure senior undergraduate engineering students’ Climate change literacy, engineering identity, career motivations, and agency through engineering. The survey was designed for students in their final senior design, or capstone course, just prior to entering the workforce. We developed the survey using prior national surveys and newly written questions categorized into six sections: (1) career goals and motivation, (2) college experiences, (3) agency, (4) climate literacy, (5) people and the planet, and (6) demographic information. We conducted focus groups with students to establish face and content validity of the survey. We collected pilot data with 200 engineering students in upper-level engineering courses to provide validity evidence for the use of these survey items to measure students and track changes across the undergraduate curriculum for our future work. In this paper, we narrate the development of the survey supported by literature and outline the next step for further validation and distribution on a national scale. Our intent is to receive feedback and input about the questions being asked and the CLIMATE instrument. Our objective is to share the nationally representative non-identifiable responses (the estimated goal is 4,000 responses) openly with education researchers interested in students understanding about climate change, their engineering identity, career motivations, and agency through engineering. Ultimately, we want this research to become a catalyst for teaching about topics related to climate change in engineering and its implications for sustainability.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
