An official website of the United States government
Synchronous, face-to-face interactions such as brainstorming are considered essential for creative tasks (the old normal). However, face-to-face interactions are difficult to arrange because of the diverse locations and conflicting availability of people—a challenge made more prominent by work-from-home practices during the COVID-19 pandemic (the new normal). In addition, face-to-face interactions are susceptible to cognitive interference. We employ crowdsourcing as an avenue to investigate creativity in asynchronous, online interactions. We choose product ideation,a natural task for the crowd since it requires human insight and creativity into what product features would be novel and useful. We compare the performance of solo crowd workers with asynchronous teams of crowd workers formed without prior coordination. Our findings suggest that, first, crowd teamwork yields fewer but more creative ideas than solo crowdwork. The enhanced team creativity results when (1) team workers reflect on each other’s ideas, and (2) teams are composed of workers of reflective, as opposed to active or mixed, personality types. Second, cognitive interference, known to inhibit creativity in face-to-face teams, may not be significant in crowd teams. Third, teamwork promotes better achievement emotions for crowd workers. These findings provide a basis for trading off creativity, quantity, and worker happiness in setting up crowdsourcing workflows for product ideation.
more »
« less
Who's in the Crowd Matters: Cognitive Factors and Beliefs Predict Misinformation Assessment Accuracy
Misinformation runs rampant on social media and has been tied to adverse health behaviors such as vaccine hesitancy. Crowdsourcing can be a means to detect and impede the spread of misinformation online. However, past studies have not deeply examined the individual characteristics - such as cognitive factors and biases - that predict crowdworker accuracy at identifying misinformation. In our study (n = 265), Amazon Mechanical Turk (MTurk) workers and university students assessed the truthfulness and sentiment of COVID-19 related tweets as well as answered several surveys on personal characteristics. Results support the viability of crowdsourcing for assessing misinformation and content stance (i.e., sentiment) related to ongoing and politically-charged topics like the COVID-19 pandemic, however, alignment with experts depends on who is in the crowd. Specifically, we find that respondents with high Cognitive Reflection Test (CRT) scores, conscientiousness, and trust in medical scientists are more aligned with experts while respondents with high Need for Cognitive Closure (NFCC) and those who lean politically conservative are less aligned with experts. We see differences between recruitment platforms as well, as our data shows university students are on average more aligned with experts than MTurk workers, most likely due to overall differences in participant characteristics on each platform. Results offer transparency into how crowd composition affects misinformation and stance assessment and have implications on future crowd recruitment and filtering practices.
more »
« less
- Award ID(s):
- 2009003
- PAR ID:
- 10602224
- Publisher / Repository:
- Association for Computing Machinery (ACM)
- Date Published:
- Journal Name:
- Proceedings of the ACM on Human-Computer Interaction
- Volume:
- 6
- Issue:
- CSCW2
- ISSN:
- 2573-0142
- Format(s):
- Medium: X Size: p. 1-18
- Size(s):
- p. 1-18
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)The ongoing pandemic has heightened the need for developing tools to flag COVID-19-related misinformation on the internet, specifically on social media such as Twitter. However, due to novel language and the rapid change of information, existing misinformation detection datasets are not effective for evaluating systems designed to detect misinformation on this topic. Misinformation detection can be divided into two sub-tasks: (i) retrieval of misconceptions relevant to posts being checked for veracity, and (ii) stance detection to identify whether the posts Agree, Disagree, or express No Stance towards the retrieved misconceptions. To facilitate research on this task, we release COVIDLies (https://ucinlp.github.io/covid19 ), a dataset of 6761 expert-annotated tweets to evaluate the performance of misinformation detection systems on 86 different pieces of COVID-19 related misinformation. We evaluate existing NLP systems on this dataset, providing initial benchmarks and identifying key challenges for future models to improve upon.more » « less
-
As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.more » « less
-
Bristol Bay in Alaska is home to the world’s largest commercial salmon fishery. During an average fishing season, the population of the Bristol Bay region more than doubles as thousands of workers from out of state converge on the fishery. In the months leading up to the 2020 commercial fishery opening, as the COVID-19 pandemic exploded worldwide, great uncertainty existed about the health risks of opening the fishery. Bristol Bay residents had not yet experienced any cases of COVID-19, yet the livelihoods of most were closely tied to the commercial fishery opening. To better understand how COVID-19 risk perceptions affected decisions to participate in the fishery, we administered an online survey to community members and fishery participants. We collected standard socioeconomic data and posed questions to gauge risk perceptions related to COVID-19. We find that COVID-19 risk perceptions vary across race/ethnic groups by residency and income. People with below median income who are members of minority groups—notably, non-resident Hispanic workers and resident Alaska Native respondents—reported the highest risk perceptions related to COVID-19. This study highlights the important linkages among risk perceptions, socioeconomic characteristics, and employment decisions during an infectious disease outbreak.more » « less
-
null (Ed.)In early 2020, colleges shifted abruptly from traditional in-person to remote distant instruction due to COVID-19 potentially exacerbating science, technology, engineering, and mathematics (STEM) students’ recruitment and retention. This preliminary study using survey methodology was conducted with STEM students at a small (700 students) private college to examine questions related to students’ perceptions of natural science careers, career decision-making factors, barriers influencing students’ career path, including effects of COVID-19 on career goals, mental health, and perceived quality of instruction. A Qualtrics® survey was sent to 180 STEM students, from which we received 53 responses (29.4% response rate). Consistent with other studies, family was one of the most important factors supporting their career path. Students had a relatively upbeat career outlook despite being in the middle of a global pandemic and were only moderately worried about the impact of COVID-19 on their future career. Despite these relatively positive outcomes, the abrupt switch to online instruction was viewed unfavorably by most respondents, who valued the hands-on learning experiences obtained with traditional in-person instruction. It is possible that respondents’ views of online instruction may improve over time as instructors become more adept at using new instructional tools. Future research should evaluate this aspect and whether students’ career goals change across time as the pandemic unfolds.more » « less
