An official website of the United States government
Abstract To design effective vaccine policies, policymakers need detailed data about who has been vaccinated, who is holding out, and why. However, existing data in the US are insufficient: reported vaccination rates are often delayed or not granular enough, and surveys of vaccine hesitancy are limited by high-level questions and self-report biases. Here we show how search engine logs and machine learning can help to fill these gaps, using anonymized Bing data from February to August 2021. First, we develop avaccine intent classifierthat accurately detects when a user is seeking the COVID-19 vaccine on Bing. Our classifier demonstrates strong agreement with CDC vaccination rates, while preceding CDC reporting by 1–2 weeks, and estimates more granular ZIP-level rates, revealing local heterogeneity in vaccine seeking. To study vaccine hesitancy, we use our classifier to identify two groups,vaccine early adoptersandvaccine holdouts. We find that holdouts, compared to early adopters matched on covariates, are 67% likelier to click on untrusted news sites, and are much more concerned about vaccine requirements, development, and vaccine myths. Even within holdouts, clusters emerge with different concerns and openness to the vaccine. Finally, we explore the temporal dynamics of vaccine concerns and vaccine seeking, and find that key indicators predict when individuals convert from holding out to seeking the vaccine.
more »
« less
Hierarchical Multi-Label Classification of Online Vaccine Concerns
Vaccine concerns are an ever-evolving target, and can shift quickly as seen during the COVID-19 pandemic. Identifying longitudinal trends in vaccine concerns and misinformation might inform the healthcare space by helping public health efforts strategically allocate resources or information campaigns. We explore the task of detecting vaccine concerns in online discourse using large language models (LLMs) in a zero-shot setting without the need for expensive training datasets. Since real-time monitoring of online sources requires large-scale inference, we explore cost-accuracy trade-offs of different prompting strategies and offer concrete takeaways that may inform choices in system designs for current applications. An analysis of different prompting strategies reveals that classifying the concerns over multiple passes through the LLM, each consisting a boolean question whether the text mentions a vaccine concern or not, works the best. Our results indicate that GPT-4 can strongly outperform crowdworker accuracy when compared to ground truth annotations provided by experts on the recently introduced VaxConcerns dataset, achieving an overall F1 score of 78.7%.
more »
« less
- Award ID(s):
- 2211526
- PAR ID:
- 10515878
- Publisher / Repository:
- 2024 International Workshop on Health Intelligence (W3PHIAI 2024)
- Date Published:
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)The arrival of the COVID-19 vaccine has been accompanied by increased discussion of vaccine hesitancy. However, it is unclear if there are shared patterns between general vaccine hesitancy and COVID-19 vaccine rejection, or if these are two different concepts. This study characterized rejection of a hypothetical COVID-19 vaccine, and compared patterns of association between general vaccine hesitancy and COVID-19 vaccine rejection. The survey was conducted online March 20-22, 2020. Participants answered questions on vaccine hesitancy and responded if they would accept the vaccine given different safety and effectiveness profiles. We assessed differences in COVID-19 rejection and general vaccine hesitancy through logistic regressions. Among 713 participants, 33.0% were vaccine hesitant, and 18.4% would reject a COVID-19 vaccine. Acceptance varied by effectiveness profile: 10.2% would reject a 95% effective COVID-19 vaccine, but 32.4% would reject a 50% effective vaccine. Those vaccine hesitant were significantly more likely to reject COVID-19 vaccination [odds ratio (OR): 5.56, 95% confidence interval (CI): 3.39, 9.11]. In multivariable logistic regression models, there were similar patterns for vaccine hesitancy and COVID-19 vaccine rejection by gender, race/ethnicity, family income, and political affiliation. But the direction of association flipped by urbanicity (P=0.0146, with rural dwellers less likely to be COVID-19 vaccine rejecters but more likely to be vaccine hesitant in general), and age (P=0.0037, with fewer pronounced differences across age for COVID-19 vaccine rejection, but a gradient of stronger vaccine hesitancy in general among younger ages). During the COVID-19 epidemic’s early phase, patterns of vaccine hesitancy and COVID-19 vaccine rejection were relatively similar. A significant minority would reject a COVID-19 vaccine, especially one with less-than-ideal effectiveness. Preparations for introducing the COVID-19 vaccine should anticipate substantial hesitation and target concerns, especially among younger adults.more » « less
-
Language models that can learn a task at inference time, called in-context learning (ICL), show increasing promise in natural language inference tasks. In ICL, a model user constructs a prompt to describe a task with a natural language instruction and zero or more examples, called demonstrations. The prompt is then input to the language model to generate a completion. In this paper, we apply ICL to the design and evaluation of satisfaction arguments, which describe how a requirement is satisfied by a system specification and associated domain knowledge. The approach builds on three prompt design patterns, including augmented generation, prompt tuning, and chain-of-thought prompting, and is evaluated on a privacy problem to check whether a mobile app scenario and associated design description satisfies eight consent requirements from the EU General Data Protection Regulation (GDPR). The overall results show that GPT-4 can be used to verify requirements satisfaction with 96.7% accuracy and dissatisfaction with 93.2% accuracy. Inverting the requirement improves verification of dissatisfaction to 97.2%. Chain-of-thought prompting improves overall GPT-3.5 performance by 9.0% accuracy. We discuss the trade-offs among templates, models and prompt strategies and provide a detailed analysis of the generated specifications to inform how the approach can be applied in practice.more » « less
-
Wardman, Jamie (Ed.)Currently, one of the most pressing public health challenges is encouraging people to get vaccinated against COVID-19. Due to limited supplies, some people have had to wait for the COVID-19 vaccine. Consumer research has suggested that people who are overlooked in initial distribution of desired goods may no longer be interested. Here, we therefore examined people’s preferences for proposed vaccine allocation strategies, as well as their anticipated responses to being overlooked. After health-care workers, most participants preferred prioritizing vaccines for high-risk individuals living in group-settings (49%) or with families (29%). We also found evidence of reluctance if passed over. After random assignment to vaccine allocation strategies that would initially overlook them, 37% of participants indicated that they would refuse the vaccine. The refusal rate rose to 42% when the vaccine allocation strategy prioritized people in areas with more COVID-19 – policies that were implemented in many areas. Even among participants who did not self-identify as vaccine hesitant, 22% said they would not want the vaccine in that case. Logistic regressions confirmed that vaccine refusal would be largest if vaccine allocation strategies targeted people who live in areas with more COVID-19 infections. In sum, once people are overlooked by vaccine allocation, they may no longer want to get vaccinated, even if they were not originally vaccine hesitant. Vaccine allocation strategies that prioritize high-infection areas and high-risk individuals in group-settings may enhance these concerns.more » « less
-
Public sentiment toward the COVID-19 vaccine as expressed on social media can interfere with communication by public health agencies on the importance of getting vaccinated. We investigated Twitter data to understand differences in sentiment, moral values, and language use between political ideologies on the COVID-19 vaccine. We estimated political ideology, conducted a sentiment analysis, and guided by the tenets of moral foundations theory (MFT), we analyzed 262,267 English language tweets from the United States containing COVID-19 vaccine-related keywords between May 2020 and October 2021. We applied the Moral Foundations Dictionary and used topic modeling and Word2Vec to understand moral values and the context of words central to the discussion of the vaccine debate. A quadratic trend showed that extreme ideologies of both Liberals and Conservatives expressed a higher negative sentiment than Moderates, with Conservatives expressing more negative sentiment than Liberals. Compared to Conservative tweets, we found the expression of Liberal tweets to be rooted in a wider set of moral values, associated with moral foundations of care (getting the vaccine for protection), fairness (having access to the vaccine), liberty (related to the vaccine mandate), and authority (trusting the vaccine mandate imposed by the government). Conservative tweets were found to be associated with harm (around safety of the vaccine) and oppression (around the government mandate). Furthermore, political ideology was associated with the expression of different meanings for the same words, e.g. “science” and “death.” Our results inform public health outreach communication strategies to best tailor vaccine information to different groups.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Workshop Report:
- The DOI is not currently available.
