skip to main content


This content will become publicly available on June 1, 2025

Title: Tailoring Vaccine Messaging with Common-Ground Opinions
One way to personalize chatbot interactions is by establishing common ground with the intended reader. A domain where establishing mutual understanding could be particularly impactful is vaccine concerns and misinformation. Vaccine interventions are forms of messaging which aim to answer concerns expressed about vaccination. Tailoring responses in this domain is difficult, since opinions often have seemingly little ideological overlap. We define the task of tailoring vaccine interventions to a Common-Ground Opinion (CGO). Tailoring responses to a CGO involves meaningfully improving the answer by relating it to an opinion or belief the reader holds. In this paper we introduce Tailor-CGO, a dataset for evaluating how well responses are tailored to provided CGOs. We benchmark several major LLMs on this task; finding GPT-4-Turbo performs significantly better than others. We also build automatic evaluation metrics, including an efficient and accurate BERT model that outperforms finetuned LLMs, investigate how to successfully tailor vaccine messaging to CGOs, and provide actionable recommendations from this investigation.  more » « less
Award ID(s):
2211526
NSF-PAR ID:
10515877
Author(s) / Creator(s):
; ; ; ; ; ; ; ;
Publisher / Repository:
Association for Computational Linguistics
Date Published:
Journal Name:
Findings of the Association for Computational Linguistics: NAACL 2024
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Vaccine concerns are an ever-evolving target, and can shift quickly as seen during the COVID-19 pandemic. Identifying longitudinal trends in vaccine concerns and misinformation might inform the healthcare space by helping public health efforts strategically allocate resources or information campaigns. We explore the task of detecting vaccine concerns in online discourse using large language models (LLMs) in a zero-shot setting without the need for expensive training datasets. Since real-time monitoring of online sources requires large-scale inference, we explore cost-accuracy trade-offs of different prompting strategies and offer concrete takeaways that may inform choices in system designs for current applications. An analysis of different prompting strategies reveals that classifying the concerns over multiple passes through the LLM, each consisting a boolean question whether the text mentions a vaccine concern or not, works the best. Our results indicate that GPT-4 can strongly outperform crowdworker accuracy when compared to ground truth annotations provided by experts on the recently introduced VaxConcerns dataset, achieving an overall F1 score of 78.7%. 
    more » « less
  2. null (Ed.)
    Development of an effective COVID-19 vaccine is widely considered as one of the best paths to ending the current health crisis. While the ability to distribute a vaccine in the short-term remains uncertain, the availability of a vaccine alone will not be sufficient to stop disease spread. Instead, policy makers will need to overcome the additional hurdle of rapid widespread adoption. In a large-scale nationally representative survey ( N = 34,200), the current work identifies monetary risk preferences as a correlate of take-up of an anticipated COVID-19 vaccine. A complementary experiment ( N = 1,003) leverages this insight to create effective messaging encouraging vaccine take-up. Individual differences in risk preferences moderate responses to messaging that provides benchmarks for vaccine efficacy (by comparing it to the flu vaccine), while messaging that describes pro-social benefits of vaccination (specifically herd immunity) speeds vaccine take-up irrespective of risk preferences. Findings suggest that policy makers should consider risk preferences when targeting vaccine-related communications. 
    more » « less
  3. Background

    The prevalence of human papillomavirus (HPV) and its related cancers is a major global concern. In the United States, routine HPV vaccination is recommended for youth aged 11 or 12 years. Despite HPV being the most common sexually transmitted infection and the vaccine’s proven efficacy, the vaccination rate among US youth remains below the recommended 80% completion rate. Mobile health (mHealth) interventions have demonstrated promise in improving health. Examining and synthesizing the current evidence about the impact of mHealth interventions on vaccination coverage in youth and intervention characteristics could guide future mHealth interventions aimed at mitigating the vaccination gap and disease burden.

    Objective

    This study aims to conduct a systematic review to assess the effectiveness of mHealth interventions on parental intent to vaccinate youth against HPV and youth’s vaccine uptake.

    Methods

    We searched empirical papers through databases including Google Scholar, PubMed, CINAHL, PsycINFO, and Cochrane Library. The inclusion criteria were the following: (1) published between January 2011 and December 2022; (2) using mHealth aimed to improve HPV vaccination rate; (3) targeted unvaccinated youth or their parents; and (4) measured HPV-related knowledge, vaccination intention, or vaccine uptake. Overall, 3 researchers screened and appraised the quality of the eligible papers using the Melnyk Levels of Evidence and the Cochrane Grading of Recommendations Assessment, Development, and Evaluation methodology. Disagreements in search results and result interpretation were resolved through consensus.

    Results

    Overall, 17 studies that met the inclusion criteria were included in the final review. Most studies were conducted in the United States (14/17, 82%), used a randomized controlled trial design (12/17, 71%), and adopted behavior change theories or a culture-centric approach (10/17, 59%). mHealth interventions included SMS text message reminders, motivational SMS text messages, computer-tailored or tablet-tailored interventions, smartphone apps, web-based tailored interventions, social media (Facebook) campaigns, digital videos, and digital storytelling interventions. Approximately 88% (15/17) of the mHealth interventions demonstrated positive effects on knowledge, intention, or behaviors related to HPV vaccination. Overall, 12% (2/17) reported limited or no intervention impact on vaccine uptake or vaccine series completion. Effective vaccine uptake was commonly seen in interventions based on behavior change theories and those that provided culturally relevant information.

    Conclusions

    This systematic review identified the impact of mHealth interventions among unvaccinated youth and their parents, which showed improvement in HPV-related knowledge, vaccination intention, or vaccine initiation. The interventions that incorporated theories and culture-centric approaches revealed the most promising results. Although these outcomes are encouraging, future studies are needed to investigate factors associated with the success of interventions using SMS text messaging or social media. More studies are also needed for a better understanding of the intervention elements that boost the responses of age-specific and ethnicity-specific populations.

     
    more » « less
  4. Mental health activities conducted by patients between therapy sessions (or "therapy homework") are a component of addressing anxiety and depression. However, to be effective, therapy homework must be tailored to the client’s needs to address the numerous barriers they encounter in everyday life. In this study, we analyze how therapists and clients tailor therapy homework to their client’s needs. We interviewed 13 therapists and 14 clients about their experiences tailoring and engaging in therapy homework. We identify criteria for tailoring homework, such as client skills, discomfort, and external barriers. We present how homework gets adapted, such as through changes in difficulty or by identifying alternatives. We discuss how technologies can better use client information for personalizing mental health interventions, such as adapting to client barriers, adjusting homework to these barriers, and creating a safer environment to support discomfort. 
    more » « less
  5. Large language models (LLMs), such as GPT-3 and GPT-4, have demonstrated exceptional performance in various natural language processing tasks and have shown the ability to solve certain reasoning problems. However, their reasoning capabilities are limited and relatively shallow, despite the application of various prompting techniques. In contrast, formal logic is adept at handling complex reasoning, but translating natural language descriptions into formal logic is a challenging task that non-experts struggle with. This paper proposes a neuro-symbolic method that combines the strengths of large language models and answer set programming. Specifically, we employ an LLM to transform natural language descriptions of logic puzzles into answer set programs. We carefully design prompts for an LLM to convert natural language descriptions into answer set programs in a step by step manner. Surprisingly, with just a few in-context learning examples, LLMs can generate reasonably complex answer set programs. The majority of errors made are relatively simple and can be easily corrected by humans, thus enabling LLMs to effectively assist in the creation of answer set programs. 
    more » « less