More Like this

1. Error messages are classifiers: a process to design and evaluate error messages

   https://doi.org/10.1145/3133850.3133862  

   Wrenn, John ; Krishnamurthi, Shriram ( January 2017 , International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software)

   This paper presents a lightweight process to guide error report authoring. We take the perspective that error reports are really classifiers of program information. They should therefore be subjected to the same measures as other classifiers (e.g., precision and recall). We formalize this perspective as a process for assessing error reports, describe our application of this process to an actual programming language, and present a preliminary study on the utility of the resulting error reports. 

   more » « less
2. Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19: A Randomized Clinical Trial

   https://doi.org/10.1001/jamanetworkopen.2021.17115  

   Torres, Carlos ; Ogbu-Nwobodo, Lucy ; Alsan, Marcella ; Stanford, Fatima Cody ; Banerjee, Abhijit ; Breza, Emily ; Chandrasekhar, Arun G. ; Eichmeyer, Sarah ; Karnani, Mohit ; Loisel, Tristan ; et al ( July 2021 , JAMA Network Open)
3. Designing Effective Tsunami Messages: Examining the Role of Short Messages and Fear in Warning Response

   https://doi.org/10.1175/WCAS-D-17-0032.1  

   Sutton, Jeannette ; Vos, Sarah C. ; Wood, Michele M. ; Turner, Monique ( January 2018 , Weather, Climate, and Society)
4. Working With AI to Persuade: Examining a Large Language Model's Ability to Generate Pro-Vaccination Messages

   https://doi.org/10.1145/3579592  

   Karinshak, Elise ; Liu, Sunny Xun ; Park, Joon Sung ; Hancock, Jeffrey T. ( April 2023 , Proceedings of the ACM on Human-Computer Interaction)

   Artificial Intelligence (AI) is a transformative force in communication and messaging strategy, with potential to disrupt traditional approaches. Large language models (LLMs), a form of AI, are capable of generating high-quality, humanlike text. We investigate the persuasive quality of AI-generated messages to understand how AI could impact public health messaging. Specifically, through a series of studies designed to characterize and evaluate generative AI in developing public health messages, we analyze COVID-19 pro-vaccination messages generated by GPT-3, a state-of-the-art instantiation of a large language model. Study 1 is a systematic evaluation of GPT-3's ability to generate pro-vaccination messages. Study 2 then observed peoples' perceptions of curated GPT-3-generated messages compared to human-authored messages released by the CDC (Centers for Disease Control and Prevention), finding that GPT-3 messages were perceived as more effective, stronger arguments, and evoked more positive attitudes than CDC messages. Finally, Study 3 assessed the role of source labels on perceived quality, finding that while participants preferred AI-generated messages, they expressed dispreference for messages that were labeled as AI-generated. The results suggest that, with human supervision, AI can be used to create effective public health messages, but that individuals prefer their public health messages to come from human institutions rather than AI sources. We propose best practices for assessing generative outputs of large language models in future social science research and ways health professionals can use AI systems to augment public health messaging.

    

   more » « less
5. Getting Meta: A Multimodal Approach for Detecting Unsafe Conversations within Instagram Direct Messages of Youth

   https://doi.org/10.1145/3579608  

   Ali, Shiza ; Razi, Afsaneh ; Kim, Seunghyun ; Alsoubai, Ashwaq ; Ling, Chen ; De Choudhury, Munmun ; Wisniewski, Pamela J. ; Stringhini, Gianluca ( April 2023 , Proceedings of the ACM on Human-Computer Interaction)

   Instagram, one of the most popular social media platforms among youth, has recently come under scrutiny for potentially being harmful to the safety and well-being of our younger generations. Automated approaches for risk detection may be one way to help mitigate some of these risks if such algorithms are both accurate and contextual to the types of online harms youth face on social media platforms. However, the imminent switch by Instagram to end-to-end encryption for private conversations will limit the type of data that will be available to the platform to detect and mitigate such risks. In this paper, we investigate which indicators are most helpful in automatically detecting risk in Instagram private conversations, with an eye on high-level metadata, which will still be available in the scenario of end-to-end encryption. Toward this end, we collected Instagram data from 172 youth (ages 13-21) and asked them to identify private message conversations that made them feel uncomfortable or unsafe. Our participants risk-flagged 28,725 conversations that contained 4,181,970 direct messages, including textual posts and images. Based on this rich and multimodal dataset, we tested multiple feature sets (metadata, linguistic cues, and image features) and trained classifiers to detect risky conversations. Overall, we found that the metadata features (e.g., conversation length, a proxy for participant engagement) were the best predictors of risky conversations. However, for distinguishing between risk types, the different linguistic and media cues were the best predictors. Based on our findings, we provide design implications for AI risk detection systems in the presence of end-to-end encryption. More broadly, our work contributes to the literature on adolescent online safety by moving toward more robust solutions for risk detection that directly takes into account the lived risk experiences of youth. 

   more » « less