More Like this

1. The unappreciated role of intent in algorithmic moderation of abusive content on social media

   https://doi.org/10.37016/mr-2020-180  

   Wang, Xinyu; Koneru, Sai; Venkit, Pranav Narayanan; Frischmann, Brett; Rajtmajer, Sarah (July 2025, Harvard Kennedy School Misinformation Review)

   A significant body of research is dedicated to developing language models that can detect various types of online abuse, for example, hate speech, cyberbullying. However, there is a disconnect between platform policies, which often consider the author's intention as a criterion for content moderation, and the current capabilities of detection models, which typically lack efforts to capture intent. This paper examines the role of intent in the moderation of abusive content. Specifically, we review state-of-the-art detection models and benchmark training datasets to assess their ability to capture intent. We propose changes to the design and development of automated detection and moderation systems to improve alignment with ethical and policy conceptualizations of these abuses. 

   more » « less
2. Online hate network spreads malicious COVID-19 content outside the control of individual social media platforms

   https://doi.org/10.1038/s41598-021-89467-y  

   Velásquez, N.; Leahy, R.; Restrepo, N. Johnson; Lupu, Y.; Sear, R.; Gabriel, N.; Jha, O. K.; Goldberg, B.; Johnson, N. F. (June 2021, Scientific Reports)

   Abstract We show that malicious COVID-19 content, including racism, disinformation, and misinformation, exploits the multiverse of online hate to spread quickly beyond the control of any individual social media platform. We provide a first mapping of the online hate network across six major social media platforms. We demonstrate how malicious content can travel across this network in ways that subvert platform moderation efforts. Machine learning topic analysis shows quantitatively how online hate communities are sharpening COVID-19 as a weapon, with topics evolving rapidly and content becoming increasingly coherent. Based on mathematical modeling, we provide predictions of how changes to content moderation policies can slow the spread of malicious content. 

   more » « less
3. Hate Raids on Twitch: Understanding Real-Time Human-Bot Coordinated Attacks in Live Streaming Communities

   https://doi.org/10.1145/3610191  

   Cai, Jie; Chowdhury, Sagnik; Zhou, Hongyang; Wohn, Donghee Yvette (September 2023, Proceedings of the ACM on Human-Computer Interaction)

   Online harassment and content moderation have been well-documented in online communities. However, new contexts and systems always bring new ways of harassment and need new moderation mechanisms. This study focuses on hate raids, a form of group attack in real-time in live streaming communities. Through a qualitative analysis of hate raids discussion in the Twitch subreddit (r/Twitch), we found that (1) hate raids as a human-bot coordinated group attack leverages the live stream system to attack marginalized streamers and other potential groups with(out) breaking the rules, (2) marginalized streamers suffer compound harms with insufficient support from the platform, (3) moderation strategies are overwhelmingly technical, but streamers still struggle to balance moderation and participation considering their marginalization status and needs. We use affordances as a lens to explain how hate raids happens in live streaming systems and propose moderation-by-design as a lens when developing new features or systems to mitigate the potential abuse of such designs. 

   more » « less
4. BotPercent: Estimating Bot Populations in Twitter Communities

   https://doi.org/10.18653/v1/2023.findings-emnlp.954  

   Tan, Zhaoxuan; Feng, Shangbin; Sclar, Melanie; Wan, Herun; Luo, Minnan; Choi, Yejin; Tsvetkov, Yulia (January 2023, Association for Computational Linguistics)

   Twitter bot detection is vital in combating misinformation and safeguarding the integrity of social media discourse. While malicious bots are becoming more and more sophisticated and personalized, standard bot detection approaches are still agnostic to social environments (henceforth, communities) the bots operate at. In this work, we introduce community-specific bot detection, estimating the percentage of bots given the context of a community. Our method{---}BotPercent{---}is an amalgamation of Twitter bot detection datasets and feature-, text-, and graph-based models, adjusted to a particular community on Twitter. We introduce an approach that performs confidence calibration across bot detection models, which addresses generalization issues in existing community-agnostic models targeting individual bots and leads to more accurate community-level bot estimations. Experiments demonstrate that BotPercent achieves state-of-the-art performance in community-level Twitter bot detection across both balanced and imbalanced class distribution settings, presenting a less biased estimator of Twitter bot populations within the communities we analyze. We then analyze bot rates in several Twitter groups, including users who engage with partisan news media, political communities in different countries, and more. Our results reveal that the presence of Twitter bots is not homogeneous, but exhibiting a spatial-temporal distribution with considerable heterogeneity that should be taken into account for content moderation and social media policy making. 

   more » « less
5. Hate Speech Classifiers Learn Normative Social Stereotypes

   https://doi.org/10.1162/tacl_a_00550  

   Davani, Aida Mostafazadeh; Atari, Mohammad; Kennedy, Brendan; Dehghani, Morteza (March 2023, Transactions of the Association for Computational Linguistics)

   Abstract Social stereotypes negatively impact individuals’ judgments about different groups and may have a critical role in understanding language directed toward marginalized groups. Here, we assess the role of social stereotypes in the automated detection of hate speech in the English language by examining the impact of social stereotypes on annotation behaviors, annotated datasets, and hate speech classifiers. Specifically, we first investigate the impact of novice annotators’ stereotypes on their hate-speech-annotation behavior. Then, we examine the effect of normative stereotypes in language on the aggregated annotators’ judgments in a large annotated corpus. Finally, we demonstrate how normative stereotypes embedded in language resources are associated with systematic prediction errors in a hate-speech classifier. The results demonstrate that hate-speech classifiers reflect social stereotypes against marginalized groups, which can perpetuate social inequalities when propagated at scale. This framework, combining social-psychological and computational-linguistic methods, provides insights into sources of bias in hate-speech moderation, informing ongoing debates regarding machine learning fairness. 

   more » « less