An official website of the United States government
With the growing ubiquity of the Internet and access to media-based social media platforms, the risks associated with media content sharing on social media and the need for safety measures against such risks have grown paramount. At the same time, risk is highly contextualized, especially when it comes to media content youth share privately on social media. In this work, we conducted qualitative content analyses on risky media content flagged by youth participants and research assistants of similar ages to explore contextual dimensions of youth online risks. The contextual risk dimensions were then used to inform semi- and self-supervised state-of-the-art vision transformers to automate the process of identifying risky images shared by youth. We found that vision transformers are capable of learning complex image features for use in automated risk detection and classification. The results of our study serve as a foundation for designing contextualized and youth-centered machine-learning methods for automated online risk detection.
more »
« less
Systemization of Knowledge (SoK): Creating a Research Agenda for Human-Centered Real-Time Risk Detection on Social Media Platforms
Accurate real-time risk identification is vital to protecting social media users from online harm, which has driven research towards advancements in machine learning (ML). While strides have been made regarding the computational facets of algorithms for “real-time” risk detection, such research has not yet evaluated these advancements through a human-centered lens. To this end, we conducted a systematic literature review of 53 peer-reviewed articles on real-time risk detection on social media. Real-time detection was mainly operationalized as “early” detection after-the-fact based on pre-defined chunks of data and evaluated based on standard performance metrics, such as timeliness. We identified several human-centered opportunities for advancing current algorithms, such as integrating human insight in feature selection, algorithms’ improvement considering human behavior, and utilizing human evaluations. This work serves as a critical call-to-action for the HCI and ML communities to work together to protect social media users before, during, and after exposure to risks.
more »
« less
- PAR ID:
- 10564914
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400703300
- Page Range / eLocation ID:
- 1 to 21
- Format(s):
- Medium: X
- Location:
- Honolulu HI USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Ensuring the online safety of youth has motivated research towards the development of machine learning (ML) methods capable of accurately detecting social media risks after-the-fact. However, for these detection models to be effective, they must proactively identify high-risk scenarios (e.g., sexual solicitations, cyberbullying) to mitigate harm. This `real-time' responsiveness is a recognized challenge within the risk detection literature. Therefore, this paper presents a novel two-level framework that first uses reinforcement learning to identify conversation stop points to prioritize messages for evaluation. Then, we optimize state-of-the-art deep learning models to accurately categorize risk priority (low, high). We apply this framework to a time-based simulation using a rich dataset of 23K private conversations with over 7 million messages donated by 194 youth (ages 13-21). We conducted an experiment comparing our new approach to a traditional conversation-level baseline. We found that the timeliness of conversations significantly improved from over 2 hours to approximately 16 minutes with only a slight reduction in accuracy (0.88 to 0.84). This study advances real-time detection approaches for social media data and provides a benchmark for future training reinforcement learning that prioritizes the timeliness of classifying high-risk conversations.more » « less
-
In addressing various risks on social media, the HCI community has advocated for teen-centered risk detection technologies over platform-based, parent-centered features. However, their real-world viability remains underexplored by secondary stakeholders beyond the family unit. Therefore, we present an evaluation of a teen-centered social media risk detection dashboard through online interviews with 33 online safety experts. While experts praised our dashboard’s clear design for teen agency, their feedback revealed five primary tensions in implementing and sustaining such technology: objective vs. context-dependent risk definition, informing risks vs. meaningful intervention, teen empowerment vs. motivation, need for data vs. data privacy, and independence vs. sustainability. These findings motivate us to rethink "teen-centered" and a shift from a "fail fast" to a "mature safely" paradigm for youth safety technology innovation.We offer design implications for addressing these tensions before system deployment with teens and strategies for aligning secondary stakeholders’ interests to deploy and sustain such technologies in the broader ecosystem of youth online safety.more » « less
-
In addressing various risks on social media, the HCI community has advocated for teen-centered risk detection technologies over platform-based, parent-centered features. However, their real-world viability remains underexplored by secondary stakeholders beyond the family unit. Therefore, we present an evaluation of a teencentered social media risk detection dashboard through online interviews with 33 online safety experts. While experts praised our dashboard’s clear design for teen agency, their feedback revealed five primary tensions in implementing and sustaining such technology: objective vs. context-dependent risk definition, informing risks vs. meaningful intervention, teen empowerment vs. motivation, need for data vs. data privacy, and independence vs. sustainability. These findings motivate us to rethink “teen-centered” and a shift from a “fail fast” to a “mature safely” paradigm for youth safety technology innovation.We offer design implications for addressing these tensions before system deployment with teens and strategies for aligning secondary stakeholders’ interests to deploy and sustain such technologies in the broader ecosystem of youth online safety.more » « less
-
null (Ed.)Cyberbullying is a prevalent concern within social computing research that has led to the development of several supervised machine learning (ML) algorithms for automated risk detection. A critical aspect of ML algorithm development is how to establish ground truth that is representative of the phenomenon of interest in the real world. Often, ground truth is determined by third-party annotators (i.e., “outsiders”) who are removed from the situational context of the interaction; therefore, they cannot fully understand the perspective of the individuals involved (i.e., “insiders”). To understand the extent of this problem, we compare “outsider” versus “insider” perspectives when annotating 2,000 posts from an online peer-support platform. We interpolate this analysis to a corpus containing over 2.3 million posts on bullying and related topics, and reveal significant gaps in ML models that use third-party annotators to detect bullying incidents. Our results indicate that models based on the insiders’ perspectives yield a significantly higher recall in identifying bullying posts and are able to capture a range of explicit and implicit references and linguistic framings, including person-specific impressions of the incidents. Our study highlights the importance of incorporating the victim’s point of view in establishing effective tools for cyberbullying risk detection. As such, we advocate for the adoption of human-centered and value-sensitive approaches for algorithm development that bridge insider-outsider perspective gaps in a way that empowers the most vulnerable.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- https://doi.org/10.1145/3613904.3642315
