More Like this

1. Sharing Photos on Social Media: Visual Attention Affects Real-World Decision Making

   https://doi.org/10.1007/978-3-030-79997-7_25  

   Fagan, Shawn E.; Wade, Lauren; Hugenberg, Kurt; Kapadia, Apu; Bertenthal, Bennett I. (January 2021, Advances in Human Factors in Robots, Unmanned Systems and Cybersecurity)

   null (Ed.)

   This study tested the effect of visual attention on decision-making in digital environments. Fifty-nine individuals were asked how likely they would be to share 40 memes (photos with superimposed captions) on social media while their eye movements were tracked. The likelihood of sharing memes increased as attention to the text of the meme increased; conversely, the likelihood of sharing decreased as attention to the image of the meme increased. In addition, increased trait levels of agreeableness predicted a greater likelihood of sharing memes. These results indicate that individual differences in personality and eye movements predict the likelihood of sharing photo-memes on social media platforms. 

   more » « less
2. Modeling User Characteristics Associated with Interdependent Privacy Perceptions on Social Media

   https://doi.org/10.1145/3577014  

   Amon, Mary Jean; Necaise, Aaron; Kartvelishvili, Nika; Williams, Aneka; Solihin, Yan; Kapadia, Apu (January 2023, ACM Transactions on Computer-Human Interaction)

   ‘Interdependent’ privacy violations occur when users share private photos and information about other people in social media without permission. This research investigated user characteristics associated with interdependent privacy perceptions, by asking social media users to rate photo-based memes depicting strangers on the degree to which they were too private to share. Users also completed questionnaires measuring social media usage and personality. Separate groups rated the memes on shareability, valence, and entertainment value. Users were less likely to share memes that were rated as private, except when the meme was entertaining or when users exhibited dark triad characteristics. Users with dark triad characteristics demonstrated a heightened awareness of interdependent privacy and increased sharing of others’ photos. A model is introduced that highlights user types and characteristics that correspond to different privacy preferences: privacy preservers, ignorers, and violators. We discuss how interventions to support interdependent privacy must effectively influence diverse users. 

   more » « less
3. Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence

   https://doi.org/10.1145/3511599  

   Singh, Nikhil; Bernal, Guillermo; Savchenko, Daria; Glassman, Elena L. (January 2022, ACM Transactions on Computer-Human Interaction)

   While developing a story, novices and published writers alike have had to look outside themselves for inspiration. Language models have recently been able to generate text fluently, producing new stochastic narratives upon request. However, effectively integrating such capabilities with human cognitive faculties and creative processes remains challenging. We propose to investigate this integration with a multimodal writing support interface that offers writing suggestions textually, visually, and aurally. We conduct an extensive study that combines elicitation of prior expectations before writing, observation and semi-structured interviews during writing, and outcome evaluations after writing. Our results illustrate individual and situational variation in machine-in-the-loop writing approaches, suggestion acceptance, and ways the system is helpful. Centrally, we report how participants perform integrative leaps , by which they do cognitive work to integrate suggestions of varying semantic relevance into their developing stories. We interpret these findings, offering modeling and design recommendations for future creative writing support technologies. 

   more » « less
4. From image to language and back again

   https://doi.org/10.1017/S1351324918000086  

   BELZ, A.; BERG, T.L.; YU, L. (May 2018, Natural Language Engineering)

   Work in computer vision and natural language processing involving images and text has been experiencing explosive growth over the past decade, with a particular boost coming from the neural network revolution. The present volume brings together five research articles from several different corners of the area: multilingual multimodal image description (Frank et al. ), multimodal machine translation (Madhyastha et al. , Frank et al. ), image caption generation (Madhyastha et al. , Tanti et al. ), visual scene understanding (Silberer et al. ), and multimodal learning of high-level attributes (Sorodoc et al. ). In this article, we touch upon all of these topics as we review work involving images and text under the three main headings of image description (Section 2), visually grounded referring expression generation (REG) and comprehension (Section 3), and visual question answering (VQA) (Section 4). 

   more » « less
5. Image and Text Feature Based Multimodal Learning for Multi-Label Classification of Radiology Images in Biomedical Literature [Image and Text Feature Based Multimodal Learning for Multi-Label Classification of Radiology Images in Biomedical Literature]

   https://doi.org/10.5220/0012438400003657  

   Hasan, Md; Jani, Md; Rahman, Md (January 2024, SCITEPRESS - Science and Technology Publications)

   Biomedical images are crucial for diagnosing and planning treatments, as well as advancing scientific understanding of various ailments. To effectively highlight regions of interest (RoIs) and convey medical concepts, annotation markers like arrows, letters, or symbols are employed. However, annotating these images with appropriate medical labels poses a significant challenge. In this study, we propose a framework that leverages multimodal input features, including text/label features and visual features, to facilitate accurate annotation of biomedical images with multiple labels. Our approach integrates state-of-the-art models such as ResNet50 and Vision Transformers (ViT) to extract informative features from the images. Additionally, we employ Generative Pre-trained Distilled-GPT2 (Transformer based Natural Language Processing architecture) to extract textual features, leveraging their natural language understanding capabilities. This combination of image and text modalities allows for a more comprehensive representation of the biomedical data, leading to improved annotation accuracy. By combining the features extracted from both image and text modalities, we trained a simplified Convolutional Neural Network (CNN) based multi-classifier to learn the image-text relations and predict multi-labels for multi-modal radiology images. We used ImageCLEFmedical 2022 and 2023 datasets to demonstrate the effectiveness of our framework. This dataset likely contains a diverse range of biomedical images, enabling the evaluation of the framework’s performance under realistic conditions. We have achieved promising results with the F1 score of 0.508. Our proposed framework exhibits potential performance in annotating biomedical images with multiple labels, contributing to improved image understanding and analysis in the medical image processing domain. 

   more » « less