More Like this

1. Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects

   https://doi.org/10.1371/journal.pcbi.1011943  

   Liu, Peng; Bo, Ke; Ding, Mingzhou; Fang, Ruogu (March 2024, PLOS Computational Biology)

   Wei, Xue-Xin (Ed.)

   Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and lesioning these neurons by setting their output to zero or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories. 

   more » « less
2. Attention Drives Emotion: Voluntary Visual Attention Increases Perceived Emotional Intensity

   https://doi.org/10.1177/0956797619844231  

   Mrkva, Kellen; Westfall, Jacob; Van_Boven, Leaf (May 2019, Psychological Science)

   Attention and emotion are fundamental psychological systems. It is well established that emotion intensifies attention. Three experiments reported here ( N = 235) demonstrated the reversed causal direction: Voluntary visual attention intensifies perceived emotion. In Experiment 1, participants repeatedly directed attention toward a target object during sequential search. Participants subsequently perceived their emotional reactions to target objects as more intense than their reactions to control objects. Experiments 2 and 3 used a spatial-cuing procedure to manipulate voluntary visual attention. Spatially cued attention increased perceived emotional intensity. Participants perceived spatially cued objects as more emotionally intense than noncued objects even when participants were asked to mentally rehearse the name of noncued objects. This suggests that the intensifying effect of attention is independent of more extensive mental rehearsal. Across experiments, attended objects were perceived as more visually distinctive, which statistically mediated the effects of attention on emotional intensity. 

   more » « less
3. MuSE: a Multimodal Dataset of Stressed Emotion

   Mimansa Jaiswal, Cristian-Paul Bara (January 2020, Proceedings of the Twelfth Language Resources and Evaluation Conference)

   Endowing automated agents with the ability to provide support, entertainment and interaction with human beings requires sensing of the users’ affective state. These affective states are impacted by a combination of emotion inducers, current psychological state, and various conversational factors. Although emotion classification in both singular and dyadic settings is an established area, the effects of these additional factors on the production and perception of emotion is understudied. This paper presents a new dataset, Multimodal Stressed Emotion (MuSE), to study the multimodal interplay between the presence of stress and expressions of affect. We describe the data collection protocol, the possible areas of use, and the annotations for the emotional content of the recordings. The paper also presents several baselines to measure the performance of multimodal features for emotion and stress classification. 

   more » « less
4. Cooperating yet distinct brain networks engaged during naturalistic paradigms: A meta-analysis of functional MRI results

   https://doi.org/10.1162/netn_a_00050  

   Bottenhorn, Katherine L.; Flannery, Jessica S.; Boeving, Emily R.; Riedel, Michael C.; Eickhoff, Simon B.; Sutherland, Matthew T.; Laird, Angela R. (January 2019, Network Neuroscience)

   Cognitive processes do not occur by pure insertion and instead depend on the full complement of co-occurring mental processes, including perceptual and motor functions. As such, there is limited ecological validity to human neuroimaging experiments that use highly controlled tasks to isolate mental processes of interest. However, a growing literature shows how dynamic, interactive tasks have allowed researchers to study cognition as it more naturally occurs. Collective analysis across such neuroimaging experiments may answer broader questions regarding how naturalistic cognition is biologically distributed throughout the brain. We applied an unbiased, data-driven, meta-analytic approach that uses k-means clustering to identify core brain networks engaged across the naturalistic functional neuroimaging literature. Functional decoding allowed us to, then, delineate how information is distributed between these networks throughout the execution of dynamical cognition in realistic settings. This analysis revealed six recurrent patterns of brain activation, representing sensory, domain-specific, and attentional neural networks that support the cognitive demands of naturalistic paradigms. Although gaps in the literature remain, these results suggest that naturalistic fMRI paradigms recruit a common set of networks that allow both separate processing of different streams of information and integration of relevant information to enable flexible cognition and complex behavior. 

   more » « less
5. Social-Emotional-Sensory Design Map for Affective Computing Informed by Neurodivergent Experiences

   https://doi.org/10.1145/3449151  

   Zolyomi, Annuska; Snyder, Jaime (April 2021, Proceedings of the ACM on Human-Computer Interaction)

   One of the grand challenges of artificial intelligence and affective computing is for technology to become emotionally-aware and thus, more human-like. Modeling human emotions is particularly complicated when we consider the lived experiences of people who are on the autism spectrum. To understand the emotional experiences of autistic adults and their attitudes towards common representations of emotions, we deployed a context study as the first phase of a Grounded Design research project. Based on community observations and interviews, this work contributes empirical evidence of how the emotional experiences of autistic adults are entangled with social interactions as well as the processing of sensory inputs. We learned that (1) the emotional experiences of autistic adults are embodied and co-constructed within the context of physical environments, social relationships, and technology use, and (2) conventional approaches to visually representing emotion in affective education and computing systems fail to accurately represent the experiences and perceptions of autistic adults. We contribute a social-emotional-sensory design map to guide designers in creating more diverse and nuanced affective computing interfaces that are enriched by accounting for neurodivergent users. 

   more » « less