More Like this

1. A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks

   https://doi.org/10.1007/s42761-021-00075-5  

   Keles, Umit ; Lin, Chujun ; Adolphs, Ralph ( September 2021 , Affective Science)

   null (Ed.)

   Abstract People spontaneously infer other people’s psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments. 

   more » « less
2. Comprehensive Social Trait Judgments From Faces in Autism Spectrum Disorder

   https://doi.org/10.1177/09567976231192236  

   Cao, Runnan ; Zhang, Na ; Yu, Hongbo ; Webster, Paula J. ; Paul, Lynn K. ; Li, Xin ; Lin, Chujun ; Wang, Shuo ( October 2023 , Psychological Science)

   Processing social information from faces is difficult for individuals with autism spectrum disorder (ASD). However, it remains unclear whether individuals with ASD make high-level social trait judgments from faces in the same way as neurotypical individuals. Here, we comprehensively addressed this question using naturalistic face images and representatively sampled traits. Despite similar underlying dimensional structures across traits, online adult participants with self-reported ASD showed different judgments and reduced specificity within each trait compared with neurotypical individuals. Deep neural networks revealed that these group differences were driven by specific types of faces and differential utilization of features within a face. Our results were replicated in well-characterized in-lab participants and partially generalized to more controlled face images (a preregistered study). By investigating social trait judgments in a broader population, including individuals with neurodevelopmental variations, we found important theoretical implications for the fundamental dimensions, variations, and potential behavioral consequences of social cognition.

    

   more » « less
3. The roles of inflammation, affect, and interoception in predicting social perception

   https://doi.org/10.1016/j.bbi.2023.05.011  

   Feldman, Mallory J. ; Jolink, Tatum A. ; Alvarez, Gabriella M. ; Fendinger, Nicholas J. ; Gaudier-Diaz, Monica M. ; Lindquist, Kristen A. ; Muscatell, Keely A. ( August 2023 , Brain, Behavior, and Immunity)

   “Sickness behavior” is an orchestrated suite of symptoms that commonly occur in the context of inflammation, and is characterized by changes in affect, social experience, and behavior. However, recent evidence suggests that inflammation may not always produce the same set of sickness behavior (e.g., fatigue, anhedonia, and social withdrawal). Rather, inflammation may be linked with different behavior across contexts and/or across in- dividuals, though research in this area is under-developed to-date. In the present study (n = 30), we evaluated the influence of affective context and individual differences in difficulty detecting bodily sensations (i.e., interoceptive difficulty) on social perception following an inflammatory challenge. Inflammation was induced using the influenza vaccine and inflammatory reactivity was operationalized as changes in circulating levels of interleukin-6 (IL-6) before the vaccine and approximately 24 h later. Twenty-four hours after administration of the influenza vaccine, we manipulated affective context using a well-validated affect misattribution task in which participants made trustworthiness judgments of individuals with neutral facial expressions following the rapid presentation of “prime” images that were positive or negative in affective content. Interoceptive difficulty was measured at baseline using a validated self-report measure. Results revealed significant interactions between inflammatory reactivity to the influenza vaccine and affective context on social perception. Specifically, in- dividuals with greater inflammatory reactivity were more biased by affective context when judging the trust- worthiness of neutral faces. In addition, interoceptive difficulty and affective context interacted to predict social perception such that individuals with greater interoceptive difficulty were more biased by affective context in these judgments. In sum, we provide some of the first evidence that inflammation may amplify the saliency of affective cues during social decision making. Our findings also replicate prior work linking interoceptive ability to the use of affect-as-information during social perception, but in the novel context of inflammation. 

   more » « less
4. Multimodal investigations of emotional face processing and social trait judgment of faces

   https://doi.org/10.1111/nyas.15084  

   Yu, Hongbo ; Lin, Chujun ; Sun, Sai ; Cao, Runnan ; Kar, Kohitij ; Wang, Shuo ( November 2023 , Annals of the New York Academy of Sciences)

   Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event‐related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single‐neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality‐dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in‐depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.

    

   more » « less
5. Atypical audiovisual temporal function in autism and schizophrenia: similar phenotype, different cause

   https://doi.org/10.1111/ejn.13911  

   Noel, Jean‐Paul ; Stevenson, Ryan A. ; Wallace, Mark T. ( April 2018 , European Journal of Neuroscience)

   Binding across sensory modalities yields substantial perceptual benefits, including enhanced speech intelligibility. The coincidence of sensory inputs across time is a fundamental cue for this integration process. Recent work has suggested that individuals with diagnoses of schizophrenia (SZ) and autism spectrum disorder (ASD) will characterize auditory and visual events as synchronous over larger temporal disparities than their neurotypical counterparts. Namely, these clinical populations possess an enlarged temporal binding window (TBW). Although patients withSZandASDshare aspects of their symptomatology, phenotypic similarities may result from distinct etiologies. To examine similarities and variances in audiovisual temporal function in these two populations, individuals diagnosed withASD(n = 46; controlsn = 40) andSZ(n = 16, controls = 16) completed an audiovisual simultaneity judgment task. In addition to standard psychometric analyses, synchrony judgments were assessed using Bayesian causal inference modeling. This approach permits distinguishing between distinct causes of an enlargedTBW: ana prioribias to bind sensory information and poor fidelity in the sensory representation. Findings indicate that bothASDandSZpopulations show deficits in multisensory temporal acuity. Importantly, results suggest that while the widerTBWs inASDmost prominently results from atypical priors, the widerTBWs inSZresults from a trend toward changes in prior and weaknesses in the sensory representations. Results are discussed in light of currentASDandSZtheories and highlight that different perceptual training paradigms focused on improving multisensory integration may be most effective in these two clinical populations and emphasize that similar phenotypes may emanate from distinct mechanistic causes.

    

   more » « less