More Like this

1. A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks

   https://doi.org/10.1007/s42761-021-00075-5  

   Keles, Umit; Lin, Chujun; Adolphs, Ralph (September 2021, Affective Science)

   null (Ed.)

   Abstract People spontaneously infer other people’s psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments. 

   more » « less
2. Infants' salivary oxytocin and positive affective reactions to people

   https://doi.org/10.1016/j.yhbeh.2024.105579  

   Zeng, Guangyu; Leung, Tiffany S; Maylott, Sarah E; Malik, Arushi; Adornato, Alexis A; Lebowitz, Mendel; Messinger, Daniel S; Szeto, Angela; Feldman, Ruth; Simpson, Elizabeth A (August 2024, Hormones and Behavior)

   Oxytocin is a neuropeptide positively associated with prosociality in adults. Here, we studied whether infants' salivary oxytocin can be reliably measured, is developmentally stable, and is linked to social behavior. We longitudinally collected saliva from 62 U.S. infants (44 % female, 56 % Hispanic/Latino, 24 % Black, 18 % non-Hispanic White, 11 % multiracial) at 4, 8, and 14 months of age and offline-video-coded the valence of their facial affect in response to a video of a smiling woman. We also captured infants' affective reactions in terms of excitement/joyfulness during a live, structured interaction with a singing woman in the Early Social Communication Scales at 14 months. We detected stable individual differences in infants' oxytocin levels over time (over minutes and months) and in infants' positive affect over months and across contexts (video-based and in live interactions). We detected no statistically significant changes in oxytocin levels between 4 and 8 months but found an increase from 8 to 14 months. Infants with higher oxytocin levels showed more positive facial affect to a smiling person video at 4 months; however, this association disappeared at 8 months, and reversed at 14 months (i.e., higher oxytocin was associated with less positive facial affect). Infant salivary oxytocin may be a reliable physiological measure of individual differences related to socio-emotional development. 

   more » « less
3. AudiFace: Multimodal Deep Learning for Depression Screening

   Ricardo Flores, ML Tlachac (December 2022, Proceedings of Machine Learning Research)

   Depression is a very common mental health disorder with a devastating social and economic impact. It can be costly and difficult to detect, traditionally requiring a significant number of hours by a trained mental health professional. Recently, machine learning and deep learning models have been trained for depression screening using modalities extracted from videos of clinical interviews conducted by a virtual agent. This complex task is challenging for deep learning models because of the multiple modalities and limited number of participants in the dataset. To address these challenges we propose AudiFace, a multimodal deep learning model that inputs temporal facial features, audio, and transcripts to screen for depression. To incorporate all three modalities, AudiFace combines multiple pre-trained transfer learning models and bidirectional LSTM with self-Attention. When compared with the state-of-the-art models, AudiFace achieves the highest F1 scores for thirteen of the fifteen different datasets. AudiFace notably improves the depression screening capabilities of general wellbeing questions. Eye gaze proved to be the most valuable of the temporal facial features, both in the unimodal and multimodal models. Our results can be used to determine the best combination of modalities, temporal facial features, as well as clinical interview questions for future depression screening applications. 

   more » « less
4. Dynamic Mode Decomposition with Control as a Model of Multimodal Behavioral Coordination

   https://doi.org/10.1145/3462244.3479916  

   Klein, Lauren; Ardulov, Victor; Gharib, Alma; Thompson, Barbara; Levitt, Pat; Matarić, Maja J. (October 2021, ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction)

   Observing how infants and mothers coordinate their behaviors can highlight meaningful patterns in early communication and infant development. While dyads often differ in the modalities they use to communicate, especially in the first year of life, it remains unclear how to capture coordination across multiple types of behaviors using existing computational models of interpersonal synchrony. This paper explores Dynamic Mode Decomposition with control (DMDc) as a method of integrating multiple signals from each communicating partner into a model of multimodal behavioral coordination. We used an existing video dataset to track the head pose, arm pose, and vocal fundamental frequency of infants and mothers during the Face-to-Face Still-Face (FFSF) procedure, a validated 3-stage interaction paradigm. For each recorded interaction, we fit both unimodal and multimodal DMDc models to the extracted pose data. The resulting dynamic characteristics of the models were analyzed to evaluate trends in individual behaviors and dyadic processes across infant age and stages of the interactions. Results demonstrate that observed trends in interaction dynamics across stages of the FFSF protocol were stronger and more significant when models incorporated both head and arm pose data, rather than a single behavior modality. Model output showed significant trends across age, identifying changes in infant movement and in the relationship between infant and mother behaviors. Models that included mothers’ audio data demonstrated similar results to those evaluated with pose data, confirming that DMDc can leverage different sets of behavioral signals from each interacting partner. Taken together, our results demonstrate the potential of DMDc toward integrating multiple behavioral signals into the measurement of multimodal interpersonal coordination. 

   more » « less
5. Multimodal Depression Classification using Articulatory Coordination Features and Hierarchical Attention Based text Embeddings

   https://doi.org/10.1109/ICASSP43922.2022.9747462  

   Seneviratne, Nadee; Espy-Wilson, Carol (May 2022, International Conference on Acoustics, Speech and Signal Processing)

   Multimodal depression classification has gained immense popularity over the recent years. We develop a multimodal depression classification system using articulatory coordination features extracted from vocal tract variables and text transcriptions obtained from an automatic speech recognition tool that yields improvements of area under the receiver operating characteristics curve compared to unimodal classifiers (7.5% and 13.7% for audio and text respectively). We show that in the case of limited training data, a segment-level classifier can first be trained to then obtain a session-wise prediction without hindering the performance, using a multi-stage convolutional recurrent neural network. A text model is trained using a Hierarchical Attention Network (HAN). The multimodal system is developed by combining embeddings from the session-level audio model and the HAN text model. 

   more » « less