Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.
Although still‐face effects are well‐studied, little is known about the degree to which the Face‐to‐Face/Still‐Face (FFSF) is associated with the production of intense affective displays. Duchenne smiling expresses more intense positive affect than non‐Duchenne smiling, while Duchenne cry‐faces express more intense negative affect than non‐Duchenne cry‐faces. Forty 4‐month‐old infants and their mothers completed the FFSF, and key affect‐indexing facial Action Units (AUs) were coded by expert Facial Action Coding System coders for the first 30 s of each FFSF episode. Computer vision software, automated facial affect recognition (AFAR), identified AUs for the entire 2‐min episodes. Expert coding and AFAR produced similar infant and mother Duchenne and non‐Duchenne FFSF effects, highlighting the convergent validity of automated measurement. Substantive AFAR analyses indicated that both infant Duchenne and non‐Duchenne smiling declined from the FF to the SF, but only Duchenne smiling increased from the SF to the RE. In similar fashion, the magnitude of mother Duchenne smiling changes over the FFSF were 2–4 times greater than non‐Duchenne smiling changes. Duchenne expressions appear to be a sensitive index of intense infant and mother affective valence that are accessible to automated measurement and may be a target for future FFSF research.
more » « less- NSF-PAR ID:
- 10481984
- Publisher / Repository:
- Wiley-Blackwell
- Date Published:
- Journal Name:
- Infancy
- Volume:
- 28
- Issue:
- 5
- ISSN:
- 1525-0008
- Format(s):
- Medium: X Size: p. 910-929
- Size(s):
- ["p. 910-929"]
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
Cerebral blood flow in 5‐ to 8‐month‐olds: Regional tissue maturity is associated with infant affectmore » « less
Abstract Infancy is marked by rapid neural and emotional development. The relation between brain function and emotion in infancy, however, is not well understood. Methods for measuring brain function predominantly rely on the BOLD signal; however, interpretation of the BOLD signal in infancy is challenging because the neuronal‐hemodynamic relation is immature. Regional cerebral blood flow (rCBF) provides a context for the infant BOLD signal and can yield insight into the developmental maturity of brain regions that may support affective behaviors. This study aims to elucidate the relations among rCBF, age, and emotion in infancy. One hundred and seven mothers reported their infants' (infant age
M ±SD = 6.14 ± 0.51 months) temperament. A subsample of infants completed MRI scans, 38 of whom produced usable perfusion MRI during natural sleep to quantify rCBF. Mother‐infant dyads completed the repeated Still‐Face Paradigm, from which infant affect reactivity and recovery to stress were quantified. We tested associations of infant age at scan, temperament factor scores, and observed affect reactivity and recovery with voxel‐wise rCBF. Infant age was positively associated with CBF in nearly all voxels, with peaks located in sensory cortices and the ventral prefrontal cortex, supporting the formulation that rCBF is an indicator of tissue maturity. Temperamental Negative Affect and recovery of positive affect following a stressor were positively associated with rCBF in several cortical and subcortical limbic regions, including the orbitofrontal cortex and inferior frontal gyrus. This finding yields insight into the nature of affective neurodevelopment during infancy. Specifically, infants with relatively increased prefrontal cortex maturity may evidence a disposition toward greater negative affect and negative reactivity in their daily lives yet show better recovery of positive affect following a social stressor. -
more » « less
Abstract Infants’ prelinguistic vocalizations reliably organize vocal turn‐taking with social partners, creating opportunities for learning to produce the sound patterns of the ambient language. This social feedback loop supporting early vocal learning is well‐documented, but its developmental origins have yet to be addressed. When do infants learn that their non‐cry vocalizations influence others? To test developmental changes in infant vocal learning, we assessed the vocalizations of 2‐ and 5‐month‐old infants in a still‐face interaction with an unfamiliar adult. During the still‐face, infants who have learned the social efficacy of vocalizing increase their babbling rate. In addition, to assess the expectations for social responsiveness that infants build from their everyday experience, we recorded caregiver responsiveness to their infants’ vocalizations during unstructured play. During the still‐face, only 5‐month‐old infants showed an increase in vocalizing (a vocal extinction burst) indicating that they had learned to expect adult responses to their vocalizations. Caregiver responsiveness predicted the magnitude of the vocal extinction burst for 5‐month‐olds. Because 5‐month‐olds show a vocal extinction burst with unfamiliar adults, they must have generalized the social efficacy of their vocalizations beyond their familiar caregiver. Caregiver responsiveness to infant vocalizations during unstructured play was similar for 2‐ and 5‐month‐olds. Infants thus learn the social efficacy of their vocalizations between 2 and 5 months of age. During this time, infants build associations between their own non‐cry sounds and the reactions of adults, which allows learning of the instrumental value of vocalizing.
-
Face registration is a major and critical step for face analysis. Existing facial activity recognition systems often employ coarse face alignment based on a few fiducial points such as eyes and extract features from equal-sized grid. Such extracted features are susceptible to variations in face pose, facial deformation, and person-specific geometry. In this work, we propose a novel face registration method named facial grid transformation to improve feature extraction for recognizing facial Action Units (AUs). Based on the transformed grid, novel grid edge features are developed to capture local facial motions related to AUs. Extensive experiments on two wellknown AU-coded databases have demonstrated that the proposed method yields significant improvements over the methods based on equal-sized grid on both posed and more importantly, spontaneous facial displays. Furthermore, the proposed method also outperforms the state-of-the-art methods using either coarse alignment or mesh-based face registration.more » « less
-
null (Ed.)Abstract People spontaneously infer other people’s psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments.more » « less
