More Like this

1. Attention Dynamics During Emotion Recognition by Deaf and Hearing Individuals

   https://doi.org/10.1093/deafed/enz036  

   Krejtz, Izabela; Krejtz, Krzysztof; Wisiecka, Katarzyna; Abramczyk, Marta; Olszanowski, Michał; Duchowski, Andrew T (October 2019, The Journal of Deaf Studies and Deaf Education)

   Abstract The enhancement hypothesis suggests that deaf individuals are more vigilant to visual emotional cues than hearing individuals. The present eye-tracking study examined ambient–focal visual attention when encoding affect from dynamically changing emotional facial expressions. Deaf (n = 17) and hearing (n = 17) individuals watched emotional facial expressions that in 10-s animations morphed from a neutral expression to one of happiness, sadness, or anger. The task was to recognize emotion as quickly as possible. Deaf participants tended to be faster than hearing participants in affect recognition, but the groups did not differ in accuracy. In general, happy faces were more accurately and more quickly recognized than faces expressing anger or sadness. Both groups demonstrated longer average fixation duration when recognizing happiness in comparison to anger and sadness. Deaf individuals directed their first fixations less often to the mouth region than the hearing group. During the last stages of emotion recognition, deaf participants exhibited more focal viewing of happy faces than negative faces. This pattern was not observed among hearing individuals. The analysis of visual gaze dynamics, switching between ambient and focal attention, was useful in studying the depth of cognitive processing of emotional information among deaf and hearing individuals. 

   more » « less
2. Emerging Adults Mirror Infants’ Emotions and Yawns

   https://doi.org/10.1002/dev.22539  

   Leung, Tiffany_S; Zeng, Guangyu; Maylott, Sarah_E; Malik, Arushi; Zhang, Shuo; McNamara, Emily_C; Jakobsen, Krisztina_V; Simpson, Elizabeth_A (August 2024, Developmental Psychobiology)

   ABSTRACT Infants’ nonverbal expressions—a broad smile or a sharp cry—are powerful at eliciting reactions. Although parents’ reactions to their own infants’ expressions are relatively well understood, here we studied whether adults more generally exhibit behavioral and physiological reactions tounfamiliarinfants producing various expressions. We recruited U.S. emerging adults (N = 84) prior to parenthood, 18–25 years old, 68% women, ethnically (20% Hispanic/Latino) and racially (7% Asian, 13% Black, 1% Middle Eastern, 70% White, 8% multiracial) diverse. They observed four 80‐s audio–video clips of unfamiliar 2‐ to 6‐month‐olds crying, smiling, yawning, and sitting calmly (emotionally neutral control). Each compilation video depicted 9 different infants (36 clips total). We found adults mirrored behaviorally and physiologically: more positive facial expressions to infants smiling, and more negative facial expressions and pupil dilation—indicating increases in arousal—to infants crying. Adults also yawned more and had more pupil dilation when observing infants yawning. Together, these findings suggest that even nonparent emerging adults are highly sensitive to unfamiliar infants’ expressions, which they naturally “catch” (i.e., behaviorally and physiologically mirror), even without instructions. Such sensitivity may have—over the course of humans’ evolutionary history—been selected for, to facilitate adults’ processing of preverbal infants’ expressions to meet their needs. 

   more » « less
3. Micro-expressions in Animated Agents

   https://doi.org/10.54941/ahfe1001081  

   Hou, Tianyu; Adamo, Nicoletta; Villani, Nicholas (January 2022, AHFE International)

   The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman &Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., &Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., &Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., &Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/ 

   more » « less
4. Pupillary and microsaccadic responses to cognitive effort and emotional arousal during complex decision making

   https://doi.org/10.16910/jemr.13.5.2  

   Krejtz, Krzysztof; Żurawska, Justyna; Duchowski, Andrew; Wichary, Szymon (May 2020, Journal of eye movement research)

   A large body of literature documents the sensitivity of pupil response to cognitive load (e.g., Krejtz et al. 2018) and emotional arousal (Bradley et al., 2008). Recent empirical evidence also showed that microsaccade characteristics and dynamics can be modulated by mental fatigue and cognitive load (e.g., Dalmaso et al. 2017). Very little is known about the sensitivity of microsaccadic characteristics to emotional arousal. The present paper demonstrates in a controlled experiment pupillary and microsaccadic responses to information processing during multi-attribute decision making under affective priming. Twenty-one psychology students were randomly assigned into three affective priming conditions (neutral, aversive, and erotic). Participants were tasked to make several discriminative decisions based on acquired cues. In line with the expectations, results showed microsaccadic rate inhibition and pupillary dilation depending on cognitive effort (number of acquired cues) prior to decision. These effects were moderated by affective priming. Aversive priming strengthened pupillary and microsaccadic response to information processing effort. In general, results suggest that pupillary response is more biased by affective priming than microsaccadic rate. The results are discussed in the light of neuropsychological mechanisms of pupillary and microsaccadic behavior generation. 

   more » « less
5. Improving Facial Expression Analysis using Histograms of Log-Transformed Nonnegative Sparse Representation with a Spatial Pyramid Structure

   https://doi.org/10.1109/FG.2013.6553774  

   Ping Liu, Shizhong Han (April 2013, Proceedings of the International Conference on Automatic Face and Gesture Recognition)

   Facial activity is the most direct signal for perceiving emotional states in people. Emotion analysis from facial displays has been attracted an increasing attention because of its wide applications from human-centered computing to neuropsychiatry. Recently, image representation based on sparse coding has shown promising results in facial expression recognition. In this paper, we introduce a novel image representation for facial expression analysis. Specifically, we propose to use the histograms of nonnegative sparse coded image features to represent a facial image. In order to capture fine appearance variations caused by facial expression, logarithmic transformation is further employed on each nonnegative sparse coded feature. In addition, the proposed Histograms of Log-Transformed Nonnegative Sparse Coding (HLNNSC) features are calculated and organized in a pyramid-like structure such that the spatial relationships among the features are captured and utilized to enhance the performance of facial expression recognition. Extensive experiments on the Cohn-Kanade database show that the proposed approach yields a significant improvement in facial expression recognition and outperforms the other sparse coding based baseline approaches. Furthermore, experimental results on the GEMEP-FERA2011 dataset demonstrate that the proposed approach is promising for recognition under less controlled and thus more challenging environment. 

   more » « less