Title: Emotion as Information in Early Social Learning
The majority of research on infants’ and children’s understanding of emotional expressions has focused on their abilities to use emotional expressions to infer how other people feel. However, an emerging body of work suggests that emotional expressions support rich, powerful inferences not just about emotional states but also about other unobserved states, such as hidden events in the physical world and mental states of other people (e.g., beliefs and desires). Here we argue that infants and children harness others’ emotional expressions as a source of information for learning about the physical and social world broadly. This “emotion as information” framework integrates affective, developmental, and computational cognitive sciences, extending the scope of signals that count as “information” in early learning. more »« less
Wu, Yang; Tessler, Michael Henry; Asaba, Mike; Zhu, Peter; Gweon, Hyo; Frank, Michael C
(, Proceedings of the Annual Conference of the Cognitive Science Society)
null
(Ed.)
Human communication involves far more than words; speak- ers’ utterances are often accompanied by various kinds of emo- tional expressions. How do listeners represent and integrate these distinct sources of information to make communicative inferences? We first show that people, as listeners, integrate both verbal and emotional information when inferring true states of the world and others’ communicative goals, and then present computational models that formalize these inferences by considering different ways in which these signals might be generated. Results suggest that while listeners understand that utterances and emotional expressions are generated by a bal- ance of speakers’ informational and social goals, they addi- tionally consider the possibility that emotional expressions are noncommunicative signals that directly reflect the speaker’s in- ternal states. These results are consistent with the predictions of a probabilistic model that integrates goal inferences with linguistic and emotional signals, moving us towards a more complete formal theory of human communicative reasoning.
Jara-Ettinger, Julian; Schachner, Adena
(, Current Directions in Psychological Science)
How do humans build and navigate their complex social world? Standard theoretical frameworks often attribute this success to a foundational capacity to analyze other people’s appearance and behavior to make inferences about their unobservable mental states. Here we argue that this picture is incomplete. Human behavior leaves traces in our physical environment that reveal our presence, our goals, and even our beliefs and knowledge. A new body of research shows that, from early in life, humans easily detect these traces—sometimes spontaneously—and readily extract social information from the physical world. From the features and placement of inanimate objects, people make inferences about past events and how people have shaped the physical world. This capacity develops early and helps explain how people have such a rich understanding of others: by drawing not only on how others act but also on the environments they have shaped. Overall, social cognition is crucial not only to our reasoning about people and actions but also to our everyday reasoning about the inanimate world.
Hou, Tianyu; Adamo, Nicoletta; Villani, Nicholas
(, AHFE International)
The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman &Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., &Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., &Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., &Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/
Saha, Koustuv; Torous, John; Caine, Eric D; De Choudhury, Munmun
(, Journal of Medical Internet Research)
null
(Ed.)
Background The COVID-19 pandemic has caused several disruptions in personal and collective lives worldwide. The uncertainties surrounding the pandemic have also led to multifaceted mental health concerns, which can be exacerbated with precautionary measures such as social distancing and self-quarantining, as well as societal impacts such as economic downturn and job loss. Despite noting this as a “mental health tsunami”, the psychological effects of the COVID-19 crisis remain unexplored at scale. Consequently, public health stakeholders are currently limited in identifying ways to provide timely and tailored support during these circumstances. Objective Our study aims to provide insights regarding people’s psychosocial concerns during the COVID-19 pandemic by leveraging social media data. We aim to study the temporal and linguistic changes in symptomatic mental health and support expressions in the pandemic context. Methods We obtained about 60 million Twitter streaming posts originating from the United States from March 24 to May 24, 2020, and compared these with about 40 million posts from a comparable period in 2019 to attribute the effect of COVID-19 on people’s social media self-disclosure. Using these data sets, we studied people’s self-disclosure on social media in terms of symptomatic mental health concerns and expressions of support. We employed transfer learning classifiers that identified the social media language indicative of mental health outcomes (anxiety, depression, stress, and suicidal ideation) and support (emotional and informational support). We then examined the changes in psychosocial expressions over time and language, comparing the 2020 and 2019 data sets. Results We found that all of the examined psychosocial expressions have significantly increased during the COVID-19 crisis—mental health symptomatic expressions have increased by about 14%, and support expressions have increased by about 5%, both thematically related to COVID-19. We also observed a steady decline and eventual plateauing in these expressions during the COVID-19 pandemic, which may have been due to habituation or due to supportive policy measures enacted during this period. Our language analyses highlighted that people express concerns that are specific to and contextually related to the COVID-19 crisis. Conclusions We studied the psychosocial effects of the COVID-19 crisis by using social media data from 2020, finding that people’s mental health symptomatic and support expressions significantly increased during the COVID-19 period as compared to similar data from 2019. However, this effect gradually lessened over time, suggesting that people adapted to the circumstances and their “new normal.” Our linguistic analyses revealed that people expressed mental health concerns regarding personal and professional challenges, health care and precautionary measures, and pandemic-related awareness. This study shows the potential to provide insights to mental health care and stakeholders and policy makers in planning and implementing measures to mitigate mental health risks amid the health crisis.
Sosea, Tiberiu; Pham, Chau; Tekle, Alexander; Caragea, Cornelia; Li, Junyi Jessy
(, Proceedings of the Language Resources and Evaluation Conference)
Understanding emotions that people express during large-scale crises helps inform policy makers and first responders about the emotional states of the population as well as provide emotional support to those who need such support. We present CovidEmo, a dataset of ~3,000 English tweets labeled with emotions and temporally distributed across 18 months. Our analyses reveal the emotional toll caused by COVID-19, and changes of the social narrative and associated emotions over time. Motivated by the time-sensitive nature of crises and the cost of large-scale annotation efforts, we examine how well large pre-trained language models generalize across domains and timeline in the task of perceived emotion prediction in the context of COVID-19. Our analyses suggest that cross-domain information transfers occur, yet there are still significant gaps. We propose semi-supervised learning as a way to bridge this gap, obtaining significantly better performance using unlabeled data from the target domain.
Wu, Yang, Schulz, Laura E., Frank, Michael C., and Gweon, Hyowon. Emotion as Information in Early Social Learning. Retrieved from https://par.nsf.gov/biblio/10438202. Current Directions in Psychological Science 30.6 Web. doi:10.1177/09637214211040779.
Wu, Yang, Schulz, Laura E., Frank, Michael C., & Gweon, Hyowon. Emotion as Information in Early Social Learning. Current Directions in Psychological Science, 30 (6). Retrieved from https://par.nsf.gov/biblio/10438202. https://doi.org/10.1177/09637214211040779
@article{osti_10438202,
place = {Country unknown/Code not available},
title = {Emotion as Information in Early Social Learning},
url = {https://par.nsf.gov/biblio/10438202},
DOI = {10.1177/09637214211040779},
abstractNote = {The majority of research on infants’ and children’s understanding of emotional expressions has focused on their abilities to use emotional expressions to infer how other people feel. However, an emerging body of work suggests that emotional expressions support rich, powerful inferences not just about emotional states but also about other unobserved states, such as hidden events in the physical world and mental states of other people (e.g., beliefs and desires). Here we argue that infants and children harness others’ emotional expressions as a source of information for learning about the physical and social world broadly. This “emotion as information” framework integrates affective, developmental, and computational cognitive sciences, extending the scope of signals that count as “information” in early learning.},
journal = {Current Directions in Psychological Science},
volume = {30},
number = {6},
author = {Wu, Yang and Schulz, Laura E. and Frank, Michael C. and Gweon, Hyowon},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
An official website of the United States government
