Training emotion recognition models has relied heavily on human annotated data, which present diversity, quality, and cost challenges. In this paper, we explore the potential of Large Language Models (LLMs), specifically GPT-4, in automating or assisting emotion annotation. We compare GPT-4 with supervised models and/or humans in three aspects: agreement with human annotations, alignment with human perception, and impact on model training. We find that common metrics that use aggregated human annotations as ground truth can underestimate GPT-4's performance, and our human evaluation experiment reveals a consistent preference for GPT-4 annotations over humans across multiple datasets and evaluators. Further, we investigate the impact of using GPT-4 as an annotation filtering process to improve model training. Together, our findings highlight the great potential of LLMs in emotion annotation tasks and underscore the need for refined evaluation methodologies.
more »
« less
Exploiting Annotators’ Typed Description of Emotion Perception to Maximize Utilization of Ratings for Speech Emotion Recognition
- Award ID(s):
- 2016719
- PAR ID:
- 10345753
- Date Published:
- Journal Name:
- IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022)
- Page Range / eLocation ID:
- 7717 to 7721
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract Friends and therapists often encourage people in distress to say how they feel (i.e., name their emotions) with the hope that identifying their emotions will help them cope. Although lay and some psychological theories posit that emotion naming should facilitate subsequent emotion regulation, there is little research directly testing this question. Here, we report on two experimental studies that test how naming the emotions evoked by aversive images impacts subsequent regulation of those emotions. In study 1 (
N = 80), participants were randomly assigned into one of four between-subjects conditions in which they either (i) passively observed aversive images, (ii) named the emotions that these images made them feel, (iii) regulated their emotions by reappraising the meaning of images, or (iv) both named and regulated their emotions. Analyses of self-reported negative affect revealed that emotion naming impeded emotion regulation via reappraisal. Participants who named their emotions before reappraising reported feelingworse than those who regulated without naming. Study 2 (N = 60) replicated these findings in a within-participants design, demonstrated that emotion naming also impeded regulation via mindful acceptance, and showed that observed effects were unrelated to a measure of social desirability, thereby mitigating the concern of experimenter demand. Together, these studies show that the impact of emotion naming on emotion regulation opposes common intuitions: instead of facilitating emotion regulation via reappraisal or acceptance, constructing an instance of a specific emotion category by giving it a name may “crystalize” one’s affective experience and make it more resistant to modification. -
more » « less
In a secularizing world, religious groups are increasingly threatened by anti‐religious groups. We present two studies investigating religious peoples’ responses to anti‐religious threats. We expected intergroup threats to shape group‐based emotions and behavioural intentions through a novel pathway whereby threat affects group‐based meta‐emotions: the ingroup’s perception of the outgroup’s emotions towards the ingroup. In Study 1, we experimentally manipulated threat and group salience with participants from two different cultures (British and Latinx/Hispanic). Study 1 demonstrated non‐interactive effects of threat increasing negative emotional responses and of group salience strengthening emotional responses. The results illustrated the role of group‐based meta‐emotions in predicting outgroup‐directed emotions and behavioural response intentions. Study 2 used a different manipulation of threat in an American sample and an identity‐based manipulation of salience to assess the impact of real‐world anti‐religious campaigns involving symbolic and realistic threats. Both threat types increased negative group‐based meta‐emotions, negative outgroup‐directed emotions, desire to respond, and opposition to the anti‐religious campaign compared to no threat. Overall, religious identity salience had little impact on outcomes. The indirect pathway through meta‐emotion replicated, suggesting the importance of considering this novel meta‐emotion pathway in intergroup relations.
-
Situations and events evoke emotions in humans, but to what extent do they inform the prediction of emotion detection models? This work investigates how well human-annotated emotion triggers correlate with features that models deemed salient in their prediction of emotions. First, we introduce a novel dataset EmoTrigger, consisting of 900 social media posts sourced from three different datasets; these were annotated by experts for emotion triggers with high agreement. Using EmoTrigger, we evaluate the ability of large language models (LLMs) to identify emotion triggers, and conduct a comparative analysis of the features considered important for these tasks between LLMs and fine-tuned models. Our analysis reveals that emotion triggers are largely not considered salient features for emotion prediction models, instead there is intricate interplay between various features and the task of emotion detection.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ICASSP43922.2022.9746990
