More Like this

1. From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs

   Niu, Minxue; Jaiswal, Mimansa; Mower_Provost, Emily (September 2024, Interspeech)

   Training emotion recognition models has relied heavily on human annotated data, which present diversity, quality, and cost challenges. In this paper, we explore the potential of Large Language Models (LLMs), specifically GPT-4, in automating or assisting emotion annotation. We compare GPT-4 with supervised models and/or humans in three aspects: agreement with human annotations, alignment with human perception, and impact on model training. We find that common metrics that use aggregated human annotations as ground truth can underestimate GPT-4's performance, and our human evaluation experiment reveals a consistent preference for GPT-4 annotations over humans across multiple datasets and evaluators. Further, we investigate the impact of using GPT-4 as an annotation filtering process to improve model training. Together, our findings highlight the great potential of LLMs in emotion annotation tasks and underscore the need for refined evaluation methodologies. 

   more » « less
2. Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion

   Singh, Smriti; Caragea, Cornelia; Li, Junyi Jessy (June 2024, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies)

   Situations and events evoke emotions in humans, but to what extent do they inform the prediction of emotion detection models? This work investigates how well human-annotated emotion triggers correlate with features that models deemed salient in their prediction of emotions. First, we introduce a novel dataset EmoTrigger, consisting of 900 social media posts sourced from three different datasets; these were annotated by experts for emotion triggers with high agreement. Using EmoTrigger, we evaluate the ability of large language models (LLMs) to identify emotion triggers, and conduct a comparative analysis of the features considered important for these tasks between LLMs and fine-tuned models. Our analysis reveals that emotion triggers are largely not considered salient features for emotion prediction models, instead there is intricate interplay between various features and the task of emotion detection. 

   more » « less
3. Exploiting Annotators’ Typed Description of Emotion Perception to Maximize Utilization of Ratings for Speech Emotion Recognition

   https://doi.org/10.1109/ICASSP43922.2022.9746990  

   Chou, Huang-Cheng; Lin, Wei-Cheng; Lee, Chi-Chun; Busso, Carlos (May 2022, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022))
4. Neuroanatomical Correlates of Emotion-Related Impulsivity

   https://doi.org/10.1016/j.biopsych.2022.07.018  

   Elliott, Matthew V.; Esmail, Serajh A.S.; Weiner, Kevin S.; Johnson, Sheri L. (March 2023, Biological Psychiatry)
5. Unsupervised Extractive Summarization of Emotion Triggers

   Sosea, Tiberiu; Zhan, Hongli; Li, Junyi Jessy; Caragea, Cornelia (May 2023, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

   Understanding what leads to emotions during large-scale crises is important as it can provide groundings for expressed emotions and subsequently improve the understanding of ongoing disasters. Recent approaches trained supervised models to both detect emotions and explain emotion triggers (events and appraisals) via abstractive summarization. However, obtaining timely and qualitative abstractive summaries is expensive and extremely time-consuming, requiring highly-trained expert annotators. In time-sensitive, high-stake contexts, this can block necessary responses. We instead pursue unsupervised systems that extract triggers from text. First, we introduce CovidET-EXT, augmenting (Zhan et al., 2022)’s abstractive dataset (in the context of the COVID-19 crisis) with extractive triggers. Second, we develop new unsupervised learning models that can jointly detect emotions and summarize their triggers. Our best approach, entitled Emotion-Aware Pagerank, incorporates emotion information from external sources combined with a language understanding module, and outperforms strong baselines. We release our data and code at https://github.com/tsosea2/CovidET-EXT. 

   more » « less