More Like this

1. Automatic Data Acquisition for Event Coreference Resolution

   https://doi.org/10.18653/v1/2021.eacl-main.101  

   Choubey, Prafulla Kumar; Huang, Ruihong (January 2021, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume)
2. Linear Cross-document Event Coreference Resolution with X-AMR

   Ahmed, S; M; Baker, G; Judge, E; Reagan, M; Wright-Bettner, K; Palmer, M; Martin, J (May 2024, ELRA and ICCL)

   Calzolari, N; Kan, M; Hoste, V; Lenci, A; Sakti, S; Xue, N (Ed.)

   Event Coreference Resolution (ECR) as a pairwise mention classification task is expensive both for automated systems and manual annotations. The task`s quadratic difficulty is exacerbated when using Large Language Models (LLMs), making prompt engineering for ECR prohibitively costly. In this work, we propose a graphical representation of events, X-AMR, anchored around individual mentions using a cross-document version of Abstract Meaning Representation. We then linearize the ECR with a novel multi-hop coreference algorithm over the event graphs. The event graphs simplify ECR, making it a) LLM cost-effective, b) compositional and interpretable, and c) easily annotated. For a fair assessment, we first enrich an existing ECR benchmark dataset with these event graphs using an annotator-friendly tool we introduce. Then, we employ GPT-4, the newest LLM by OpenAI, for these annotations. Finally, using the ECR algorithm, we assess GPT-4 against humans and analyze its limitations. Through this research, we aim to advance the state-of-the-art for efficient ECR and shed light on the potential shortcomings of current LLMs at this task. Code and annotations: https://github.com/ahmeshaf/gpt_coref 

   more » « less
3. Identifying the Most Dominant Event in a News Article by Mining Event Coreference Relations

   https://doi.org/10.18653/v1/N18-2055  

   Choubey, Prafulla Kumar; Raju, Kaushik; Huang, Ruihong (January 2018, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers))
4. Cross-Document Event Coreference Resolution: Instruct Humans or Instruct GPT?

   https://doi.org/10.18653/v1/2023.conll-1.38  

   Zhao, Jin; Xue, Nianwen; Min, Bonan (December 2023, Association for Computational Linguistics)

   Jiang, Jing; Reitter, David; Deng, Shumin (Ed.)

   This paper explores utilizing Large Language Models (LLMs) to perform Cross-Document Event Coreference Resolution (CDEC) annotations and evaluates how they fare against human annotators with different levels of training. Specifically, we formulate CDEC as a multi-category classification problem on pairs of events that are represented as decontextualized sentences, and compare the predictions of GPT-4 with the judgment of fully trained annotators and crowdworkers on the same data set. Our study indicates that GPT-4 with zero-shot learning outperformed crowd-workers by a large margin and exhibits a level of performance comparable to trained annotators. Upon closer analysis, GPT-4 also exhibits tendencies of being overly confident, and force annotation decisions even when such decisions are not warranted due to insufficient information. Our results have implications on how to perform complicated annotations such as CDEC in the age of LLMs, and show that the best way to acquire such annotations might be to combine the strengths of LLMs and trained human annotators in the annotation process, and using untrained or undertrained crowdworkers is no longer a viable option to acquire high-quality data to advance the state of the art for such problems. 

   more » « less
5. Integrated Annotation of Event Structure, Object States, and Entity Coreference

   https://doi.org/10.18653/v1/2023.crac-main.9  

   Rim, Kyeongmin; Pustejovsky, James (January 2023, Association for Computational Linguistics)