More Like this

1. Expanding Russian PropBank: Challenges and Insights for Developing new SRL Resources

   Myers, Skatje; Khamov, Roman; Pollins, Adam; Tozier, Rebekah; Babko-Malaya, Olga; Palmer, Martha (May 2024, ELRA and ICCL)

   Bonial, Claire; Bonn, Julia; Hwang, Jena D (Ed.)

   Semantic role labeling (SRL) resources, such as Proposition Bank (PropBank), provide useful input to downstream applications. In this paper we present some challenges and insights we learned while expanding the previously developed Russian PropBank. This new effort involved annotation and adjudication of all predicates within a subset of the prior work in order to provide a test corpus for future applications. We discuss a number of new issues that arose while developing our PropBank for Russian as well as our solutions. Framing issues include: distinguishing between morphological processes that warrant new frames, differentiating between modal verbs and predicate verbs, and maintaining accurate representations of a given language’s semantics. Annotation issues include disagreements derived from variability in Universal Dependency parses and semantic ambiguity within the text. Finally, we demonstrate how Russian sentence structures reveal inherent limitations to PropBank’s ability to capture semantic data. These discussions should prove useful to anyone developing a PropBank or similar SRL resources for a new language. 

   more » « less
2. Russian PropBank

   Moeller, Sarah; Wagner, Irina; Palmer, Martha; Conger, Kathryn; Myers, Skatje (May 2020, Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020),)

   This paper presents a proposition bank for Russian (RuPB), a resource for semantic role labeling (SRL). The motivating goal for this resource is to automatically project semantic role labels from English to Russian. This paper describes frame creation strategies, coverage, and the process of sense disambiguation. It discusses language-specific issues that complicated the process of building the PropBank and how these challenges were exploited as language-internal guidance for consistency and coherence. 

   more » « less
3. Mapping PropBank Argument Labels to Czech Verbal Valency

   Hajič, Jan; Fučíková, Eva; Lopatkova, Marketa; Urešová, Zdeňka (May 2024, ELRA and ICCL)

   Bonial, Claire; Bonn, Julia; Hwang, Jena D (Ed.)

   For many years, there has been attempts to compare predicate-argument labeling schemas between formalism, typically under the dependency assumptions (even if the annotation by these schemas could have been performed on either constituent-based specifications or dependency ones). Given the growing number of resources that link various lexical resources to one another, as well as thanks to parallel annotated corpora (with or without annotation), it is now possible to do more in-depth studies of those correspondences. We present here a high-coverage pilot study of mapping the labeling system used in PropBank (for English) to Czech, which has so far used mainly valency lexicons (in several closely related forms) for annotation projects, under a different level of specification and different theoretical assumptions. The purpose of this study is both theoretical (comparing the argument labeling schemes) and practical (to be able to annotate Czech under the standard UMR specifications). 

   more » « less
4. ClearTAC: Verb Tense, Aspect, and Form Classification Using Neural Nets

   Myers, Skatje; Palmer, Martha (July 2019, Proceedings of the 1st Designing Meaning Representations Workshop, DMR-2019, held in conjunction with ACL,)

   This paper proposes using a Bidirectional LSTM-CRF model in order to identify the tense and aspect of verbs. The information that this classifier outputs can be useful for ordering events and can provide a pre-processing step to improve efficiency of annotating this type of information. This neural network architecture has been successfully employed for other sequential labeling tasks, and we show that it significantly outperforms the rule-based tool TMV-annotator on the PropBank I dataset. 

   more » « less
5. Adjudicating LLMs as PropBank Adjudicators

   Bonn, Julia; Madabushi, Harish Tayyar; Hwang, Jena D; Bonial, Claire (May 2024, ELRA and ICCL)

   Bonial, Claire; Bonn, Julia; Hwang, Jena D (Ed.)

   We evaluate the ability of large language models (LLMs) to provide PropBank semantic role label annotations across different realizations of the same verbs in transitive, intransitive, and middle voice constructions. In order to assess the meta-linguistic capabilities of LLMs as well as their ability to glean such capabilities through in-context learning, we evaluate the models in a zero-shot setting, in a setting where it is given three examples of another verb used in transitive, intransitive, and middle voice constructions, and finally in a setting where it is given the examples as well as the correct sense and roleset information. We find that zero-shot knowledge of PropBank annotation is almost nonexistent. The largest model evaluated, GPT-4, achieves the best performance in the setting where it is given both examples and the correct roleset in the prompt, demonstrating that larger models can ascertain some meta-linguistic capabilities through in-context learning. However, even in this setting, which is simpler than the task of a human in PropBank annotation, the model achieves only 48% accuracy in marking numbered arguments correctly. To ensure transparency and reproducibility, we publicly release our dataset and model responses. 

   more » « less