More Like this

1. Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence

   Hoyle, A.; Goel, P.; Hian-Cheong. A.; Peskov, D.; Boyd-Graber, J.; & Resnik, P. (January 2021, Advances in neural information processing systems)

   Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, the field has coalesced around automated estimates of topic coherence, which rely on the frequency of word co-occurrences in a reference corpus. Contemporary neural topic models surpass classical ones according to these metrics. At the same time, topic model evaluation suffers from a validation gap: automated coherence, developed for classical models, has not been validated using human experimentation for neural models. In addition, a meta-analysis of topic modeling literature reveals a substantial standardization gap in automated topic modeling benchmarks. To address the validation gap, we compare automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion. To address the standardization gap, we systematically evaluate a dominant classical model and two state-of-the-art neural models on two commonly used datasets. Automated evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments. 

   more » « less
2. Automatic Evaluation of Local Topic Quality

   https://doi.org/10.18653/v1/P19-1076  

   Lund, Jeffrey; Armstrong, Piper; Fearn, Wilson; Cowley, Stephen; Byun, Courtni; Boyd-Graber, Jordan; Seppi, Kevin (January 2019, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics)

   null (Ed.)

   Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improve the quality of these token-level topic assignments, have been evaluated only with respect to global metrics. We propose a task designed to elicit human judgments of token-level topic assignments. We use a variety of topic model types and parameters and discover that global metrics agree poorly with human assignments. Since human evaluation is expensive we propose a variety of automated metrics to evaluate topic models at a local level. Finally, we correlate our proposed metrics with human judgments from the task on several datasets. We show that an evaluation based on the percent of topic switches correlates most strongly with human judgment of local topic quality. We suggest that this new metric, which we call consistency, be adopted alongside global metrics such as topic coherence when evaluating new topic models. 

   more » « less
3. Improving Neural Topic Models using Knowledge Distillation

   https://doi.org/10.18653/v1/2020.emnlp-main.137  

   Hoyle, Alexander Miserlis; Goel, Pranav; Resnik, Philip (November 2020, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP))

   null (Ed.)

   Abstract: Topic models are often used to identify human-interpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics. 

   more » « less
4. Which Evaluations Uncover Sense Representations that Actually Make Sense?

   Jordan Boyd-Graber, Fenfei Guo (May 2020, Proceedings of the 12th Language Resources and Evaluation Conference)

   null (Ed.)

   Text representations are critical for modern natural language processing. One form of text representation, sense-specific embeddings, reflect a word’s sense in a sentence better than single-prototype word embeddings tied to each type. However, existing sense representations are not uniformly better: although they work well for computer-centric evaluations, they fail for human-centric tasks like inspecting a language’s sense inventory. To expose this discrepancy, we propose a new coherence evaluation for sense embeddings. We also describe a minimal model (Gumbel Attention for Sense Induction) optimized for discovering interpretable sense representations that are more coherent than existing sense embeddings. 

   more » « less
5. SimQA: Detecting Simultaneous MT Errors through Word-by-Word Question Answering

   https://doi.org/10.18653/v1/2022.emnlp-main.378  

   Han, HyoJung; Carpuat, Marine; Boyd-Graber, Jordan (January 2022, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing)

   Detractors of neural machine translation admit that while its translations are fluent, it sometimes gets key facts wrong. This is particularly important in simultaneous interpretation where translations have to be provided as fast as possible: before a sentence is complete. Yet, evaluations of simultaneous machine translation (SimulMT) fail to capture if systems correctly translate the most salient elements of a question: people, places, and dates. To address this problem, we introduce a downstream word-by-word question answering evaluation task (SimQA): given a source language question, translate the question word by word into the target language, and answer as soon as possible. SimQA jointly measures whether the SimulMT models translate the question quickly and accurately, and can reveal shortcomings in existing neural systems—hallucinating or omitting facts. 

   more » « less