More Like this

1. Neural Machine Translation of Text from Non-Native Speakers

   Anastasopoulos, Antonios; Lui, Alison; Nguyen, Toan Q.; Chiang, David (June 2019, Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics)

   Neural Machine Translation (NMT) systems are known to degrade when confronted with noisy data, especially when the system is trained only on clean data. In this paper, we show that augmenting training data with sentences containing artificially-introduced grammatical errors can make the system more robust to such errors. In combination with an automatic grammar error correction system, we can recover 1.0 BLEU out of 2.4 BLEU lost due to grammatical errors. We also present a set of Spanish translations of the JFLEG grammar error correction corpus, which allows for testing NMT robustness to real grammatical errors. 

   more » « less
2. BLiMP: The Benchmark of Linguistic Minimal Pairs for English

   https://doi.org/10.1162/tacl_a_00321  

   Warstadt, Alex; Parrish, Alicia; Liu, Haokun; Mohananey, Anhad; Peng, Wei; Wang, Sheng-Fu; Bowman, Samuel R. (December 2020, Transactions of the Association for Computational Linguistics)

   null (Ed.)

   We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP), 1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands. 

   more » « less
3. Augmented Spanish-Persian Neural Machine Translation [Augmented Spanish-Persian Neural Machine Translation]

   https://doi.org/10.5220/0010369804820488  

   Ahmadnia, Benyamin; Aranovich, Raul (January 2021, Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021))

   null (Ed.)

   Neural Machine Translation (NMT) performs training of a neural network employing an encoder-decoder architecture. However, the quality of the neural-based translations predominantly depends on the availability of a large amount of bilingual training dataset. In this paper, we explore the performance of translations predicted by attention-based NMT systems for Spanish to Persian low-resource language pairs. We analyze the errors of NMT systems that occur in the Persian language and provide an in-depth comparison of the performance of the system based on variations in sentence length and size of the training dataset. We evaluate our translation results using BLEU and human evaluation measures based on the adequacy, fluency, and overall rating. 

   more » « less
4. Can Synthetic Translations Improve Bitext Quality?

   https://doi.org/10.18653/v1/2022.acl-long.326  

   Briakou, Eleftheria; Carpuat, Marine (May 2022, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers))

   Synthetic translations have been used for a wide range of NLP tasks primarily as a means of data augmentation. This work explores, instead, how synthetic translations can be used to revise potentially imperfect reference translations in mined bitext. We find that synthetic samples can improve bitext quality without any additional bilingual supervision when they replace the originals based on a semantic equivalence classifier that helps mitigate NMT noise. The improved quality of the revised bitext is confirmed intrinsically via human evaluation and extrinsically through bilingual induction and MT tasks. 

   more » « less
5. Simul-LLM: A Framework for Exploring High-Quality Simultaneous Translation with Large Language Models

   Agostinelli, Victor; Wild, Max; Raffel, Matthew; Fuad, Kazi; Chen, Lizhong (August 2024, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL))

   Large language models (LLMs) with billions of parameters and pretrained on massive amounts of data are now capable of near or better than state-of-the-art performance in a variety of downstream natural language processing tasks. Neural machine translation (NMT) is one such task that LLMs have been applied to with great success. However, little research has focused on applying LLMs to the more difficult subset of NMT called simultaneous translation (SimulMT), where translation begins before the entire source context is available to the model. In this paper, we address key challenges facing LLMs fine-tuned for SimulMT, validate classical SimulMT concepts and practices in the context of LLMs, explore adapting LLMs that are fine-tuned for NMT to the task of SimulMT, and introduce Simul-LLM, the first open-source fine-tuning and evaluation pipeline development framework for LLMs focused on SimulMT. 

   more » « less