More Like this

1. Neural Network Acceptability Judgments

   https://doi.org/10.1162/tacl_a_00290  

   Warstadt, Alex; Singh, Amanpreet; Bowman, Samuel R. (March 2019, Transactions of the Association for Computational Linguistics)

   This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions. 

   more » « less
2. Multiple grammars within linguistic populations: Distributions and theoretical implications

   https://doi.org/10.1075/lab.24052.pol  

   Polinsky, Maria (July 2025, Linguistic Approaches to Bilingualism)

   Abstract This paper explores the concept of multiple grammars (MGs) and their implications for linguistic theory, language acquisition, and bilingual language knowledge. Drawing on evidence from phenomena such as scope interactions, verb raising, and agreement patterns, I argue that seemingly identical surface structures can be undergirded by different grammatical analyses that may compete within speaker populations. I then propose a typology of MG distributions, includingshared MGs, competing MGs,andpartial MGs, each with distinct consequences for acquisition and use. Contrary to expectations of simplification, bilingualism can sometimes lead to an expansion of grammatical analyses and does not always lead to the elimination of MGs. The paper discusses methods for predicting environments conducive to MGs, considering factors such as structural ambiguity and silent elements. The examination of MGs compels us to explore how learners navigate underdetermined input, especially in bilingual contexts, and to examine the interplay between gradient acceptability judgments and categorical grammatical distinctions. The study of MGs offers valuable insights into language variation, change, and the nature of linguistic competence. 

   more » « less
3. When Do You Need Billions of Words of Pretraining Data?

   Zhang, Yian; Warstadt, Alex; Li, Haau-Sing; Bowman, Samuel R. (January 2021, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics)

   null (Ed.)

   NLP is currently dominated by language models like RoBERTa which are pretrained on billions of words. But what exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? To explore this question, we adopt five styles of evaluation: classifier probing, information-theoretic probing, unsupervised relative acceptability judgments, unsupervised language model knowledge probing, and fine-tuning on NLU tasks. We then draw learning curves that track the growth of these different measures of model ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that these LMs require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features we test. They need a much larger quantity of data in order to acquire enough commonsense knowledge and other skills required to master typical downstream NLU tasks. The results suggest that, while the ability to encode linguistic features is almost certainly necessary for language understanding, it is likely that other, unidentified, forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models. 

   more » « less
4. TDLR: Top Semantic-Down Syntactic Language Representation

   Kaushik Roy, Vipula Rawte (December 2022, NeurIPS'22 Workshop on All Things Attention: Bridging Different Perspectives on Attention)

   Language understanding involves processing text with both the grammatical and 2 common-sense contexts of the text fragments. The text “I went to the grocery store 3 and brought home a car” requires both the grammatical context (syntactic) and 4 common-sense context (semantic) to capture the oddity in the sentence. Contex5 tualized text representations learned by Language Models (LMs) are expected to 6 capture a variety of syntactic and semantic contexts from large amounts of training 7 data corpora. Recent work such as ERNIE has shown that infusing the knowl8 edge contexts, where they are available in LMs, results in significant performance 9 gains on General Language Understanding (GLUE) benchmark tasks. However, 10 to our knowledge, no knowledge-aware model has attempted to infuse knowledge 11 through top-down semantics-driven syntactic processing (Eg: Common-sense to 12 Grammatical) and directly operated on the attention mechanism that LMs leverage 13 to learn the data context. We propose a learning framework Top-Down Language 14 Representation (TDLR) to infuse common-sense semantics into LMs. In our 15 implementation, we build on BERT for its rich syntactic knowledge and use the 16 knowledge graphs ConceptNet and WordNet to infuse semantic knowledge. 

   more » « less
5. Investigating representations of verb bias in neural language models

   https://doi.org/10.18653/v1/2020.emnlp-main.376  

   Hawkins, Robert D; Yamakoshi, Takateru; Griffiths, Thomas; Goldberg, Adele (January 2020, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP))

   null (Ed.)

   Languages typically provide more than one grammatical construction to express certain types of messages. A speaker’s choice of construction is known to depend on multiple factors, including the choice of main verb – a phenomenon known as verb bias. Here we introduce DAIS, a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. This dataset includes 200 unique verbs and systematically varies the definiteness and length of arguments. We use this dataset, as well as an existing corpus of naturally occurring data, to evaluate how well recent neural language models capture human preferences. Results show that larger models perform better than smaller models, and transformer architectures (e.g. GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under comparable parameter and training settings. Additional analyses of internal feature representations suggest that transformers may better integrate specific lexical information with grammatical constructions. 

   more » « less