More Like this

1. What Can String Probability Tell Us About Grammaticality?

   https://doi.org/10.1162/TACL.a.611  

   Hu, Jennifer; Wilcox, Ethan Gotlieb; Song, Siyuan; Mahowald, Kyle; Levy, Roger P (January 2026, Transactions of the Association for Computational Linguistics)

   Abstract What have language models (LMs) learned about grammar? This question remains hotly debated, with major ramifications for linguistic theory. However, since probability and grammaticality are distinct notions in linguistics, it is not obvious what string probabilities can reveal about an LM’s underlying grammatical knowledge. We present a theoretical analysis of the relationship between grammar, meaning, and string probability, based on simple assumptions about the generative process of corpus data. Our framework makes three predictions, which we validate empirically using 280K sentence pairs in English and Chinese: (1) correlation between the probability of strings within minimal pairs, i.e., string pairs with minimal semantic differences; (2) correlation between models’ and humans’ deltas within minimal pairs; and (3) poor separation in probability space between unpaired grammatical and ungrammatical strings. Our analyses give theoretical grounding for using probability to learn about LMs’ structural knowledge, and suggest directions for future work in LM grammatical evaluation. 

   more » « less
2. How Linguistics Learned to Stop Worrying and Love the Language Models

   https://doi.org/10.1017/S0140525X2510112X  

   Futrell, Richard; Mahowald, Kyle (July 2025, Behavioral and Brain Sciences)

   Abstract Language models can produce fluent, grammatical text. Nonetheless, some maintain that language models don’t really learn language and also that, even if they did, that would not be informative for the study of human learning and processing. On the other side, there have been claims that the success of LMs obviates the need for studying linguistic theory and structure. We argue that both extremes are wrong. LMs can contribute to fundamental questions about linguistic structure, language processing, and learning. They force us to rethink arguments and ways of thinking that have been foundational in linguistics. While they do not replace linguistic structure and theory, they serve as model systems and working proofs of concept for gradient, usage-based approaches to language. We offer an optimistic take on the relationship between language models and linguistics. 

   more » « less
3. Neural Network Acceptability Judgments

   https://doi.org/10.1162/tacl_a_00290  

   Warstadt, Alex; Singh, Amanpreet; Bowman, Samuel R. (March 2019, Transactions of the Association for Computational Linguistics)

   This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions. 

   more » « less
4. Multiple grammars within linguistic populations: Distributions and theoretical implications

   https://doi.org/10.1075/lab.24052.pol  

   Polinsky, Maria (July 2025, Linguistic Approaches to Bilingualism)

   Abstract This paper explores the concept of multiple grammars (MGs) and their implications for linguistic theory, language acquisition, and bilingual language knowledge. Drawing on evidence from phenomena such as scope interactions, verb raising, and agreement patterns, I argue that seemingly identical surface structures can be undergirded by different grammatical analyses that may compete within speaker populations. I then propose a typology of MG distributions, includingshared MGs, competing MGs,andpartial MGs, each with distinct consequences for acquisition and use. Contrary to expectations of simplification, bilingualism can sometimes lead to an expansion of grammatical analyses and does not always lead to the elimination of MGs. The paper discusses methods for predicting environments conducive to MGs, considering factors such as structural ambiguity and silent elements. The examination of MGs compels us to explore how learners navigate underdetermined input, especially in bilingual contexts, and to examine the interplay between gradient acceptability judgments and categorical grammatical distinctions. The study of MGs offers valuable insights into language variation, change, and the nature of linguistic competence. 

   more » « less
5. When Do You Need Billions of Words of Pretraining Data?

   Zhang, Yian; Warstadt, Alex; Li, Haau-Sing; Bowman, Samuel R. (January 2021, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics)

   null (Ed.)

   NLP is currently dominated by language models like RoBERTa which are pretrained on billions of words. But what exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? To explore this question, we adopt five styles of evaluation: classifier probing, information-theoretic probing, unsupervised relative acceptability judgments, unsupervised language model knowledge probing, and fine-tuning on NLU tasks. We then draw learning curves that track the growth of these different measures of model ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that these LMs require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features we test. They need a much larger quantity of data in order to acquire enough commonsense knowledge and other skills required to master typical downstream NLU tasks. The results suggest that, while the ability to encode linguistic features is almost certainly necessary for language understanding, it is likely that other, unidentified, forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models. 

   more » « less