More Like this

1. Learning to Scale Multilingual Representations for Vision-Language Tasks

   Andrea Burns, Donghyun Kim (January 2020, European Conference on Computer Vision)

   null (Ed.)

   Current multilingual vision-language models either require a large number of additional parameters for each supported language, or suffer performance degradation as languages are added. In this paper, we-9*6 propose a Scalable Multilingual Aligned Language Representation (SMALR) that supports many languages with few model parameters without sacrificing downstream task performance. SMALR learns a fixed size language-agnostic representation for most words in a multilingual vocabulary, keeping language-specific features for just a few. We use a masked cross-language modeling loss to align features with context from other languages. Additionally, we propose a cross-lingual consistency module that ensures predictions made for a query and its machine translation are comparable. The effectiveness of SMALR is demonstrated with ten diverse languages, over twice the number supported in vision-language tasks to date. We evaluate on multilingual image-sentence retrieval and outperform prior work by 3–4% with less than 1/5th the training parameters compared to other word embedding methods. 

   more » « less
2. Learning to Scale Multilingual Representations for Vision-Language Tasks

   https://doi.org/10.1007/978-3-030-58548-8  

   Andrea Burns, Donghyun Kim (January 2020, European Conference on Computer Vision)

   null (Ed.)

   Current multilingual vision-language models either require a large number of additional parameters for each supported language, or suffer performance degradation as languages are added. In this paper, we-9*6 propose a Scalable Multilingual Aligned Language Representation (SMALR) that supports many languages with few model parameters without sacrificing downstream task performance. SMALR learns a fixed size language-agnostic representation for most words in a multilingual vocabulary, keeping language-specific features for just a few. We use a masked cross-language modeling loss to align features with context from other languages. Additionally, we propose a cross-lingual consistency module that ensures predictions made for a query and its machine translation are comparable. The effectiveness of SMALR is demonstrated with ten diverse languages, over twice the number supported in vision-language tasks to date. We evaluate on multilingual image-sentence retrieval and outperform prior work by 3–4% with less than 1/5th the training parameters compared to other word embedding methods. 

   more » « less
3. Testing the Predictions of Surprisal Theory in 11 Languages

   https://doi.org/10.1162/tacl_a_00612  

   Wilcox, Ethan G.; Pimentel, Tiago; Meister, Clara; Cotterell, Ryan; Levy, Roger P. (January 2023, Transactions of the Association for Computational Linguistics)

   Abstract Surprisal theory posits that less-predictable words should take more time to process, with word predictability quantified as surprisal, i.e., negative log probability in context. While evidence supporting the predictions of surprisal theory has been replicated widely, much of it has focused on a very narrow slice of data: native English speakers reading English texts. Indeed, no comprehensive multilingual analysis exists. We address this gap in the current literature by investigating the relationship between surprisal and reading times in eleven different languages, distributed across five language families. Deriving estimates from language models trained on monolingual and multilingual corpora, we test three predictions associated with surprisal theory: (i) whether surprisal is predictive of reading times, (ii) whether expected surprisal, i.e., contextual entropy, is predictive of reading times, and (iii) whether the linking function between surprisal and reading times is linear. We find that all three predictions are borne out crosslinguistically. By focusing on a more diverse set of languages, we argue that these results offer the most robust link to date between information theory and incremental language processing across languages. 

   more » « less
4. The dynamics of bilingualism in language shift ecologies

   https://doi.org/10.1075/lab.22035.gre  

   Grenoble, Lenore A; Osipov, Boris (February 2023, Linguistic Approaches to Bilingualism)

   Abstract A large percentage of the world’s languages – anywhere from 50 to 90% – are currently spoken in what we call shift ecologies, situations of unstable bi- or multilingualism where speakers, and in particular younger speakers, do not use their ancestral language but rather speak the majority language. The present paper addresses several interrelated questions with regard to the linguistic effects of bilingualism in such shift ecologies. These language ecologies are dynamic: language choices and preferences change, as do speakers’ proficiency levels. One result is multiple kinds of variation in these endangered language communities. Understanding change and shift requires a methodology for establishing a baseline; descriptive grammars rarely provide information about usage and multilingual language practices. An additional confounder is a range of linguistic variation: regional (dialectal); generational (language-internal change without contact or shift); contact-based (contact with or without shift); and proficiency-based (variation which develops as a result of differing levels of input and usage). Widespread, ongoing language shift today provides opportunities to examine the linguistic changes exhibited by shifting speakers, that is, to zero in on language change and loss in process, rather than as an end product. 

   more » « less
5. StarCoder: May the Source be With You!

   Li, R.; Allal, L. B.; Zi, Y.; Muennighoff, N.; Kocetkov, D.; Mou, C.; Marone, M.; Akiki, C.; Li, J.; Chim, J.; et al (December 2023, Transactions on machine learning research)

   Durrett, G (Ed.)

   The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context length, infilling capabilities and fast large-batch inference enabled by multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced from The Stack, a large collection of permissively licensed GitHub repositories with inspection tools and an opt-out process. We fine-tuned StarCoderBase on 35B Python tokens, resulting in the creation of StarCoder. We perform the most comprehensive evaluation of Code LLMs to date and show that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms every model that is fine-tuned on Python, can be prompted to achieve 40% pass@1 on HumanEval, and still retains its performance on other programming languages. We take several important steps towards a safe open-access model release, including an improved PII redaction pipeline and a novel attribution tracing tool, and make the StarCoder models publicly available under a more commercially viable version of the Open Responsible AI Model license. 

   more » « less