More Like this

1. Learning Phonotactics from Linguistic Informants

   Breiss, Canaan; Ross, Alexis; Maina-Kilaas, Amani; Levy, Roger; Andreas, Jacob (June 2024, Society for Computation in Linguistics)

   We propose an interactive approach to language learning that utilizes linguistic acceptability judgments from an informant (a competent lan- guage user) to learn a grammar. Given a gram- mar formalism and a framework for synthesiz- ing data, our model iteratively selects or synthe- sizes a data-point according to one of a range of information-theoretic policies, asks the in- formant for a binary judgment, and updates its own parameters in preparation for the next query. We demonstrate the effectiveness of our model in the domain of phonotactics, the rules governing what kinds of sound-sequences are acceptable in a language, and carry out two experiments, one with typologically-natural linguistic data and another with a range of procedurally-generated languages. We find that the information-theoretic policies that our model uses to select items to query the infor- mant achieve sample efficiency comparable to, and sometimes greater than, fully supervised approaches. 

   more » « less
2. A transition-based parser for unscoped episdoc logical form

   Kim, Gene Louis; Duong, Viet; Lu, Xin; Schubert, Lenhart (June 2021, Fourteenth Int. Conf. on Computational Semantics (IWCS 2021))

   null (Ed.)

   “Episodic Logic: Unscoped Logical Form” (EL-ULF) is a semantic representation capturing predicate-argument structure as well as more challenging aspects of language within the Episodic Logic formalism. We present the first learned approach for parsing sentences into ULFs, using a growing set of annotated examples. The results provide a strong baseline for future improvement. Our method learns a sequence-to-sequence model for predicting the transition action sequence within a modified cache transition system. We evaluate the efficacy of type grammar-based constraints, a word-to-symbol lexicon, and transition system state features in this task. Our system is availableat https://github.com/genelkim/ ulf-transition-parser. We also present the first official annotated ULF dataset at https://www.cs.rochester.edu/u/ gkim21/ulf/resources/. 

   more » « less
3. Send to me first: Priority in synchronous message-passing

   https://doi.org/10.1017/S0956796822000119  

   CHUANG, CHENG-EN; IRACI, GRANT; ZIAREK, LUKASZ (January 2022, Journal of Functional Programming)

   Abstract In this paper, we introduce a tiered-priority scheme for a synchronous message-passing language with support for selective communication and first-class communication protocols. Crucially, our scheme allows higher priority threads to communicate with lower priority threads, providing the ability to express programs that would be rejected by classic priority mechanisms that disallow any (potentially) blocking interactions between threads of differing priorities. We formalize our scheme in a novel semantic framework featuring a collection of actions to represent possible communications. Utilizing our formalism, we prove several important and desirable properties of our priority scheme. We also provide a prototype implementation of our tiered-priority scheme capable of expressing Concurrent ML and built in the MLton SML compiler and runtime. We evaluate the viability of our implementation through three case studies: a prioritized buyer-seller protocol and predictable shutdown mechanisms in the Swerve web server and eXene windowing toolkit. Our experiments show that priority can be easily added to existing CML programs without degrading performance. Our system exhibits negligible overheads on more modest workloads. 

   more » « less
4. A Kernel-Based View of Language Model Fine-Tuning

   Malladi, Sadhika; Wettig, Alexander; Yu, Dingli; Chen, Danqi; Arora, Sanjeev. (July 2023, Proceedings of the 40th International Conference on Machine Learning)

   It has become standard to solve NLP tasks by fine-tuning pre-trained language models (LMs), especially in low-data settings. There is minimal theoretical understanding of empirical success, e.g., why fine-tuning a model with $10^8$ or more parameters on a couple dozen training points does not result in overfitting. We investigate whether the Neural Tangent Kernel (NTK)—which originated as a model to study the gradient descent dynamics of infinitely wide networks with suitable random initialization—describes fine-tuning of pre-trained LMs. This study was inspired by the decent performance of NTK for computer vision tasks (Wei et al., 2022). We extend the NTK formalism to Adam and use Tensor Programs (Yang, 2020) to characterize conditions under which the NTK lens may describe fine-tuning updates to pre-trained language models. Extensive experiments on 14 NLP tasks validate our theory and show that formulating the downstream task as a masked word prediction problem through prompting often induces kernel-based dynamics during fine-tuning. Finally, we use this kernel view to propose an explanation for the success of parameter-efficient subspace-based fine-tuning methods. 

   more » « less
5. An exact imaginary-time path-integral phase-space formulation of multi-time correlation functions

   https://doi.org/10.1063/5.0137898  

   Videla, Pablo E.; Batista, Victor S. (March 2023, The Journal of Chemical Physics)

   An exact representation of quantum mechanics using the language of phase-space variables provides a natural starting point to introduce and develop semiclassical approximations for the calculation of time correlation functions. Here, we introduce an exact path-integral formalism for calculations of multi-time quantum correlation functions as canonical averages over ring-polymer dynamics in imaginary time. The formulation provides a general formalism that exploits the symmetry of path integrals with respect to permutations in imaginary time, expressing correlations as products of imaginary-time-translation-invariant phase-space functions coupled through Poisson bracket operators. The method naturally recovers the classical limit of multi-time correlation functions and provides an interpretation of quantum dynamics in terms of “interfering trajectories” of the ring-polymer in phase space. The introduced phase-space formulation provides a rigorous framework for the future development of quantum dynamics methods that exploit the invariance of imaginary time path integrals to cyclic permutations. 

   more » « less