More Like this

1. Global convergence of neuron birth-death dynamics

   Rotskoff, Grant; Jelassi, S; Bruna, Joan; Vanden-Eijnden, Eric. (June 2019, International Conference on Machine Learning)

   Neural networks with a large number of units ad- mit a mean-field description, which has recently served as a theoretical explanation for the favor- able training properties of “overparameterized” models. In this regime, gradient descent obeys a deterministic partial differential equation (PDE) that converges to a globally optimal solution for networks with a single hidden layer under appro- priate assumptions. In this work, we propose a non-local mass transport dynamics that leads to a modified PDE with the same minimizer. We im- plement this non-local dynamics as a stochastic neuronal birth-death process and we prove that it accelerates the rate of convergence in the mean- field limit. We subsequently realize this PDE with two classes of numerical schemes that converge to the mean-field equation, each of which can easily be implemented for neural networks with finite numbers of units. We illustrate our algorithms with two models to provide intuition for the mech- anism through which convergence is accelerated 

   more » « less
2. Dynamics of finite width Kernel and prediction fluctuations in mean field neural networks

   https://doi.org/10.1088/1742-5468/ad642b  

   Bordelon, Blake; Pehlevan, Cengiz (October 2024, Journal of Statistical Mechanics: Theory and Experiment)

   Abstract We analyze the dynamics of finite width effects in wide but finite feature learning neural networks. Starting from a dynamical mean field theory description of infinite width deep neural network kernel and prediction dynamics, we provide a characterization of the $O\left(1/\sqrt{\text{width}}\right)$fluctuations of the dynamical mean field theory order parameters over random initializations of the network weights. Our results, while perturbative in width, unlike prior analyses, are non-perturbative in the strength of feature learning. We find that once the mean field/µP parameterization is adopted, the leading finite size effect on the dynamics is to introduce initialization variance in the predictions and feature kernels of the networks. In the lazy limit of network training, all kernels are random but static in time and the prediction variance has a universal form. However, in the rich, feature learning regime, the fluctuations of the kernels and predictions are dynamically coupled with a variance that can be computed self-consistently. In two layer networks, we show how feature learning can dynamically reduce the variance of the final tangent kernel and final network predictions. We also show how initialization variance can slow down online learning in wide but finite networks. In deeper networks, kernel variance can dramatically accumulate through subsequent layers at large feature learning strengths, but feature learning continues to improve the signal-to-noise ratio of the feature kernels. In discrete time, we demonstrate that large learning rate phenomena such as edge of stability effects can be well captured by infinite width dynamics and that initialization variance can decrease dynamically. For convolutional neural networks trained on CIFAR-10, we empirically find significant corrections to both the bias and variance of network dynamics due to finite width. 

   more » « less
3. The Refractory Period Matters: Unifying Mechanisms of Macroscopic Brain Waves

   https://doi.org/10.1162/neco_a_01371  

   Weistuch, Corey; Mujica-Parodi, Lilianne R.; Dill, Ken (April 2021, Neural Computation)

   Abstract The relationship between complex brain oscillations and the dynamics of individual neurons is poorly understood. Here we utilize maximum caliber, a dynamical inference principle, to build a minimal yet general model of the collective (mean field) dynamics of large populations of neurons. In agreement with previous experimental observations, we describe a simple, testable mechanism, involving only a single type of neuron, by which many of these complex oscillatory patterns may emerge. Our model predicts that the refractory period of neurons, which has often been neglected, is essential for these behaviors. 

   more » « less
4. Bring the pain: wounding reveals a transition from cortical excitability to epithelial excitability in Xenopus embryos

   https://doi.org/10.3389/fcell.2023.1295569  

   Sepaniac, Leslie A; Davenport, Nicholas R; Bement, William M (February 2024, Frontiers in Cell and Developmental Biology)

   The cell cortex plays many critical roles, including interpreting and responding to internal and external signals. One behavior which supports a cell’s ability to respond to both internal and externally-derived signaling is cortical excitability, wherein coupled positive and negative feedback loops generate waves of actin polymerization and depolymerization at the cortex. Cortical excitability is a highly conserved behavior, having been demonstrated in many cell types and organisms. One system well-suited to studying cortical excitability isXenopus laevis, in which cortical excitability is easily monitored for many hours after fertilization. Indeed, recent investigations usingX. laevishave furthered our understanding of the circuitry underlying cortical excitability and how it contributes to cytokinesis. Here, we describe the impact of wounding, which represents both a chemical and a physical signal, on cortical excitability. In early embryos (zygotes to early blastulae), we find that wounding results in a transient cessation (“freezing”) of wave propagation followed by transport of frozen waves toward the wound site. We also find that wounding near cell-cell junctions results in the formation of an F-actin (actin filament)-based structure that pulls the junction toward the wound; at least part of this structure is based on frozen waves. In later embryos (late blastulae to gastrulae), we find that cortical excitability diminishes and is progressively replaced by epithelial excitability, a process in which wounded cells communicate with other cells via wave-like increases of calcium and apical F-actin. While the F-actin waves closely follow the calcium waves in space and time, under some conditions the actin wave can be uncoupled from the calcium wave, suggesting that they may be independently regulated by a common upstream signal. We conclude that as cortical excitability disappears from the level of the individual cell within the embryo, it is replaced by excitability at the level of the embryonic epithelium itself. 

   more » « less
5. Phase transition in a kinetic mean-field game model of inertial self-propelled agents

   https://doi.org/10.1063/5.0230729  

   Grover, Piyush; Huo, Mandy (December 2024, Chaos: An Interdisciplinary Journal of Nonlinear Science)

   The framework of mean-field games (MFGs) is used for modeling the collective dynamics of large populations of non-cooperative decision-making agents. We formulate and analyze a kinetic MFG model for an interacting system of non-cooperative motile agents with inertial dynamics and finite-range interactions, where each agent is minimizing a biologically inspired cost function. By analyzing the associated coupled forward–backward in a time system of nonlinear Fokker–Planck and Hamilton–Jacobi–Bellman equations, we obtain conditions for closed-loop linear stability of the spatially homogeneous MFG equilibrium that corresponds to an ordered state with non-zero mean speed. Using a combination of analysis and numerical simulations, we show that when energetic cost of control is reduced below a critical value, this equilibrium loses stability, and the system transitions to a traveling wave solution. Our work provides a game-theoretic perspective to the problem of collective motion in non-equilibrium biological and bio-inspired systems. 

   more » « less