skip to main content


Title: Depth-Separation with Multilayer Mean-Field Networks
Mean-field limit has been successfully applied to neural networks, leading to many results in optimizing overparametrized networks. However, existing works often focus on two-layer networks and/or require large number of neurons. We give a new framework for extending the mean-field limit to multilayer network, and show that a polynomial-size three-layer network in our framework can learn the function constructed by Safran et al. (2019) – which is known to be not approximable by any two-layer networks  more » « less
Award ID(s):
1845171
NSF-PAR ID:
10335916
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
International Conference on Learning Representations
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. In this letter, we propose an epidemic model over temporal networks that explicitly encapsulates two different control actions. We develop our model within the theoretical framework of activity driven networks (ADNs), which have emerged as a valuable tool to capture the complexity of dynamical processes on networks, coevolving at a comparable time scale to the temporal network formation. Specifically, we complement a susceptible–infected–susceptible epidemic model with features that are typical of nonpharmaceutical interventions in public health policies: i) actions to promote awareness, which induce people to adopt self-protective behaviors, and ii) confinement policies to reduce the social activity of infected individuals. In the thermodynamic limit of large-scale populations, we use a mean-field approach to analytically derive the epidemic threshold, which offers viable insight to devise containment actions at the early stages of the outbreak. Through the proposed model, it is possible to devise an optimal epidemic control policy as the combination of the two strategies, arising from the solution of an optimization problem. Finally, the analytical computation of the epidemic prevalence in endemic diseases on homogeneous ADNs is used to optimally calibrate control actions toward mitigating an endemic disease. Simulations are provided to support our theoretical results. 
    more » « less
  2. Beck, Jeff (Ed.)
    Characterizing metastable neural dynamics in finite-size spiking networks remains a daunting challenge. We propose to address this challenge in the recently introduced replica-mean-field (RMF) limit. In this limit, networks are made of infinitely many replicas of the finite network of interest, but with randomized interactions across replicas. Such randomization renders certain excitatory networks fully tractable at the cost of neglecting activity correlations, but with explicit dependence on the finite size of the neural constituents. However, metastable dynamics typically unfold in networks with mixed inhibition and excitation. Here, we extend the RMF computational framework to point-process-based neural network models with exponential stochastic intensities, allowing for mixed excitation and inhibition. Within this setting, we show that metastable finite-size networks admit multistable RMF limits, which are fully characterized by stationary firing rates. Technically, these stationary rates are determined as the solutions of a set of delayed differential equations under certain regularity conditions that any physical solutions shall satisfy. We solve this original problem by combining the resolvent formalism and singular-perturbation theory. Importantly, we find that these rates specify probabilistic pseudo-equilibria which accurately capture the neural variability observed in the original finite-size network. We also discuss the emergence of metastability as a stochastic bifurcation, which can be interpreted as a static phase transition in the RMF limits. In turn, we expect to leverage the static picture of RMF limits to infer purely dynamical features of metastable finite-size networks, such as the transition rates between pseudo-equilibria. 
    more » « less
  3. Neural networks with a large number of units ad- mit a mean-field description, which has recently served as a theoretical explanation for the favor- able training properties of “overparameterized” models. In this regime, gradient descent obeys a deterministic partial differential equation (PDE) that converges to a globally optimal solution for networks with a single hidden layer under appro- priate assumptions. In this work, we propose a non-local mass transport dynamics that leads to a modified PDE with the same minimizer. We im- plement this non-local dynamics as a stochastic neuronal birth-death process and we prove that it accelerates the rate of convergence in the mean- field limit. We subsequently realize this PDE with two classes of numerical schemes that converge to the mean-field equation, each of which can easily be implemented for neural networks with finite numbers of units. We illustrate our algorithms with two models to provide intuition for the mech- anism through which convergence is accelerated 
    more » « less
  4. Blackwell, Kim T. (Ed.)
    From the action potentials of neurons and cardiac cells to the amplification of calcium signals in oocytes, excitability is a hallmark of many biological signalling processes. In recent years, excitability in single cells has been related to multiple-timescale dynamics through canards , special solutions which determine the effective thresholds of the all-or-none responses. However, the emergence of excitability in large populations remains an open problem. Here, we show that the mechanism of excitability in large networks and mean-field descriptions of coupled quadratic integrate-and-fire (QIF) cells mirrors that of the individual components. We initially exploit the Ott-Antonsen ansatz to derive low-dimensional dynamics for the coupled network and use it to describe the structure of canards via slow periodic forcing. We demonstrate that the thresholds for onset and offset of population firing can be found in the same way as those of the single cell. We combine theoretical analysis and numerical computations to develop a novel and comprehensive framework for excitability in large populations, applicable not only to models amenable to Ott-Antonsen reduction, but also to networks without a closed-form mean-field limit, in particular sparse networks. 
    more » « less
  5. Abstract

    Enhancing our understanding of the intricate interplay among hydro‐climatic processes is crucial for a comprehensive assessment of water availability and climate extremes across global land regions. Here, we propose an integrated framework to investigate networks of the global fields of multiple hydrological variables (Precipitation, Evapotranspiration, Soil Moisture). We apply a two‐layer complex network concept to formulate the independent networks of each hydrological variable and their interactions. Intra‐ (Single‐layer) and cross‐ (two‐layer) network coefficients are derived from the formulated hydrological network to quantify the linkage, spatial connection density, and scale for the independent hydrological fields (or variables) and their interactions. The joint distribution of the intra‐network coefficients reveals multiple spatial scales of connectivity for a moderately well‐connected location in case of evapotranspiration and soil moisture. With increasing global mean temperature, spatially synchronized evapotranspiration over such a large scale may lead to multi‐continental droughts and heatwaves. Furthermore, the (cross‐) network coefficients have identified regions acting as “bottlenecks” for moisture flow and the water‐dominated areas with less evaporative actions. The contrasting features of two‐layer network coefficients have provided a qualitative picture of moisture circulation and recirculation over many hydrological hotspot regions, such as the Amazonian basin, Indian subcontinents, and the Sahel region. The derived results can be employed to gain insights into the global water cycle’s multiple interacting processes (e.g., land‐atmosphere interactions).

     
    more » « less