More Like this

1. Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces

   Liu, Hao; Yang, Haizhao; Chen, Minshuo; Zhao, Tuo; Liao, Wenjing (January 2024, Journal of machine learning research)

   Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively. 

   more » « less
2. Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces

   Liu, Hao; Yang, Haizhao; Chen, Minshuo; Zhao, Tuo; Liao, Wenjing (January 2024, Journal of Machine Learning Research)

   Raginsky, Maxim (Ed.)

   Learning operators between infinitely dimensional spaces is an important learning task arising in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively. 

   more » « less
3. Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices

   Potapczynski, A; Qiu, S; Finzi, M; Ferri, C; Chen, Z; Goldblum, M; Bruss, B; De_Sa, C; Wilson, AG (December 2024, Advances in Neural Information Processing Systems)

   Dense linear layers are the dominant computational bottleneck in large neural networks, presenting a critical need for more efficient alternatives. Previous efforts focused on a small number of hand-crafted structured matrices and neglected to investigate whether these structures can surpass dense layers in terms of compute-optimal scaling laws when both the model size and training examples are optimally allocated. In this work, we present a unifying framework that enables searching among all linear operators expressible via an Einstein summation. This framework encompasses many previously proposed structures, such as low-rank, Kronecker, Tensor-Train, Block Tensor-Train (BTT), and Monarch, along with many novel structures. To analyze the framework, we develop a taxonomy of all such operators based on their computational and algebraic properties and show that differences in the compute-optimal scaling laws are mostly governed by a small number of variables that we introduce. Namely, a small ω (which measures parameter sharing) and large ψ (which measures the rank) reliably led to better scaling laws. Guided by the insight that full-rank structures that maximize parameters per unit of compute perform the best, we propose BTT-MoE, a novel Mixture-of-Experts (MoE) architecture obtained by sparsifying computation in the BTT structure. In contrast to the standard sparse MoE for each entire feed-forward network, BTT-MoE learns an MoE in every single linear layer of the model, including the projection matrices in the attention blocks. We find BTT-MoE provides a substantial compute-efficiency gain over dense layers and standard MoE. 

   more » « less
4. Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective

   Ba, Jimmy; Erdogdu, Murat A; Suzuki, Taiji; Wang, Zhichao; Wu, Denny (December 2023, Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

   Oh, A; Naumann, T; Globerson, A; Saenko, K; Hardt, M; Levine, S (Ed.)

   We consider the problem of learning a single-index target function f∗ : Rd → R under the spiked covariance data: f∗(x) = σ∗   √ 1 1+θ ⟨x,μ⟩   , x ∼ N(0, Id + θμμ⊤), θ ≍ dβ for β ∈ [0, 1), where the link function σ∗ : R → R is a degree-p polynomial with information exponent k (defined as the lowest degree in the Hermite expansion of σ∗), and it depends on the projection of input x onto the spike (signal) direction μ ∈ Rd. In the proportional asymptotic limit where the number of training examples n and the dimensionality d jointly diverge: n, d → ∞, n/d → ψ ∈ (0,∞), we ask the following question: how large should the spike magnitude θ be, in order for (i) kernel methods, (ii) neural networks optimized by gradient descent, to learn f∗? We show that for kernel ridge regression, β ≥ 1 − 1 p is both sufficient and necessary. Whereas for two-layer neural networks trained with gradient descent, β > 1 − 1 k suffices. Our results demonstrate that both kernel methods and neural networks benefit from low-dimensional structures in the data. Further, since k ≤ p by definition, neural networks can adapt to such structures more effectively. 

   more » « less
5. Molecular dynamics simulations to explore the structure and rheological properties of normal and hyperconcentrated airway mucus

   https://doi.org/10.1111/sapm.12433  

   Ford, Andrew G.; Cao, Xue‐Zheng; Papanikolas, Micah J.; Kato, Takafumi; Boucher, Richard C.; Markovetz, Matthew R.; Hill, David B.; Freeman, Ronit; Forest, Mark Gregory (August 2021, Studies in Applied Mathematics)

   Abstract We develop the first molecular dynamics model of airway mucus based on the detailed physical properties and chemical structure of the predominant gel‐forming mucin MUC5B. Our airway mucus model leverages the LAMMPS open‐source code [https://lammps.sandia.gov], based on the statistical physics of polymers, from single molecules to networks. On top of the LAMMPS platform, the chemical structure of MUC5B is used to superimpose proximity‐based, noncovalent, transient interactions within and between the specific domains of MUC5B polymers. We explore feasible ranges of hydrophobic and electrostatic interaction strengths between MUC5B domains with 9 nm spatial and 1 ns temporal resolution. Our goal here is to propose and test a mechanistic hypothesis for a striking clinical observation with respect to airway mucus: a 10‐fold increase in nonswellable, dense structures called flakes during progression of cystic fibrosis disease. Among the myriad possible effects that might promote self‐organization of MUC5B networks into flake structures, we hypothesize and confirm that the clinically confirmed increase in mucin concentration, from 1.5 to 5 mg/ml, alone is sufficient to drive the structure changes observed with scanning electron microscopy images from experimental samples. We postprocess the LAMMPS simulated data sets at 1.5 and 5 mg/ml, both to image the structure transition and compare with scanning electron micrographs and to show that the 3.33‐fold increase in concentration induces closer proximity of interacting electrostatic and hydrophobic domains, thereby amplifying the proximity‐based strength of the interactions. 

   more » « less