More Like this

1. On Energy-Based Models with Overparametrized Shallow Neural Networks

   Domingo-Enrich, C; Bietti, A; Vanden-Eijnden, E; Bruna, J (July 2021, Proceedings of Machine Learning Research)

   null (Ed.)

   Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called “active” regime provide a statistical advantage over their associated “lazy” or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data. 

   more » « less
2. Deep Learning and Likelihood Approaches for Viral Phylogeography Converge on the Same Answers Whether the Inference Model Is Right or Wrong

   https://doi.org/10.1093/sysbio/syad074  

   Thompson, Ammon; Liebeskind, Benjamin_J; Scully, Erik_J; Landis, Michael_J; Perez, ed., Manolo (January 2024, Systematic Biology)

   Abstract Analysis of phylogenetic trees has become an essential tool in epidemiology. Likelihood-based methods fit models to phylogenies to draw inferences about the phylodynamics and history of viral transmission. However, these methods are often computationally expensive, which limits the complexity and realism of phylodynamic models and makes them ill-suited for informing policy decisions in real-time during rapidly developing outbreaks. Likelihood-free methods using deep learning are pushing the boundaries of inference beyond these constraints. In this paper, we extend, compare, and contrast a recently developed deep learning method for likelihood-free inference from trees. We trained multiple deep neural networks using phylogenies from simulated outbreaks that spread among 5 locations and found they achieve close to the same levels of accuracy as Bayesian inference under the true simulation model. We compared robustness to model misspecification of a trained neural network to that of a Bayesian method. We found that both models had comparable performance, converging on similar biases. We also implemented a method of uncertainty quantification called conformalized quantile regression that we demonstrate has similar patterns of sensitivity to model misspecification as Bayesian highest posterior density (HPD) and greatly overlap with HPDs, but have lower precision (more conservative). Finally, we trained and tested a neural network against phylogeographic data from a recent study of the SARS-Cov-2 pandemic in Europe and obtained similar estimates of region-specific epidemiological parameters and the location of the common ancestor in Europe. Along with being as accurate and robust as likelihood-based methods, our trained neural networks are on average over 3 orders of magnitude faster after training. Our results support the notion that neural networks can be trained with simulated data to accurately mimic the good and bad statistical properties of the likelihood functions of generative phylogenetic models. 

   more » « less
3. Neural networks estimate muscle force in dynamic conditions better than Hill-type muscle models

   https://doi.org/10.1242/jeb.250268  

   Athanasiadou, Maria Eleni; Daley, Monica A; Koelewijn, Anne D (November 2025, Journal of Experimental Biology)

   ABSTRACT Hill-type muscle models are widely used, even though they do not accurately represent the relationship between activation and force in dynamic contractions. We explored the use of neural networks as an alternative approach to capture features of dynamic muscle function, without a priori assumptions about force–length–velocity relationships. We trained neural networks using an existing dataset of two guinea fowl muscles to estimate muscle force from activation, fascicle length and velocity. Training data were recorded using sonomicrometry, electromyography and a tendon buckle. First, we compared the neural networks with Hill-type muscle models, using the same data for network training and model optimization. Second, we trained neural networks on larger datasets, in a more realistic machine learning scenario. We found that neural networks generally yielded higher coefficients of determination and lower errors than Hill-type muscle models. Neural networks performed better when estimating forces on the muscle used for training, but on another bird, than on a different muscle of the same bird, likely due to inaccuracies in activation and force scaling. We extracted force–length and force–velocity relationships from the trained neural networks and found that both effects were underestimated and the relationships were not well replicated outside the training data distribution. We discuss suggested experimental designs and the challenge of collecting suitable training data. Given a suitable training dataset, neural networks could provide a useful alternative to Hill-type muscle models, particularly for modeling muscle dynamics in faster movements; however, scaling of the training data should be comparable between muscles and animals. 

   more » « less
4. Expressivity of Neural Networks with Random Weights and Learned Biases

   Williams, Ezekiel; Payeur, Alexandre; Ryoo, Avery Hee-Woon; Jiralerspong, Thomas; Perich, Matthew G; Mazzucato, Luca; Lajoie, Guillaume (January 2025, International Conference on Learning Representations)

   Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neural networks as learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks in which a smaller subset of weights (e.g., output weights) are tuned, leaving other parameters random. However, it remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and AI that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can approximate any continuous function on compact sets. We further show an analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix-based approaches. 

   more » « less
5. Generative PointNet: deep energy-based learning on unordered point sets for 3D generation, reconstruction and classification

   Xie, J.; Xu, Y.; Zheng, Z.; Zhu, S. C.; Wu, Y. N. (January 2021, IEEE Conference on Computer Vision and Pattern Recognition)

   We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input permutation- invariant bottom-up neural network. The energy function learns a coordinate encoding of each point and then aggregates all individual point features into an energy for the whole point cloud. We call our model the Generative PointNet because it can be derived from the discriminative PointNet. Our model can be trained by MCMC based maximum likelihood learning (as well as its variants), without the help of any assisting networks like those in GANs and VAEs. Unlike most point cloud generators that rely on hand-crafted distance metrics, our model does not require any hand-crafted distance metric for the point cloud generation, because it synthesizes point clouds by matching observed examples in terms of statistical properties defined by the energy function. Furthermore, we can learn a short run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpolation. The learned point cloud representation can be useful for point cloud classification. Experiments demonstrate the advantages of the proposed generative model of point clouds. 

   more » « less