More Like this

1. Neural Manifold Ordinary Differential Equations

   Lou, Aaron; Lim, Derek; Katsman, Isay; Huang, Leo; Jiang, Qingxuan; Lim, Ser Nam (January 2021, 2020 Advances in Neural Information Processing Systems (NeurIPS 2020))

   Larochelle, Hugo; Ranzato, Marc'Aurelio; Hadsell, Raia; Balcan, Maria-Florina; Lin, Hsuan-Tien (Ed.)

   To better conform to data geometry, recent deep generative modelling techniques adapt Euclidean constructions to non-Euclidean spaces. In this paper, we study normalizing flows on manifolds. Previous work has developed flow models for specific cases; however, these advancements hand craft layers on a manifold-by-manifold basis, restricting generality and inducing cumbersome design constraints. We overcome these issues by introducing Neural Manifold Ordinary Differential Equations, a manifold generalization of Neural ODEs, which enables the construction of Manifold Continuous Normalizing Flows (MCNFs). MCNFs require only local geometry (therefore generalizing to arbitrary manifolds) and compute probabilities with continuous change of variables (allowing for a simple and expressive flow construction). We find that leveraging continuous manifold dynamics produces a marked improvement for both density estimation and downstream tasks. 

   more » « less
2. Neural differential recurrent neural network with adaptive time steps

   Tan, Yixuan; Xie, Liyan; Cheng, Xiuyuan (June 2023, Short version published at the Symbiosis of Deep Learning and Differential Equations III (DLDE III) at NeurIPS 2023.)

   The neural Ordinary Differential Equation (ODE) model has shown success in learning complex continuous-time processes from observations on discrete time stamps. In this work, we consider the modeling and forecasting of time series data that are non-stationary and may have sharp changes like spikes. We propose an RNN-based model, called RNN-ODE-Adap, that uses a neural ODE to represent the time development of the hidden states, and we adaptively select time steps based on the steepness of changes of the data over time so as to train the model more efficiently for the "spike-like" time series. Theoretically, RNN-ODE-Adap yields provably a consistent estimation of the intensity function for the Hawkes-type time series data. We also provide an approximation analysis of the RNN-ODE model showing the benefit of adaptive steps. The proposed model is demonstrated to achieve higher prediction accuracy with reduced computational cost on simulated dynamic system data and point process data and on a real electrocardiography dataset. 

   more » « less
3. DDPNOpt: Differential Dynamic Programming Neural Optimizer

   Guan-Horng, Liu; Chen, Tianrong; Evangelos. A, Theodorou (January 2021, International Conference on Learning Representations)

   null (Ed.)
4. Heavy Ball Neural Ordinary Differential Equations

   Xia, Hedi; Suliafu, Vai; Ji, Hangjie; Nguyen, Tan; Bertozzi, Andrea; Osher, Stanley; Wang, Bao. (January 2021, Advances in neural information processing systems)

   We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly reducing the number of function evaluations (NFEs) and improving the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long-term dependencies from complex sequential data. We verify the advantages of HBNODEs over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling. Our method requires remarkably fewer forward and backward NFEs, is more accurate, and learns long-term dependencies more effectively than the other ODE-based neural network models. Code is available at https://github.com/hedixia/HeavyBallNODE. 

   more » « less
5. Stochastic physics-informed neural ordinary differential equations

   https://doi.org/10.1016/j.jcp.2022.111466  

   O'Leary, Jared; Paulson, Joel A.; Mesbah, Ali (November 2022, Journal of Computational Physics)