More Like this

1. Gated recurrent units viewed through the lens of continuous time dynamical systems

   https://doi.org/10.3389/fncom.2021.678158  

   Jordan, Ian; Sokol, Piotr; Park, Il Memming (January 2021, Frontiers in computational neuroscience)

   null (Ed.)

   In recent years, the efficacy of using artificial recurrent neural networks to model cortical dynamics has been a topic of interest. Gated recurrent units (GRUs) are specialized memory elements for building these recurrent neural networks. Despite their incredible success in natural language, speech, video processing, and extracting dynamics underlying neural data, little is understood about the specific dynamics representable in a GRU network, and how these dynamics play a part in performance and generalization. As a result, it is both difficult to know a priori how successful a GRU network will perform on a given task, and also their capacity to mimic the underlying behavior of their biological counterparts. Using a continuous time analysis, we gain intuition on the inner workings of GRU networks. We restrict our presentation to low dimensions, allowing for a comprehensive visualization. We found a surprisingly rich repertoire of dynamical features that includes stable limit cycles (nonlinear oscillations), multi-stable dynamics with various topologies, and homoclinic bifurcations. At the same time GRU networks are limited in their inability to produce continuous attractors, which are hypothesized to exist in biological neural networks. We contextualize the usefulness of different kinds of observed dynamics and support our claims experimentally. 

   more » « less
2. Forecasting the Evolution of Hydropower Generation

   https://doi.org/10.1145/3394486.3403337  

   Zhou, Fan; Li, Liang; Zhang, Kunpeng; Trajcevski, Goce; Yao, Fuming; Huang, Ying; Zhong, Ting; Wang, Jiahao; Liu, Qiao (July 2020, The 26th {ACM} {SIGKDD} Conference on Knowledge Discovery and Data Mining, CA, August 2020)

   null (Ed.)

   Hydropower is the largest renewable energy source for electricity generation in the world, with numerous benefits in terms of: environment protection (near-zero air pollution and climate impact), cost-effectiveness (long-term use, without significant impacts of market fluctuation), and reliability (quickly respond to surge in demand). However, the effectiveness of hydropower plants is affected by multiple factors such as reservoir capacity, rainfall, temperature and fluctuating electricity demand, and particularly their complicated relationships, which make the prediction/recommendation of station operational output a difficult challenge. In this paper, we present DeepHydro, a novel stochastic method for modeling multivariate time series (e.g., water inflow/outflow and temperature) and forecasting power generation of hydropower stations. DeepHydro captures temporal dependencies in co-evolving time series with a new conditioned latent recurrent neural networks, which not only considers the hidden states of observations but also preserves the uncertainty of latent variables. We introduce a generative network parameterized on a continuous normalizing flow to approximate the complex posterior distribution of multivariate time series data, and further use neural ordinary differential equations to estimate the continuous-time dynamics of the latent variables constituting the observable data. This allows our model to deal with the discrete observations in the context of continuous dynamic systems, while being robust to the noise. We conduct extensive experiments on real-world datasets from a large power generation company consisting of cascade hydropower stations. The experimental results demonstrate that the proposed method can effectively predict the power production and significantly outperform the possible candidate baseline approaches. 

   more » « less
3. Nerve Theorems for Fixed Points of Neural Networks

   https://doi.org/10.1007/978-3-030-95519-9_6  

   Santander, D. E.; Ebli, S.; Patania, A.; Sanderson, N.; Burtscher, F.; Morrison, K.; Curto, C. (May 2022, Association for Women in Mathematics series)

   Gasparovic, E.; Robins, V.; Turner, K. (Ed.)

   Nonlinear network dynamics are notoriously difficult to understand. Here we study a class of recurrent neural networks called combinatorial threshold-linear networks (CTLNs) whose dynamics are determined by the structure of a directed graph. They are a special case of TLNs, a popular framework for modeling neural activity in computational neuroscience. In prior work, CTLNs were found to be surprisingly tractable mathematically. For small networks, the fixed points of the network dynamics can often be completely determined via a series of graph rules that can be applied directly to the underlying graph. For larger networks, it remains a challenge to understand how the global structure of the network interacts with local properties. In this work, we propose a method of covering graphs of CTLNs with a set of smaller directional graphs that reflect the local flow of activity. While directional graphs may or may not have a feedforward architecture, their fixed point structure is indicative of feedforward dynamics. The combinatorial structure of the graph cover is captured by the nerve of the cover. The nerve is a smaller, simpler graph that is more amenable to graphical analysis. We present three nerve theorems that provide strong constraints on the fixed points of the underlying network from the structure of the nerve. We then illustrate the power of these theorems with some examples. Remarkably, we find that the nerve not only constrains the fixed points of CTLNs, but also gives insight into the transient and asymptotic dynamics. This is because the flow of activity in the network tends to follow the edges of the nerve. 

   more » « less
4. Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling

   Djeumou, Franck.; Neary, Cyrus.; Goubault, Eric.; Putot, Sylvie.; Topcu, Ufuk. (January 2022, 4th Annual Conference on Learning for Dynamics and Control)

   Firoozi, R.; Mehr, N.; Yel, E.; Antonova, R.; Bohg, J.; Schwager, M.; Kochenderfer, M. (Ed.)

   Effective inclusion of physics-based knowledge into deep neural network models of dynamical sys- tems can greatly improve data efficiency and generalization. Such a priori knowledge might arise from physical principles (e.g., conservation laws) or from the system’s design (e.g., the Jacobian matrix of a robot), even if large portions of the system dynamics remain unknown. We develop a framework to learn dynamics models from trajectory data while incorporating a priori system knowledge as inductive bias. More specifically, the proposed framework uses physics-based side information to inform the structure of the neural network itself, and to place constraints on the values of the outputs and the internal states of the model. It represents the system’s vector field as a composition of known and unknown functions, the latter of which are parametrized by neural networks. The physics-informed constraints are enforced via the augmented Lagrangian method during the model’s training. We experimentally demonstrate the benefits of the proposed approach on a variety of dynamical systems – including a benchmark suite of robotics environments featur- ing large state spaces, non-linear dynamics, external forces, contact forces, and control inputs. By exploiting a priori system knowledge during training, the proposed approach learns to predict the system dynamics two orders of magnitude more accurately than a baseline approach that does not include prior knowledge, given the same training dataset. 

   more » « less
5. State-space modeling for degrading systems with stochastic neural networks and dynamic Bayesian layers

   https://doi.org/10.1080/24725854.2023.2185323  

   Farhat, Md Tanzin; Moghaddass, Ramin (February 2023, IISE Transactions)

   To monitor the dynamic behavior of degrading systems over time, a flexible hierarchical discrete-time state-space model (SSM) is introduced that can mathematically characterize the stochastic evolution of the latent states (discrete, continuous, or hybrid) of degrading systems, dynamic measurements collected from condition monitoring sources (e.g., sensors with mixed-type out-puts), and the failure process. This flexible SSM is inspired by Bayesian hierarchical modeling and recurrent neural networks without imposing prior knowledge regarding the stochastic structure of the system dynamics and its variables. The temporal behavior of degrading systems and the relationship between variables of the corresponding system dynamics are fully characterized by stochastic neural networks without having to define parametric relationships/distributions between deterministic and stochastic variables. A Bayesian filtering-based learning method is introduced to train the structure of the proposed framework with historical data. Also, the steps to utilize the proposed framework for inference and prediction of the latent states and sensor outputs are dis-cussed. Numerical experiments are provided to demonstrate the application of the proposed framework for degradation system modeling and monitoring. 

   more » « less