skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Domain-driven models yield better predictions at lower cost than reservoir computers in Lorenz systems
Recent advances in computing algorithms and hardware have rekindled interest in developing high-accuracy, low-cost surrogate models for simulating physical systems. The idea is to replace expensive numerical integration of complex coupled partial differential equations at fine time scales performed on supercomputers, with machine-learned surrogates that efficiently and accurately forecast future system states using data sampled from the underlying system. One particularly popular technique being explored within the weather and climate modelling community is the echo state network (ESN), an attractive alternative to other well-known deep learning architectures. Using the classical Lorenz 63 system, and the three tier multi-scale Lorenz 96 system (Thornes T, Duben P, Palmer T. 2017 Q. J. R. Meteorol. Soc. 143 , 897–908. ( doi:10.1002/qj.2974 )) as benchmarks, we realize that previously studied state-of-the-art ESNs operate in two distinct regimes, corresponding to low and high spectral radius (LSR/HSR) for the sparse, randomly generated, reservoir recurrence matrix. Using knowledge of the mathematical structure of the Lorenz systems along with systematic ablation and hyperparameter sensitivity analyses, we show that state-of-the-art LSR-ESNs reduce to a polynomial regression model which we call Domain-Driven Regularized Regression (D2R2). Interestingly, D2R2 is a generalization of the well-known SINDy algorithm (Brunton SL, Proctor JL, Kutz JN. 2016 Proc. Natl Acad. Sci. USA 113 , 3932–3937. ( doi:10.1073/pnas.1517384113 )). We also show experimentally that LSR-ESNs (Chattopadhyay A, Hassanzadeh P, Subramanian D. 2019 ( http://arxiv.org/abs/1906.08829 )) outperform HSR ESNs (Pathak J, Hunt B, Girvan M, Lu Z, Ott E. 2018 Phys. Rev. Lett. 120 , 024102. ( doi:10.1103/PhysRevLett.120.024102 )) while D2R2 dominates both approaches. A significant goal in constructing surrogates is to cope with barriers to scaling in weather prediction and simulation of dynamical systems that are imposed by time and energy consumption in supercomputers. Inexact computing has emerged as a novel approach to helping with scaling. In this paper, we evaluate the performance of three models (LSR-ESN, HSR-ESN and D2R2) by varying the precision or word size of the computation as our inexactness-controlling parameter. For precisions of 64, 32 and 16 bits, we show that, surprisingly, the least expensive D2R2 method yields the most robust results and the greatest savings compared to ESNs. Specifically, D2R2 achieves 68 × in computational savings, with an additional 2 × if precision reductions are also employed, outperforming ESN variants by a large margin. This article is part of the theme issue ‘Machine learning for weather and climate modelling’.  more » « less
Award ID(s):
1707400
PAR ID:
10294655
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
Volume:
379
Issue:
2194
ISSN:
1364-503X
Page Range / eLocation ID:
20200246
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Lee, Jonghyun; Darve, Eric F.; Kitanidis, Peter K.; Mahoney, Michael W.; Karpatne, Anuj; Farthing, Matthew W.; Hesser, Tyler (Ed.)
    Modern design, control, and optimization often require multiple expensive simulations of highly nonlinear stiff models. These costs can be amortized by training a cheap surrogate of the full model, which can then be used repeatedly. Here we present a general data-driven method, the continuous time echo state network (CTESN), for generating surrogates of nonlinear ordinary differential equations with dynamics at widely separated timescales. We empirically demonstrate the ability to accelerate a physically motivated scalable model of a heating system by 98x while maintaining relative error of within 0.2 %. We showcase the ability for this surrogate to accurately handle highly stiff systems which have been shown to cause training failures with common surrogate methods such as Physics-Informed Neural Networks (PINNs), Long Short Term Memory (LSTM) networks, and discrete echo state networks (ESN). We show that our model captures fast transients as well as slow dynamics, while demonstrating that fixed time step machine learning techniques are unable to adequately capture the multi-rate behavior. Together this provides compelling evidence for the ability of CTESN surrogates to predict and accelerate highly stiff dynamical systems which are unable to be directly handled by previous scientific machine learning techniques. 
    more » « less
  2. Abstract Long‐lead forecasting for spatio‐temporal systems can entail complex nonlinear dynamics that are difficult to specify a priori. Current statistical methodologies for modeling these processes are often highly parameterized and, thus, challenging to implement from a computational perspective. One potential parsimonious solution to this problem is a method from the dynamical systems and engineering literature referred to as an echo state network (ESN). ESN models usereservoir computingto efficiently compute recurrent neural network forecasts. Moreover, multilevel (deep) hierarchical models have recently been shown to be successful at predicting high‐dimensional complex nonlinear processes, particularly those with multiple spatial and temporal scales of variability (such as those we often find in spatio‐temporal environmental data). Here, we introduce a deep ensemble ESN (D‐EESN) model. Despite the incorporation of a deep structure, the presented model is computationally efficient. We present two versions of this model for spatio‐temporal processes that produce forecasts and associated measures of uncertainty. The first approach utilizes a bootstrap ensemble framework, and the second is developed within a hierarchical Bayesian framework (BD‐EESN). This more general hierarchical Bayesian framework naturally accommodates non‐Gaussian data types and multiple levels of uncertainties. The methodology is first applied to a data set simulated from a novel non‐Gaussian multiscale Lorenz‐96 dynamical system simulation model and, then, to a long‐lead United States (U.S.) soil moisture forecasting application. Across both applications, the proposed methodology improves upon existing methods in terms of both forecast accuracy and quantifying uncertainty. 
    more » « less
  3. This paper reports on the effects of shear rate and interface modeling parameters on the hydrodynamic slip length (LS) for water–graphite interfaces calculated using non-equilibrium molecular dynamics. Five distinct non-bonded solid–liquid interaction parameters were considered to assess their impact on LS. The interfacial force field derivations included sophisticated electronic structure calculation-informed and empirically determined parameters. All interface models exhibited a similar and bimodal LS response when varying the applied shear rate. LS in the low shear rate regime (LSR) is in good agreement with previous calculations obtained through equilibrium molecular dynamics. As the shear rate increases, LS sharply increases and asymptotes to a constant value in the high shear regime (HSR). It is noteworthy that LS in both the LSR and HSR can be characterized by the density depletion length, whereas solid–liquid adhesion metrics failed to do so. For all interface models, LHSR calculations were, on average, ∼28% greater than LLSR, and this slip jump was confirmed using the SPC/E and TIP4P/2005 water models. To address the LS transition from the LSR to the HSR, the viscosity of water and the interfacial friction coefficient were investigated. It was observed that in the LSR, the viscosity and friction coefficient decreased at a similar rate, while in the LSR-to-HSR transition, the friction coefficient decreased at a faster rate than the shear viscosity until they reached a new equilibrium, hence explaining the LS-bimodal behavior. This study provides valuable insights into the interplay between interface modeling parameters, shear rate, and rheological properties in understanding hydrodynamic slip behavior. 
    more » « less
  4. Sugarcane croplands account for ~70% of global sugar production and ~60% of global ethanol production. Monitoring and predicting gross primary production (GPP) and transpiration (T) in these fields is crucial to improve crop yield estimation and management. While moderate-spatial-resolution (MSR, hundreds of meters) satellite images have been employed in several models to estimate GPP and T, the potential of high-spatial-resolution (HSR, tens of meters) imagery has been considered in only a few publications, and it is underexplored in sugarcane fields. Our study evaluated the efficacy of MSR and HSR satellite images in predicting daily GPP and T for sugarcane plantations at two sites equipped with eddy flux towers: Louisiana, USA (subtropical climate) and Sao Paulo, Brazil (tropical climate). We employed the Vegetation Photosynthesis Model (VPM) and Vegetation Transpiration Model (VTM) with C4 photosynthesis pathway, integrating vegetation index data derived from satellite images and on-ground weather data, to calculate daily GPP and T. The seasonal dynamics of vegetation indices from both MSR images (MODIS sensor, 500 m) and HSR images (Landsat, 30 m; Sentinel-2, 10 m) tracked well with the GPP seasonality from the EC flux towers. The enhanced vegetation index (EVI) from the HSR images had a stronger correlation with the tower-based GPP. Our findings underscored the potential of HSR imagery for estimating GPP and T in smaller sugarcane plantations. 
    more » « less
  5. Abstract We report the first direct detection of molecular hydrogen associated with the Galactic nuclear wind. The Far-Ultraviolet Spectroscopic Explorer spectrum of LS 4825, a B1 Ib–II star at l , b = 1.67°,−6.63° lying d = 9.9 − 0.8 + 1.4 kpc from the Sun, ∼1 kpc below the Galactic plane near the Galactic center, shows two high-velocity H 2 components at v LSR = −79 and −108 km s −1 . In contrast, the FUSE spectrum of the nearby (∼0.6° away) foreground star HD 167402 at d = 4.9 − 0.7 + 0.8 kpc reveals no H 2 absorption at these velocities. Over 60 lines of H 2 from rotational levels J = 0 to 5 are identified in the high-velocity clouds. For the v LSR = −79 km s −1 cloud we measure total log N (H 2 ) ≥ 16.75 cm −2 , molecular fraction f H 2 ≥ 0.8%, and T 01 ≥ 97 and T 25 ≤ 439 K for the ground- and excited-state rotational excitation temperatures. At v LSR = −108 km s −1 , we measure log N (H 2 ) = 16.13 ± 0.10 cm −2 , f H 2 ≥ 0.5%, and T 01 = 77 − 18 + 34 and T 25 = 1092 − 117 + 149 K, for which the excited-state ortho- to para-H 2 is 1.0 − 0.1 + 0.3 , much less than the equilibrium value of 3 expected for gas at this temperature. This nonequilibrium ratio suggests that the −108 km s −1 cloud has been recently excited and has not yet had time to equilibrate. As the LS 4825 sight line passes close by a tilted section of the Galactic disk, we propose that we are probing a boundary region where the nuclear wind is removing gas from the disk. 
    more » « less