More Like this

1. Sparsity-promoting algorithms for the discovery of informative Koopman-invariant subspaces

   https://doi.org/10.1017/jfm.2021.271  

   Pan, Shaowu; Arnold-Medabalimi, Nicholas; Duraisamy, Karthik (June 2021, Journal of Fluid Mechanics)

   null (Ed.)

   Koopman decomposition is a nonlinear generalization of eigen-decomposition, and is being increasingly utilized in the analysis of spatio-temporal dynamics. Well-known techniques such as the dynamic mode decomposition (DMD) and its linear variants provide approximations to the Koopman operator, and have been applied extensively in many fluid dynamic problems. Despite being endowed with a richer dictionary of nonlinear observables, nonlinear variants of the DMD, such as extended/kernel dynamic mode decomposition (EDMD/KDMD) are seldom applied to large-scale problems primarily due to the difficulty of discerning the Koopman-invariant subspace from thousands of resulting Koopman eigenmodes. To address this issue, we propose a framework based on a multi-task feature learning to extract the most informative Koopman-invariant subspace by removing redundant and spurious Koopman triplets. In particular, we develop a pruning procedure that penalizes departure from linear evolution. These algorithms can be viewed as sparsity-promoting extensions of EDMD/KDMD. Furthermore, we extend KDMD to a continuous-time setting and show a relationship between the present algorithm, sparsity-promoting DMD and an empirical criterion from the viewpoint of non-convex optimization. The effectiveness of our algorithm is demonstrated on examples ranging from simple dynamical systems to two-dimensional cylinder wake flows at different Reynolds numbers and a three-dimensional turbulent ship-airwake flow. The latter two problems are designed such that very strong nonlinear transients are present, thus requiring an accurate approximation of the Koopman operator. Underlying physical mechanisms are analysed, with an emphasis on characterizing transient dynamics. The results are compared with existing theoretical expositions and numerical approximations. 

   more » « less
2. Tensor-Based Koopman Operator and Its Application to Optimal Control Problems

   https://doi.org/10.2514/1.G008913  

   Xu, Zhi; Dai, Ran; Howell, Kathleen C (June 2025, Journal of Guidance, Control, and Dynamics)

   The Koopman operator theory provides a global linearization framework for general nonlinear dynamics, offering significant advantages for system analysis and control. However, practical applications typically involve approximating the infinite-dimensional Koopman operator in a lifted space spanned by a finite set of observable functions. The accuracy of this approximation is the key to effective Koopman operator-based analysis and control methods, generally improving as the dimension of the observables increases. Nonetheless, this increase in dimensionality significantly escalates both storage requirements and computational complexity, particularly for high-dimensional systems, thereby limiting the applicability of these methods in real-world problems. In this paper, we address this problem by reformulating the Koopman operator in tensor format to break the curse of dimensionality associated with its approximation through tensor decomposition techniques. This effective reduction in complexity enables the selection of high-dimensional observable functions and the handling of large-scale datasets, which leads to a precise linear prediction model utilizing the tensor-based Koopman operator. Furthermore, we propose an optimal control framework with the tensor-based Koopman operator, which adeptly addresses the nonlinear dynamics and constraints by linear reformulation in the lifted space and significantly reduces the computational complexity through separated representation of the tensor structure. 

   more » « less
3. Convex Geometry and Duality of Over-parameterized Neural Networks

   Ergen, Tolga; Pilanci, Mert (January 2000, International Conference on Artificial Intelligence and Statistics (AISTATS 2020))

   null (Ed.)

   We develop a convex analytic framework for ReLU neural networks which elucidates the inner workings of hidden neurons and their function space characteristics. We show that neural networks with rectified linear units act as convex regularizers, where simple solutions are encouraged via extreme points of a certain convex set. For one dimensional regression and classification, as well as rank-one data matrices, we prove that finite two-layer ReLU networks with norm regularization yield linear spline interpolation. We characterize the classification decision regions in terms of a closed form kernel matrix and minimum L1 norm solutions. This is in contrast to Neural Tangent Kernel which is unable to explain neural network predictions with finitely many neurons. Our convex geometric description also provides intuitive explanations of hidden neurons as auto encoders. In higher dimensions, we show that the training problem for two-layer networks can be cast as a finite dimensional convex optimization problem with infinitely many constraints. We then provide a family of convex relaxations to approximate the solution, and a cutting-plane algorithm to improve the relaxations. We derive conditions for the exactness of the relaxations and provide simple closed form formulas for the optimal neural network weights in certain cases. We also establish a connection to ℓ0-ℓ1 equivalence for neural networks analogous to the minimal cardinality solutions in compressed sensing. Extensive experimental results show that the proposed approach yields interpretable and accurate models. 

   more » « less
4. Beyond expectations: residual dynamic mode decomposition and variance for stochastic dynamical systems

   https://doi.org/10.1007/s11071-023-09135-w  

   Colbrook, Matthew_J; Li, Qin; Raut, Ryan_V; Townsend, Alex (December 2023, Nonlinear Dynamics)

   Abstract Koopman operators linearize nonlinear dynamical systems, making their spectral information of crucial interest. Numerous algorithms have been developed to approximate these spectral properties, and dynamic mode decomposition (DMD) stands out as the poster child of projection-based methods. Although the Koopman operator itself is linear, the fact that it acts in an infinite-dimensional space of observables poses challenges. These include spurious modes, essential spectra, and the verification of Koopman mode decompositions. While recent work has addressed these challenges for deterministic systems, there remains a notable gap in verified DMD methods for stochastic systems, where the Koopman operator measures the expectation of observables. We show that it is necessary to go beyond expectations to address these issues. By incorporating variance into the Koopman framework, we address these challenges. Through an additional DMD-type matrix, we approximate the sum of a squared residual and a variance term, each of which can be approximated individually using batched snapshot data. This allows verified computation of the spectral properties of stochastic Koopman operators, controlling the projection error. We also introduce the concept of variance-pseudospectra to gauge statistical coherency. Finally, we present a suite of convergence results for the spectral information of stochastic Koopman operators. Our study concludes with practical applications using both simulated and experimental data. In neural recordings from awake mice, we demonstrate how variance-pseudospectra can reveal physiologically significant information unavailable to standard expectation-based dynamical models. 

   more » « less
5. Nuclear Norm Regularization for Deep Learning

   Scarvelis, Christopher; Solomon, Justin (December 2024, NeurIPS Proceedings)

   Penalizing the nuclear norm of a function's Jacobian encourages it to locally behave like a low-rank linear map. Such functions vary locally along only a handful of directions, making the Jacobian nuclear norm a natural regularizer for machine learning problems. However, this regularizer is intractable for high-dimensional problems, as it requires computing a large Jacobian matrix and taking its SVD. We show how to efficiently penalize the Jacobian nuclear norm using techniques tailor-made for deep learning. We prove that for functions parametrized as compositions f=g∘h, one may equivalently penalize the average squared Frobenius norm of Jg and Jh. We then propose a denoising-style approximation that avoids the Jacobian computations altogether. Our method is simple, efficient, and accurate, enabling Jacobian nuclear norm regularization to scale to high-dimensional deep learning problems. We complement our theory with an empirical study of our regularizer's performance and investigate applications to denoising and representation learning. 

   more » « less