skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Learning Object-Centric Dynamic Modes from Video and Emerging Properties
One of the long-term objectives of Machine Learning is to endow machines with the capacity of structuring and interpreting the world as we do. This is particularly challenging in scenes involving time series, such as video sequences, since seemingly different data can correspond to the same underlying dynamics. Recent approaches seek to decompose video sequences into their composing objects, attributes and dynamics in a self-supervised fashion, thus simplifying the task of learning suitable features that can be used to analyze each component. While existing methods can successfully disentangle dynamics from other components, there have been relatively few efforts in learning parsimonious representations of these underlying dynamics. In this paper, motivated by recent advances in non-linear identification, we propose a method to decompose a video into moving objects, their attributes and the dynamic modes of their trajectories. We model video dynamics as the output of a Koopman operator to be learned from the available data. In this context, the dynamic information contained in the scene is encapsulated in the eigenvalues and eigenvectors of the Koopman operator, providing an interpretable and parsimonious representation. We show that such decomposition can be used for instance to perform video analytics, predict future frames or generate synthetic video. We test our framework in a variety of datasets that encompass different dynamic scenarios, while illustrating the novel features that emerge from our dynamic modes decomposition: Video dynamics interpretation and user manipulation at test-time. We successfully forecast challenging object trajectories from pixels, achieving competitive performance while drawing useful insights.  more » « less
Award ID(s):
2038493 1814631
PAR ID:
10466350
Author(s) / Creator(s):
; ; ; ; ;
Editor(s):
Matni, N; Morari, M; Pappas, G.J.
Date Published:
Journal Name:
Proceedings of Machine Learning Research
Volume:
211
ISSN:
2640-3498
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; (, Journal of Fluid Mechanics)
    null (Ed.)
    Koopman decomposition is a nonlinear generalization of eigen-decomposition, and is being increasingly utilized in the analysis of spatio-temporal dynamics. Well-known techniques such as the dynamic mode decomposition (DMD) and its linear variants provide approximations to the Koopman operator, and have been applied extensively in many fluid dynamic problems. Despite being endowed with a richer dictionary of nonlinear observables, nonlinear variants of the DMD, such as extended/kernel dynamic mode decomposition (EDMD/KDMD) are seldom applied to large-scale problems primarily due to the difficulty of discerning the Koopman-invariant subspace from thousands of resulting Koopman eigenmodes. To address this issue, we propose a framework based on a multi-task feature learning to extract the most informative Koopman-invariant subspace by removing redundant and spurious Koopman triplets. In particular, we develop a pruning procedure that penalizes departure from linear evolution. These algorithms can be viewed as sparsity-promoting extensions of EDMD/KDMD. Furthermore, we extend KDMD to a continuous-time setting and show a relationship between the present algorithm, sparsity-promoting DMD and an empirical criterion from the viewpoint of non-convex optimization. The effectiveness of our algorithm is demonstrated on examples ranging from simple dynamical systems to two-dimensional cylinder wake flows at different Reynolds numbers and a three-dimensional turbulent ship-airwake flow. The latter two problems are designed such that very strong nonlinear transients are present, thus requiring an accurate approximation of the Koopman operator. Underlying physical mechanisms are analysed, with an emphasis on characterizing transient dynamics. The results are compared with existing theoretical expositions and numerical approximations. 
    more » « less
  2. ; (, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences)
    Approximating the Koopman operator from data is numerically challenging when many lifting functions are considered. Even low-dimensional systems can yield unstable or ill-conditioned results in a high-dimensional lifted space. In this paper, Extended Dynamic Mode Decomposition (DMD) and DMD with control, two methods for approximating the Koopman operator, are reformulated as convex optimization problems with linear matrix inequality constraints. Asymptotic stability constraints and system norm regularizers are then incorporated as methods to improve the numerical conditioning of the Koopman operator. Specifically, the H ∞   norm is used to penalize the input–output gain of the Koopman system. Weighting functions are then applied to penalize the system gain at specific frequencies. These constraints and regularizers introduce bilinear matrix inequality constraints to the regression problem, which are handled by solving a sequence of convex optimization problems. Experimental results using data from an aircraft fatigue structural test rig and a soft robot arm highlight the advantages of the proposed regression methods. 
    more » « less
  3. ; ; (, IEEE Transactions on Automatic Control)
    We devise a novel formulation and propose the concept of modal participation factors to nonlinear dynamical systems. The original definition of modal participation factors (or simply participation factors) provides a simple yet effective metric. It finds use in theory and practice, quantifying the interplay between states and modes of oscillation in a linear time-invariant (LTI) system. In this paper, with the Koopman operator framework, we present the results of participation factors for nonlinear dynamical systems with an asymptotically stable equilibrium point or limit cycle. We show that participation factors are defined for the entire domain of attraction, beyond the vicinity of an attractor, where the original definition of participation factors for LTI systems is a special case. Finally, we develop a numerical method to estimate participation factors using time series data from the underlying nonlinear dynamical system. The numerical method can be implemented by leveraging a well-established numerical scheme in the Koopman operator framework called dynamic mode decomposition. 
    more » « less
  4. ; (, Ocean Engineering)
    Developing an accurate dynamic model for an Autonomous Underwater Vehicle (AUV) is challenging due to the diverse array of forces exerted on it in an underwater environment. These forces include hydrodynamic effects such as drag, buoyancy, and added mass. Consequently, achieving precision in predicting the AUV's behavior requires a comprehensive understanding of these dynamic forces and their interplay. In our research, we have devised a linear data-driven dynamic model rooted in Koopman's theory. The cornerstone of leveraging Koopman theory lies in accurately estimating the Koopman operator. To achieve this, we employ the dynamic mode decomposition (DMD) method, which enables the generation of the Koopman operator. We have developed a Fractional Sliding Mode Control (FSMC) method to provide robustness and high tracking performance for AUV systems. The efficacy of the proposed controller has been verified through simulation results. 
    more » « less
  5. ; (, IEEE)
    The modeling of nonlinear dynamics based on Koopman operator theory, originally applicable only to autonomous systems with no control, is extended to nonautonomous control system without approximation of the input matrix. Prevailing methods using a least square estimate of the input matrix may result in an erroneous input matrix, misinforming the controller. Here, a new method for constructing a Koopman model that yields the exact input matrix is presented. A set of state variables are introduced so that the control inputs are linearly involved in the dynamics of actuators. With these variables, a lifted linear model with the exact input matrix, called a Control-Coherent Koopman Model, is constructed by superposing control input terms, which are linear in local actuator dynamics, to the Koopman operator of the associated autonomous nonlinear system. As an example, the proposed method is applied to multi degree-of-freedom robotic arms, which are controlled with Model Predictive Control (MPC). It is demonstrated that the prevailing Dynamic Mode Decomposition with Control (DMDc) using an approximate input matrix does not provide a satisfactory result, while the Control-Coherent Koopman Model performs well with the correct input matrix, even performing better than the bilinear formulation of the Koopman operator. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email