An official website of the United States government
This work introduces a data-driven control approach for stabilizing high-dimensional dynamical systems from scarce data. The proposed context-aware controller inference approach is based on the observation that controllers need to act locally only on the unstable dynamics to stabilize systems. This means it is sufficient to learn the unstable dynamics alone, which are typically confined to much lower dimensional spaces than the high-dimensional state spaces of all system dynamics and thus few data samples are sufficient to identify them. Numerical experiments demonstrate that context-aware controller inference learns stabilizing controllers from orders of magnitude fewer data samples than traditional data-driven control techniques and variants of reinforcement learning. The experiments further show that the low data requirements of context-aware controller inference are especially beneficial in data-scarce engineering problems with complex physics, for which learning complete system dynamics is often intractable in terms of data and training costs.
more »
« less
Learning to stabilize high-dimensional unknown systems using Lyapunov-guided exploration
Designing stabilizing controllers is a fundamental challenge in autonomous systems, particularly for high-dimensional, nonlinear systems that can hardly be accurately modeled with differential equations. The Lyapunov theory offers a solution for stabilizing control systems, still, current methods relying on Lyapunov functions require access to complete dynamics or samples of system executions throughout the entire state space. Consequently, they are impractical for high-dimensional systems. This paper introduces a novel framework, LYapunov-Guided Exploration (LYGE), for learning stabilizing controllers tailored to high-dimensional, unknown systems. LYGE employs Lyapunov theory to iteratively guide the search for samples during exploration while simultaneously learning the local system dynamics, control policy, and Lyapunov functions. We demonstrate its scalability on highly complex systems, including a high-fidelity F-16 jet model featuring a 16D state space and a 4D input space. Experiments indicate that, compared to prior works in reinforcement learning, imitation learning, and neural certificates, LYGE reduces the distance to the goal by 50% while requiring only 5% to 32% of the samples. Furthermore, we demonstrate that our algorithm can be extended to learn controllers guided by other certificate functions for unknown systems.
more »
« less
- Award ID(s):
- 2238030
- PAR ID:
- 10554286
- Publisher / Repository:
- Proceedings of the 6th Annual Learning for Dynamics & Control Conference, PMLR
- Date Published:
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Learning a dynamical system requires stabilizing the unknown dynamics to avoid state blow-ups. However, the standard reinforcement learning (RL) methods lack formal stabilization guarantees, which limits their applicability for the control of real-world dynamical systems. We propose a novel policy optimization method that adopts Krasovskii's family of Lyapunov functions as a stability constraint. We show that solving this stability-constrained optimization problem using a primal-dual approach recovers a stabilizing policy for the underlying system even under modeling error. Combining this method with model learning, we propose a model-based RL framework with formal stability guarantees, Krasovskii-Constrained Reinforcement Learning (KCRL). We theoretically study KCRL with kernel-based feature representation in model learning and provide a sample complexity guarantee to learn a stabilizing controller for the underlying system. Further, we empirically demonstrate the effectiveness of KCRL in learning stabilizing policies in online voltage control of a distributed power system. We show that KCRL stabilizes the system under various real-world solar and electricity demand profiles, whereas standard RL methods often fail to stabilize.more » « less
-
Learning policies for contact-rich manipulation is a challenging problem due to the presence of multiple contact modes with different dynamics, which complicates state and action exploration. Contact-rich motion planning uses simplified dynamics to reduce the search space dimension, but the found plans are then difficult to execute under the true object-manipulator dynamics. This paper presents an algorithm for learning controllers based on guided policy search, where motion plans based on simplified dynamics define rewards and sampling distributions for policy gradient-based learning. We demonstrate that our guided policy search method improves the ability to learn manipulation controllers, through a task involving pushing a box over a step.more » « less
-
Abstarct This work presents a theoretical framework for the safety‐critical control of time delay systems. The theory of control barrier functions, that provides formal safety guarantees for delay‐free systems, is extended to systems with state delay. The notion of control barrier functionals is introduced, to attain formal safety guarantees by enforcing the forward invariance of safe sets defined in the infinite dimensional state space. The proposed framework is able to handle multiple delays and distributed delays both in the dynamics and in the safety condition, and provides an affine constraint on the control input that yields provable safety. This constraint can be incorporated into optimization problems to synthesize pointwise optimal and provable safe controllers. The applicability of the proposed method is demonstrated by numerical simulation examples.more » « less
-
Perfect adaptation, the ability to regulate and maintain gene expression to its desired value despite disturbances, is important in the development of organisms. Building biological controllers to endow engineered biological systems with such perfect adaptation capability is a key goal in synthetic biology. Model-guided exploration of such synthetic circuits has been effective in designing such systems. However, theoretical analysis to guarantee controller properties with nonlinear models, such as Hill functions, remains challenging, while use of linear models fails to capture the inherent nonlinear dynamics of gene expression systems. Here, we propose a reverse engineering approach to infer the kinetic parameters for nonlinear Hill function-type models from analysis of linear models and apply our method to design controllers, which achieve perfect adaptation. Focusing on three biological network motif-based controllers, we demonstrate via simulation the efficacy of the proposed approach in combining linear system theories with nonlinear modelling, to design multiple gene circuits that could deliver perfect adaptation. Given the ubiquitous use of Hill functions in describing the dynamics of biological regulatory networks, we anticipate the proposed reverse engineering approach to benefit a wide range of systems and synthetic biology applications.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
