More Like this

1. Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions

   Choi, Jason; Castañeda, Fernando; Tomlin, Claire J.; Sreenath, Koushil (July 2020, Robotics: Science and Systems (RSS))

   In this paper, the issue of model uncertainty in safety-critical control is addressed with a data-driven approach. For this purpose, we utilize the structure of an input-output linearization controller based on a nominal model along with a Control Barrier Function and Control Lyapunov Function based Quadratic Program (CBF-CLF-QP). Specifically, we propose a novel reinforcement learning framework which learns the model uncertainty present in the CBF and CLF constraints, as well as other control-affine dynamic constraints in the quadratic program. The trained policy is combined with the nominal model based CBF-CLF-QP, resulting in the Reinforcement Learning based CBF-CLF-QP (RL-CBF-CLF-QP), which addresses the problem of model uncertainty in the safety constraints. The performance of the proposed method is validated by testing it on an underactuated nonlinear bipedal robot walking on randomly spaced stepping stones with one step preview, obtaining stable and safe walking under model uncertainty. 

   more » « less
2. Model Predictive Control Barrier Functions: Guaranteed Safety with Reduced Conservatism and Shortened Horizon

   https://doi.org/10.23919/ACC60939.2024.10644741  

   Abdi, Hossein; Zhao, Pan; Hovakimyan, Naira; Ghabcheloo, Reza (July 2024, IEEE)

   In this study, we address the problem of safe control in systems subject to state and input constraints by integrating the Control Barrier Function (CBF) into the Model Predictive Control (MPC) formulation. While CBF offers a conservative policy and traditional MPC lacks the safety guarantee beyond the finite horizon, the proposed scheme takes advantage of both MPC and CBF approaches to provide a guaranteed safe control policy with reduced conservatism and a shortened horizon. The proposed methodology leverages the sum-of-square (SOS) technique to construct CBFs that make forward invariant safe sets in the state space that are then used as a terminal constraint on the last predicted state. CBF invariant sets cover the state space around system fixed points. These islands of forward invariant CBF sets will be connected to each other using MPC. To do this, we proposed a technique to handle the MPC optimization problem subject to the combination of intersections and union of constraints. Our approach, termed Model Predictive Control Barrier Functions (MPCBF), is validated using numerical examples to demonstrate its efficacy, showing improved performance compared to classical MPC and CBF. 

   more » « less
3. Data‐Driven Distributed Safe Control Design for Multi‐Agent Systems

   https://doi.org/10.1002/acs.4088  

   Khaledi, Marjan; Kiumarsi, Bahare (January 2026, International Journal of Adaptive Control and Signal Processing)

   ABSTRACT This paper presents a data‐driven control barrier function (CBF) technique for ensuring safe control of multi‐agent systems (MASs) with uncertain linear dynamics. A data‐driven quadratic programming (QP) optimization is first developed for CBF‐based safe control of single‐agent systems using a nonlinear controller. This approach is then extended to the distributed safe control of MASs. To bypass system identification, the closed‐loop dynamics are represented using collected data, and the safety constraints are imposed on this closed‐loop representation. This data‐efficient representation is subsequently integrated into QP optimizations, resulting in data‐driven QP formulations that learn the closed‐loop systems and their corresponding controllers, ensuring the safety of the MASs. As a result, the presented CBF‐based approach designs a safe controller based solely on input, state, and state‐derivative measurements without requiring knowledge of the underlying dynamics of the agents. Furthermore, for the special case of linear controllers, we show that the need for state‐derivative measurements can be eliminated. We also show that the sample complexity of learning closed‐loop dynamics is less than that of its model‐based counterpart, which relies on open‐loop system identification. The simulation results for a safe formation control problem demonstrate the efficacy of the proposed approach for the MASs. 

   more » « less
4. Safe and Efficient Reinforcement Learning Using Disturbance-Observer-Based Control Barrier Functions

   Cheng, Yikun; Zhao, Pan; Hovakimyan, Naira (June 2023, Proceedings of Machine Learning Research)

   Safe reinforcement learning (RL) with assured satisfaction of hard state constraints during training has recently received a lot of attention. Safety filters, e.g., based on control barrier functions (CBFs), provide a promising way for safe RL via modifying the unsafe actions of an RL agent on the fly. Existing safety filter-based approaches typically involve learning of uncertain dynamics and quantifying the learned model error, which leads to conservative filters before a large amount of data is collected to learn a good model, thereby preventing efficient exploration. This paper presents a method for safe and efficient RL using disturbance observers (DOBs) and control barrier functions (CBFs). Unlike most existing safe RL methods that deal with hard state constraints, our method does not involve model learning, and leverages DOBs to accurately estimate the pointwise value of the uncertainty, which is then incorporated into a robust CBF condition to generate safe actions. The DOB-based CBF can be used as a safety filter with model-free RL algorithms by minimally modifying the actions of an RL agent whenever necessary to ensure safety throughout the learning process. Simulation results on a unicycle and a 2D quadrotor demonstrate that the proposed method outperforms a state-of-the-art safe RL algorithm using CBFs and Gaussian processes-based model learning, in terms of safety violation rate, and sample and computational efficiency. 

   more » « less
5. Safe and Efficient Reinforcement Learning using Disturbance-Observer-Based Control Barrier Functions

   Cheng, Yikun; Zhao, Pan; Hovakimyan, Naira (June 2023, Proceedings of The 5th Annual Learning for Dynamics and Control Conference)

   Matni, Nikolai; Morari, Manfred; Pappas, George J. (Ed.)

   Safe reinforcement learning (RL) with assured satisfaction of hard state constraints during training has recently received a lot of attention. Safety filters, e.g., based on control barrier functions (CBFs), provide a promising way for safe RL via modifying the unsafe actions of an RL agent on the fly. Existing safety filter-based approaches typically involve learning of uncertain dynamics and quantifying the learned model error, which leads to conservative filters before a large amount of data is collected to learn a good model, thereby preventing efficient exploration. This paper presents a method for safe and efficient RL using disturbance observers (DOBs) and control barrier functions (CBFs). Unlike most existing safe RL methods that deal with hard state constraints, our method does not involve model learning, and leverages DOBs to accurately estimate the pointwise value of the uncertainty, which is then incorporated into a robust CBF condition to generate safe actions. The DOB-based CBF can be used as a safety filter with model-free RL algorithms by minimally modifying the actions of an RL agent whenever necessary to ensure safety throughout the learning process. Simulation results on a unicycle and a 2D quadrotor demonstrate that the proposed method outperforms a state-of-the-art safe RL algorithm using CBFs and Gaussian processes-based model learning, in terms of safety violation rate, and sample and computational efficiency. 

   more » « less