More Like this

1. Guiding Safe Exploration with Weakest Preconditions

   Greg Anderson ; Swarat Chaudhuri ; Isil Dillig ( January 2023 , International Conference on Learning Representations)

   In reinforcement learning for safety-critical settings, it is often desirable for the agent to obey safety constraints at all points in time, including during training. We present a novel neurosymbolic approach called SPICE to solve this safe exploration problem. SPICE uses an online shielding layer based on symbolic weakest preconditions to achieve a more precise safety analysis than existing tools without unduly impacting the training process. We evaluate the approach on a suite of continuous control benchmarks and show that it can achieve comparable performance to existing safe learning techniques while incurring fewer safety violations. Additionally, we present theoretical results showing that SPICE converges to the optimal safe policy under reasonable assumptions. 

   more » « less
2. Safe adaptive output‐feedback optimal control of a class of linear systems

   https://doi.org/10.1002/rnc.7334  

   Mahmud, S_M Nahid ; Abudia, Moad ; Nivison, Scott A ; Bell, Zachary I ; Kamalapurkar, Rushikesh ( July 2024 , International Journal of Robust and Nonlinear Control)

   The objective of this research is to enable safety‐critical systems to simultaneously learn and execute optimal control policies in a safe manner to achieve complex autonomy. Learning optimal policies via trial and error, that is, traditional reinforcement learning, is difficult to implement in safety‐critical systems, particularly when task restarts are unavailable. Safe model‐based reinforcement learning techniques based on a barrier transformation have recently been developed to address this problem. However, these methods rely on full‐state feedback, limiting their usability in a real‐world environment. In this work, an output‐feedback safe model‐based reinforcement learning technique based on a novel barrier‐aware dynamic state estimator has been designed to address this issue. The developed approach facilitates simultaneous learning and execution of safe control policies for safety‐critical linear systems. Simulation results indicate that barrier transformation is an effective approach to achieve online reinforcement learning in safety‐critical systems using output feedback. 

   more » « less
3. CAT: Closed-loop Adversarial Training for Safe End-to-End Driving

   Zhang, Linrui ; Peng, Zhenghao ; Li, Quanyi ; Zhou, Bolei. ( November 2023 , Conference on Robot Learning (CoRL) 2023)

   Driving safety is a top priority for autonomous vehicles. Orthogonal to prior work handling accident-prone traffic events by algorithm designs at the policy level, we investigate a Closed-loop Adversarial Training (CAT) framework for safe end-to-end driving in this paper through the lens of environment augmentation. CAT aims to continuously improve the safety of driving agents by training the agent on safety-critical scenarios that are dynamically generated over time. A novel resampling technique is developed to turn log-replay real-world driving scenarios into safety-critical ones via probabilistic factorization, where the adversarial traffic generation is modeled as the multiplication of standard motion prediction sub-problems. Consequently, CAT can launch more efficient physical attacks compared to existing safety-critical scenario generation methods and yields a significantly less computational cost in the iterative learning pipeline. We incorporate CAT into the MetaDrive simulator and validate our approach on hundreds of driving scenarios imported from real-world driving datasets. Experimental results demonstrate that CAT can effectively generate adversarial scenarios countering the agent being trained. After training, the agent can achieve superior driving safety in both log-replay and safety-critical traffic scenarios on the held- out test set. Code and data are available at https://metadriverse.github.io/cat. 

   more » « less
4. Safe and Efficient Reinforcement Learning using Disturbance-Observer-Based Control Barrier Functions

   Cheng, Yikun ; Zhao, Pan ; Hovakimyan, Naira ( June 2023 , Proceedings of The 5th Annual Learning for Dynamics and Control Conference)

   Matni, Nikolai ; Morari, Manfred ; Pappas, George J. (Ed.)

   Safe reinforcement learning (RL) with assured satisfaction of hard state constraints during training has recently received a lot of attention. Safety filters, e.g., based on control barrier functions (CBFs), provide a promising way for safe RL via modifying the unsafe actions of an RL agent on the fly. Existing safety filter-based approaches typically involve learning of uncertain dynamics and quantifying the learned model error, which leads to conservative filters before a large amount of data is collected to learn a good model, thereby preventing efficient exploration. This paper presents a method for safe and efficient RL using disturbance observers (DOBs) and control barrier functions (CBFs). Unlike most existing safe RL methods that deal with hard state constraints, our method does not involve model learning, and leverages DOBs to accurately estimate the pointwise value of the uncertainty, which is then incorporated into a robust CBF condition to generate safe actions. The DOB-based CBF can be used as a safety filter with model-free RL algorithms by minimally modifying the actions of an RL agent whenever necessary to ensure safety throughout the learning process. Simulation results on a unicycle and a 2D quadrotor demonstrate that the proposed method outperforms a state-of-the-art safe RL algorithm using CBFs and Gaussian processes-based model learning, in terms of safety violation rate, and sample and computational efficiency. 

   more » « less
5. Safe and Efficient Reinforcement Learning Using Disturbance-Observer-Based Control Barrier Functions

   Cheng, Yikun ; Zhao, Pan ; Hovakimyan, Naira ( June 2023 , Proceedings of Machine Learning Research)

   Safe reinforcement learning (RL) with assured satisfaction of hard state constraints during training has recently received a lot of attention. Safety filters, e.g., based on control barrier functions (CBFs), provide a promising way for safe RL via modifying the unsafe actions of an RL agent on the fly. Existing safety filter-based approaches typically involve learning of uncertain dynamics and quantifying the learned model error, which leads to conservative filters before a large amount of data is collected to learn a good model, thereby preventing efficient exploration. This paper presents a method for safe and efficient RL using disturbance observers (DOBs) and control barrier functions (CBFs). Unlike most existing safe RL methods that deal with hard state constraints, our method does not involve model learning, and leverages DOBs to accurately estimate the pointwise value of the uncertainty, which is then incorporated into a robust CBF condition to generate safe actions. The DOB-based CBF can be used as a safety filter with model-free RL algorithms by minimally modifying the actions of an RL agent whenever necessary to ensure safety throughout the learning process. Simulation results on a unicycle and a 2D quadrotor demonstrate that the proposed method outperforms a state-of-the-art safe RL algorithm using CBFs and Gaussian processes-based model learning, in terms of safety violation rate, and sample and computational efficiency. 

   more » « less