Reinforcement learning (RL) has demonstrated impressive performance in robotic control in recent years. Many real-world systems are subject to safety constraints. As a result, safe RL has recently received a lot of attention, although there are different denitions of "safety" Garcıa and Fernández (2015); Brunke et al. (2022). We limit our discussion to safe RL that aims to ensure satisfaction of hard state constraints all the time during both training and deployment.
Among different safe RL paradigms, a commonly used one is to leverage safety lters (SFs) to constrain the actions of RL agents and modify them whenever necessary to ensure satisfaction of safety constraints. The advantages of this paradigm mainly lie in its exibility, i.e., a safety lter can often work with many existing RL algorithms without (many) modications to the RL algorithms. Along this line, researchers have proposed different safety lters based on shielding Alshiekh et al. (2018), control barrier functions (CBFs) Cheng et al. (2019); Ohnishi et al. (2019); Emam et al. (2021), Hamilton-Jacobi reachability (HJR) Fisac et al. (2018), and model predictive safety certication (MPSC) Wabersich et al. (2021). Among these different SFs, the shielding SF of Alshiekh Assumption 1 There exist positive constants l d and b d such that for any x, y ∈ X , the following inequalities hold:
(3)
Moreover, the constants l d and b d are known.
Remark 1 Assumption 1 is a fairly standard assumption in nonlinear systems stating that the uncertain function d(x) is locally Lipschitz continuous with a known bound on the Lipschitz constant in the compact set X and is bounded by a known constant at the origin. The Lipschitz constant and uniform bounds will affect the estimation error bound when using a DOB to estimate the uncertainty d(x), as characterized in Lemma 4.
Reinforcement learning aims to nd an optimal policy π * in an environment which can be formulated as a Markov decision process (MDP). In this work, an MDP is dened by a tuple (S, A, p, r), where the state space S and the action space A are continuous. Given the current state s t ∈ S and action a t ∈ A, the transition function p : S × S × A → [0, ∞) represents the probability density of the succeeding state s t+1 ∈ S. The reward function r : S × A → [r min , r max ] determines a bounded reward for each transition.
Our proposed safe RL scheme can work with any model-free RL algorithm. For illustration and experimental demonstration in Section 4, we choose soft actor-critic (SAC) Haarnoja et al. (2018), a state-of-the-art model-free RL algorithm. SAC uses an off-policy formulation that reuses historical data to improve sample efciency and utilizes entropy maximization to improve the stability of the training process. In general, SAC aims to nd a policy to maximize an entropy objective which is formed as
, where H(•) is the entropy term that incentivizes exploration, α is a positive parameter to determine the relative importance of the entropy term against the reward, ρ π denotes the states and actions distribution induced by the policy π, and T is the termination time.
The CBFs are introduced in Ames et al. (2016) to synthesize control laws to ensure forward invariance of some sets (often related to safety) for nonlinear control-afne systems. They are often used as safety lters to modify a baseline control law to ensure that the system stays in a safety set. Consider a set
where h(x) is a continuously differentiable function h. A function β : (-b, a) → (-∞, ∞) is said to belong to extended class K for some a, b > 0 if it is strictly increasing and β(0) = 0.
Denition 2 (CBF Ames et al. ( 2016)). Given a set C dened using h(x) via (4), h(x) is a control barrier function for (1) if there exists an extended class
where
. We dene the input relative degree (disturbance relative degree) of a differentiable function h : R n → R with respect to (1) as the number of times we need to differentiate it along (1) until the input u (the disturbance d) explicitly shows up. Condition (5) works only for constraints with input relative degrees (IRDs) of one. To handle constraints with higher IRDs, high-order CBFs are introduced in Xiao and Belta (2021). Before introducing high-order CBFs, we make the following assumption, which indicates that the input u and the disturbance d show up together when differentiating h.
The disturbance relative degree is equal to the input relative degree.
Dene a sequence of functions ϕ i : R n → R, i ∈ {1, ..., m} as:
Furthermore, dene an associate sequence of sets as:
Denition 3 (High-Order CBF under Perturbed System Dynamics). Consider a sequential function ϕ i (x) dened in ( 6) and a sequential set C i , i ∈ {1, ..., m} dened in ( 7). Under Assumption 2, an m th -order differentiable function h :
where
The true uncertainty d in Denitions 2 and 3 is not accessible in practice. Therefore, it is impossible to evaluate whether a function h(x) obeys the constraints in ( 5) or ( 8). One solution is to derive a sufcient condition for (5) or (8) using a uniform bound for the uncertainty d(x), as adopted in Zhao et al. (2020); Nguyen and Sreenath (2016). In the following, we will derive an alternative sufcient condition to dene the so-called DOB-CBFs.
Disturbance observers have been widely used in control of uncertain systems Chen et al. (2015). All different types of DOBs share a common idea, i.e., lumping all the uncertainties (that may consist of unknown parameters, unmodeled dynamics and external disturbances) together as a total disturbance and estimate its value at each time instant. In this work, we leverage the DOB presented in Zhao et al. (2020), which is inspired by the piecewise-constant (PC) adaptive law Wang et al. (2017) and (Hovakimyan and Cao, 2010, Section 3.3), and was used in L 1 adaptive control of manned aircraft Ackerman et al. (2017Ackerman et al. ( , 2019)), and in learning-enabled control Gahlawat et al. (2020); Cheng et al. (2022b). The DOB contains two components, i.e., a state predictor and a PC estimation law to estimate the disturbance. For the disturbed system (1), the state predictor is given by
where x = xx denotes the prediction error, a > 0 is a constant, and d(t) is the estimated disturbance. The disturbance estimation is updated according to
where T is the estimation sampling time. The estimation error bound associated with the DOB is given next.
Lemma 4 (Estimation Error Bound Zhao et al. ( 2020)) Given the uncertain system (1) subject to Assumption 1, and the DOB dened via ( 9) and ( 10), the estimation error can be bounded as
where η ≜ l d (max x∈X ,u∈U ∥f (x) + g(x)u∥ + θ). Moreover, lim T →0 γ(T ) = 0.
Remark 5 Lemma 4 implies that the estimated disturbance can be made arbitrarily accurate for t ≥ T , by reducing T , the latter only subject to hardware limitations, including measurement noises.
Remark 6 As seen from Lemma 4, larger values of the Lipschitz bound l d and the uniform bound b d (introduced in Assumption 1) lead to larger estimation error bounds. On the other hand, we found that the estimation error bound in (11) could be quite conservative in most cases, potentially due to the frequent use of Lipschitz continuity properties and triangular inequalities. Therefore, for practical implementation one could leverage some empirical study, e.g., doing simulations under a few user-selected functions of d(x) and determining a tighter bound than that dened in (11). The conservativeness also indicates that even if the constants l d and b d violate the conditions (3) to a certain extent, the error bound in (11) most likely still holds in practice.
In this section, we rst introduce high-order DOB-based CBFs (DOB-CBFs), by extending the result in Zhao et al. (2020), which only considers constraints with IRDs of one. The work in Das ¸and Murray (2022) considers high IRD constraints using exponential CBFs in the presence of matched uncertainties (which are injected to the system through the same channel as control inputs). Compared to Das ¸and Murray (2022), we do not constrain the uncertainties to be matched, and leverage high-order CBFs Xiao and Belta (2021) which are generalizations of exponential CBFs. Then, we introduce our DOB-CBF based safe RL scheme (DOB-CBF-RL).
Given the disturbance bound in Assumption 1 and the estimation error bound in Lemma 4, we have
The detailed proof could be found in the extended version of this paper Cheng et al. (2022a). Therefore, we have the following denition.
Denition 7 (DOB-CBF) Consider the system in (1), the DOB dened via (9) and (10) with an estimation error bound given by (11), and a sequential function ϕ i (x) dened in ( 6) and a sequential set C i , i ∈ {1, ..., m} dened in (7). Under Assumption 2, an m th -order differentiable function h : R n → R is a high-order DOB-based control barrier function of relative degree m for (1) if there exist extended class K functions β i , i ∈ {1, ..., m}, such that
It is obvious that if a control input u is a solution for (12), it also satises (8). We next dene
Then, the main theorem of the proposed approach is introduced as follows.
Theorem 8 Suppose the condition ( 12) holds. Then, the condition
is a sufcient condition for (8). It is also a necessary condition for ( 8) for any t ≥ T when T → 0.
The proof of Theorem 8 could be found in Cheng et al. (2022a).
Using the condition in ( 14) that depends on the estimated disturbance d(t), we can compute the safe control inputs via solving a quadratic programming (QP) problem dened as
where P is a positive-denite weighting matrix, u RL is the action of RL policy and u safe is the nal control input applied to the system (1) during both policy training and policy deployment. In case u RL satises the constraints of ( 15) and is therefore safe, we have u safe =u RL ; otherwise, (15) produces safe inputs that are mostly close to u RL . With the DOB-CBF-QP in (15), our proposed DOB-CBF-RL scheme is summarized in Algorithm 1. At each step during training, the vanilla RL policy determines a potentially unsafe action. This action is then modied by the DOB-CBF-QP in (15) to produce a safe action u safe that is applied to the environment. It is worth mentioning that the tuple (x t , u safe t , x t+1 , r t ) involving the safe action is added to the reply buffer D and used to update the policy. Using safe actions for policy training will promote the agent to learn a safe and optimal policy (although not guaranteed), which the DOB-CBF as a safety lter does not need to (frequently) intervene with.
Algorithm 1: DOB-CBF based safe RL Input: Initial SAC policy π θ , number of episodes N , number of steps per episode M , number of policy update G, nominal dynamics ẋ = f (x) + g(x)u, DOB dened via ( 9) and ( 10) with estimation error bound γ, for i = 1, ...., N do for t = 1, ..., M do Obtain action u RL t from policy π θ Obtain disturbance estimation dt from the DOB dened via ( 9) and ( 10
We use a unicycle and a 2D quadrotor to validate the efcacy of the proposed DOB-CBF-RL method. For comparison, we also implemented a state-of-the-art safe RL method based on CBFs and GPR-based model learning (denoted as GP-CBF-RL) in Cheng et al. (2019). The policy training was performed on a desktop with an INTEL i9-9980XE and an NVIDIA 3090Ti GPU and 64GB RAM. The codes are available at https://github.com/Adriancyk/Safe-RL-with-Disturbance-Observer.
A unicycle model was borrowed from Emam et al. (2021) and adapted. The state x = [p x , p y , θ] T , where p x and p y denote the robot position along the x-axis and y-axis, respectively, and θ is the counterclockwise angle between the positive direction of the x-axis and the head direction of the robot. The control inputs are the linear velocity v and angular velocity ω of the system. The goal is to navigate the unicycle from the red dot to the yellow dot without colliding with any obstacles, as shown in Figure 2 (Right). The vehicle's equations of motion are given by
where the uncertainty d m is introduced to mimic the slippery ground that causes the vehicle to lose partial control efciency. We train DOB-CBF-RL policy and the GP-CBF-RL policy separately and dene h i (x) = 1 2 ((∥p i,obs ∥ - p 2 x + p 2 y ) 2r 2 i,obs ), where i = 1, 2, 3, p i,obs denotes the location of ith obstacle in xy-plane, and r i,obs is the radius of the ith obstacle. The parameters of the system dynamics and the details of the RL training are included in Cheng et al. (2022a).
We can see from Figure 2 (Left) that the DOB-CBF-RL policy consistently converges within 60 episodes. In comparison, it takes at least 90 episodes for the GP-CBF-RL to nd an equally good