More Like this

1. Learning-Based Adaptive Optimal Output Regulation of Discrete-Time Linear Systems

   https://doi.org/10.1016/j.ifacol.2023.10.912  

   Chakraborty, Sayan; Gao, Weinan; Cui, Leilei; Lewis, Frank L; Jiang, Zhong-Ping (October 2023, IFAC-PapersOnLine)

   In this paper, we address the problem of model-free optimal output regulation of discrete-time systems that aims at achieving asymptotic tracking and disturbance rejection when we have no exact knowledge of the system parameters. Insights from reinforcement learning and adaptive dynamic programming are used to solve this problem. An interesting discovery is that the model-free discrete-time output regulation differs from the continuous-time counterpart in terms of the persistent excitation condition required to ensure the uniqueness and convergence of the policy iteration. In this work, it is shown that this persistent excitation condition must be carefully established in order to ensure the uniqueness and convergence properties of the policy iteration. 

   more » « less
2. A Lyapunov characterization of robust policy optimization

   https://doi.org/10.1007/s11768-023-00163-w  

   Cui, Leilei; Jiang, Zhong-Ping (August 2023, Control Theory and Technology)

   In this paper, we study the robustness property of policy optimization (particularly Gauss–Newton gradient descent algorithm which is equivalent to the policy iteration in reinforcement learning) subject to noise at each iteration. By invoking the concept of input-to-state stability and utilizing Lyapunov’s direct method, it is shown that, if the noise is sufficiently small, the policy iteration algorithm converges to a small neighborhood of the optimal solution even in the presence of noise at each iteration. Explicit expressions of the upperbound on the noise and the size of the neighborhood to which the policies ultimately converge are provided. Based on Willems’ fundamental lemma, a learning-based policy iteration algorithm is proposed. The persistent excitation condition can be readily guaranteed by checking the rank of the Hankel matrix related to an exploration signal. The robustness of the learning-based policy iteration to measurement noise and unknown system disturbances is theoretically demonstrated by the input-to-state stability of the policy iteration. Several numerical simulations are conducted to demonstrate the efficacy of the proposed method. 

   more » « less
3. Optimal Secure Control with Linear Temporal Logic Constraints

   Niu, L; Clark, A (July 2019, IEEE transactions on automatic control)

   Prior work on automatic control synthesis for cyberphysical systems under logical constraints has primarily focused on environmental disturbances or modeling uncertainties, however, the impact of deliberate and malicious attacks has been less studied. In this paper, we consider a discrete-time dynamical system with a linear temporal logic (LTL) constraint in the presence of an adversary, which is modeled as a stochastic game. We assume that the adversary observes the control policy before choosing an attack strategy. We investigate two problems. In the first problem, we synthesize a robust control policy for the stochastic game that maximizes the probability of satisfying the LTL constraint. A value iteration based algorithm is proposed to compute the optimal control policy. In the second problem, we focus on a subclass of LTL constraints, which consist of an arbitrary LTL formula and an invariant constraint. We then investigate the problem of computing a control policy that minimizes the expected number of invariant constraint violations while maximizing the probability of satisfying the arbitrary LTL constraint. We characterize the optimality condition for the desired control policy. A policy iteration based algorithm is proposed to compute the control policy. We illustrate the proposed approaches using two numerical case studies. 

   more » « less
4. Data-Driven Combined Longitudinal and Lateral Control for the Car Following Problem

   https://doi.org/10.1109/TCST.2025.3539216  

   Cui, Leilei; Chakraborty, Sayan; Ozbay, Kaan; Jiang, Zhong-Ping (May 2025, IEEE Transactions on Control Systems Technology)

   This paper studies the problem of data-driven combined longitudinal and lateral control of autonomous vehicles (AVs) such that the AV can stay within a safe but minimum distance from its leading vehicle and, at the same time, in the lane. Most of the existing methods for combined longitudinal and lateral control are either model-based or developed by purely data-driven methods such as reinforcement learning. Traditional model-based control approaches are insufficient to address the adaptive optimal control design issue for AVs in dynamically changing environments and subject to model uncertainty. Moreover, the conventional reinforcement learning approaches require a large volume of data, and cannot guarantee the stability of the vehicle. These limitations are addressed by integrating the advanced control theory with reinforcement learning techniques. To be more specific, by utilizing adaptive dynamic programming techniques and using the motion data collected from the vehicles, a policy iteration algorithm is proposed such that the control policy is iteratively optimized in the absence of the precise knowledge of the AV’s dynamical model. Furthermore, the stability of the AV is guaranteed with the control policy generated at each iteration of the algorithm. The efficiency of the proposed approach is validated by SUMO simulation, a microscopic traffic simulation platform, for different traffic scenarios. 

   more » « less
5. An adaptive learning and control architecture for mitigating sensor and actuator attacks in connected autonomous vehicle platoons

   https://doi.org/10.1002/acs.3032  

   Jin, Xu; Haddad, Wassim M.; Jiang, Zhong‐Ping; Kanellopoulos, Aris; Vamvoudakis, Kyriakos G. (June 2019, International Journal of Adaptive Control and Signal Processing)

   Summary In this paper, we develop an adaptive control algorithm for addressing security for a class of networked vehicles that comprise a formation ofhuman‐driven vehicles sharing kinematic data and an autonomous vehicle in the aft of the vehicle formation receiving data from the preceding vehicles through wireless vehicle‐to‐vehicle communication devices. Specifically, we develop an adaptive controller for mitigating time‐invariant state‐dependent adversarial sensor and actuator attacks while guaranteeing uniform ultimate boundedness of the closed‐loop networked system. Furthermore, an adaptive learning framework is presented for identifying the state space model parameters based on input‐output data. This learning technique utilizes previously stored data as well as current data to identify the system parameters using a relaxed persistence of excitation condition. The effectiveness of the proposed approach is demonstrated by an illustrative numerical example involving a platoon of connected vehicles. 

   more » « less