More Like this

1. Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

   https://doi.org/10.1109/TAC.2023.3273786  

   Cui, Leilei; Pang, Bo; Jiang, Zhong-Ping (May 2023, IEEE Transactions on Automatic Control)

   This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning (RL) and adaptive dynamic programming (ADP) and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving. 

   more » « less
2. Learning-based adaptive optimal control of linear time-delay systems: A value iteration approach

   https://doi.org/10.1016/j.automatica.2024.111944  

   Cui, Leilei; Pang, Bo; Krstić, Miroslav; Jiang, Zhong-Ping (January 2025, Automatica)

   This paper proposes a novel learning-based adaptive optimal controller design method for a class of continuous-time linear time-delay systems. A key strategy is to exploit the state-of-the-art reinforcement learning (RL) techniques and adaptive dynamic programming (ADP), and propose a data-driven method to learn the near-optimal controller without the precise knowledge of system dynamics. Specifically, a value iteration (VI) algorithm is proposed to solve the infinite-dimensional Riccati equation for the linear quadratic optimal control problem of time-delay systems using finite samples of input-state trajectory data. It is rigorously proved that the proposed VI algorithm converges to the near-optimal solution. Compared with the previous literature, the nice features of the proposed VI algorithm are that it is directly developed for continuous-time systems without discretization and an initial admissible controller is not required for implementing the algorithm. The efficacy of the proposed methodology is demonstrated by two practical examples of metal cutting and autonomous driving. 

   more » « less
3. A Value Iteration Approach to Adaptive Optimal Control of Linear Time-Delay Systems

   https://doi.org/10.1016/j.ifacol.2023.10.524  

   Cui, Leilei; Pang, Bo; Jiang, Zhong-Ping (October 2023, IFAC-PapersOnLine)

   This paper presents a novel learning-based adaptive optimal controller design for linear time-delay systems described by delay differential equations (DDEs). A key strategy is to exploit the value iteration (VI) approach to solve the linear quadratic optimal control problem for time-delay systems. However, previous learning-based control methods are all exclusively devoted to discrete-time time-delay systems. In this article, we aim to fill in the gap by developing a learning-based VI approach to solve the infinite-dimensional algebraic Riccati equation (ARE) for continuous-time time-delay systems. One nice feature of the proposed VI approach is that an initial admissible controller is not required to start the algorithm. The efficacy of the proposed methodology is demonstrated by the example of autonomous driving. 

   more » « less
4. Adaptive Optimal Control of Linear Periodic Systems: An Off-Policy Value Iteration Approach

   https://doi.org/10.1109/TAC.2020.2987313  

   Pang, Bo; Jiang, Zhong-Ping (April 2020, IEEE Transactions on Automatic Control)

   This paper studies the infinite-horizon adaptive optimal control of continuous-time linear periodic (CTLP) systems. A novel value iteration (VI) based off-policy ADP algorithm is proposed for a general class of CTLP systems, so that approximate optimal solutions can be obtained directly from the collected data, without the exact knowledge of system dynamics. Under mild conditions, the proofs on uniform convergence of the proposed algorithm to the optimal solutions are given for both the model-based and model-free cases. The VI-based ADP algorithm is able to find suboptimal controllers without assuming the knowledge of an initial stabilizing controller. Application to the optimal control of a triple inverted pendulum subjected to a periodically varying load demonstrates the feasibility and effectiveness of the proposed method. 

   more » « less
5. Reinforcement Learning for Adaptive Periodic Linear Quadratic Control

   Bo Pang, Zhong-Ping Jiang (December 2019, 2019 IEEE 58th Conference on Decision and Control (CDC))

   This paper presents a first solution to the problem of adaptive LQR for continuous-time linear periodic systems. Specifically, reinforcement learning and adaptive dynamic programming (ADP) techniques are used to develop two algorithms to obtain near-optimal controllers. Firstly, the policy iteration (PI) and value iteration (VI) methods are proposed when the model is known. Then, PI-based and VI-based off-policy ADP algorithms are derived to find near-optimal solutions directly from input/state data collected along the system trajectories, without the exact knowledge of system dynamics. The effectiveness of the derived algorithms is validated using the well-known lossy Mathieu equation. 

   more » « less