More Like this

1. Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

   https://doi.org/10.1109/TAC.2023.3273786  

   Cui, Leilei ; Pang, Bo ; Jiang, Zhong-Ping ( May 2023 , IEEE Transactions on Automatic Control)

   This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning (RL) and adaptive dynamic programming (ADP) and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving. 

   more » « less
2. A Value Iteration Approach to Adaptive Optimal Control of Linear Time-Delay Systems

   https://doi.org/10.1016/j.ifacol.2023.10.524  

   Cui, Leilei ; Pang, Bo ; Jiang, Zhong-Ping ( October 2023 , IFAC-PapersOnLine)

   This paper presents a novel learning-based adaptive optimal controller design for linear time-delay systems described by delay differential equations (DDEs). A key strategy is to exploit the value iteration (VI) approach to solve the linear quadratic optimal control problem for time-delay systems. However, previous learning-based control methods are all exclusively devoted to discrete-time time-delay systems. In this article, we aim to fill in the gap by developing a learning-based VI approach to solve the infinite-dimensional algebraic Riccati equation (ARE) for continuous-time time-delay systems. One nice feature of the proposed VI approach is that an initial admissible controller is not required to start the algorithm. The efficacy of the proposed methodology is demonstrated by the example of autonomous driving. 

   more » « less
3. Adaptive Optimal Control of Linear Periodic Systems: An Off-Policy Value Iteration Approach

   https://doi.org/10.1109/TAC.2020.2987313  

   Pang, Bo ; Jiang, Zhong-Ping ( April 2020 , IEEE Transactions on Automatic Control)

   This paper studies the infinite-horizon adaptive optimal control of continuous-time linear periodic (CTLP) systems. A novel value iteration (VI) based off-policy ADP algorithm is proposed for a general class of CTLP systems, so that approximate optimal solutions can be obtained directly from the collected data, without the exact knowledge of system dynamics. Under mild conditions, the proofs on uniform convergence of the proposed algorithm to the optimal solutions are given for both the model-based and model-free cases. The VI-based ADP algorithm is able to find suboptimal controllers without assuming the knowledge of an initial stabilizing controller. Application to the optimal control of a triple inverted pendulum subjected to a periodically varying load demonstrates the feasibility and effectiveness of the proposed method. 

   more » « less
4. Reinforcement Learning for Adaptive Periodic Linear Quadratic Control

   Bo Pang, Zhong-Ping Jiang ( December 2019 , 2019 IEEE 58th Conference on Decision and Control (CDC))

   This paper presents a first solution to the problem of adaptive LQR for continuous-time linear periodic systems. Specifically, reinforcement learning and adaptive dynamic programming (ADP) techniques are used to develop two algorithms to obtain near-optimal controllers. Firstly, the policy iteration (PI) and value iteration (VI) methods are proposed when the model is known. Then, PI-based and VI-based off-policy ADP algorithms are derived to find near-optimal solutions directly from input/state data collected along the system trajectories, without the exact knowledge of system dynamics. The effectiveness of the derived algorithms is validated using the well-known lossy Mathieu equation. 

   more » « less
5. Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems

   https://doi.org/10.1109/TAC.2022.3172250  

   Pang, Bo ; Jiang, Zhong-Ping ( April 2023 , IEEE Transactions on Automatic Control)

   Alessandro Astolfi (Ed.)

   This article studies the adaptive optimal stationary control of continuous-time linear stochastic systems with both additive and multiplicative noises, using reinforcement learning techniques. Based on policy iteration, a novel off-policy reinforcement learning algorithm, named optimistic least-squares-based policy iteration, is proposed, which is able to find iteratively near-optimal policies of the adaptive optimal stationary control problem directly from input/state data without explicitly identifying any system matrices, starting from an initial admissible control policy. The solutions given by the proposed optimistic least-squares-based policy iteration are proved to converge to a small neighborhood of the optimal solution with probability one, under mild conditions. The application of the proposed algorithm to a triple inverted pendulum example validates its feasibility and effectiveness. 

   more » « less