More Like this

1. Learning Optimal Controllers by Policy Gradient: Global Optimality via Convex Parameterization

   https://doi.org/10.1109/CDC45484.2021.9682821  

   Sun, Yue ; Fazel, Maryam ( January 2021 , 60th IEEE Conference on Decision and Control (CDC),)

   Common reinforcement learning methods seek optimal controllers for unknown dynamical systems by searching in the "policy" space directly. A recent line of research, starting with [1], aims to provide theoretical guarantees for such direct policy-update methods by exploring their performance in classical control settings, such as the infinite horizon linear quadratic regulator (LQR) problem. A key property these analyses rely on is that the LQR cost function satisfies the "gradient dominance" property with respect to the policy parameters. Gradient dominance helps guarantee that the optimal controller can be found by running gradient-based algorithms on the LQR cost. The gradient dominance property has so far been verified on a case-by-case basis for several control problems including continuous/discrete time LQR, LQR with decentralized controller, H2/H∞ robust control.In this paper, we make a connection between this line of work and classical convex parameterizations based on linear matrix inequalities (LMIs). Using this, we propose a unified framework for showing that gradient dominance indeed holds for a broad class of control problems, such as continuous- and discrete-time LQR, minimizing the L2 gain, and problems using system-level parameterization. Our unified framework provides insights into the landscape of the cost function as a function of the policy, and enables extending convergence results for policy gradient descent to a much larger class of problems. 

   more » « less
2. Event-Triggered Robust Stabilization of Nonlinear Input-Constrained Systems Using Single Network Adaptive Critic Designs

   https://doi.org/10.1109/TSMC.2018.2853089  

   Xiong Yang, Haibo He ( January 2018 , IEEE transactions on systems, man, and cybernetics. Systems)

   In this paper, we study the event-triggered robust stabilization problem of nonlinear systems subject to mismatched perturbations and input constraints. First, with the introduction of an infinite-horizon cost function for the auxiliary system, we transform the robust stabilization problem into a constrained optimal control problem. Then, we prove that the solution of the event-triggered Hamilton–Jacobi–Bellman (ETHJB) equation, which arises in the constrained optimal control problem, guarantees original system states to be uniformly ultimately bounded (UUB). To solve the ETHJB equation, we present a single network adaptive critic design (SN-ACD). The critic network used in the SN-ACD is tuned through the gradient descent method. By using Lyapunov method, we demonstrate that all the signals in the closed-loop auxiliary system are UUB. Finally, we provide two examples, including the pendulum system, to validate the proposed event-triggered control strategy. 

   more » « less
3. Uniqueness of the Riccati operator of the non-standard ARE of a third order dynamics with boundary control

   https://doi.org/10.2478/candc-2022-0013  

   Lasiecka, Irena ; Triggiani, Roberto ( June 2022 , Control and Cybernetics)

   Abstract The Moore-Gibson-Thompson [MGT] dynamics is considered. This third order in time evolution arises within the context of acoustic wave propagation with applications in high frequency ultrasound technology. The optimal boundary feedback control is constructed in order to have on-line regulation. The above requires wellposedness of the associated Algebraic Riccati Equation. The paper by Lasiecka and Triggiani (2022) recently contributed a comprehensive study of the Optimal Control Problem for the MGT-third order dynamics with boundary control, over an infinite time-horizon. A critical missing point in such a study is the issue of uniqueness (within a specific class) of the corresponding highly non-standard Algebraic Riccati Equation. The present note resolves this problem in the positive, thus completing the study of Lasiecka and Triggiani (2022) with the final goal of having on line feedback control, which is also optimal. 

   more » « less
4. A min-plus fundamental solution semigroup for a class of approximate infinite dimensional optimal control problems

   McEneaney, William M ; Dower, Peter M. ( July 2020 , Proceedings of the American Control Conference)

   By exploiting min-plus linearity, semiconcavity, and semigroup properties of dynamic programming, a fundamental solution semigroup for a class of approximate finite horizon linear infinite dimensional optimal control problems is constructed. Elements of this fundamental solution semigroup are parameterized by the time horizon, and can be used to approximate the solution of the corresponding finite horizon optimal control problem for any terminal cost. They can also be composed to compute approximations on longer horizons. The value function approximation provided takes the form of a min-plus convolution of a kernel with the terminal cost. A general construction for this kernel is provided, along with a spectral representation for a restricted class of sub-problems. 

   more » « less
5. Optimal Control of a Large Population of Randomly Interrogated Interacting Agents

   https://doi.org/10.1109/TAC.2022.3202838  

   Komaee, Arash ( August 2022 , IEEE Transactions on Automatic Control)

   This article investigates a stochastic optimal control problem with linear Gaussian dynamics, quadratic performance measure, but non-Gaussian observations. The linear Gaussian dynamics characterizes a large number of interacting agents evolving under a centralized control and external disturbances. The aggregate state of the agents is only partially known to the centralized controller by means of the samples taken randomly in time and from anonymous randomly selected agents. Due to removal of the agent identity from the samples, the observation set has a non-Gaussian structure, and as a consequence, the optimal control law that minimizes a quadratic cost is essentially nonlinear and infinite-dimensional, for any finite number of agents. For infinitely many agents, however, this paper shows that the optimal control law is the solution to a reduced order, finite-dimensional linear quadratic Gaussian problem with Gaussian observations sampled only in time. For this problem, the separation principle holds and is used to develop an explicit optimal control law by combining a linear quadratic regulator with a separately designed finite-dimensional minimum mean square error state estimator. Conditions are presented under which this simple optimal control law can be adopted as a suboptimal control law for finitely many agents. 

   more » « less