More Like this

1. Learning Optimal Controllers by Policy Gradient: Global Optimality via Convex Parameterization

   https://doi.org/10.1109/CDC45484.2021.9682821  

   Sun, Yue; Fazel, Maryam (January 2021, 60th IEEE Conference on Decision and Control (CDC),)

   Common reinforcement learning methods seek optimal controllers for unknown dynamical systems by searching in the "policy" space directly. A recent line of research, starting with [1], aims to provide theoretical guarantees for such direct policy-update methods by exploring their performance in classical control settings, such as the infinite horizon linear quadratic regulator (LQR) problem. A key property these analyses rely on is that the LQR cost function satisfies the "gradient dominance" property with respect to the policy parameters. Gradient dominance helps guarantee that the optimal controller can be found by running gradient-based algorithms on the LQR cost. The gradient dominance property has so far been verified on a case-by-case basis for several control problems including continuous/discrete time LQR, LQR with decentralized controller, H2/H∞ robust control.In this paper, we make a connection between this line of work and classical convex parameterizations based on linear matrix inequalities (LMIs). Using this, we propose a unified framework for showing that gradient dominance indeed holds for a broad class of control problems, such as continuous- and discrete-time LQR, minimizing the L2 gain, and problems using system-level parameterization. Our unified framework provides insights into the landscape of the cost function as a function of the policy, and enables extending convergence results for policy gradient descent to a much larger class of problems. 

   more » « less
2. Frequency Regulation using Data-Driven Controllers in Power Grids with Variable Inertia due to Renewable Energy

   https://doi.org/10.1109/PESGM40551.2019.8973437  

   Hidalgo-Gonzalez, Patricia; Henriquez-Auba, Rodrigo; Callaway, Duncan S.; Tomlin, Claire J. (August 2019, 2019 IEEE Power & Energy Society General Meeting (PESGM))

   With the increasing penetration of non-synchronous variable renewable energy sources (RES) in power grids, the system's inertia decreases and varies over time, affecting the capability of current control schemes to handle frequency regulation. Providing virtual inertia to power systems has become an interesting topic of research, since it may provide a reasonable solution to address this new issue. However, power dynamics are usually modeled as time-invariant, without including the effect of varying inertia due to the presence of RES. This paper presents a framework to design a fixed learned controller based on datasets of optimal time-varying LQR controllers. In our scheme, we model power dynamics as a hybrid system with discrete modes representing different rotational inertia regimes of the grid. We test the performance of our controller in a twelve-bus system using different fixed inertia modes. We also study our learned controller as the inertia changes over time. By adding virtual inertia we can guarantee stability of high-renewable (low-inertia) modes. The novelty of our work is to propose a design framework for a stable controller with fixed gains for time-varying power dynamics. This is relevant because it would be simpler to implement a proportional controller with fixed gains compared to a time-varying control. 

   more » « less
3. Data-Driven Optimal Control of Traffic Signals for Urban Road Networks

   https://doi.org/10.1109/CDC51059.2022.9992876  

   Liu, Tong; Wang, Hong; Jiang, Zhong-Ping (December 2022, IEEE)

   This paper studies the issue of data-driven optimal control design for traffic signals of oversaturated urban road networks. The signal control system based on the store and forward model is generally uncontrollable for which the controllable decomposition is needed. Instead of identifying the unknown parameters like saturation rates and turning ratios, a finite number of measured trajectories can be used to parametrize the system and help directly construct a transformation matrix for Kalman controllable decomposition through the fundamental lemma of J. C. Willems. On top of that, an infinite-horizon linear quadratic regulator (LQR) problem is formulated considering the constraints of green times for traffic signals. The problem can be solved through a two-phase data-driven learning process, where one solves an infinite-horizon unconstrained LQR problem and the other solves a finite-horizon constrained LQR problem. The simulation result shows the theoretical analysis is effective and the proposed data-driven controller can yield desired performance for reducing traffic congestion. 

   more » « less
4. Deep Reinforcement Learning for Adaptive Traffic Signal Control

   https://doi.org/10.1115/DSCC2019-9076  

   Tan, Kai Liang; Poddar, Subhadipto; Sarkar, Soumik; Sharma, Anuj (October 2019, ASME Dynamic Systems and Control Conference)

   Abstract Many existing traffic signal controllers are either simple adaptive controllers based on sensors placed around traffic intersections, or optimized by traffic engineers on a fixed schedule. Optimizing traffic controllers is time consuming and usually require experienced traffic engineers. Recent research has demonstrated the potential of using deep reinforcement learning (DRL) in this context. However, most of the studies do not consider realistic settings that could seamlessly transition into deployment. In this paper, we propose a DRL-based adaptive traffic signal control framework that explicitly considers realistic traffic scenarios, sensors, and physical constraints. In this framework, we also propose a novel reward function that shows significantly improved traffic performance compared to the typical baseline pre-timed and fully-actuated traffic signals controllers. The framework is implemented and validated on a simulation platform emulating real-life traffic scenarios and sensor data streams. 

   more » « less
5. Reducing Urban Traffic Congestion Using Deep Learning and Model Predictive Control

   https://doi.org/10.1109/TNNLS.2023.3264709  

   Yin, Zhun; Liu, Tong; Wang, Chieh; Wang, Hong; Jiang, Zhong-Ping (January 2023, IEEE Transactions on Neural Networks and Learning Systems)

   This article proposes a deep learning (DL)-based control algorithm—DL velocity-based model predictive control (VMPC)—for reducing traffic congestion with slowly time-varying traffic signal controls. This control algorithm consists of system identification using DL and traffic signal control using VMPC. For the training process of DL, we established a modeling error entropy loss as the criteria inspired by the theory of stochastic distribution control (SDC) originated by the fourth author. Simulation results show that the proposed algorithm can reduce traffic congestion with a slowly varying traffic signal control input. Results of an ablation study demonstrate that this algorithm compares favorably to other model-based controllers in terms of prediction error, signal varying speed, and control effectiveness. 

   more » « less