More Like this

1. A finite horizon optimal stochastic impulse control problem with a decision lag

   Li, C.; Yong, J. (April 2021, Dynamics of continuous discrete and impulsive systems)

   null (Ed.)

   This paper studies an optimal stochastic impulse control problem in a finite time horizon with a decision lag, by which we mean that after an impulse is made, a fixed number units of time has to be elapsed before the next impulse is allowed to be made. The continuity of the value function is proved. A suitable version of dynamic programming principle is established, which takes into account the dependence of state process on the elapsed time. The corresponding Hamilton-Jacobi-Bellman (HJB) equation is derived, which exhibits some special feature of the problem. The value function of this optimal impulse control problem is characterized as the unique viscosity solution to the corresponding HJB equation. An optimal impulse control is constructed provided the value function is given. Moreover, a limiting case with the waiting time approaching 0 is discussed. 

   more » « less
2. Viscosity Solutions for McKean–Vlasov Control on a Torus

   https://doi.org/10.1137/22M1543732  

   Soner, H Mete; Yan, Qinxin (April 2024, SIAM Journal on Control and Optimization)

   An optimal control problem in the space of probability measures, and the viscosity solu- tions of the corresponding dynamic programming equations defined using the intrinsic linear derivative are studied. The value function is shown to be Lipschitz continuous with respect to a novel smooth Fourier-Wasserstein metric. A comparison result between the Lipschitz viscosity sub and super solutions of the dynamic programming equation is proved using this metric, characterizing the value function as the unique Lipschitz viscosity solution. 

   more » « less
3. Enhancing value function estimation through first-order state-action dynamics in offline reinforcement learning

   Lien, Yun-Hsuan; Hsieh, Ping-Chun; Li, Tzu-Mao; Wang, Yu-Shuen (June 2024, International Conference of Machine Learning)

   In offline reinforcement learning (RL), updating the value function with the discrete-time Bellman Equation often encounters challenges due to the limited scope of available data. This limitation stems from the Bellman Equation, which cannot accurately predict the value of unvisited states. To address this issue, we have introduced an innovative solution that bridges the continuousand discrete-time RL methods, capitalizing on their advantages. Our method uses a discrete-time RL algorithm to derive the value function from a dataset while ensuring that the function’s first derivative aligns with the local characteristics of states and actions, as defined by the HamiltonJacobi-Bellman equation in continuous RL. We provide practical algorithms for both deterministic policy gradient methods and stochastic policy gradient methods. Experiments on the D4RL dataset show that incorporating the first-order information significantly improves policy performance for offline RL problems. 

   more » « less
4. Learning Safety Critics via a Non-Contractive Binary Bellman Operator

   https://doi.org/10.1109/IEEECONF59524.2023.10476995  

   Castellano, Agustin; Min, Hancheng; Bazerque, Juan Andrés; Mallada, Enrique (October 2023, IEEE)

   The inability to naturally enforce safety in Reinforcement Learning (RL), with limited failures, is a core challenge impeding its use in real-world applications. One notion of safety of vast practical relevance is the ability to avoid (unsafe) regions of the state space. Though such a safety goal can be captured by an action-value-like function, a.k.a. safety critics, the associated operator lacks the desired contraction and uniqueness properties that the classical Bellman operator enjoys. In this work, we overcome the non-contractiveness of safety critic operators by leveraging that safety is a binary property. To that end, we study the properties of the binary safety critic associated with a deterministic dynamical system that seeks to avoid reaching an unsafe region. We formulate the corresponding binary Bellman equation (B2E) for safety and study its properties. While the resulting operator is still non-contractive, we fully characterize its fixed points representing--except for a spurious solution--maximal persistently safe regions of the state space that can always avoid failure. We provide an algorithm that, by design, leverages axiomatic knowledge of safe data to avoid spurious fixed points. 

   more » « less
5. Convex Q-Learning in Continuous Time with Application to Dispatch of Distributed Energy Resources

   https://doi.org/10.1109/CDC49753.2023.10383620  

   Lu, Fan; Mathias, Joel; Meyn, Sean; Kalsi, Karanjit (December 2023, IEEE)

   Convex Q-learning is a recent approach to reinforcement learning, motivated by the possibility of a firmer theory for convergence, and the possibility of making use of greater a priori knowledge regarding policy or value function structure. This paper explores algorithm design in the continuous time domain, with a finite-horizon optimal control objective. The main contributions are (i) The new Q-ODE: a model-free characterization of the Hamilton-Jacobi-Bellman equation. (ii) A formulation of Convex Q-learning that avoids approximations appearing in prior work. The Bellman error used in the algorithm is defined by filtered measurements, which is necessary in the presence of measurement noise. (iii) Convex Q-learning with linear function approximation is a convex program. It is shown that the constraint region is bounded, subject to an exploration condition on the training input. (iv) The theory is illustrated in application to resource allocation for distributed energy resources, for which the theory is ideally suited. 

   more » « less