More Like this

1. Conversion of Certain Stochastic Control Problems into Deterministic Control Problems

   McEneaney, William M; Dower, Peter M. (July 2020, 21st IFAC World Congress)

   A class of nonlinear, stochastic staticization control problems (including minimization problems with smooth, convex, coercive payoffs) driven by diffusion dynamics with constant diffusion coefficient is considered. A fundamental solution form is obtained where the same solution can be used for a limited variety of terminal costs without re-solution of the problem. One may convert this fundamental solution form from a stochastic control problem form to a deterministic control problem form. This yields an equivalence between certain second-order (in space) Hamilton-Jacobi partial differential equations (HJ PDEs) and associated first-order HJ PDEs. This reformulation has substantial numerical implications. 

   more » « less
2. A min-plus fundamental solution semigroup for a class of approximate infinite dimensional optimal control problems

   McEneaney, William M; Dower, Peter M. (July 2020, Proceedings of the American Control Conference)

   By exploiting min-plus linearity, semiconcavity, and semigroup properties of dynamic programming, a fundamental solution semigroup for a class of approximate finite horizon linear infinite dimensional optimal control problems is constructed. Elements of this fundamental solution semigroup are parameterized by the time horizon, and can be used to approximate the solution of the corresponding finite horizon optimal control problem for any terminal cost. They can also be composed to compute approximations on longer horizons. The value function approximation provided takes the form of a min-plus convolution of a kernel with the terminal cost. A general construction for this kernel is provided, along with a spectral representation for a restricted class of sub-problems. 

   more » « less
3. Finite-horizon covariance control of linear time-varying systems

   https://doi.org/10.1109/CDC.2017.8264189  

   Goldshtein, Maxim; Tsiotras, Panagiotis (December 2017, Conference on Decision and Control)

   We consider the problem of finite-horizon optimal control of a discrete linear time-varying system subject to a stochastic disturbance and fully observable state. The initial state of the system is drawn from a known Gaussian distribution, and the final state distribution is required to reach a given target Gaussian distribution, while minimizing the expected value of the control effort. We derive the linear optimal control policy by first presenting an efficient solution for the diffusion-less case, and we then solve the case with diffusion by reformulating the system as a superposition of diffusion-less systems. We show that the resulting solution coincides with a LQG problem with particular terminal cost weight matrix. 

   more » « less
4. A Neural Network Approach for Stochastic Optimal Control

   Li, Xingjian; Verma, Deepanshu; Ruthotto, Lars (June 2024, SIAM journal on scientific computing)

   We present a neural network approach for approximating the value function of high- dimensional stochastic control problems. Our training process simultaneously updates our value function estimate and identifies the part of the state space likely to be visited by optimal trajectories. Our approach leverages insights from optimal control theory and the fundamental relation between semi-linear parabolic partial differential equations and forward-backward stochastic differential equations. To focus the sampling on relevant states during neural network training, we use the stochastic Pontryagin maximum principle (PMP) to obtain the optimal controls for the current value function estimate. By design, our approach coincides with the method of characteristics for the non-viscous Hamilton-Jacobi-Bellman equation arising in deterministic control problems. Our training loss consists of a weighted sum of the objective functional of the control problem and penalty terms that enforce the HJB equations along the sampled trajectories. Importantly, training is unsupervised in that it does not require solutions of the control problem. Our numerical experiments highlight our scheme’s ability to identify the relevant parts of the state space and produce meaningful value estimates. Using a two-dimensional model problem, we demonstrate the importance of the stochastic PMP to inform the sampling and compare to a finite element approach. With a nonlinear control affine quadcopter example, we illustrate that our approach can handle complicated dynamics. For a 100-dimensional benchmark problem, we demonstrate that our approach improves accuracy and time-to-solution and, via a modification, we show the wider applicability of our scheme. 

   more » « less
5. Extended Mean Field Control Problems: stochastic maximum principle and transport perspective

   Beatrice Acciaio, Julio Backhoff-Veraguas (June 2018, arXiv.org)

   We study Mean Field stochastic control problems where the cost function and the state dynamics depend upon the joint distribution of the controlled state and the control process. We prove suitable versions of the Pontryagin stochastic maximum principle, both in necessary and in sufficient form, which extend the known conditions to this general framework. Furthermore, we suggest a variational approach to study a weak formulation of these control problems. We show a natural connection between this weak formulation and optimal transport on path space, which inspires a novel discretization scheme. 

   more » « less