More Like this

1. Maximum Principle for Mean Field Type Control Problems with General Volatility Functions

   Bensoussan, Alain; Huang, Ziyu; Yam, Sheung (July 2024, International game theory review)

   In this paper, we study the maximum principle of mean field type control problems when the volatility function depends on the state and its measure and also the control, by using our recently developed method in [Bensoussan, A., Huang, Z. and Yam, S. C. P. [2023] Control theory on Wasserstein space: A new approach to optimality conditions, Ann. Math. Sci. Appl.; Bensoussan, A., Tai, H. M. and Yam, S. C. P. [2023] Mean field type control problems, some Hilbert-space-valued FBSDEs, and related equations, preprint (2023), arXiv:2305.04019; Bensoussan, A. and Yam, S. C. P. [2019] Control problem on space of random variables and master equation, ESAIM Control Optim. Calc. Var. 25, 10]. Our method is to embed the mean field type control problem into a Hilbert space to bypass the evolution in the Wasserstein space. We here give a necessary condition and a sufficient condition for these control problems in Hilbert spaces, and we also derive a system of forward–backward stochastic differential equations. 

   more » « less
2. Mean field approach to stochastic control with partial information

   https://doi.org/10.1051/cocv/2021085  

   Bensoussan, Alain; Yam, Sheung Chi (January 2021, ESAIM: Control, Optimisation and Calculus of Variations)

   Buttazzo, G.; Casas, E.; de Teresa, L.; Glowinski, R.; Leugering, G.; Trélat, E.; Zhang, X. (Ed.)

   In our present article, we follow our way of developing mean field type control theory in our earlier works [Bensoussanet al., Mean Field Games and Mean Field Type Control Theory.Springer, New York (2013)], by first introducing the Bellman and then master equations, the system of Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck (FP) equations, and then tackling them by looking for the semi-explicit solution for the linear quadratic case, especially with an arbitrary initial distribution; such a problem, being left open for long, has not been specifically dealt with in the earlier literature, such as Bensoussan [Stochastic Control of Partially Observable Systems. Cambridge University Press, (1992)] and Nisio [Stochastic control theory: Dynamic programming principle. Springer (2014)], which only tackled the linear quadratic setting with Gaussian initial distributions. Thanks to the effective mean-field theory, we propose a solution to this long standing problem of the general non-Gaussian case. Besides, our problem considered here can be reduced to the model in Bandiniet al.[Stochastic Process. Appl.129(2019) 674–711], which is fundamentally different from our present proposed framework. 

   more » « less
3. Control theory on Wasserstein space: a new approach to optimality conditions

   https://doi.org/10.4310/AMSA.2023.v8.n3.a6  

   Bensoussan, Alain; Huang, Ziyu; Yam, Sheung_Chi Phillip (January 2023, Annals of Mathematical Sciences and Applications)

   We study the deterministic control problem in the Wasserstein space, following the recent works of Bonnet and Frankowska, but with a new approach. One of the major advantages of our approach is that it reconciles the closed loop and the open loop approaches, without the technicalities of the traditional feedback control methodology. It allows also to embed the control problem in the Wasserstein space into a control problem in a Hilbert space, similar to the lifting method introduced by P. L. Lions, used already in our previous works. The Hilbert space is different from that proposed by P. L. Lions, and it allows to recover the control problem in the Wasserstein space as a particular case. 

   more » « less
4. Viscosity Solutions of the Eikonal Equation on the Wasserstein Space

   https://doi.org/10.1007/s00245-024-10145-2  

   Soner, H Mete; Yan, Qinxin (August 2024, Applied Mathematics & Optimization)

   Dynamic programming equations for mean field control problems with a separable structure are Eikonal type equations on the Wasserstein space. Standard differentiation using linear derivatives yield a direct extension of the classical viscosity theory. We use Fourier representation of the Sobolev norms on the space of measures, together with the standard techniques from the finite dimensional theory to prove a comparison result among semi-continuous sub and super solutions, obtaining a unique characterization of the value function. 

   more » « less
5. Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

   Carmona, Rene; Lauriere, Mathieu (October 2019, ArXivorg)

   We develop a general reinforcement learning framework for mean field control (MFC) problems. Such problems arise for instance as the limit of collaborative multi-agent control problems when the number of agents is very large. The asymptotic problem can be phrased as the optimal control of a non-linear dynamics. This can also be viewed as a Markov decision process (MDP) but the key difference with the usual RL setup is that the dynamics and the reward now depend on the state's probability distribution itself. Alternatively, it can be recast as a MDP on the Wasserstein space of measures. In this work, we introduce generic model-free algorithms based on the state-action value function at the mean field level and we prove convergence for a prototypical Q-learning method. We then implement an actor-critic method and report numerical results on two archetypal problems: a finite space model motivated by a cyber security application and a continuous space model motivated by an application to swarm motion. 

   more » « less