An official website of the United States government
We develop a general reinforcement learning framework for mean field control (MFC) problems. Such problems arise for instance as the limit of collaborative multi-agent control problems when the number of agents is very large. The asymptotic problem can be phrased as the optimal control of a non-linear dynamics. This can also be viewed as a Markov decision process (MDP) but the key difference with the usual RL setup is that the dynamics and the reward now depend on the state's probability distribution itself. Alternatively, it can be recast as a MDP on the Wasserstein space of measures. In this work, we introduce generic model-free algorithms based on the state-action value function at the mean field level and we prove convergence for a prototypical Q-learning method. We then implement an actor-critic method and report numerical results on two archetypal problems: a finite space model motivated by a cyber security application and a continuous space model motivated by an application to swarm motion.
more »
« less
A machine learning framework for solving high-dimensional mean field game and mean field control problems
Mean field games (MFG) and mean field control (MFC) are critical classes of multiagent models for the efficient analysis of massive populations of interacting agents. Their areas of application span topics in economics, finance, game theory, industrial engineering, crowd motion, and more. In this paper, we provide a flexible machine learning framework for the numerical solution of potential MFG and MFC models. State-of-the-art numerical methods for solving such problems utilize spatial discretization that leads to a curse of dimensionality. We approximately solve high-dimensional problems by combining Lagrangian and Eulerian viewpoints and leveraging recent advances from machine learning. More precisely, we work with a Lagrangian formulation of the problem and enforce the underlying Hamilton–Jacobi–Bellman (HJB) equation that is derived from the Eulerian formulation. Finally, a tailored neural network parameterization of the MFG/MFC solution helps us avoid any spatial discretization. Our numerical results include the approximate solution of 100-dimensional instances of optimal transport and crowd motion problems on a standard work station and a validation using a Eulerian solver in two dimensions. These results open the door to much-anticipated applications of MFG and MFC models that are beyond reach with existing numerical methods.
more »
« less
- Award ID(s):
- 1751636
- PAR ID:
- 10143814
- Publisher / Repository:
- Proceedings of the National Academy of Sciences
- Date Published:
- Journal Name:
- Proceedings of the National Academy of Sciences
- Volume:
- 117
- Issue:
- 17
- ISSN:
- 0027-8424
- Page Range / eLocation ID:
- p. 9183-9193
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Mean-field games (MFGs) provide a statistical physics-inspired modelling framework for decision-making in large populations of strategic, non-cooperative agents. Mathematically, these systems consist of a forwards–backwards in time-system of two coupled nonlinear partial differential equations (PDEs), namely, the Fokker–Plank (FP) and the Hamilton–Jacobi–Bellman (HJB) equations, governing the agent state and control distribution, respectively. In this work, we study a finite-time MFG with a rich global bifurcation structure using a reduced-order model (ROM). The ROM is a four-dimensional (4D) two-point boundary value problem (BVP) obtained by restricting the controlled dynamics to the first two moments of the agent state distribution, i.e. the mean and the variance. Phase space analysis of the ROM reveals that the invariant manifolds of periodic orbits around the so-called ‘ergodic MFG equilibrium’ play a crucial role in determining the bifurcation diagram and imparting a topological signature to various solution branches. We show a qualitative agreement of these results with numerical solutions of the full-order MFG PDE system.more » « less
-
We consider the problem of representing collective behavior of large popula- tions and predicting the evolution of a population distribution over a discrete state space. A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individ- ual actions and predicting the temporal evolution of population distributions. We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP. This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning. Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.more » « less
-
Abstract In this paper, we introduce a bilevel optimization framework for addressing inverse mean-field games, alongside an exploration of numerical methods tailored for this bilevel problem. The primary benefit of our bilevel formulation lies in maintaining the convexity of the objective function and the linearity of constraints in the forward problem. Our paper focuses on inverse mean-field games characterized by unknown obstacles and metrics. We show numerical stability for these two types of inverse problems. More importantly, we, for the first time, establish the identifiability of the inverse mean-field game with unknown obstacles via the solution of the resultant bilevel problem. The bilevel approach enables us to employ an alternating gradient-based optimization algorithm with a provable convergence guarantee. To validate the effectiveness of our methods in solving the inverse problems, we have designed comprehensive numerical experiments, providing empirical evidence of its efficacy.more » « less
-
We consider a class of deterministic mean field games, where the state associated with each player evolves according to an ODE which is linear w.r.t. the control. Existence, uniqueness, and stability of solutions are studied from the point of viewof generic theory. Within a suitable topological space of dynamics and cost functionals, we prove that, for “nearly all” mean field games (in the Baire category sense) the best reply map is single-valued for a.e. player. As a consequence, the mean field game admits a strong (not randomized) solution. Examples are given of open sets of games admitting a single solution, and other open sets admitting multiple solutions. Further examples show the existence of an open set of MFG having a unique solution which is asymptotically stable w.r.t. the best reply map, and another open set of MFG having a unique solution which is unstable. We conclude with an example of a MFG with terminal constraints which does not have any solution, not even in the mild sense with randomized strategies.more » « less
