More Like this

1. Physics-Informed Graph Neural Operator for Mean Field Games on Graph: A Scalable Learning Approach

   https://doi.org/10.3390/g15020012  

   Chen, Xu; Liu, Shuo; Di, Xuan (April 2024, Games)

   Mean-field games (MFGs) are developed to model the decision-making processes of a large number of interacting agents in multi-agent systems. This paper studies mean-field games on graphs (G-MFGs). The equilibria of G-MFGs, namely, mean-field equilibria (MFE), are challenging to solve for their high-dimensional action space because each agent has to make decisions when they are at junction nodes or on edges. Furthermore, when the initial population state varies on graphs, we have to recompute MFE, which could be computationally challenging and memory-demanding. To improve the scalability and avoid repeatedly solving G-MFGs every time their initial state changes, this paper proposes physics-informed graph neural operators (PIGNO). The PIGNO utilizes a graph neural operator to generate population dynamics, given initial population distributions. To better train the neural operator, it leverages physics knowledge to propagate population state transitions on graphs. A learning algorithm is developed, and its performance is evaluated on autonomous driving games on road networks. Our results demonstrate that the PIGNO is scalable and generalizable when tested under unseen initial conditions. 

   more » « less
2. A Black-box Approach for Non-stationary Multi-agent Reinforcement Learning

   Jiang, Haozhe; Cui, Qiwen; Xiong, Zhihan; Fazel, Maryam; Du, Simon S (January 2024, Proceedings of the International Conference on Learning Representations)

   We investigate learning the equilibria in non-stationary multi-agent systems and address the challenges that differentiate multi-agent learning from single-agent learning. Specifically, we focus on games with bandit feedback, where testing an equilibrium can result in substantial regret even when the gap to be tested is small, and the existence of multiple optimal solutions (equilibria) in stationary games poses extra challenges. To overcome these obstacles, we propose a versatile black-box approach applicable to a broad spectrum of problems, such as general-sum games, potential games, and Markov games, when equipped with appropriate learning and testing oracles for stationary environments. Our algorithms can achieve O(∆^1/4 T^3/4) regret when the degree of nonstationarity, as measured by total variation ∆, is known, and O(∆^1/5 T^4/5) regret when ∆ is unknown, where T is the number of rounds. Meanwhile, our algorithm inherits the favorable dependence on number of agents from the oracles. As a side contribution that may be independent of interest, we show how to test for various types of equilibria by a black-box reduction to single-agent learning, which includes Nash equilibria, correlated equilibria, and coarse correlated equilibria. 

   more » « less
3. Context-Aware Bayesian Network Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning

   Chen, Dingyang; Zhang, Qi (October 2023, Proceedings of the 40th International Conference on Machine Learning)

   Andreas Krause, Emma Brunskill (Ed.)

   Executing actions in a correlated manner is a common strategy for human coordination that often leads to better cooperation, which is also potentially beneficial for cooperative multi-agent reinforcement learning (MARL). However, the recent success of MARL relies heavily on the convenient paradigm of purely decentralized execution, where there is no action correlation among agents for scalability considerations. In this work, we introduce a Bayesian network to inaugurate correlations between agents’ action selections in their joint policy. Theoretically, we establish a theoretical justification for why action dependencies are beneficial by deriving the multi-agent policy gradient formula under such a Bayesian network joint policy and proving its global convergence to Nash equilibria under tabular softmax policy parameterization in cooperative Markov games. Further, by equipping existing MARL algorithms with a recent method of differentiable directed acyclic graphs (DAGs), we develop practical algorithms to learn the context-aware Bayesian network policies in scenarios with partial observability and various difficulty. We also dynamically decrease the sparsity of the learned DAG throughout the training process, which leads to weakly or even purely independent policies for decentralized execution. Empirical results on a range of MARL benchmarks show the benefits of our approach. 

   more » « less
4. Multi-scale games: Representing and solving games on networks with group structure

   Jin, Kun; Vorobeychik, Yevgeniy; Liu, Mingyan (January 2021, Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   Network games provide a natural machinery to compactly represent strategic interactions among agents whose payoffs exhibit sparsity in their dependence on the actions of others. Besides encoding interaction sparsity, however, real networks often exhibit a multi-scale structure, in which agents can be grouped into communities, those communities further grouped, and so on, and where interactions among such groups may also exhibit sparsity. We present a general model of multi-scale network games that encodes such multi-level structure. We then develop several algorithmic approaches that leverage this multi-scale structure, and derive sufficient conditions for convergence of these to a Nash equilibrium. Our numerical experiments demonstrate that the proposed approaches enable orders of magnitude improvements in scalability when computing Nash equilibria in such games. For example, we can solve previously intractable instances involving up to 1 million agents in under 15 minutes. 

   more » « less
5. Coordinated Relative Attitude Control and Synchronization of a Multi-body Network of Vehicles

   https://doi.org/10.23919/ACC63710.2025.11108024  

   Srinivasu, Neon; Hashkavaei, Nazanin S; Sanyal, Amit K; Butcher, Eric A (July 2025, IEEE)

   This work analyzes and develops some fundamental results for attitude consensus control of a network of rigid-body vehicles, considered a multi-agent rigid body system (MARBS). The system is analyzed using a full rigid body dynamics model on TSO(3) for each vehicle (agent) in the network. Therefore, the state space of the system is TSO(3)^N, where N is the number of vehicles. Attitude synchronization control laws for each vehicle to reach a consensus attitude with zero angular velocity for a particular type of network are obtained, using a Morse-Lyapunov function. Some fundamental results on equilibria of the network under these attitude consensus control laws are obtained. We show that unlike cooperative control of multi-agent systems with highly simplified dynamics models for agents, like point particles or unicycles where the state space of the dynamics is modeled as a vector space, there are multiple equilibrium solutions possible for attitude consensus control laws for a MARBS with dynamics on TSO(3)^N. Further, the number of equilibria depends on the network graph topology. This is followed by numerical simulation results for two different network graphs, which show this network control framework to be effective in obtaining attitude consensus. 

   more » « less