More Like this

1. What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?

   Han, Songyang; Su, Sanbao; He, Sihong; Han, Shuo; Yang, Haizhao; Zou, Shaofeng; Miao, Fei (February 2024, Transactions on machine learning research)

   Various methods for Multi-Agent Reinforcement Learning (MARL) have been developed with the assumption that agents' policies are based on accurate state information. However, policies learned through Deep Reinforcement Learning (DRL) are susceptible to adversarial state perturbation attacks. In this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to investigate different solution concepts of MARL under state uncertainties. Our analysis shows that the commonly used solution concepts of optimal agent policy and robust Nash equilibrium do not always exist in SAMGs. To circumvent this difficulty, we consider a new solution concept called robust agent policy, where agents aim to maximize the worst-case expected state value. We prove the existence of robust agent policy for finite state and finite action SAMGs. Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C) algorithm to learn robust policies for MARL agents under state uncertainties. Our experiments demonstrate that our algorithm outperforms existing methods when faced with state perturbations and greatly improves the robustness of MARL policies. Our code is public on https://songyanghan.github.io/what_is_solution/. 

   more » « less
2. What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?

   Han, Songyang; Su, Sanbao; He, Sihong; Han, Shuo; Yang, Haizhao; Zou, Shaofeng; Miao, Fei (February 2024, Transactions on Machine Learning Research)

   Various methods for Multi-Agent Reinforcement Learning (MARL) have been developed with the assumption that agents’ policies are based on accurate state information. However, policies learned through Deep Reinforcement Learning (DRL) are susceptible to adversarial state perturbation attacks. In this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to investigate different solution concepts of MARL under state uncertainties. Our analysis shows that the commonly used solution concepts of optimal agent policy and robust Nash equilibrium do not always exist in SAMGs. To circumvent this difficulty, we consider a new solution concept called robust agent policy, where agents aim to maximize the worst-case expected state value. We prove the existence of robust agent policy for finite state and finite action SAMGs. Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C) algorithm to learn robust policies for MARL agents under state uncertainties. Our experiments demonstrate that our algorithm outperforms existing methods when faced with state perturbations and greatly improves the robustness of MARL policies. Our code is public on https://songyanghan.github.io/what_is_solution/. 

   more » « less
3. PACE: A Framework for Learning and Control in Linear Incomplete-Information Differential Games

   Soltanian, Seyed Yousef; Zhang, Wenlong (May 2025, Proceedings of Machine Learning Research)

   In this paper, we address the problem of a two-player linear quadratic differential game with incomplete information, a scenario commonly encountered in multi-agent control, human-robot interaction (HRI), and approximation methods to solve general-sum differential games. While solutions to such linear differential games are typically obtained through coupled Riccati equations, the complexity increases when agents have incomplete information, particularly when neither is aware of the other’s cost function. To tackle this challenge, we propose a model-based Peer-Aware Cost Estimation (PACE) framework for learning the cost parameters of the other agent. In PACE, each agent treats its peer as a learning agent rather than a stationary optimal agent, models their learning dynamics, and leverages this dynamic to infer the cost function parameters of the other agent. This approach enables agents to infer each other’s objective function in real time based solely on their previous state observations and dynamically adapt their control policies. Furthermore, we provide a theoretical guarantee for the convergence of parameter estimation and the stability of system states in PACE. Additionally, using numerical studies, we demonstrate how modeling the learning dynamics of the other agent benefits PACE, compared to approaches that approximate the other agent as having complete information, particularly in terms of stability and convergence speed. 

   more » « less
4. Density-Driven Optimal Control for Efficient and Collaborative Multiagent Nonuniform Coverage

   https://doi.org/10.1109/TSMC.2025.3622075  

   Seo, Sungjun; Lee, Kooktae (December 2025, IEEE Transactions on Systems, Man, and Cybernetics: Systems)

   This paper addresses the critical challenge of non-uniform area coverage in multi-agent systems, where certain regions require higher priority based on mission objectives. Traditional uniform coverage methods are often inadequate in many area coverage scenarios, as they may overlook real-world complexities where uniform coverage is neither feasible nor desirable. Although several non-uniform coverage methods exist, they frequently lack guarantees of optimality or fail to consider essential real-world constraints such as agent dynamics, the number of agents, available operation time, and decentralized control. To overcome these limitations, a novel control scheme called Density-Driven Optimal Control (D$^2$OC) is proposed. The key innovation of D$^2$OC lies in the integration of optimal transport theory with multi-agent control, allowing agents to dynamically adjust their coverage according to a reference density distribution that reflects the mission’s priorities. Optimality is guaranteed by solving an optimization problem that incorporates the aforementioned real-world constraints. The control law is derived from the Lagrangian associated with the objective cost, utilizing optimal transport for both linear and nonlinear systems. Guarantees for global optimality and the existence of the optimal control input for linear systems are provided through an analytic solution. Additionally, an efficient data-sharing algorithm for decentralized control among multiple agents is proposed. To evaluate the efficacy of the proposed control scheme, various simulation results are presented to compare its performance with existing methods. 

   more » « less
5. STL-GO: Spatio-Temporal Logic with Graph Operators for Distributed Systems with Multiple Network Topologies

   https://doi.org/10.1145/3760258  

   Zhao, Yiqi; Yu, Xinyi; Hoxha, Bardh; Fainekos, Georgios; Deshmukh, Jyotirmoy; Lindemann, Lars (November 2025, ACM Transactions on Embedded Computing Systems)

   Multi-agent systems (MASs) consisting of a number of autonomous agents that communicate, coordinate, and jointly sense the environment to achieve complex missions can be found in a variety of applications such as robotics, smart cities, and internet-of-things applications. Modeling and monitoring MAS requirements to guarantee overall mission objectives, safety, and reliability is an important problem. Such requirements implicitly require reasoning about diverse sensing and communication modalities between agents, analysis of the dependencies between agent tasks, and the spatial or virtual distance between agents. To capture such rich MAS requirements, we model agent interactions via multiple directed graphs, and introduce a new logic –Spatio-Temporal Logic with Graph Operators(STL-GO). The key innovation in STL-GO are graph operators that enable us to reason about the number of agents along either the incoming or outgoing edges of the underlying interaction graph that satisfy a given property of interest; for example, the requirement that an agent should sense at least two neighboring agents whose task graphs indicate the ability to collaborate. We then propose novel distributed monitoring conditions for individual agents that use only local information to determine whether or not an STL-GO specification is satisfied. We compare the expressivity of STL-GO against existing spatio-temporal logic formalisms, and demonstrate the utility of STL-GO and our distributed monitors in a bike-sharing and a multi-drone case study. 

   more » « less