%AHan, Songyang%ASu, Sanbao%AHe, Sihong%AHan, Shuo%AYang, Haizhao%AZou, Shaofeng%AMiao, Fei%BJournal Name: Transactions on Machine Learning Research
%D2024%Iopenreview
%JJournal Name: Transactions on Machine Learning Research
%K
%MOSTI ID: 10509715
%PMedium: X
%TWhat is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?
%XVarious methods for Multi-Agent Reinforcement Learning (MARL) have been developed
with the assumption that agentsâ€™ policies are based on accurate state information. However,
policies learned through Deep Reinforcement Learning (DRL) are susceptible to adversarial
state perturbation attacks. In this work, we propose a State-Adversarial Markov Game
(SAMG) and make the first attempt to investigate different solution concepts of MARL under
state uncertainties. Our analysis shows that the commonly used solution concepts of optimal
agent policy and robust Nash equilibrium do not always exist in SAMGs. To circumvent this
difficulty, we consider a new solution concept called robust agent policy, where agents aim to
maximize the worst-case expected state value. We prove the existence of robust agent policy
for finite state and finite action SAMGs. Additionally, we propose a Robust Multi-Agent
Adversarial Actor-Critic (RMA3C) algorithm to learn robust policies for MARL agents under
state uncertainties. Our experiments demonstrate that our algorithm outperforms existing
methods when faced with state perturbations and greatly improves the robustness of MARL
policies. Our code is public on https://songyanghan.github.io/what_is_solution/.
%0Journal Article