More Like this

1. Decentralized graph-based multi-agent reinforcement learning using reward machines

   https://doi.org/10.1016/j.neucom.2023.126974  

   Hu, Jueming ; Xu, Zhe ; Wang, Weichang ; Qu, Guannan ; Pang, Yutian ; Liu, Yongming ( January 2024 , Neurocomputing)

   In multi-agent reinforcement learning (MARL), it is challenging for a collection of agents to learn complex temporally extended tasks. The difficulties lie in computational complexity and how to learn the high-level ideas behind reward functions. We study the graph-based Markov Decision Process (MDP), where the dynamics of neighboring agents are coupled. To learn complex temporally extended tasks, we use a reward machine (RM) to encode each agent’s task and expose reward function internal structures. RM has the capacity to describe high-level knowledge and encode non-Markovian reward functions. We propose a decentralized learning algorithm to tackle computational complexity, called decentralized graph-based reinforcement learning using reward machines (DGRM), that equips each agent with a localized policy, allowing agents to make decisions independently based on the information available to the agents. DGRM uses the actor-critic structure, and we introduce the tabular Q-function for discrete state problems. We show that the dependency of the Q-function on other agents decreases exponentially as the distance between them increases. To further improve efficiency, we also propose the deep DGRM algorithm, using deep neural networks to approximate the Q-function and policy function to solve large-scale or continuous state problems. The effectiveness of the proposed DGRM algorithm is evaluated by three case studies, two wireless communication case studies with independent and dependent reward functions, respectively, and COVID-19 pandemic mitigation. Experimental results show that local information is sufficient for DGRM and agents can accomplish complex tasks with the help of RM. DGRM improves the global accumulated reward by 119% compared to the baseline in the case of COVID-19 pandemic mitigation. 

   more » « less
2. Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learning.

   Yuchen Xiao, Xueguang Lyu ( January 2021 , the International Symposium on Multi-Robot and Multi-Agent Systems)

   Policy gradient methods have become popular in multi-agent reinforcement learning, but they suffer from high variance due to the presence of environmental stochasticity and exploring agents (i.e., non-stationarity), which is potentially worsened by the difficulty in credit assignment. As a result, there is a need for a method that is not only capable of efficiently solving the above two problems but also robust enough to solve a variety of tasks. To this end, we propose a new multi-agent policy gradient method, called Robust Local Advantage (ROLA) Actor-Critic. ROLA allows each agent to learn an individual action-value function as a local critic as well as ameliorating environment non-stationarity via a novel centralized training approach based on a centralized critic. By using this local critic, each agent calculates a baseline to reduce variance on its policy gradient estimation, which results in an expected advantage action-value over other agents’ choices that implicitly improves credit assignment. We evaluate ROLA across diverse benchmarks and show its robustness and effectiveness over a number of state-of-the-art multi-agent policy gradient algorithms. 

   more » « less
3. An Efficient Distributed Reinforcement Learning for Enhanced Multi-Microgrid Management

   https://doi.org/10.1109/IJCNN55064.2022.9892754  

   Das, Avijit ; Ni, Zhen ; Wu, Di ( September 2022 , 2022 International Joint Conference on Neural Networks (IJCNN))

   Economic dispatch in a multi-microgrid (MMG) system involves an increasing number of states from distributed energy resources (DERs) compared to a single microgrid. In these cases, traditional reinforcement learning (RL) approaches may become computationally expensive or less effective in finding the least-cost solution. This paper presents a novel RL approach that employs local learning agents to interact with individual microgrid environments in a distributed manner and a global agent to search for actions to minimize system cost at the MMG system level. The proposed distributed RL framework is more efficient in learning the dispatch policy compared to conventional approaches. Case studies are performed on a 3-microgrid system with different types of DERs. Results substantiate the effectiveness of the proposed approach in comparison with conventional methods in terms of operation costs, computation time, and peak-to-average ratio. 

   more » « less
4. Semicentralized Deep Deterministic Policy Gradient in Cooperative StarCraft Games

   https://doi.org/10.1109/TNNLS.2020.3042943  

   Xie, Dong ; Zhong, Xiangnan ( December 2020 , IEEE Transactions on Neural Networks and Learning Systems)

   null (Ed.)

   In this article, we propose a novel semicentralized deep deterministic policy gradient (SCDDPG) algorithm for cooperative multiagent games. Specifically, we design a two-level actor-critic structure to help the agents with interactions and cooperation in the StarCraft combat. The local actor-critic structure is established for each kind of agents with partially observable information received from the environment. Then, the global actor-critic structure is built to provide the local design an overall view of the combat based on the limited centralized information, such as the health value. These two structures work together to generate the optimal control action for each agent and to achieve better cooperation in the games. Comparing with the fully centralized methods, this design can reduce the communication burden by only sending limited information to the global level during the learning process. Furthermore, the reward functions are also designed for both local and global structures based on the agents' attributes to further improve the learning performance in the stochastic environment. The developed method has been demonstrated on several scenarios in a real-time strategy game, i.e., StarCraft. The simulation results show that the agents can effectively cooperate with their teammates and defeat the enemies in various StarCraft scenarios. 

   more » « less
5. Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning

   Xiao, Yuchen ; Tan, Weihao ; Amato, Christopher ( January 2022 , Advances in neural information processing systems)

   Synchronizing decisions across multiple agents in realistic settings is problematic since it requires agents to wait for other agents to terminate and communicate about termination reliably. Ideally, agents should learn and execute asynchronously instead. Such asynchronous methods also allow temporally extended actions that can take different amounts of time based on the situation and action executed. Unfortunately, current policy gradient methods are not applicable in asynchronous settings, as they assume that agents synchronously reason about action selection at every time step. To allow asynchronous learning and decision-making, we formulate a set of asynchronous multi-agent actor-critic methods that allow agents to directly optimize asynchronous policies in three standard training paradigms: decentralized learning, centralized learning, and centralized training for decentralized execution. Empirical results (in simulation and hardware) in a variety of realistic domains demonstrate the superiority of our approaches in large multi-agent problems and validate the effectiveness of our algorithms for learning high-quality and asynchronous solutions. 

   more » « less