More Like this

1. Evolutionary Constraints on Connectivity Patterns in the Mammalian Suprachiasmatic Nucleus

   https://doi.org/10.3389/fnetp.2021.716883  

   Spencer, Connor ; Tripp, Elizabeth ; Fu, Feng ; Pauls, Scott ( August 2021 , Frontiers in Network Physiology)

   The mammalian suprachiasmatic nucleus (SCN) comprises about 20,000 interconnected oscillatory neurons that create and maintain a robust circadian signal which matches to external light cues. Here, we use an evolutionary game theoretic framework to explore how evolutionary constraints can influence the synchronization of the system under various assumptions on the connection topology, contributing to the understanding of the structure of interneuron connectivity. Our basic model represents the SCN as a network of agents each with two properties—a phase and a flag that determines if it communicates with its neighbors or not. Communication comes at a cost to the agent, but synchronization of phases with its neighbors bears a benefit. Earlier work shows that when we have “all-to-all” connectivity, where every agent potentially communicates with every other agent, there is often a simple trade-off that leads to complete communication and synchronization of the system: the benefit must be greater than twice the cost. This trade-off for all-to-all connectivity gives us a baseline to compare to when looking at other topologies. Using simulations, we compare three plausible topologies to the all-to-all case, finding that convergence to synchronous dynamics occurs in all considered topologies under similar benefit and cost trade-offs. Consequently, sparser, less biologically costly topologies are reasonable evolutionary outcomes for organisms that develop a synchronizable oscillatory network. Our simulations also shed light on constraints imposed by the time scale on which we observe the SCN to arise in mammals. We find two conditions that allow for a synchronizable system to arise in relatively few generations. First, the benefits of connectivity must outweigh the cost of facilitating the connectivity in the network. Second, the game at the core of the model needs to be more cooperative than antagonistic games such as the Prisoner’s Dilemma. These results again imply that evolutionary pressure may have driven the system towards sparser topologies, as they are less costly to create and maintain. Last, our simulations indicate that models based on the mutualism game fare the best in uptake of communication and synchronization compared to more antagonistic games such as the Prisoner’s Dilemma. 

   more » « less
2. Towards Situated Communication in Multi-Step Interactions: Time is a Key Pressure in Communication Emergence

   Kalinowska, Aleksandra ; Davoodi, Elnaz ; Mathewson, Kory W. ; Murphey, Todd ; Pilarski, Patrick M. ( October 2022 , Annual Meeting of the Cognitive Science Society)

   Enabling efficient communication in artificial agents brings us closer to machines that can cooperate with each other and with human partners. Hand-engineered approaches have substantial limitations, leading to increased interest in methods for communication to emerge autonomously between artificial agents. Most of the research in the field explores unsituated communication in one-step referential tasks. The tasks are not temporally interactive and lack time pressures typically present in natural communication and language learning. In these settings, agents can successfully learn what to communicate but not when or whether to communicate. Here, we extend the literature by assessing emergence of communication between reinforcement learning agents in a temporally interactive, cooperative task of navigating a gridworld environment. We show that, through multi-step interactions, agents develop just-in-time messaging protocols that enable them to successfully solve the task. With memory—which provides flexibility around message timing—agent pairs converge to a look-ahead communication protocol, finding an optimal solution to the task more quickly than without memory. Lastly, we explore situated communication, enabling the acting agent to choose when and whether to communicate. With the opportunity cost of forgoing an action to communicate, the acting agent learns to solicit information sparingly, in line with the Gricean Maxim of quantity. Our results point towards the importance of studying language emergence through situated communication in multi-step interactions. 

   more » « less
3. Adaptive Agent Architecture for Real-time Human-Agent Teaming

   Ni, T. ( January 2021 , AAAI Workshop on Plan, Activity, and Intent Recognition)

   Teamwork is a set of interrelated reasoning, actions and behaviors of team members that facilitate common objectives. Teamwork theory and experiments have resulted in a set of states and processes for team effectiveness in both human-human and agent-agent teams. However, human-agent teaming is less well studied because it is so new and involves asymmetry in policy and intent not present in human teams. To optimize team performance in human-agent teaming, it is critical that agents infer human intent and adapt their polices for smooth coordination. Most literature in human-agent teaming builds agents referencing a learned human model. Though these agents are guaranteed to perform well with the learned model, they lay heavy assumptions on human policy such as optimality and consistency, which is unlikely in many real-world scenarios. In this paper, we propose a novel adaptive agent architecture in human-model-free setting on a two-player cooperative game, namely Team Space Fortress (TSF). Previous human-human team research have shown complementary policies in TSF game and diversity in human players’ skill, which encourages us to relax the assumptions on human policy. Therefore, we discard learning human models from human data, and instead use an adaptation strategy on a pre-trained library of exemplar policies composed of RL algorithms or rule-based methods with minimal assumptions of human behavior. The adaptation strategy relies on a novel similarity metric to infer human policy and then selects the most complementary policy in our library to maximize the team performance. The adaptive agent architecture can be deployed in real-time and generalize to any off-the-shelf static agents. We conducted human-agent experiments to evaluate the proposed adaptive agent framework, and demonstrated the suboptimality, diversity, and adaptability of human policies in human-agent teams. 

   more » « less
4. Stable and Efficient Shapley Value-Based Reward Reallocation for Multi-Agent Reinforcement Learning of Autonomous Vehicles

   Songyang Han, He Wang ( May 2022 , 2022 IEEE International Conference on Robotics and Automation)

   With the development of sensing and communica- tion technologies in networked cyber-physical systems (CPSs), multi-agent reinforcement learning (MARL)-based methodolo- gies are integrated into the control process of physical systems and demonstrate prominent performance in a wide array of CPS domains, such as connected autonomous vehicles (CAVs). However, it remains challenging to mathematically characterize the improvement of the performance of CAVs with commu- nication and cooperation capability. When each individual autonomous vehicle is originally self-interest, we can not assume that all agents would cooperate naturally during the training process. In this work, we propose to reallocate the system’s total reward efficiently to motivate stable cooperation among autonomous vehicles. We formally define and quantify how to reallocate the system’s total reward to each agent under the proposed transferable utility game, such that communication- based cooperation among multi-agents increases the system’s total reward. We prove that Shapley value-based reward reallocation of MARL locates in the core if the transferable utility game is a convex game. Hence, the cooperation is stable and efficient and the agents should stay in the coalition or the cooperating group. We then propose a cooperative policy learning algorithm with Shapley value reward reallocation. In experiments, compared with several literature algorithms, we show the improvement of the mean episode system reward of CAV systems using our proposed algorithm. 

   more » « less
5. Intelligent Distributed Swarm Control for Large-Scale Multi-UAV Systems: A Hierarchical Learning Approach

   https://doi.org/10.3390/electronics12010089  

   Dey, Shawon ; Xu, Hao ( January 2023 , Electronics)

   In this paper, a distributed swarm control problem is studied for large-scale multi-agent systems (LS-MASs). Different than classical multi-agent systems, an LS-MAS brings new challenges to control design due to its large number of agents. It might be more difficult for developing the appropriate control to achieve complicated missions such as collective swarming. To address these challenges, a novel mixed game theory is developed with a hierarchical learning algorithm. In the mixed game, the LS-MAS is represented as a multi-group, large-scale leader–follower system. Then, a cooperative game is used to formulate the distributed swarm control for multi-group leaders, and a Stackelberg game is utilized to couple the leaders and their large-scale followers effectively. Using the interaction between leaders and followers, the mean field game is used to continue the collective swarm behavior from leaders to followers smoothly without raising the computational complexity or communication traffic. Moreover, a hierarchical learning algorithm is designed to learn the intelligent optimal distributed swarm control for multi-group leader–follower systems. Specifically, a multi-agent actor–critic algorithm is developed for obtaining the distributed optimal swarm control for multi-group leaders first. Furthermore, an actor–critic–mass method is designed to find the decentralized swarm control for large-scale followers. Eventually, a series of numerical simulations and a Lyapunov stability proof of the closed-loop system are conducted to demonstrate the performance of the developed scheme. 

   more » « less