An official website of the United States government
The advent of autonomous mobile multi-robot systems has driven innovation in both the industrial and defense sectors. The integration of such systems in safety- and security- critical applications has raised concern over their resilience to attack. In this work, we investigate the security problem of a stealthy adversary masquerading as a properly functioning agent. We show that conventional multi-agent pathfinding solutions are vulnerable to these physical masquerade attacks. Furthermore, we provide a constraint-based formulation of multi-agent pathfinding that yields multi-agent plans that are provably resilient to physical masquerade attacks. This formalization leverages inter-agent observations to facilitate introspective monitoring to guarantee resilience.
more »
« less
This content will become publicly available on December 1, 2026
Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Development
Agentic AI and Multi-Agent Systems are poised to dominate industry and society imminently. Powered by goal- driven autonomy, they represent a powerful form of generative AI, marking a transition from reactive content generation into proactive multitasking capabilities. As an exemplar, we propose an architecture of a multi-agent system for the implementation phase of the software engineering process. We also present a comprehensive threat model for the proposed system. We demon- strate that while such systems can generate code quite accurately, they are vulnerable to attacks, including code injection. Due to their autonomous design and lack of humans in the loop, these systems cannot identify and respond to attacks by themselves. This paper analyzes the vulnerability of multi-agent systems and concludes that the coder-reviewer-tester architecture is more resilient than both the coder and coder-tester architectures, but is less efficient at writing code. We find that by adding a security analysis agent, we mitigate the loss in efficiency while achieving even better resiliency. We conclude by demonstrating that the security analysis agent is vulnerable to advanced code injection attacks, showing that embedding poisonous few-shot examples in the injected code can increase the attack success rate from 0% to 71.95%.
more »
« less
- Award ID(s):
- 2349452
- PAR ID:
- 10655815
- Publisher / Repository:
- International Conference on Machine Learning Applications (ICMLA)
- Date Published:
- Subject(s) / Keyword(s):
- Agentic AI, Threat Model, Software Engineering Process
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
RISC-V is a promising open source architecture that targets low-power embedded devices and SoCs. However, there is a dearth of practical and low-overhead security solutions in the RISC-V architecture. Programs compiled using RISC-V toolchains are still vulnerable to code injection and code reuse attacks such as buffer overflow and return-oriented programming (ROP). In this paper, we propose two hardware implemented security extensions to RISC-V that provides a defense mechanism against such attacks. We first employ a Physically Unclonable Function (PUF)-based randomized canary generation technique that removes the need to store the sensitive canary words in memory or CPU registers, thereby being more secure, while incurring low overheads. We implement the proposed Canary Engine in RISC-V RocketChip with Rocket Custom Coprocessor (RoCC). Simulation results show 2.2% average execution overhead with a single buffer protection, while a 10X increase in buffer count only increases the overhead by 1.5X when protection is extended to all buffers. We further improve upon this with a dedicated security coprocessor FIXER, implemented on the RoCC. FIXER enforces fine-grained control-flow integrity (CFI) of running programs on backward edges (returns) and forward edges (calls) without requiring any architectural modifications to the processor core. Compared to software-based solutions, FIXER reduces energy overhead by 60% at minimal execution time (1.5%) and area (2.9%) overheads.more » « less
-
Most proposals for securing control systems are heuristic in nature, and while they increase the protection of their target, the security guarantees they provide are unclear. This paper proposes a new way of modeling the security guarantees of a Cyber-Physical System (CPS) against arbitrary false command attacks. As our main case study, we use the most popular testbed for control systems security. We first propose a detailed formal model of this testbed and then show how the original configuration is vulnerable to a single-actuator attack. We then propose modifications to the control system and prove that our modified system is secure against arbitrary, single-actuator attacks.more » « less
-
Controller Area Network (CAN) is the de-facto standard in-vehicle network system. Despite its wide adoption by automobile manufacturers, the lack of security design makes it vulnerable to attacks. For instance, broadcasting packets without authentication allows the impersonation of electronic control units (ECUs). Prior mitigations, such as message authentication or intrusion detection systems, fail to address the compatibility requirement with legacy ECUs, stealthy and sporadic malicious messaging, or guaranteed attack detection. We propose a novel authentication system called ShadowAuth that overcomes the aforementioned challenges by offering backward-compatible packet authentication to ECUs without requiring ECU firmware source code. Specifically, our authentication scheme provides transparent CAN packet authentication without modifying existing CAN packet definitions (e.g., J1939) via automatic ECU firmware instrumentation technique to locate CAN packet transmission code, and instrument authentication code based on the CAN packet behavioral transmission patterns. ShadowAuth enables vehicles to detect state-of-the-art CAN attacks, such as bus-off and packet injection, responsively within 60ms without false positives. ShadowAuth provides a sound and deployable solution for real-world ECUs.more » « less
-
Cyber-Physical Systems (CPS) have been increasingly subject to cyber-attacks including code injection attacks. Zero day attacks further exasperate the threat landscape by requiring a shift to defense in depth approaches. With the tightly coupled nature of cyber components with the physical domain, these attacks have the potential to cause significant damage if safety-critical applications such as automobiles are compromised. Moving target defense techniques such as instruction set randomization (ISR) have been commonly proposed to address these types of attacks. However, under current implementations an attack can result in system crashing which is unacceptable in CPS. As such, CPS necessitate proper control reconfiguration mechanisms to prevent a loss of availability in system operation. This paper addresses the problem of maintaining system and security properties of a CPS under attack by integrating ISR, detection, and recovery capabilities that ensure safe, reliable, and predictable system operation. Specifically, we consider the problem of detecting code injection attacks and reconfiguring the controller in real-time. The developed framework is demonstrated with an autonomous vehicle case study.more » « less
