An official website of the United States government
he Simplex Architecture is a runtime assurance framework where control authority may switch from an unverified and potentially unsafe advanced controller to a backup baseline controller in order to maintain the safety of an autonomous cyber-physical system. In this work, we show that runtime checks can replace the requirement to statically verify safety of the baseline controller. This is important as there are many powerful control techniques, such as model-predictive control and neural network controllers, that work well in practice but are difficult to statically verify. Since the method does not use internal information about the advanced or baseline controller, we call the approach the Black-Box Simplex Architecture. We prove the architecture is safe and present two case studies where (i) model-predictive control provides safe multi-robot coordination, and (ii) neural networks provably prevent collisions in groups of F-16 aircraft, despite the controllers occasionally outputting unsafe commands.
more »
« less
The black-box simplex architecture for runtime assurance of multi-agent CPS
The Simplex Architecture is a runtime assurance framework where control authority may switch from an unverified and potentially unsafe advanced controller to a backup baseline controller in order to maintain the safety of an autonomous cyber-physical system. In this work, we show that runtime checks can replace the requirement to statically verify safety of the baseline controller. This is important as there are many powerful control techniques, such as model-predictive control and neural network controllers, that work well in practice but are difficult to statically verify. Since the method does not use internal information about the advanced or baseline controller, we call the approach the Black-Box Simplex Architecture. We prove the architecture is safe and present two case studies where (i) modelpredictive control provides safe multi-robot coordination, and (ii) neural networks provably prevent collisions in groups of F-16 aircraft, despite the controllers occasionally outputting unsafe commands. We further show how to safely blend commands from the advanced and baseline controllers in multiagent systems, reducing the performance impact when switching is necessary to preserve safety.
more »
« less
- Award ID(s):
- 1954837
- PAR ID:
- 10508985
- Publisher / Repository:
- Springer
- Date Published:
- Journal Name:
- Innovations in Systems and Software Engineering
- ISSN:
- 1614-5046
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This paper presents Barrier-based Simplex (Bb-Simplex), a new, provably correct design for runtime assurance of continuous dynamic systems. Bb-Simplex is centered around the Simplex control architecture, which consists of a high-performance advanced controller that is not guaranteed to maintain safety of the plant, a verified-safe baseline controller, and a decision module that switches control of the plant between the two controllers to ensure safety without sacrificing performance. In Bb-Simplex, Barrier certificates are used to prove that the baseline controller ensures safety. Furthermore, Bb-Simplex features a new automated method for deriving, from the barrier certificate, the conditions for switching between the controllers. Our method is based on the Taylor expansion of the barrier certificate and yields computationally inexpensive switching conditions. We also propose extensions to Bb-Simplex to enable its use in hybrid systems, which have multiple modes each with its own dynamics, and to support its use when only approximate dynamics (not exact dynamics) are available, for both continuous and hybrid dynamic systems. We consider significant applications of Bb-Simplex to microgrids featuring advanced controllers in the form of neural networks trained using reinforcement learning. These microgrids are modeled in RTDS, an industry-standard high-fidelity, real-time power systems simulator. Our results demonstrate that Bb-Simplex can automatically derive switching conditions for complex continuous and hybrid dynamic systems, the switching conditions are not overly conservative, and Bb-Simplex ensures safety even in the presence of adversarial attacks on the neural controller when only approximate dynamics (with an error bound) are available. Our experiments to evaluate conservativeness of the switching conditions show that switches from an unsafe advanced controller to a baseline controller occur on average only a few control periods before a safety violation would have occurred.more » « less
-
Dang, Thao; Stolz, Volker (Ed.)We present Barrier-based Simplex (Bb-Simplex), a new, provably correct design for runtime assurance of continuous dynamical systems. Bb-Simplex is centered around the Simplex Control Architecture, which consists of a high-performance advanced controller which is not guaranteed to maintain safety of the plant, a verified-safe baseline controller, and a decision module that switches control of the plant between the two controllers to ensure safety without sacrificing performance. In Bb-Simplex, Barrier certificates are used to prove that the baseline controller ensures safety. Furthermore, Bb-Simplex features a new automated method for deriving, from the barrier certificate, the conditions for switching between the controllers. Our method is based on the Taylor expansion of the barrier certificate and yields computationally inexpensive switching conditions. We consider a significant application of Bb-Simplex to a microgrid featuring an advanced controller in the form of a neural network trained using reinforcement learning. The microgrid is modeled in RTDS, an industry-standard high-fidelity, real-time power systems simulator. Our results demonstrate that Bb-Simplex can automatically derive switching conditions for complex systems, the switching conditions are not overly conservative, and Bb-Simplex ensures safety even in the presence of adversarial attacks on the neural controller.more » « less
-
We present the Distributed Simplex Architecture (DSA), a new runtime assurance technique that provides safety guarantees for multi-agent systems (MASs). DSA is inspired by the Simplex control architecture of Sha et al., but with some significant differences. The traditional Simplex approach is limited to single-agent systems or a MAS with a centralized control scheme. DSA addresses this limitation by extending the scope of Simplex to include MASs under distributed control. In DSA, each agent runs a local instance of traditional Simplex such that the preservation of safety in the local instances implies safety for the entire MAS. Control Barrier Functions play a critical role. They are used to define DSA’s core components (the baseline controller and the decision module’s switching logic between advanced and baseline controllers) and to verify the safety of a DSA instance in a distributed manner. We provide a general proof of safety for DSA, and present experimental results for several case studies, including flocking with collision avoidance, safe navigation of ground rovers through way-points, and the safe operation of a microgrid.more » « less
-
Large language models (LLMs) provide a promising interface for mapping natural-language user instructions to robot actions, but unconstrained generation can produce invalid plans and unsafe behaviors when deployed on physical platforms. In this work, we present a three-level, defense-in-depth architecture that uses constrained behavior trees (BTs) as an intermediate, verifiable control mechanism for LLM-driven robot control. At the high-level layer, an LLM produces a task-level BT candidate in a restricted JSON format, limited to an allowed set of node types, actions, and parameter ranges. The candidate is then statically validated using human-defined rules, including schema checks and semantic safety constraints. Invalid candidates are rejected and re-generated up to a retry limit. At the mid-level layer, a BT interpreter executes the validated tree reactively using symbolic world-state updates and outputs motion primitives. A runtime safety monitor detects hazards such as obstacle proximity and execution stalls, overriding the command stream to trigger STOP or a safe fallback behavior. At the low-level layer, the system maps sensor data from the robot’s perception into symbolic state features and converts motion commands from the upper layer into actuator control signals. Experiments in small-scale navigation scenarios demonstrate reliable task execution while enforcing user-specified safety constraints through validation and runtime monitoring.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1007/s11334-024-00553-6
