An official website of the United States government
As modern cyber-physical systems (CPS) increas- ingly embed machine learning in perception, prediction, and control, understanding why a system behaves as it does has become essential for safety, debugging, and trustworthiness. However, existing explainable AI (XAI) methods focus on isolated models or decision components, providing fragmented insights that fail to capture the full causal chain of system operation. To address these challenges, we present X-CPS, a graph-structured framework for explaining learning-enabled CPS behaviors. X- CPS introduces a formal CPS specification interface from which a graph-structured representation is constructed offline, encod- ing the system architecture, inter-component dependencies, and component-specific analysis logic. At runtime, a Large Language Model (LLM) leverages this structured representation to process diverse natural language queries, including why, why not, and what-if questions, and generate explanations that trace decisions from high-level outputs through intermediate layers to raw sensor observations. A user study on autonomous vehicle behaviors shows that X-CPS significantly outperforms the attribution- based baseline across all evaluation metrics, demonstrating that graph-structured architectural context enhances user trust and interpretability.
more »
« less
This content will become publicly available on April 20, 2027
Cognition Envelopes for Bounded Decision Making in Autonomous UAS Operations
Cyber-physical systems increasingly rely on foundational models, such as Large Language Models (LLMs) and Vision-Language Models (VLMs) to increase autonomy through enhanced perception, inference, and planning. However, these models also introduce new types of errors, such as hallucinations, over-generalizations, and context misalignments, resulting in incorrect and flawed decisions. To address this, we introduce the concept of Cognition Envelopes, designed to establish reasoning boundaries that constrain AI-generated decisions while complementing the use of meta-cognition and traditional safety envelopes. As with safety envelopes, Cognition Envelopes require practical guidelines and systematic processes for their definition, validation, and assurance. In this paper we describe an LLM/VLM-supported pipeline for dynamic clue analysis within the domain of small autonomous Uncrewed Aerial Systems deployed on Search and Rescue (SAR) missions, and a Cognition Envelope based on probabilistic reasoning and resource analysis. We evaluate the approach through assessing decisions made by our Clue Analysis Pipeline in a series of SAR missions. Finally, we identify key software engineering challenges for systematically designing, implementing, and validating Cognition Envelopes for AI-supported decisions in cyber-physical systems.
more »
« less
- PAR ID:
- 10668139
- Publisher / Repository:
- IEEE/ACM Conference on AI for Software Engineering
- Date Published:
- Volume:
- 2026
- Subject(s) / Keyword(s):
- cognition envelope, LLM, VLM, autonomous decisions, probabilistic lost person models, small uncrewed aerial systems, search and rescue
- Format(s):
- Medium: X
- Location:
- Rio de Janeiro, Brazil
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The adoption of digital technology in industrial control systems (ICS) enables improved control over operation, ease of system diagnostics and reduction in cost of maintenance of cyber physical systems (CPS). However, digital systems expose CPS to cyber-attacks. The problem is grave since these cyber-attacks can lead to cascading failures affecting safety in CPS. Unfortunately, the relationship between safety events and cyber-attacks in ICS is ill-understood and how cyber-attacks can lead to cascading failures affecting safety. Consequently, CPS operators are ill-prepared to handle cyber-attacks on their systems. In this work, we envision adopting Explainable AI to assist CPS oper-ators in analyzing how a cyber-attack can trigger safety events in CPS and then interactively determining potential approaches to mitigate those threats. We outline the design of a formal framework, which is based on the notion of transition systems, and the associated toolsets for this purpose. The transition system is represented as an AI Planning problem and adopts the causal formalism of human reasoning to asssit CPS operators in their analyses. We discuss some of the research challenges that need to be addressed to bring this vision to fruition.more » « less
-
Hierarchical decisions in natural environments require processing uncertainty across multiple levels, but existing models struggle to explain how animals perform flexible, goal-directed behaviors under such conditions. Here we introduce CogLinks, biologically grounded neural architectures that combine corticostriatal circuits for reinforcement learning and frontal thalamocortical networks for executive control. Through mathematical analysis and targeted lesion, we show that these systems specialize in different forms of uncertainty, and their interaction supports hierarchical decisions by regulating efficient exploration, and strategy switching. We apply CogLinks to a computational psychiatry problem, linking neural dysfunction in schizophrenia to atypical reasoning patterns in decision making. Overall, CogLink fills an important gap in the computational landscape, providing a bridge from neural substrates to higher cognition.more » « less
-
Safe reinforcement learning (safe RL) has been applied to synthesize control policies that maximize task rewards while adhering to safety constraints within simulated secure cyber-physical systems. However, the vulnerability of safe RL to adversarial attacks remains largely unexplored. We argue that understanding the safety vulnerabilities of learned control policies is crucial for ensuring true safety in real-world scenarios. To address this gap, we first formally define the safe RL problem with formal language (Signal temporal logic), and demonstrate that even optimal policies are susceptible to observation perturbations. We then introduce novel safety violation attacks that exploit adversarial models trained with reversed safety constraints to induce unsafe behaviors. Lastly, through both theoretical analysis and experimental results, we demonstrate that our approach is more effective at violating safety constraints than existing adversarial RL methods, which primarily focus on reducing task rewards rather than compromising safety.more » « less
-
As AI systems progress, we rely more on them to make decisions with us and for us. To ensure that such decisions are aligned with human values, it is imperative for us to understand not only what decisions they make but also how they come to those decisions. Reasoning language models, which provide both final responses and partially transparent intermediate thinking traces, present a timely opportunity to study AI procedural reasoning. Unlike math and code problems which often have objectively correct answers, moral dilemmas are an excellent testbed for process-focused evaluation because they allow for multiple defensible conclusions. We present MoReBench: 1,000 moral scenarios, each paired with rubric criteria experts consider essential to include or avoid when reasoning about the scenarios. MoReBench contains over 23,000 criteria covering moral considerations, trade-offs, and actionable recommendations. We also curate MoReBench-Theory, 150 examples testing reasoning under five major frameworks in normative ethics. Results show that scaling laws and existing benchmarks on math, code, and scientific reasoning fail to predict moral reasoning ability, and models show partiality toward specific moral frameworks.more » « less
