An official website of the United States government
When dealing with safety–critical systems, various regulations, standards, and guidelines stipulate stringent requirements for certification and traceability of artifacts, but typically lack details with regards to the corresponding software engineering process. Given the industrial practice of only using semi-formal notations for describing engineering processes – with the lack of proper tool mapping – engineers and developers need to invest a significant amount of time and effort to ensure that all steps mandated by quality assurance are followed. The sheer size and complexity of systems and regulations make manual, timely feedback from Quality Assurance (QA) engineers infeasible. In order to address these issues, in this paper, we propose a novel framework for tracking, and “passively” executing processes in the background, automatically checking QA constraints depending on process progress, and informing the developer of unfulfilled QA constraints. We evaluate our approach by applying it to three case studies: a safety–critical open-source community system, a safety–critical system in the air-traffic control domain, and a non-safety–critical, web-based system. Results from our analysis confirm that trace links are often corrected or completed after the work step has been considered finished, and the engineer has already moved on to another step. Thus, support for timely and automated constraint checking has significant potential to reduce rework as the engineer receives continuous feedback already during their work step.
more »
« less
GRuM — A flexible model-driven runtime monitoring framework and its application to automated aerial and ground vehicles
When dealing with safety-critical systems, various regulations, standards, and guidelines stipulate stringent requirements for certification and traceability of artifacts, but typically lack \rev{details} with regards to the corresponding software engineering process. Given the industrial practice of only using semi-formal notations for describing engineering processes with the lack of proper tool mapping engineers and developers need to invest a significant amount of time and effort to ensure that all steps mandated by quality assurance are followed. The sheer size and complexity of systems and regulations make manual, timely feedback from Quality Assurance (QA) engineers infeasible. In order to address these issues, in this paper, we propose a novel framework for tracking, and ``passively'' executing processes in the background, automatically checking QA constraints depending on process progress, and informing the developer of unfulfilled QA constraints. We evaluate our approach by applying it to three case studies: a safety-critical open-source community system, a safety-critical system in the air-traffic control domain, and a non-safety-critical, web-based system. Results from our analysis confirm that trace links are often corrected or completed after the work step has been considered finished, and the engineer has already moved on to another step. Thus, support for timely and automated constraint checking has significant potential to reduce rework as the engineer receives continuous feedback already during their work step.
more »
« less
- Award ID(s):
- 1931962
- PAR ID:
- 10468153
- Publisher / Repository:
- Elsevier
- Date Published:
- Journal Name:
- Journal of Systems and Software
- Volume:
- 203
- Issue:
- C
- ISSN:
- 0164-1212
- Page Range / eLocation ID:
- 111733
- Subject(s) / Keyword(s):
- Cyber-Physical Systems Runtime Monitoring Model-Driven Engineering
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Regulations, standards, and guidelines for safety-critical systems stipulate stringent traceability but do not prescribe the corresponding, detailed software engineering process. Given the industrial practice of using only semi-formal notations to describe engineering processes, processes are rarely ``executable'' and developers have to spend significant manual effort in ensuring that they follow the steps mandated by quality assurance. The size and complexity of systems and regulations makes manual, timely feedback from Quality Assurance (QA) engineers infeasible. In this paper we propose a novel framework for tracking processes in the background, automatically checking QA constraints depending on process progress, and informing the developer of unfulfilled QA constraints. We evaluate our approach by applying it to two different case studies; one open source community system and a safety-critical system in the air-traffic control domain. Results from the analysis show that trace links are often corrected or completed after the fact and thus timely and automated constraint checking support has significant potential on reducing rework.more » « less
-
Emergency response services are critical to public safety, with 9-1-1 call-takers playing a key role in ensuring timely and effective emergency operations. To ensure call-taking performance consistency, quality assurance is implemented to evaluate and refine call-takers' skillsets. However, traditional human-led evaluations struggle with high call volumes, leading to low coverage and delayed assessments. We introduce LogiDebrief, an AI-driven framework that automates traditional 9-1-1 call debriefing by integrating Signal-Temporal Logic (STL) with Large Language Models (LLMs) for fully-covered rigorous performance evaluation. LogiDebrief formalizes call-taking requirements as logical specifications, enabling systematic assessment of 9-1-1 calls against procedural guidelines. It employs a three-step verification process: (1) contextual understanding to identify responder types, incident classifications, and critical conditions; (2) STL-based runtime checking with LLM integration to ensure compliance; and (3) automated aggregation of results into quality assurance reports. Beyond its technical contributions, LogiDebrief has demonstrated real-world impact. Successfully deployed at Metro Nashville Department of Emergency Communications, it has assisted in debriefing 1,701 real-world calls, saving 311.85 hours of active engagement. Empirical evaluation with real-world data confirms its accuracy, while a case study and extensive user study highlight its effectiveness in enhancing call-taking performance.more » « less
-
As assurance cases have grown in popularity for safety-critical systems, so too has their complexity and thus the need for methods to systematically build them. Assurance cases can grow too large and too abstract for anyone but the original builders to understand, making reuse difficult. Reuse is important because different systems might have identical or similar components, and a good solution for one system should be applicable to similar systems. Prior research has shown engineers can alleviate some of the complexity issues through modularity and identifying common patterns which are more easily understood for reuse across different systems. However, we believe these patterns are too complicated for users who lack expertise in software engineering or assurance cases. This paper suggests the concept of lower-level patterns which we call recipes. We use the safety-critical field of synthetic biology, as an example discipline to demonstrate how a recipe can be built and applied.more » « less
-
We present a principal-agent model of a one-shot, shallow, systems engineering process. The process is "one-shot" in the sense that decisions are made during a one-time step and that they are final. The term "shallow" refers to a one-layer hierarchy of the process. Specifically, we assume that the systems engineer has already decomposed the problem in subsystems and that each subsystem is assigned to a different subsystem engineer. Each subsystem engineer works independently to maximize their own expected payoff. The goal of the systems engineer is to maximize the system-level payoff by incentivizing the subsystem engineers. We restrict our attention to requirements-based system-level payoffs, i.e., the systems engineer makes a profit only if all the design requirements are met. We illustrate the model using the design of an Earth-orbiting satellite system where the systems engineer determines the optimum incentive structures and requirements for two subsystems: the propulsion subsystem and the power subsystem. The model enables the analysis of a systems engineer's decisions about optimal passed-down requirements and incentives for sub-system engineers under different levels of task difficulty and associated costs. Sample results, for the case of risk-neutral systems and subsystems engineers, show that it is not always in the best interest of the systems engineer to pass down the true requirements. As expected, the model predicts that for small to moderate task uncertainties the optimal requirements are higher than the true ones, effectively eliminating the probability of failure for the systems engineer. In contrast, the model predicts that for large task uncertainties the optimal requirements should be smaller than the true ones in order to lure the subsystem engineers into participation.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1016/j.jss.2023.111733
