An official website of the United States government
A number of criteria have been proposed to judge test suite adequacy. While search-based test generation has improved greatly at criteria coverage, the produced suites are still often ineffective at detecting faults. Efficacy may be limited by the single-minded application of one criterion at a time when generating suites - a sharp contrast to human testers, who simultaneously explore multiple testing strategies. We hypothesize that automated generation can be improved by selecting and simultaneously exploring multiple criteria. To address this hypothesis, we have generated multi-criteria test suites, measuring efficacy against the Defects4J fault database. We have found that multi-criteria suites can be up to 31.15% more effective at detecting complex, real-world faults than suites generated to satisfy a single criterion and 70.17% more effective than the default combination of all eight criteria. Given a fixed search budget, we recommend pairing a criterion focused on structural exploration - such as Branch Coverage - with targeted supplemental strategies aimed at the type of faults expected from the system under test. Our findings offer lessons to consider when selecting such combinations.
more »
« less
Node Monitoring as a Fault Detection Countermeasure against Information Leakage within a RISC-V Microprocessor
Advanced, superscalar microprocessors (μP) are highly susceptible to wear-out failures because of their highly complex, densely packed circuit structure and extreme operational frequencies. Although many types of fault detection and mitigation strategies have been proposed, none have addressed the specific problem of detecting faults that lead to information leakage events on I/O channels of the μP. Information leakage can be defined very generally as any type of output that the executing program did not intend to produce. In this work, we restrict this definition to output that represents a security concern, and in particular, to the leakage of plaintext or encryption keys, and propose a counter-based countermeasure to detect faults that cause this type of leakage event. Fault injection (FI) experiments are carried out on two RISC-V microprocessors emulated as soft cores on a Xilinx multi-processor System-on-chip (MPSoC) FPGA. The μP designs are instrumented with a set of counters that records the number of transitions that occur on internal nodes. The transition counts are collected from all internal nodes under both fault-free and faulty conditions, and are analyzed to determine which counters provide the highest fault coverage and lowest latency for detecting leakage faults. We show that complete coverage of all leakage faults is possible using only a single counter strategically placed within the branch compare logic of the μPs.
more »
« less
- Award ID(s):
- 1814804
- PAR ID:
- 10392151
- Date Published:
- Journal Name:
- Cryptography
- Volume:
- 6
- Issue:
- 3
- ISSN:
- 2410-387X
- Page Range / eLocation ID:
- 38
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Functional broadside tests were developed to avoid overtesting of delay faults. The tests achieve this goal by creating functional operation conditions during their functional capture cycles. To increase the achievable fault coverage, close-to-functional scan-based tests are allowed to deviate from functional operation conditions. This article suggests that a more comprehensive functional broadside test set can be obtained by replacing target faults that cannot be detected with faults that have similar (but not identical) detection conditions. A more comprehensive functional broadside test set has the advantage that it still maintains functional operation conditions. It covers the test holes created when target faults cannot be detected by detecting similar faults. The article considers the case where the target faults are transition faults. When a standard transition fault, with an extra delay of a single clock cycle, cannot be detected, an unspecified transition fault is used instead. An unspecified transition fault captures the behaviors of transition faults with different extra delays. When this fault cannot be detected, a stuck-at fault is used instead. A stuck-at fault has some of the detection conditions of a transition fault. Multicycle functional broadside tests are used to allow unspecified transition faults to be detected. As a by-product, test compaction also occurs. The structure of the test generation procedure accommodates the complexity of producing functional broadside tests by considering the target as well as replacement faults together. Experimental results for benchmark circuits demonstrate the fault coverage improvements achieved, and the effect on the number of tests.more » « less
-
null (Ed.)This paper shows how to use bounded-time recovery (BTR) to defend distributed systems against non-crash faults and attacks. Unlike many existing fault-tolerance techniques, BTR does not attempt to completely mask all symptoms of a fault; instead, it ensures that the system returns to the correct behavior within a bounded amount of time. This weaker guarantee is sufficient, e.g., for many cyber-physical systems, where physical properties - such as inertia and thermal capacity - prevent quick state changes and thus limit the damage that can result from a brief period of undefined behavior. We present an algorithm called REBOUND that can provide BTR for the Byzantine fault model. REBOUND works by detecting faults and then reconfiguring the system to exclude the faulty nodes. This supports very fine-grained responses to faults: for instance, the system can move or replace existing tasks, or drop less critical tasks entirely to conserve resources. REBOUND can take useful actions even when a majority of the nodes is compromised, and it requires less redundancy than full fault-tolerance.more » « less
-
null (Ed.)The use of multicycle tests, with several functional capture cycles between scan operations, contributes significantly to the ability to compact a test set. Multicycle tests have the added benefit that they can contribute to the detection of defects with complex behaviors that are not detected by single-cycle or two-cycle tests. To ensure that this benefit is materialized when test compaction is applied to transition faults, this article suggests to incorporate into the test compaction procedure an additional fault model whose fault coverage increases when multicycle tests are used. To ensure that the computational complexity of test compaction is not increased by a fault model with a large number of faults, or faults with complex behaviors, the added fault model is required to have the same characteristics as the transition fault model. A type of transition fault called unspecified transition fault satisfies these requirements. The article describes a test compaction procedure for transition faults that incorporates unspecified transition faults, and presents experimental results for benchmark circuits to demonstrate the levels of test compaction and fault coverage that can be achieved.more » « less
-
Ghosh, Sudipto; Troubitsyna, Elena; Chen, Zhenyu (Ed.)When we quantify the effectiveness of a test suite by its mutation coverage, we are in fact equating test suite effectiveness with fault detection: to the extent that mutations are faithful proxies of actual faults, it is sensible to consider that the effectiveness of a test suite to kill mutants reflects its ability to detect faults. But there is another way to measure the effectiveness of a test suite: by its ability to expose the failures of an incorrect program. The relationship between failures and faults is tenuous at best: a fault is the adjudged or hypothesized cause of a failure. The same failure may be attributed to more than one fault. This raises the question: what is the relationship between detecting faults and exposing failures. In this paper, we discuss an empirical experiment in which we explore this relationship.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/cryptography6030038
