An official website of the United States government
Summary Search‐based unit test generation, if effective at fault detection, can lower the cost of testing. Such techniques rely on fitness functions to guide the search. Ultimately, such functions represent test goals that approximate—but do not ensure—fault detection. The need to rely on approximations leads to two questions—can fitness functions produce effective tests and, if so, which should be used to generate tests?To answer these questions, we have assessed the fault‐detection capabilities of unit test suites generated to satisfy eight white‐box fitness functions on 597 real faults from the Defects4J database. Our analysis has found that the strongest indicators of effectiveness are a high level of code coverage over the targeted class and high satisfaction of a criterion's obligations. Consequently, the branch coverage fitness function is the most effective. Our findings indicate that fitness functions that thoroughly explore system structure should be used as primary generation objectives—supported by secondary fitness functions that explore orthogonal, supporting scenarios. Our results also provide further evidence that future approaches to test generation should focus on attaining higher coverage of private code and better initialization and manipulation of class dependencies.
more »
« less
To call, or not to call: contrasting direct and indirect branch coverage in test generation
While adequacy criteria offer an end-point for testing, they do not mandate how targets are covered. Branch Coverage may be attained through direct calls to methods, or through indirect calls between methods. Automated generation is biased towards the rapid gains offered by indirect coverage. Therefore, even with the same end-goal, humans and automation produce very different tests. Direct coverage may yield tests that are more understandable, and that detect faults missed by traditional approaches. However, the added burden for the generation framework may result in lower coverage and faults that emerge through method interactions may be missed. To compare the two approaches, we have generated test suites for both, judging efficacy against real faults. We have found that requiring direct coverage results in lower achieved coverage and likelihood of fault detection. However, both forms of Branch Coverage cover code and detect faults that the other does not. By isolating methods, Direct Branch Coverage is less constrained in the choice of input. However, traditional Branch Coverage is able to leverage method interactions to discover faults. Ultimately, both are situationally applicable within the context of a broader testing strategy.
more »
« less
- Award ID(s):
- 1657299
- PAR ID:
- 10082003
- Date Published:
- Journal Name:
- SBST '18 Proceedings of the 11th International Workshop on Search-Based Software Testing
- Page Range / eLocation ID:
- 43 to 50
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Coral reefs comprise some of the most biodiverse habitats on the planet. These ecosystems face a range of stressors, making quantifying community assemblages and potential changes vital to effective management. To understand short- and long-term changes in biodiversity and detect early warning signals of decline, new methods for quantifying biodiversity at scale are necessary. Acoustic monitoring techniques have proven useful in observing species activities and biodiversity on coral reefs through aggregate approaches (i.e. energy as a proxy). However, few studies have ground-truthed these acoustic analyses with human-based observations. In this study, we sought to expand these passive acoustic methods by investigating biological sounds and fish call rates on a healthy reef, providing a unique set of human-confirmed, labeled acoustic observations. We analyzed acoustic data from Tektite Reef, St. John, US Virgin Islands, over a 2 mo period. A subset of acoustic files was manually inspected to identify recurring biotic sounds and quantify reef activity throughout the day. We found a high variety of acoustic signals in this soundscape. General patterns of call rates across time conformed to expectations, with dusk and dawn showing important and significantly elevated peaks in soniferous fish activity. The data reflected high variability in call rates across days and lunar phases. Call rates did not correspond to sound pressure levels, suggesting that certain call types may drive crepuscular trends in sound levels while lower-level critical calls, likely key for estimating biodiversity and behavior, may be missed by gross sound level analyses.more » « less
-
The paper develops a methodology for the online built-in self-testing of deep neural network (DNN) accelerators to validate the correct operation with respect to their functional specifications. The DNN of interest is realized in the hardware to perform in-memory computing using non-volatile memory cells as computational units. Assuming a functional fault model, we develop methods to generate pseudorandom and structured test patterns to detect hardware faults. We also develop a test-sequencing strategy that combines these different classes of tests to achieve high fault coverage. The testing methodology is applied to a broad class of DNNs trained to classify images from the MNIST, Fashion-MNIST, and CIFAR-10 datasets. The goal is to expose hardware faults which may lead to the incorrect classification of images. We achieve an average fault coverage of 94% for these different architectures, some of which are large and complex.more » « less
-
After more than 30 years of development, combinatorial testing has become an essential method in the field of software testing, which has always been an active field. Through combined coverage, combinatorial testing can detect faults caused by various parameters and their interactions in a software under test. This paper aims to review the development of combinatorial testing and briefly introduce its basic applications. We classify today's combinatorial testing applications, including those of traditional industry and those of the IT industry. The research-and-application progress of combinatorial testing was investigated with respect to various fields of application, and potential application directions for the future were proposed to provide ideas its the extensive application.more » « less
-
Concolic testing is a scalable solution for automated generation of directed tests for validation of hardware designs. Unfortunately, concolic testing fails to cover complex corner cases such as hard-to-activate branches. In this article, we propose an incremental concolic testing technique to cover hard-to-activate branches in register-transfer level (RTL) models. We show that a complex branch condition can be viewed as a sequence of easy-to-activate events. We map the branch coverage problem to the coverage of a sequence of events. We propose an efficient algorithm to cover the sequence of events using concolic testing. Specifically, the test generated to activate the current event is used as the starting point to activate the next event in the sequence. Experimental results demonstrate that our approach can be used to generate directed tests to cover complex corner cases in RTL models while state-of-the-art methods fail to activate them.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3194718.3194719
