skip to main content


Title: To call, or not to call: contrasting direct and indirect branch coverage in test generation
While adequacy criteria offer an end-point for testing, they do not mandate how targets are covered. Branch Coverage may be attained through direct calls to methods, or through indirect calls between methods. Automated generation is biased towards the rapid gains offered by indirect coverage. Therefore, even with the same end-goal, humans and automation produce very different tests. Direct coverage may yield tests that are more understandable, and that detect faults missed by traditional approaches. However, the added burden for the generation framework may result in lower coverage and faults that emerge through method interactions may be missed. To compare the two approaches, we have generated test suites for both, judging efficacy against real faults. We have found that requiring direct coverage results in lower achieved coverage and likelihood of fault detection. However, both forms of Branch Coverage cover code and detect faults that the other does not. By isolating methods, Direct Branch Coverage is less constrained in the choice of input. However, traditional Branch Coverage is able to leverage method interactions to discover faults. Ultimately, both are situationally applicable within the context of a broader testing strategy.  more » « less
Award ID(s):
1657299
NSF-PAR ID:
10082003
Author(s) / Creator(s):
Date Published:
Journal Name:
SBST '18 Proceedings of the 11th International Workshop on Search-Based Software Testing
Page Range / eLocation ID:
43 to 50
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Coral reefs comprise some of the most biodiverse habitats on the planet. These ecosystems face a range of stressors, making quantifying community assemblages and potential changes vital to effective management. To understand short- and long-term changes in biodiversity and detect early warning signals of decline, new methods for quantifying biodiversity at scale are necessary. Acoustic monitoring techniques have proven useful in observing species activities and biodiversity on coral reefs through aggregate approaches (i.e. energy as a proxy). However, few studies have ground-truthed these acoustic analyses with human-based observations. In this study, we sought to expand these passive acoustic methods by investigating biological sounds and fish call rates on a healthy reef, providing a unique set of human-confirmed, labeled acoustic observations. We analyzed acoustic data from Tektite Reef, St. John, US Virgin Islands, over a 2 mo period. A subset of acoustic files was manually inspected to identify recurring biotic sounds and quantify reef activity throughout the day. We found a high variety of acoustic signals in this soundscape. General patterns of call rates across time conformed to expectations, with dusk and dawn showing important and significantly elevated peaks in soniferous fish activity. The data reflected high variability in call rates across days and lunar phases. Call rates did not correspond to sound pressure levels, suggesting that certain call types may drive crepuscular trends in sound levels while lower-level critical calls, likely key for estimating biodiversity and behavior, may be missed by gross sound level analyses. 
    more » « less
  2. Summary

    Search‐based unit test generation, if effective at fault detection, can lower the cost of testing. Such techniques rely on fitness functions to guide the search. Ultimately, such functions represent test goals that approximate—but do not ensure—fault detection. The need to rely on approximations leads to two questions—can fitness functions produce effective tests and, if so, which should be used to generate tests?To answer these questions, we have assessed the fault‐detection capabilities of unit test suites generated to satisfy eight white‐box fitness functions on 597 real faults from the Defects4J database. Our analysis has found that the strongest indicators of effectiveness are a high level of code coverage over the targeted class and high satisfaction of a criterion's obligations. Consequently, the branch coverage fitness function is the most effective. Our findings indicate that fitness functions that thoroughly explore system structure should be used as primary generation objectives—supported by secondary fitness functions that explore orthogonal, supporting scenarios. Our results also provide further evidence that future approaches to test generation should focus on attaining higher coverage of private code and better initialization and manipulation of class dependencies.

     
    more » « less
  3. After more than 30 years of development, combinatorial testing has become an essential method in the field of software testing, which has always been an active field. Through combined coverage, combinatorial testing can detect faults caused by various parameters and their interactions in a software under test. This paper aims to review the development of combinatorial testing and briefly introduce its basic applications. We classify today's combinatorial testing applications, including those of traditional industry and those of the IT industry. The research-and-application progress of combinatorial testing was investigated with respect to various fields of application, and potential application directions for the future were proposed to provide ideas its the extensive application. 
    more » « less
  4. While the existing methods for testing XACML policies have varying levels of effectiveness, none of them can reveal the majority of policy faults. The undisclosed faults may lead to unauthorized access and denial of service. This paper presents an approach to strong mutation testing of XACML policies that automatically generates tests from the mutants of a given policy. Such mutants represent the targeted faults that may appear in the policy. In this approach, we first compose the strong mutation constraints that capture the semantic difference between each mutant and its original policy. Then, we use a constraint solver to derive an access request (i.e., test). The test suite generated from all the mutants of a policy can achieve a perfect mutation score, thus uncover all hypothesized faults or demonstrate their absence. Based on the mutation-based approach, this paper further explores optimal test suite that achieves a perfect mutation score without duplicate tests. To evaluate the proposed approach, our experiments have included all the subject policies in the relevant literature and used a number of new policies. The results demonstrate that: (1) it is scalable to generate a mutation-based test suite to achieve a perfect mutation score, (2) it can be impractical to generate the optimal test suite due to the expensive removal of duplicate tests, (3) different from the results of the existing study, the modified-condition/decision coverage-based method, currently the most effective one, has low mutation scores for several policies. 
    more » « less
  5. Abstract

    Introduced species can have cascading effects on ecological communities, but indirect effects of species introductions are rarely the focus of ecological studies. For example, managed honey bees (Apis mellifera) have been widely introduced outside their native range and are increasingly dominant floral visitors. Multiple studies have documented how honey bees impact native bee communities through floral resource competition, but few have quantified how these competitive interactions indirectly affect pollination and plant reproduction. Such indirect effects are hard to detect because honey bees are themselves pollinators and may directly impact pollination through their own floral visits. The potentially huge but poorly understood impacts that non‐native honey bees have on native plant populations combined with increased pressure from beekeepers to place hives in U.S. National Parks and Forests makes exploring impacts of honey bee introductions on native plant pollination of pressing concern. In this study, we used experimental hive additions, field observations, as well as single‐visit and multiple‐visit pollination effectiveness trials across multiple years to untangle the direct and indirect impacts of increasing honey bee abundance on the pollination of an ecologically important wildflower,Camassia quamash. We found compelling evidence that honey bee introductions indirectly decrease pollination by reducing nectar and pollen availability and competitively excluding visits from more effective native bees. In contrast, the direct impact of honey bee visits on pollination was negligible, and, if anything, negative. Honey bees were ineffective pollinators, and increasing visit quantity could not compensate for inferior visit quality. Indeed, although the effect was not statistically significant, increased honey bee visits had a marginally negative impact on seed production. Thus, honey bee introductions may erode longstanding plant‐pollinator mutualisms, with negative consequences for plant reproduction. Our study calls for a more thorough understanding of the indirect effects of species introductions and more careful coordination of hive placements.

     
    more » « less