The paradigm shift of deploying applications to
the cloud has introduced both opportunities and challenges.
Although clouds use elasticity to scale resource usage at runtime
to help meet an application’s performance requirements,
developers are still challenged by unpredictable performance,
little control of execution environment, and differences among
cloud service providers, all while being charged for their cloud
usages. Application performance stability is particularly affected
by multi-tenancy in which the hardware is shared among varying
applications and virtual machines. Developers porting their
applications need to meet performance requirements, but testing
on the cloud under the effects of performance uncertainty is
difficult and expensive, due to high cloud usage costs.
This paper presents a first approach to testing an application
with typical inputs for how its performance will be affected by
performance uncertainty, without incurring undue costs of bruteforce
testing in the cloud. We specify cloud uncertainty testing
criteria, design a test-based strategy to characterize the blackbox
cloud’s performance distributions using these testing criteria,
and support execution of tests to characterize the resource usage
and cloud baseline performance of the application to be deployed.
Importantly, we developed a smart test oracle that estimates the
application’s performance with certain confidence levels using
the above characterization test results and determines whether
it will meet its performance requirements. We evaluated our
testing approach on both the Chameleon cloud and Amazon web
services; results indicate that this testing strategy shows promise
as a cost-effective approach to test for performance effects of
cloud uncertainty when porting an application to the cloud.
more »
« less
Ensuring the Observability of Structural Test Obligations
Test adequacy criteria are widely used to guide test creation. However, many of these criteria are sensitive to statement structure or the choice of test oracle. This is because such criteria ensure that execution reaches the element of interest, but impose no constraints on the execution path after this point. We are not guaranteed to observe a failure just because a fault is triggered. To address this issue, we have proposed the concept of observability—an extension to coverage criteria based on Boolean expressions that combines the obligations of a host criterion with an additional path condition that increases the likelihood that a fault encountered will propagate to a monitored variable.
Our study, conducted over five industrial systems and an additional forty open-source systems, has revealed that adding observability tends to improve efficacy over satisfaction of the traditional criteria, with average improvements of 125.98% in mutation detection with the common output-only test oracle and per-model improvements of up to 1760.52%. Ultimately, there is merit to our hypothesis—observability reduces sensitivity to the choice of oracle and to the program structure.
more »
« less
- Award ID(s):
- 1657299
- NSF-PAR ID:
- 10082000
- Date Published:
- Journal Name:
- IEEE Transactions on Software Engineering
- ISSN:
- 0098-5589
- Page Range / eLocation ID:
- 1 to 1
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Biological small-angle X-ray solution scattering (BioSAXS) is now widely used to gain information on biomolecules in the solution state. Often, however, it is not obvious in advance whether a particular sample will scatter strongly enough to give useful data to draw conclusions under practically achievable solution conditions. Conformational changes that appear to be large may not always produce scattering curves that are distinguishable from each other at realistic concentrations and exposure times. Emerging technologies such as time-resolved SAXS (TR-SAXS) pose additional challenges owing to small beams and short sample path lengths. Beamline optics vary in brilliance and degree of background scatter, and major upgrades and improvements to sources promise to expand the reach of these methods. Computations are developed to estimate BioSAXS sample intensity at a more detailed level than previous approaches, taking into account flux, energy, sample thickness, window material, instrumental background, detector efficiency, solution conditions and other parameters. The results are validated with calibrated experiments using standard proteins on four different beamlines with various fluxes, energies and configurations. The ability of BioSAXS to statistically distinguish a variety of conformational movements under continuous-flow time-resolved conditions is then computed on a set of matched structure pairs drawn from the Database of Macromolecular Motions (http://molmovdb.org). The feasibility of experiments is ranked according to sample consumption, a quantity that varies by over two orders of magnitude for the set of structures. In addition to photon flux, the calculations suggest that window scattering and choice of wavelength are also important factors given the short sample path lengths common in such setups.more » « less
-
more » « less
Summary Researchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and various platforms (e.g., desktop, web, or mobile applications). The generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit‐testing of libraries versus system‐testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved by systematically executing large‐scale evaluations of different generators. However, executing such empirical evaluations is not trivial and requires substantial effort to select appropriate benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this Software Note, we present our
JUnit Generation Benchmarking Infrastructure (JUGE ) supporting generators (search‐based, random‐based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall benchmarking effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, several editions of a unit testing tool competition, co‐located with the Search‐Based Software Testing Workshop, have taken place whereJUGE was used and evolved. As a result, an increasing amount of tools (over 10) from academia and industry have been evaluated onJUGE , matured over the years, and allowed the identification of future research directions. Based on the experience gained from the competitions, we discuss the expected impact ofJUGE in improving the knowledge transfer on tools and approaches for test generation between academia and industry. Indeed, theJUGE infrastructure demonstrated an implementation design that is flexible enough to enable the integration of additional unit test generation tools, which is practical for developers and allows researchers to experiment with new and advanced unit testing tools and approaches. -
Real-time systems are susceptible to adversarial factors such as faults and attacks, leading to severe consequences. This paper presents an optimal checkpoint scheme to bolster fault resilience in real-time systems, addressing both logical consistency and timing correctness. First, we partition message-passing processes into a directed acyclic graph (DAG) based on their dependencies, ensuring checkpoint logical consistency. Then, we identify the DAG’s critical path, representing the longest sequential path, and analyze the optimal checkpoint strategy along this path to minimize overall execution time, including checkpointing overhead. Upon fault detection, the system rolls back to the nearest valid checkpoints for recovery. Our algorithm derives the optimal checkpoint count and intervals, and we evaluate its performance through extensive simulations and a case study. Results show a 99.97% and 67.86% reduction in execution time compared to checkpoint-free systems in simulations and the case study, respectively. Moreover, our proposed strategy outperforms prior work and baseline methods, increasing deadline achievement rates by 31.41% and 2.92% for small-scale tasks and 78.53% and 4.15% for large-scale tasks.more » « less
-
With the growing effort to reduce power consumption in machines, fault tolerance becomes more of a concern. This holds particularly for large-scale computing, where execution failures due to soft faults waste excessive time and resources. These large-scale applications are normally parallel in nature and rely on control structures tailored specifically for parallel computing, such as locks and barriers. While there are many studies on resilient software, to our knowledge none of them focus on protecting these parallel control structures. In this work, we present a method of ensuring the correct operation of both locks and barriers in parallel applications. Our method tracks the memory locations used within parallel sections and detects a violation of the control structures. Upon detecting any violation, the violating thread is rolled back to the beginning of the structure and reattempts it, similar to rollback mechanisms in transactional memory systems. We test the method on representative samples of the BigDataBench kernels and find it exhibits a mean error reduction of 93.6% for basic mutex locks and barriers with a mean 6.55% execution time overhead at 64 threads. Additionally, we provide a comparison to transactional memory methods and demonstrate up to a mean 57.5% execution time overhead reduction.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1109/TSE.2018.2869146
