While the existing methods for testing XACML policies have varying
levels of effectiveness, none of them can reveal the majority of
policy faults. The undisclosed faults may lead to unauthorized access
and denial of service. This paper presents an approach to strong
mutation testing of XACML policies that automatically generates
tests from the mutants of a given policy. Such mutants represent the
targeted faults that may appear in the policy. In this approach, we
first compose the strong mutation constraints that capture the semantic
difference between each mutant and its original policy. Then,
we use a constraint solver to derive an access request (i.e., test). The
test suite generated from all the mutants of a policy can achieve
a perfect mutation score, thus uncover all hypothesized faults or
demonstrate their absence. Based on the mutation-based approach,
this paper further explores optimal test suite that achieves a perfect
mutation score without duplicate tests. To evaluate the proposed approach,
our experiments have included all the subject policies in the
relevant literature and used a number of new policies. The results
demonstrate that: (1) it is scalable to generate a mutation-based test
suite to achieve a perfect mutation score, (2) it can be impractical
to generate the optimal test suite due to the expensive removal of
duplicate tests, (3) different from the results of the existing study,
the modified-condition/decision coverage-based method, currently
the most effective one, has low mutation scores for several policies.
more »
« less
Mutation Analysis of NGAC Policies
The NGAC (Next Generation Access Control) standard for attribute-based access control (ABAC) allows for run-time changes of the permission and prohibition configurations through administrative obligations triggered by access events. It makes access control more fine-grained and dynamic. However, it raises challenges for assuring the correctness of NGAC policies. As policy testing is an important technique for quality assurance, this paper presents an approach to mutation analysis of NGAC policies. It can evaluate the effectiveness of a testing method and reveal potential faults in an inadequately tested policy. The mutation analysis covers various types of potential faults in the assignments, associations, prohibitions, and obligations of NGAC policies. This paper also proposes an incremental testing approach that first validates the initial configuration of a policy and then the policy as a whole. It helps determine whether faults appear in the configuration or the obligations. To evaluate the work, we have developed four working policies and their test suites based on the current NGAC reference implementation. The empirical studies show that the mutation analysis can shed light on the strengths and weaknesses of the test suites. They also demonstrate the need for developing more cost-effective testing methods.
more »
« less
- Award ID(s):
- 1954327
- PAR ID:
- 10376451
- Date Published:
- Journal Name:
- Proc. of the 26h ACM Symposium on Access Control Models and Technologies
- Page Range / eLocation ID:
- 71 to 82
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Policy testing is an important means for quality assurance of access control policies. Experimental studies on the testing methods of XACML policies have shown their varying levels of effectiveness. However, there is a lack of explanation for why they are unable to detect certain types of faults. It is unclear what is essential to the fault detection capability. To address this issue, we propose a theory on policy testing by formalizing the fault detection conditions with respect to a comprehensive fault model of XACML policies. The detection condition of a policy fault, composed of the reachability, necessity, and propagation constraints, is sufficient and necessary for revealing the fault. The formalized fault detection conditions can qualify the inherent strengths and limitations of testing methods. We have applied the formalization to the qualitative evaluations of five testing methods for the current version of the XACML standard. The results show that, for each method, there are certain types of faults that can always or never be revealed, while the detection of other faults may depend on the particular policy structure.more » « less
-
Greybox fuzzing and mutation testing are two popular but mostly independent fields of software testing research that have so far had limited overlap. Greybox fuzzing, generally geared towards searching for new bugs, predominantly uses code coverage for selecting inputs to save. Mutation testing is primarily used as a stronger alternative to code coverage in assessing the quality of regression tests; the idea is to evaluate tests for their ability to identify artificially injected faults in the target program. But what if we wanted to use greybox fuzzing to synthesize high-quality regression tests? In this paper, we develop and evaluate Mu2, a Java-based framework for incorporating mutation analysis in the greybox fuzzing loop, with the goal of producing a test-input corpus with a high mutation score. Mu2 makes use of a differential oracle for identifying inputs that exercise interesting program behavior without causing crashes. This paper describes several dynamic optimizations implemented in Mu2 to overcome the high cost of performing mutation analysis with every fuzzer-generated input. These optimizations introduce trade-offs in fuzzing throughput and mutation killing ability, which we evaluate empirically on five real-world Java benchmarks. Overall, variants of Mu2 are able to synthesize test-input corpora with a higher mutation score than state-of-the-art Java fuzzer Zest.more » « less
-
Code coverage is the most widely adopted criteria for measuring test effectiveness in software quality assurance. The performance of coverage criteria (in indicating test suites' effectiveness) has been widely studied in prior work. Most of the studies use randomly constructed pseudo test suites to facilitate data collection for correlation analysis, yet no previous work has systematically studied whether pseudo test suites would lead to inflated correlation results. This paper focuses on the potentially wide-spread threat with a study over 123 real-world Java projects. Following the typical experimental process of studying coverage criteria, we investigate the correlation between statement/assertion coverage and mutation score using both pseudo and original test suites. Except for direct correlation analysis, we control the number of assertions and the test suite size to conduct partial correlation analysis. The results reveal that 1) the correlation (between coverage criteria and mutation score) derived from pseudo test suites is much higher than from original test suites (from 0.21 to 0.39 higher in Kendall value); 2) contrary to previously reported, statement coverage has a stronger correlation with mutation score than assertion coverage.more » « less
-
The physical design of a robot and the policy that controls its motion are inherently coupled, and should be determined according to the task and environment. In an increasing number of applications, data-driven and learning-based approaches, such as deep reinforcement learning, have proven effective at designing control policies. For most tasks, the only way to evaluate a physical design with respect to such control policies is empirical---i.e., by picking a design and training a control policy for it. Since training these policies is time-consuming, it is computationally infeasible to train separate policies for all possible designs as a means to identify the best one. In this work, we address this limitation by introducing a method that jointly optimizes over the physical design and control network. Our approach maintains a distribution over designs and uses reinforcement learning to optimize a control policy to maximize expected reward over the design distribution. We give the controller access to design parameters to allow it to tailor its policy to each design in the distribution. Throughout training, we shift the distribution towards higher-performing designs, eventually converging to a design and control policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel designs and walking gaits, outperforming baselines across different settings.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3450569.3463563
