Infrastructure as code (IaC) scripts, such as Ansible scripts, are used to provision computing infrastructure at scale. Existence of bugs in IaC test scripts, such as, configuration and security bugs, can be consequential for the provisioned computing infrastructure. A characterization study of bugs in IaC test scripts is the first step to understand the quality concerns that arise during testing of IaC scripts, and also provide recommendations for practitioners on quality assurance. We conduct an empirical study with 4,831 Ansible test scripts mined from 104 open source software (OSS) repositories where we quantify bug frequency, and categorize bugs in test scripts. We further categorize testing patterns, i.e., recurring coding patterns in test scripts, which also correlate with appearance of bugs. From our empirical study, we observe 1.8% of 4,831 Ansible test scripts to include a bug, and 45.2% of the 104 repositories to contain at least one test script that includes bugs. We identify 7 categories of bugs, which includes security bugs and performance bugs that are related with metadata extraction. We also identify 3 testing patterns that correlate with appearance of bugs: 'assertion roulette’, 'local only testing’, and 'remote mystery guest‘. Based on our findings, we advocate for detection and mitigation of the 3 testing patterns as these patterns can have negative implications for troubleshooting failures, reproducible deployments of software, and provisioning of computing infrastructure.
more »
« less
Security Smells in Ansible and Chef Scripts: A Replication Study
Context: Security smells are recurring coding patterns that are indicative of security weakness and require further inspection. As infrastructure as code (IaC) scripts, such as Ansible and Chef scripts, are used to provision cloud-based servers and systems at scale, security smells in IaC scripts could be used to enable malicious users to exploit vulnerabilities in the provisioned systems. Goal: The goal of this article is to help practitioners avoid insecure coding practices while developing infrastructure as code scripts through an empirical study of security smells in Ansible and Chef scripts. Methodology: We conduct a replication study where we apply qualitative analysis with 1,956 IaC scripts to identify security smells for IaC scripts written in two languages: Ansible and Chef. We construct a static analysis tool called Security Linter for Ansible and Chef scripts (SLAC) to automatically identify security smells in 50,323 scripts collected from 813 open source software repositories. We also submit bug reports for 1,000 randomly selected smell occurrences. Results: We identify two security smells not reported in prior work: missing default in case statement and no integrity check. By applying SLAC we identify 46,600 occurrences of security smells that include 7,849 hard-coded passwords. We observe agreement for 65 of the responded 94 bug reports, which suggests the relevance of security smells for Ansible and Chef scripts amongst practitioners. Conclusion: We observe security smells to be prevalent in Ansible and Chef scripts, similarly to that of the Puppet scripts. We recommend practitioners to rigorously inspect the presence of the identified security smells in Ansible and Chef scripts using (i) code review, and (ii) static analysis tools.
more »
« less
- Award ID(s):
- 2026869
- NSF-PAR ID:
- 10273685
- Date Published:
- Journal Name:
- ACM Transactions on Software Engineering and Methodology
- Volume:
- 30
- Issue:
- 1
- ISSN:
- 1049-331X
- Page Range / eLocation ID:
- 1 to 31
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Infrastructure as Code (IaC) scripts, such as Puppet scripts, provide practitioners the opportunity to provision computing infrastructure automatically at scale. Poorly written IaC scripts impact various facets of quality (such as security and maintainability) and, in turn, may lead to serious consequences. Many of the ill-effects can be avoided or rectified easily by following recommendations derived from research and best practices gleaned from experience. While researchers have investigated methods to improve quality aspects of Puppet scripts, such research needs to be summarized and synthesized for industry practitioners. In this article, we summarize recent research in the IaC domain by discussing key quality issues, specifically security and maintainability smells, that may arise in an IaC script. We also mine open-source repositories from three organizations (Mozilla, Openstack, and Wikimedia) and report our observations on the identified smells. Furthermore, we also synthesize recommendations from the literature for software practitioners that could improve the quality of IaC scripts. Software development teams dealing with large computing infrastructure can get benefited from the actionable recommended practices. In addition, researchers in the domain may use this study to find opportunities to improve the state-of-the-art.more » « less
-
The practice of infrastructure as code (IaC) recommends automated management of computing infrastructure with application of quality assurance, such as linting and testing. To that end, researchers recently have investigated quality concerns in IaC test manifests by deriving a catalog of test smells. The relevance of the identified smells need to be quantified by obtaining feedback from practitioners. Such feedback can help the IaC community understand if smells have relevance amongst practitioners, and derive future research directions. We survey 30 practitioners to assess the relevance of three Ansible test smell categories namely, assertion roulette, local only testing, and remote mystery guest. We observe local only testing to be the most agreed upon test smell category, whereas, assertion roulette is the least agreed upon test smell category. Our findings provide a nuanced perspective of test smells for IaC, and lays the groundwork for future research.more » « less
-
Defects in infrastructure as code (IaC) scripts can have serious consequences, for example, creating large-scale system outages. A taxonomy of IaC defects can be useful for understanding the nature of defects, and identifying activities needed to fix and prevent defects in IaC scripts. The goal of this paper is to help practitioners improve the quality of infrastructure as code (IaC) scripts by developing a defect taxonomy for IaC scripts through qualitative analysis. We develop a taxonomy of IaC defects by applying qualitative analysis on 1,448 defect-related commits collected from open source software (OSS) repositories of the Openstack organization. We conduct a survey with 66 practitioners to assess if they agree with the identified defect categories included in our taxonomy. We quantify the frequency of identified defect categories by analyzing 80,425 commits collected from 291 OSS repositories spanning across 2005 to 2019. Our defect taxonomy for IaC consists of eight categories, including a category specific to IaC called idempotency (i.e., defects that lead to incorrect system provisioning when the same IaC script is executed multiple times). We observe the surveyed 66 practitioners to agree most with idempotency. The most frequent defect category is configuration data i.e., providing erroneous configuration data in IaC scripts. Our taxonomy and the quantified frequency of the defect categories may help in advancing the science of IaC script quality.more » « less
-
Despite being beneficial in automated provisioning of computing infrastructure at scale, infrastructure as code (IaC) scripts are susceptible to containing secrets, such as hard-coded passwords. A derivation of practices related to secret management for IaC can help practitioners to secure their secrets, potentially aiding them to securely develop IaC scripts. The goal of the paper is to help practitioners in secure development of infrastructure as code (IaC) scripts by identifying practices for secret management in IaC. We conduct a grey literature review with 38 Internet artifacts to identify 12 practices. We identify practices that are applicable for all IaC languages, e.g., prioritized encryption, as well as language-specific practices, such as state separation for Terraform. Our findings can be beneficial for (i) practitioners who can apply the identified practices to secure secrets in IaC development, and (ii) researchers who can investigate how the secret management process can be improved to facilitate secure development of IaC scripts.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1145/3408897
