- Award ID(s):
- 1814798
- PAR ID:
- 10139112
- Date Published:
- Journal Name:
- 42nd ACM/IEEE International Conference on Software Engineering (ICSE)
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
Infrastructure as code (IaC) scripts, such as Ansible scripts, are used to provision computing infrastructure at scale. Existence of bugs in IaC test scripts, such as, configuration and security bugs, can be consequential for the provisioned computing infrastructure. A characterization study of bugs in IaC test scripts is the first step to understand the quality concerns that arise during testing of IaC scripts, and also provide recommendations for practitioners on quality assurance. We conduct an empirical study with 4,831 Ansible test scripts mined from 104 open source software (OSS) repositories where we quantify bug frequency, and categorize bugs in test scripts. We further categorize testing patterns, i.e., recurring coding patterns in test scripts, which also correlate with appearance of bugs. From our empirical study, we observe 1.8% of 4,831 Ansible test scripts to include a bug, and 45.2% of the 104 repositories to contain at least one test script that includes bugs. We identify 7 categories of bugs, which includes security bugs and performance bugs that are related with metadata extraction. We also identify 3 testing patterns that correlate with appearance of bugs: 'assertion roulette’, 'local only testing’, and 'remote mystery guest‘. Based on our findings, we advocate for detection and mitigation of the 3 testing patterns as these patterns can have negative implications for troubleshooting failures, reproducible deployments of software, and provisioning of computing infrastructure.more » « less
-
Infrastructure as code (IaC) is the practice of automatically managing computing infrastructure at scale. Despite yielding multiple benefits for organizations, the practice of IaC is susceptible to quality concerns, which can lead to large-scale consequences. While researchers have studied quality concerns in IaC manifests, quality aspects of infrastructure orchestrators, i.e., tools that implement the practice of IaC, remain an under-explored area. A systematic investigation of defects in infrastructure orchestrators can help foster further research in the domain of IaC. From our empirical study with 22,445 commits mined from the Ansible infrastructure orchestrator we observe (i) a defect density of 17.9 per KLOC, (ii) 12 categories of Ansible components for which defects appear, and (iii) the ‘Module’ component to include more defects than the other 11 components. Based on our empirical study, we provide recommendations for researchers to conduct future research to enhance the quality of infrastructure orchestrators.more » « less
-
Infrastructure as Code (IaC) scripts, such as Puppet scripts, provide practitioners the opportunity to provision computing infrastructure automatically at scale. Poorly written IaC scripts impact various facets of quality (such as security and maintainability) and, in turn, may lead to serious consequences. Many of the ill-effects can be avoided or rectified easily by following recommendations derived from research and best practices gleaned from experience. While researchers have investigated methods to improve quality aspects of Puppet scripts, such research needs to be summarized and synthesized for industry practitioners. In this article, we summarize recent research in the IaC domain by discussing key quality issues, specifically security and maintainability smells, that may arise in an IaC script. We also mine open-source repositories from three organizations (Mozilla, Openstack, and Wikimedia) and report our observations on the identified smells. Furthermore, we also synthesize recommendations from the literature for software practitioners that could improve the quality of IaC scripts. Software development teams dealing with large computing infrastructure can get benefited from the actionable recommended practices. In addition, researchers in the domain may use this study to find opportunities to improve the state-of-the-art.more » « less
-
In infrastructure as code (IaC), state reconciliation is the process of querying and comparing the infrastructure state prior to changing the infrastructure. As state reconciliation is pivotal to manage IaC-based computing infrastructure at scale, defects related to state reconciliation can create large-scale consequences. A categorization of state reconciliation defects, i.e., defects related to state reconciliation, can aid in understanding the nature of state reconciliation defects. We conduct an empirical study with 5,110 state reconciliation defects where we apply qualitative analysis to categorize state reconciliation defects. From the identified defect categories, we derive heuristics to design prompts for a large language model (LLM), which in turn are used for validation of state reconciliation. From our empirical study, we identify 8 categories of state reconciliation defects, amongst which 3 have not been reported for previously-studied software systems. The most frequently occurring defect category is inventory, i.e., the category of defects that occur when managing infrastructure inventory. Using an LLM with heuristics-based paragraph style prompts, we identify 9 previously unknown state reconciliation defects of which 7 have been accepted as valid defects, and 4 have already been fixed. Based on our findings, we conclude the paper by providing a set of recommendations for researchers and practitioners.
-
null (Ed.)Context: Security smells are recurring coding patterns that are indicative of security weakness and require further inspection. As infrastructure as code (IaC) scripts, such as Ansible and Chef scripts, are used to provision cloud-based servers and systems at scale, security smells in IaC scripts could be used to enable malicious users to exploit vulnerabilities in the provisioned systems. Goal: The goal of this article is to help practitioners avoid insecure coding practices while developing infrastructure as code scripts through an empirical study of security smells in Ansible and Chef scripts. Methodology: We conduct a replication study where we apply qualitative analysis with 1,956 IaC scripts to identify security smells for IaC scripts written in two languages: Ansible and Chef. We construct a static analysis tool called Security Linter for Ansible and Chef scripts (SLAC) to automatically identify security smells in 50,323 scripts collected from 813 open source software repositories. We also submit bug reports for 1,000 randomly selected smell occurrences. Results: We identify two security smells not reported in prior work: missing default in case statement and no integrity check. By applying SLAC we identify 46,600 occurrences of security smells that include 7,849 hard-coded passwords. We observe agreement for 65 of the responded 94 bug reports, which suggests the relevance of security smells for Ansible and Chef scripts amongst practitioners. Conclusion: We observe security smells to be prevalent in Ansible and Chef scripts, similarly to that of the Puppet scripts. We recommend practitioners to rigorously inspect the presence of the identified security smells in Ansible and Chef scripts using (i) code review, and (ii) static analysis tools.more » « less