skip to main content

Title: How does combinatorial testing perform in the real world: an empirical study
Studies have shown that combinatorial testing (CT) can be effective for detecting faults in software systems. By focusing on the interactions between different factors of a system, CT shows its potential for detecting faults, especially those that can be revealed only by the specific combinations of values of multiple factors (multi-factor faults). However, is CT practical enough to be applied in the industry? Can it be more effective than other industry-favored techniques? Are there any challenges when applying CT in practice? These research questions remain in the context of industrial settings. In this paper, we present an empirical study of CT on five industrial systems with real faults. The details of the input space model (ISM) construction, such as factor identification and value assignment, are included. We compared the faults detected by CT with those detected by the inhouse testing teams using other methods, and the results suggest that despite some challenges, CT is an effective technique to detect real faults, especially multi-factor faults, of software systems in industrial settings. Observations and lessons learned are provided to further improve the fault detection effectiveness and overcome various challenges.
Authors:
Award ID(s):
1822137
Publication Date:
NSF-PAR ID:
10194693
Journal Name:
Empirical software engineering
Volume:
25
Page Range or eLocation-ID:
2661-2693
ISSN:
1382-3256
Sponsoring Org:
National Science Foundation
More Like this
  1. Context: Addressing women's under-representation in the soft-ware industry, a widely recognized concern, requires attracting as well as retaining more women. Hearing from women practitioners, particularly those positioned in multi-cultural settings, about their challenges and and adopting their lived experienced solutions can support the design of programs to resolve the under-representation issue. Goal: We investigated the challenges women face in global software development teams, particularly what motivates women to leave their company; how those challenges might break down according to demographics; and strategies to mitigate the identified challenges. Method: To achieve this goal, we conducted an ex-ploratory case study in Ericsson,more »a global technology company. We surveyed 94 women and employed mixed-methods to analyze the data. Results: Our findings reveal that women face socio-cultural challenges, including work-life balance issues, benevolent and hos-tile sexism, lack of recognition and peer parity, impostor syndrome, glass ceiling bias effects, the prove-it-again phenomenon, and the maternal wall. The participants of our research provided different suggestions to address/mitigate the reported challenges, including sabbatical policies, flexibility of location and time, parenthood support, soft skills training for managers, equality of payment and opportunities between genders, mentoring and role models to sup-port career growth, directives to hire more women, inclusive groups and events, women's empowerment, and recognition for women's success. The framework of challenges and suggestions can inspire further initiatives both in academia and industry to onboard and retain women. Women represent less than 24% of employees in software development industry and experience various types of prejudice and bias. Even in companies that care about Diversity & Inclusion, “untying the mooring ropes” of socio-cultural problems is hard. Hearing from women, especially those working in a multi-cultural organization, about their challenges and adopting their suggestions can be vital to design programs and resolve the under-representation issue. In this work we work closely with a large software development or-ganization which invests and believes in diversity and inclusion. We listened to women and the challenges they face in global soft-ware development teams of this company and what these women suggest reduce the problems and increase retention. Our research showed that women face work-life balance issues and encounter invisible barriers that prevent them from rising to top positions. They also suffer micro-aggression and sexism, need to show com-petence constantly, be supervised in essential tasks, and receive less work after becoming mothers. Moreover, women miss having more female colleagues, lack self-confidence and recognition. The women from the company suggested sabbatical policies, the flexibil-ity of location and time, parenthood support, soft skills training for managers, equality of opportunities, role models to support career growth, directives to hire more women, support groups, and more interaction between women, inclusive groups and events, women's empowerment by publishing their success stories in media and recognizing their achievements. Our results had been shared with the company Human Resources department and management and they considered the diagnosis helpful and will work on actions to mitigate the challenges that women still perceive.« less
  2. A number of criteria have been proposed to judge test suite adequacy. While search-based test generation has improved greatly at criteria coverage, the produced suites are still often ineffective at detecting faults. Efficacy may be limited by the single-minded application of one criterion at a time when generating suites - a sharp contrast to human testers, who simultaneously explore multiple testing strategies. We hypothesize that automated generation can be improved by selecting and simultaneously exploring multiple criteria. To address this hypothesis, we have generated multi-criteria test suites, measuring efficacy against the Defects4J fault database. We have found that multi-criteria suitesmore »can be up to 31.15% more effective at detecting complex, real-world faults than suites generated to satisfy a single criterion and 70.17% more effective than the default combination of all eight criteria. Given a fixed search budget, we recommend pairing a criterion focused on structural exploration - such as Branch Coverage - with targeted supplemental strategies aimed at the type of faults expected from the system under test. Our findings offer lessons to consider when selecting such combinations.« less
  3. Abstract A substantial body of research now exists demonstrating sensitivities of marine organisms to ocean acidification (OA) in laboratory settings. However, corresponding in situ observations of marine species or ecosystem changes that can be unequivocally attributed to anthropogenic OA are limited. Challenges remain in detecting and attributing OA effects in nature, in part because multiple environmental changes are co-occurring with OA, all of which have the potential to influence marine ecosystem responses. Furthermore, the change in ocean pH since the industrial revolution is small relative to the natural variability within many systems, making it difficult to detect, and in somemore »cases, has yet to cross physiological thresholds. The small number of studies that clearly document OA impacts in nature cannot be interpreted as a lack of larger-scale attributable impacts at the present time or in the future but highlights the need for innovative research approaches and analyses. We summarize the general findings in four relatively well-studied marine groups (seagrasses, pteropods, oysters, and coral reefs) and integrate overarching themes to highlight the challenges involved in detecting and attributing the effects of OA in natural environments. We then discuss four potential strategies to better evaluate and attribute OA impacts on species and ecosystems. First, we highlight the need for work quantifying the anthropogenic input of CO2 in coastal and open-ocean waters to understand how this increase in CO2 interacts with other physical and chemical factors to drive organismal conditions. Second, understanding OA-induced changes in population-level demography, potentially increased sensitivities in certain life stages, and how these effects scale to ecosystem-level processes (e.g. community metabolism) will improve our ability to attribute impacts to OA among co-varying parameters. Third, there is a great need to understand the potential modulation of OA impacts through the interplay of ecology and evolution (eco–evo dynamics). Lastly, further research efforts designed to detect, quantify, and project the effects of OA on marine organisms and ecosystems utilizing a comparative approach with long-term data sets will also provide critical information for informing the management of marine ecosystems.« less
  4. With increasing demands for high performance in structural systems, Smart Structures Technologies (SST), which includes advanced sensing, modern control, smart materials, optimization and novel testing, is receiving considerable attention as it has the potential to transform many fields in engineering, including civil, mechanical, aerospace, and geotechnical engineering. Currently, there is a significant gap between the engineering science with fundamental research in academia and engineering practice with potential application in the industry. To respond to this challenge, San Francisco State University and the University of South Carolina will collaborate with industrial partners to establish a Research Experiences for Undergraduates (REU) Sitemore »program, focusing on academia-industry collaborations in SST. This REU program will train undergraduate students to serve as the catalysts to facilitate the research infusion between academic and industrial partners. This student-driven joint venture between academia and industry will establish a virtuous circle for knowledge exchange and contribute to advancing both fundamental research and implementation of SST. The program will feature: formal training, workshops, and supplemental activities in the conduct of research in academia and industry; innovative research experience through engagement in projects with scientific and practical merits in both academic and industrial environments; experience in conducting laboratory experiments; and opportunities to present the research outcomes to the broader community at professional settings. This REU program will provide engineering undergraduate students a unique research experience in both academic and industrial settings through cooperative research projects. Experiencing research in both worlds is expected to help students transition from a relatively dependent status to an independent status as their competence level increases. The joint efforts among two institutions and industry partners provide the project team with extensive access to valuable resources, such as expertise to offer a wider-range of informative training workshops, advanced equipment, valuable data sets, experienced undergraduate mentors, and professional connections, that will facilitate a meaningful REU experience. Recruitment of participants will target 20 collaborating minority and primarily undergraduate institutions (15 of them are Hispanic-Serving Institutions, HSI) with limited science, technology, engineering, and mathematics (STEM) research capabilities. The model developed through this program may help to exemplify the establishment of a sustainable collaboration model between academia and industry that helps address the nation's need for mature, independent, informed, and globally competitive STEM professionals and is adapted to other disciplines. In this poster, the details of the program will be described. The challenges and lesson-learned on the collaboration between the two participating universities, communications with industrial partners, recruitment of the students, set up of the evaluation plans, and development of the program will be discussed.« less
  5. With increasing demands for high performance in structural systems, Smart Structures Technologies (SST), which includes advanced sensing, modern control, smart materials, optimization and novel testing, is receiving considerable attention as it has the potential to transform many fields in engineering, including civil, mechanical, aerospace, and geotechnical engineering. Currently, there is a significant gap between the engineering science with fundamental research in academia and engineering practice with potential application in the industry. To respond to this challenge, San Francisco State University and the University of South Carolina will collaborate with industrial partners to establish a Research Experiences for Undergraduates (REU) Sitemore »program, focusing on academia-industry collaborations in SST. This REU program will train undergraduate students to serve as the catalysts to facilitate the research infusion between academic and industrial partners. This student-driven joint venture between academia and industry will establish a virtuous circle for knowledge exchange and contribute to advancing both fundamental research and implementation of SST. The program will feature: formal training, workshops, and supplemental activities in the conduct of research in academia and industry; innovative research experience through engagement in projects with scientific and practical merits in both academic and industrial environments; experience in conducting laboratory experiments; and opportunities to present the research outcomes to the broader community at professional settings. This REU program will provide engineering undergraduate students a unique research experience in both academic and industrial settings through cooperative research projects. Experiencing research in both worlds is expected to help students transition from a relatively dependent status to an independent status as their competence level increases. The joint efforts among two institutions and industry partners provide the project team with extensive access to valuable resources, such as expertise to offer a wider-range of informative training workshops, advanced equipment, valuable data sets, experienced undergraduate mentors, and professional connections, that will facilitate a meaningful REU experience. Recruitment of participants will target 20 collaborating minority and primarily undergraduate institutions (15 of them are Hispanic-Serving Institutions, HSI) with limited science, technology, engineering, and mathematics (STEM) research capabilities. The model developed through this program may help to exemplify the establishment of a sustainable collaboration model between academia and industry that helps address the nation's need for mature, independent, informed, and globally competitive STEM professionals and is adapted to other disciplines. In this poster, the details of the program will be described. The challenges and lesson-learned on the collaboration between the two participating universities, communications with industrial partners, recruitment of the students, set up of the evaluation plans, and development of the program will be discussed.« less