An official website of the United States government
To teach students software engineering skills, instructors must be able to explain what quality looks like and be able to assess it. Operationally, this means that we need comprehensive rubrics for assessing artifacts such as commits, issues, pull requests and code reviews. For software engineering education research, such rubrics are also needed for assessing the effectiveness of various interventions. Ideally, these rubrics would be rooted in some ground truth, rather than reflecting the subjective opinions of individual instructors; this is particularly important if rubrics are used in empirical studies. In this paper, we seek to establish a ``ground truth'' for the assessment of commits by using the Delphi process, an empirical method for establishing expert consensus around a complex question. We assembled a diverse panel of 16 experts in software engineering with voices from both industry and academia. By applying the Delphi process through four rounds of questions to our panel, we derived a set of rubric items for commits that represent the consensus view of the expert panel. Our contribution is both the rubric items themselves and a roadmap by which future researchers can arrive at similar rubrics for other software engineering artifacts.
more »
« less
Quality Assessment for Large-Scale Industrial Software Systems: Experience Report at Alibaba
To assure high software quality for large-scale industrial software systems, traditional approaches of software quality assurance, such as software testing and performance engineering, have been widely used within Alibaba, the world's largest retailer, and one of the largest Internet companies in the world. However, there still exists a high demand for software quality assessment to achieve high sustainability of business growth and engineering culture in Alibaba. To address this issue, we develop an industrial solution for software quality assessment by following the GQM paradigm in an industrial setting. Moreover, we integrate multiple assessment methods into our solution, ranging from metric selection to rating aggregation. Our solution has been implemented, deployed, and adopted at Alibaba: (1) used by Alibaba's Business Platform Unit to continually monitor the quality for 60+ core software systems; (2) used by Alibaba's R&D Efficiency Unit to support group-wide quality-aware code search and automatic code inspection. This paper presents our proposed industrial solution, including its techniques and industrial adoption, along with the lessons learned during the development and deployment of our solution.
more »
« less
- Award ID(s):
- 1816615
- PAR ID:
- 10190940
- Date Published:
- Journal Name:
- Quality Assessment for Large-Scale Industrial Software Systems: Experience Report at Alibaba
- Page Range / eLocation ID:
- 142 to 149
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
To teach students software engineering skills, instructors must be able to explain what quality looks like and be able to assess it. Operationally, this means that we need comprehensive rubrics for assessing artifacts such as commits, issues, pull requests and code reviews. For software engineering education research, such rubrics are also needed for assessing the effectiveness of various interventions. Ideally, these rubrics would be rooted in some ground truth, rather than reflecting the subjective opinions of individual instructors; this is particularly important if rubrics are used in empirical studies. In this paper, we seek to establish a ``ground truth'' for the assessment of commits by using the Delphi process, an empirical method for establishing expert consensus around a complex question. We assembled a diverse panel of 16 experts in software engineering with voices from both industry and academia. By applying the Delphi process through four rounds of questions to our panel, we derived a set of rubric items for commits that represent the consensus view of the expert panel. Our contribution is both the rubric items themselves and a roadmap by which future researchers can arrive at similar rubrics for other software engineering artifacts.more » « less
-
To teach students software engineering skills, instructors must be able to explain what quality looks like and be able to assess it. Operationally, this means that we need comprehensive rubrics for assessing artifacts such as commits, issues, pull requests and code reviews. For software engineering education research, such rubrics are also needed for assessing the effectiveness of various interventions. Ideally, these rubrics would be rooted in some ground truth, rather than reflecting the subjective opinions of individual instructors; this is particularly important if rubrics are used in empirical studies. In this paper, we seek to establish a ``ground truth'' for the assessment of commits by using the Delphi process, an empirical method for establishing expert consensus around a complex question. We assembled a diverse panel of 16 experts in software engineering with voices from both industry and academia. By applying the Delphi process through four rounds of questions to our panel, we derived a set of rubric items for commits that represent the consensus view of the expert panel. Our contribution is both the rubric items themselves and a roadmap by which future researchers can arrive at similar rubrics for other software engineering artifacts.more » « less
-
null (Ed.)Unit testing focuses on verifying the functions of individual units of a software system. It is challenging due to the high inter-dependencies among software units. Developers address this by mocking-replacing the dependency by a "faked" object. Despite the existence of powerful, dedicated mocking frameworks, developers often turn to a "hand-rolled" approach-inheritance. That is, they create a subclass of the dependent class and mock its behavior through method overriding. However, this requires tedious implementation and compromises the design quality of unit tests. This work contributes a fully automated refactoring framework to identify and replace the usage of inheritance by using Mockito-a well received mocking framework. Our approach is built upon the empirical experience from five open source projects that use inheritance for mocking. We evaluate our approach on four other projects. Results show that our framework is efficient, generally applicable to new datasets, mostly preserves test case behaviors in detecting defects (in the form of mutants), and decouples test code from production code. The qualitative evaluation by experienced developers suggests that the auto-refactoring solutions generated by our framework improve the quality of the unit test cases in various aspects, such as making test conditions more explicit, as well as improved cohesion, readability, understandability, and maintainability with test cases.more » « less
-
Software documentation supports a broad set of software maintenance tasks; however, creating and maintaining high-quality, multi-level software documentation can be incredibly time-consuming and therefore many code bases suffer from a lack of adequate documentation. We address this problem through presenting HGEN, a fully automated pipeline that leverages LLMs to transform source code through a series of six stages into a well-organized hierarchy of formatted documents. We evaluate HGEN both quantitatively and qualitatively. First, we use it to generate documentation for three diverse projects, and engage key developers in comparing the quality of the generated documentation against their own previously produced manually-crafted documentation. We then pilot HGEN in nine different industrial projects using diverse datasets provided by each project. We collect feedback from project stakeholders, and analyze it using an inductive approach to identify recurring themes. Results show that HGEN produces artifact hierarchies similar in quality to manually constructed documentation, with much higher coverage of the core concepts than the baseline approach. Stakeholder feedback highlights HGEN's commercial impact potential as a tool for accelerating code comprehension and maintenance tasks.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- https://doi.org/10.1109/APSEC48747.2019.00028
