More Like this

1. CodeDJ: Reproducible Queries over Large-Scale Software Repositories

   Maj, P; Siek, K; Kovalenko, A; Vitek, Jan (July 2021, European Conference on Object Oriented Programming)

   Moller, A; Sridharan, M (Ed.)

   Analyzing massive code bases is a staple of modern software engineering research – a welcome side-effect of the advent of large-scale software repositories such as GitHub. Selecting which projects one should analyze is a labor-intensive process, and a process that can lead to biased results if the selection is not representative of the population of interest. One issue faced by researchers is that the interface exposed by software repositories only allows the most basic of queries. CodeDJ is an infrastructure for querying repositories composed of a persistent datastore, constantly updated with data acquired from GitHub, and an in-memory database with a Rust query interface. CodeDJ supports reproducibility, historical queries are answered deterministically using past states of the datastore; thus researchers can reproduce published results. To illustrate the benefits of CodeDJ, we identify biases in the data of a published study and, by repeating the analysis with new data, we demonstrate that the study’s conclusions were sensitive to the choice of projects. 

   more » « less
2. TiQi: A natural language interface for querying software project data

   https://doi.org/10.1109/ASE.2017.8115714  

   Lin, Jinfeng; Liu, Yalin; Guo, Jin; Cleland-Huang, Jane; Goss, William; Liu, Wenchuang; Lohar, Sugandha; Monaikul, Natawut; Rasin, Alexander (October 2017, Automated Software Engineering (ASE), 2017 32nd IEEE/ACM International Conference on)

   Software projects produce large quantities of data such as feature requests, requirements, design artifacts, source code, tests, safety cases, release plans, and bug reports. If leveraged effectively, this data can be used to provide project intelligence that supports diverse software engineering activities such as release planning, impact analysis, and software analytics. However, project stakeholders often lack skills to formulate complex queries needed to retrieve, manipulate, and display the data in meaningful ways. To address these challenges we introduce TiQi, a natural language interface, which allows users to express software-related queries verbally or written in natural language. TiQi is a web-based tool. It visualizes available project data as a prompt to the user, accepts Natural Language (NL) queries, transforms those queries into SQL, and then executes the queries against a centralized or distributed database. Raw data is stored either directly in the database or retrieved dynamically at runtime from case tools and repositories such as Github and Jira. The transformed query is visualized back to the user as SQL and augmented UML, and raw data results are returned. Our tool demo can be found on YouTube at the following link:http://tinyurl.com/TIQIDemo. 

   more » « less
3. QuerTCI: A Tool Integrating GitHub Issue Querying with Comment Classification

   https://doi.org/10.5281/zenodo.6115404  

   Paing, Ye; Castro-Vélez, Tatiana; Khatchadourian, Raffi (January 2022, Zenodo)

   Issue tracking systems enable users and developers to comment on problems plaguing a software system. Empirical Software Engineering (ESE) researchers study (open-source) project issues and the comments and threads within to discover---among others---challenges developers face when, e.g., incorporating new technologies, platforms, and programming language constructs. However, issue discussion threads accumulate over time and thus can become unwieldy, hindering any insight that researchers may gain.  While existing approaches alleviate this burden by classifying issue thread comments, there is a gap between searching popular open-source software repositories (e.g., those on GitHub) for issues containing particular keywords and feeding the results into a classification model. In this paper, we demonstrate a research infrastructure tool called QuerTCI that bridges this gap by integrating the GitHub issue comment search API with the classification models found in existing approaches. Using queries, ESE researchers can retrieve GitHub issues containing particular keywords, e.g., those related to a certain programming language construct, and subsequently classify the kinds of discussions occurring in those issues.  Using our tool, our hope is that ESE researchers can uncover challenges related to particular technologies using certain keywords through popular open-source repositories more seamlessly than previously possible. A tool demonstration video may be found at: https://youtu.be/fADKSxn0QUk.</p> 

   more » « less
4. GitHub Actions: The Impact on the Pull Request Process

   https://doi.org/10.1007/s10664-023-10369-w  

   Wessel, Mairieli; Vargovich, Joseph; Gerosa, Marco_A; Treude, Christoph (September 2023, Empirical Software Engineering)

   Abstract Software projects frequently use automation tools to perform repetitive activities in the distributed software development process. Recently, GitHub introducedGitHub Actions, a feature providing automated workflows for software projects. Understanding and anticipating the effects of adopting such technology is important for planning and management. Our research investigates how projects useGitHub Actions, what the developers discuss about them, and how project activity indicators change after their adoption. Our results indicate that 1,489 out of 5,000 most popular repositories (almost 30% of our sample) adoptGitHub Actionsand that developers frequently ask for help implementing them. Our findings also suggest that the adoption ofGitHub Actionsleads to more rejections of pull requests (PRs), more communication in accepted PRs and less communication in rejected PRs, fewer commits in accepted PRs and more commits in rejected PRs, and more time to accept a PR. We found similar results when segmenting our results by categories ofGitHub Actions. We suggest practitioners consider these effects when adoptingGitHub Actionson their projects. 

   more » « less
5. The Fault in Our Stars: Designing Reproducible Large-scale Code Analysis Experiments

   https://doi.org/10.4230/LIPIcs.ECOOP.2024.27  

   Maj, Petr; Muroya, Stefanie; Siek, Konrad; Di_Grazia, Luca; Vitek, Jan (January 2024, Schloss Dagstuhl – Leibniz-Zentrum für Informatik)

   Aldrich, Jonathan; Salvaneschi, Guido (Ed.)

   Large-scale software repositories are a source of insights for software engineering. They offer an unmatched window into the software development process at scale. Their sheer number and size holds the promise of broadly applicable results. At the same time, that very size presents practical challenges for scaling tools and algorithms to millions of projects. A reasonable approach is to limit studies to representative samples of the population of interest. Broadly applicable conclusions can then be obtained by generalizing to the entire population. The contribution of this paper is a standardized experimental design methodology for choosing the inputs of studies working with large-scale repositories. We advocate for a methodology that clearly lays out what the population of interest is, how to sample it, and that fosters reproducibility. Along the way, we discourage researchers from using extrinsic attributes of projects such as stars, that measure some unclear notion of popularity. 

   more » « less