More Like this

1. Toward Predicting Architectural Significance of Implementation Issues

   Shahbazian, Arman; Nam, Daye; Medvidovic, Nenad (May 2018, International Conference on Mining Software Repositories)

   In a software system’s development lifecycle, engineers make numerous design decisions that subsequently cause architectural change in the system. Previous studies have shown that, more often than not, these architectural changes are unintentional by-products of continual software maintenance tasks. The result of inadvertent architectural changes is accumulation of technical debt and deterioration of software quality. Despite their important implications, there is a relative shortage of techniques, tools, and empirical studies pertaining to architectural design decisions. In this paper, we take a step toward addressing that scarcity by using the information in the issue and code repositories of open-source software systems to investigate the cause and frequency of such architectural design decisions. Furthermore, building on these results, we develop a predictive model that is able to identify the architectural significance of newly submitted issues, thereby helping engineers to prevent the adverse effects of architectural decay. The results of this study are based on the analysis of 21,062 issues affecting 301 versions of 5 large open-source systems for which the code changes and issues were publicly accessible. 

   more » « less
2. Recovering Architectural Design Decisions

   Shahbazian, Arman; Lee, Youn Kyu; Le, Duc; Brun, Yuriy; Medvidovic, Nenad (April 2018, International Conference on Software Architecture)

   Designing and maintaining a software system’s architecture typically involve making numerous design decisions, each potentially affecting the system’s functional and nonfunctional properties. Understanding these design decisions can help inform future decisions and implementation choices and can avoid introducing regressions and architectural inefficiencies later. Unfortunately, design decisions are rarely well documented and are typically a lost artifact of the architecture creation and maintenance process. The loss of this information can thus hurt development. To address this shortcoming, we develop RecovAr, a technique for automatically recovering design decisions from the project’s readily available history artifacts, such as an issue tracker and version control repository. RecovAr uses state-ofthe- art architectural recovery techniques on a series of version control commits and maps those commits to issues to identify decisions that affect system architecture. While some decisions can still be lost through this process, our evaluation on Hadoop and Struts, two large open-source systems with over 8 years of development each and, on average, more than 1 million lines of code, shows that RecovAr has the recall of 75% and a precision of 77%. Our work formally defines architectural design decisions and develops an approach for tracing such decisions in project histories. Additionally, the work introduces methods to classify whether decisions are architectural and to map decisions to code elements. Finally, our work contributes a methodology engineers can follow to preserve design-decision knowledge in their projects. 

   more » « less
3. The untold impact of learning approaches on software fault-proneness predictions: an analysis of temporal aspects

   https://doi.org/10.1007/s10664-024-10454-8  

   Ahmad, Mohammad Jamil; Goseva-Popstojanova, Katerina; Lutz, Robyn R (July 2024, Empirical Software Engineering)

   This paper aims to improve software fault-proneness prediction by investigating the unex- plored effects on classification performance of the temporal decisions made by practitioners and researchers regarding (i) the interval for which they will collect longitudinal features (soft- ware metrics data), and (ii) the interval for which they will predict software bugs (the target variable). We call these specifics of the data used for training and of the target variable being predicted the learning approach, and explore the impact of the two most common learning approaches on the performance of software fault-proneness prediction, both within a single release of a software product and across releases. The paper presents empirical results from a study based on data extracted from 64 releases of twelve open-source projects. Results show that the learning approach has a substantial, and typically unacknowledged, impact on classi- fication performance. Specifically, we show that one learning approach leads to significantly better performance than the other, both within-release and across-releases. Furthermore, this paper uncovers that, for within-release predictions, the difference in classification perfor- mance is due to different levels of class imbalance in the two learning approaches. Our findings show that improved specification of the learning approach is essential to under- standing and explaining the performance of fault-proneness prediction models, as well as to avoiding misleading comparisons among them. The paper concludes with some practical recommendations and research directions based on our findings toward improved software fault-proneness prediction. This preprint has not undergone any post-submission improvements or corrections. The Version of Record of this article is published in Empirical Software Engineering, and is available online at https://doi.org/10.1007/s10664-024-10454-8. 

   more » « less
4. Evaluating Engineering Students’ Moral Sensitivity in a Natural Disaster Context

   Lapatin, M.; Barrens, S.; & Kim, K.; Padgett Walsh, K.; Feinstein, S.; Rutherford, C.; Nguyen, L.; Faust, K. (August 2022, ASEE PEER)

   Engineered systems are designed to serve societal needs, from bridges providing mobility to communication systems enabling the transfer of information. It is essential that engineers recognize the social impact of their work to ensure they provide equitable benefits across communities when implementing such systems. In times of crisis, such as after natural disasters, these ethical considerations and awareness of community needs are especially important. Ethical development must begin when engineers are still students so that they can be trained to consider ethical issues before they begin working. Ethical development can be observed using James Rest’s Four-Component Model of Morality: moral sensitivity, moral judgement, moral motivation, and moral behavior. Previous work has focused largely on the second stage, moral judgement, which describes the ability to determine which action is morally right when confronted with an ethical issue. Here, however, we focus on the first stage, moral sensitivity, emphasizing one’s ability to recognize a moral issue. Studies show that while moral sensitivity does not always lead to moral behavior; moral sensitivity can help explain variances in moral behavior. Researchers argue that pinpointing students’ gaps in moral sensitivity can help educators identify gaps in engineering ethics curriculum. Towards this goal, we interviewed undergraduate engineering students to evaluate their moral sensitivity, using a current event, the 2021 Hurricane Ida in Southern Louisiana, as background. This natural disaster provided a useful context to evaluate moral sensitivity due to the complex effects of such a crisis on engineered, natural, and social systems. The story is framed using Lind’s Indicators of Ethical Sensitivity, providing the story characteristics, stakeholders, and consequences. We asked interviewees to provide the final indicator—ethical issues. Using a qualitative content analysis, we found that interviewees connected several ethical issues with the primary consequence of socioeconomic inequities. Identified ethical issues included topics of climate change, infrastructure, disaster planning, and corporate/government accountability. Implications of this study include recommendations for future moral sensitivity research and applications to improve classroom learning. 

   more » « less
5. Issues in the Reproducibility of Deep Learning Results

   https://doi.org/10.1109/SPMB47826.2019.9037840  

   Jean-Paul, S.; Elseify, T.; Obeid, I.; Picone, J. (December 2019, IEEE Signal Processing in Medicine and Biology Symposium (SPMB))

   Obeid, I.; Selesnik, I.; Picone, J. (Ed.)

   The Neuronix high-performance computing cluster allows us to conduct extensive machine learning experiments on big data [1]. This heterogeneous cluster uses innovative scheduling technology, Slurm [2], that manages a network of CPUs and graphics processing units (GPUs). The GPU farm consists of a variety of processors ranging from low-end consumer grade devices such as the Nvidia GTX 970 to higher-end devices such as the GeForce RTX 2080. These GPUs are essential to our research since they allow extremely compute-intensive deep learning tasks to be executed on massive data resources such as the TUH EEG Corpus [2]. We use TensorFlow [3] as the core machine learning library for our deep learning systems, and routinely employ multiple GPUs to accelerate the training process. Reproducible results are essential to machine learning research. Reproducibility in this context means the ability to replicate an existing experiment – performance metrics such as error rates should be identical and floating-point calculations should match closely. Three examples of ways we typically expect an experiment to be replicable are: (1) The same job run on the same processor should produce the same results each time it is run. (2) A job run on a CPU and GPU should produce identical results. (3) A job should produce comparable results if the data is presented in a different order. System optimization requires an ability to directly compare error rates for algorithms evaluated under comparable operating conditions. However, it is a difficult task to exactly reproduce the results for large, complex deep learning systems that often require more than a trillion calculations per experiment [5]. This is a fairly well-known issue and one we will explore in this poster. Researchers must be able to replicate results on a specific data set to establish the integrity of an implementation. They can then use that implementation as a baseline for comparison purposes. A lack of reproducibility makes it very difficult to debug algorithms and validate changes to the system. Equally important, since many results in deep learning research are dependent on the order in which the system is exposed to the data, the specific processors used, and even the order in which those processors are accessed, it becomes a challenging problem to compare two algorithms since each system must be individually optimized for a specific data set or processor. This is extremely time-consuming for algorithm research in which a single run often taxes a computing environment to its limits. Well-known techniques such as cross-validation [5,6] can be used to mitigate these effects, but this is also computationally expensive. These issues are further compounded by the fact that most deep learning algorithms are susceptible to the way computational noise propagates through the system. GPUs are particularly notorious for this because, in a clustered environment, it becomes more difficult to control which processors are used at various points in time. Another equally frustrating issue is that upgrades to the deep learning package, such as the transition from TensorFlow v1.9 to v1.13, can also result in large fluctuations in error rates when re-running the same experiment. Since TensorFlow is constantly updating functions to support GPU use, maintaining an historical archive of experimental results that can be used to calibrate algorithm research is quite a challenge. This makes it very difficult to optimize the system or select the best configurations. The overall impact of all of these issues described above is significant as error rates can fluctuate by as much as 25% due to these types of computational issues. Cross-validation is one technique used to mitigate this, but that is expensive since you need to do multiple runs over the data, which further taxes a computing infrastructure already running at max capacity. GPUs are preferred when training a large network since these systems train at least two orders of magnitude faster than CPUs [7]. Large-scale experiments are simply not feasible without using GPUs. However, there is a tradeoff to gain this performance. Since all our GPUs use the NVIDIA CUDA® Deep Neural Network library (cuDNN) [8], a GPU-accelerated library of primitives for deep neural networks, it adds an element of randomness into the experiment. When a GPU is used to train a network in TensorFlow, it automatically searches for a cuDNN implementation. NVIDIA’s cuDNN implementation provides algorithms that increase the performance and help the model train quicker, but they are non-deterministic algorithms [9,10]. Since our networks have many complex layers, there is no easy way to avoid this randomness. Instead of comparing each epoch, we compare the average performance of the experiment because it gives us a hint of how our model is performing per experiment, and if the changes we make are efficient. In this poster, we will discuss a variety of issues related to reproducibility and introduce ways we mitigate these effects. For example, TensorFlow uses a random number generator (RNG) which is not seeded by default. TensorFlow determines the initialization point and how certain functions execute using the RNG. The solution for this is seeding all the necessary components before training the model. This forces TensorFlow to use the same initialization point and sets how certain layers work (e.g., dropout layers). However, seeding all the RNGs will not guarantee a controlled experiment. Other variables can affect the outcome of the experiment such as training using GPUs, allowing multi-threading on CPUs, using certain layers, etc. To mitigate our problems with reproducibility, we first make sure that the data is processed in the same order during training. Therefore, we save the data from the last experiment and to make sure the newer experiment follows the same order. If we allow the data to be shuffled, it can affect the performance due to how the model was exposed to the data. We also specify the float data type to be 32-bit since Python defaults to 64-bit. We try to avoid using 64-bit precision because the numbers produced by a GPU can vary significantly depending on the GPU architecture [11-13]. Controlling precision somewhat reduces differences due to computational noise even though technically it increases the amount of computational noise. We are currently developing more advanced techniques for preserving the efficiency of our training process while also maintaining the ability to reproduce models. In our poster presentation we will demonstrate these issues using some novel visualization tools, present several examples of the extent to which these issues influence research results on electroencephalography (EEG) and digital pathology experiments and introduce new ways to manage such computational issues. 

   more » « less