An official website of the United States government
Software developers have difficulty understanding the rationale and intent behind original developers' design decisions. Code histories aim to provide richer contexts for code changes over time, but can introduce a large amount of information to the already cognitively demanding task of code comprehension. Storytelling has shown benefits in communicating complex, time-dependent information, yet programmers are reluctant to write stories for their code changes. We explored the utility of narratives made by generative AI. We conducted a within-subjects study comparing the performance of 16 programmers when recalling code history information from a list-view format versus a comparable AI-generated narrative format. Our study found that when using the story-view, participants were 16\% more successful at recalling code history information, and had 30\% less error when assessing the correctness of their responses. We did not find any significant differences in programmer's perceived mental effort or their attitudes towards reuse when using narrative code stories.
more »
« less
An Exploratory Study of Programmers' Analogical Reasoning and Software History Usage During Code Re-Purposing
Background: Software development relies on collaborative problem-solving. Understanding previously addressed problems in software is crucial for developers to identify and repurpose functionalities for new problem-solving contexts. Objective: We explore the barriers programmers encounter during code repurposing and investigate how access to historical context about the original developer's goals may affect this process. Method: We present an exploratory study of 16 programmers who completed two code repurposing tasks in different code bases. Participants completed these tasks both with and without access to the historical information of the original developer's goals. We explore how programmers use analogical reasoning to identify and apply existing software artifacts to new goals. Results: We show that programmers often failed to notice analogies, made false analogies, and underestimated the value of reuse. Even when useful analogies were made, programmers struggled to find the relevant code. We also describe the patterns of how participants utilized code histories. Conclusion: We highlight the barriers programmers face in noticing and applying analogies during code reuse. We suggest design recommendations for future tools to allow lightweight evaluation of code to help programmers identify reuse opportunities.
more »
« less
- Award ID(s):
- 2128128
- PAR ID:
- 10545444
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400705335
- Page Range / eLocation ID:
- 109 to 120
- Format(s):
- Medium: X
- Location:
- Lisbon Portugal
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
When studying how software developers perform security tasks, researchers often ask participants to write code. These studies can be challenging because programming can be time-consuming and frustrating. This paper explores whether alternatives to code-writing can yield scientifically valid results while reducing participant stress. We conducted a remote study in which Python programmers completed two encryption tasks using an assigned library by either writing code from scratch, reading existing code and identifying issues, or fixing issues in existing code. We found that the read and fix conditions were less effective than the write condition in revealing security problems with APIs and their documentation, but still provided useful insights. Meanwhile, the read and especially fix conditions generally resulted in more positive participant experiences. Based on these findings, we make preliminary recommendations for how and when researchers might best use all three study design methods; we also recommend future work to further explore the uses and trade-offs of these approaches.more » « less
-
While AI programming tools hold the promise of increasing programmers’ capabilities and productivity to a remarkable degree, they often exclude users from essential decision making processes, causing many to effectively “turn off their brains” and over-rely on solutions provided by these systems. These behaviors can have severe consequences in critical domains, like software security. We propose Human-in-the-Loop Decoding, a novel interaction technique that allows users to observe and directly influence LLM decisions during code generation, in order to align the model’s output with their personal requirements. We implement this technique in HILDE, a code completion assistant that highlights critical decisions made by the LLM and provides local alternatives for the user to explore. In a within-subjects study (N=18) on security-related tasks, we found that HILDE led participants to generate significantly fewer vulnerabilities and better align code generation with their goals compared to a traditional code completion assistant.more » « less
-
While AI programming tools hold the promise of increasing programmers’ capabilities and productivity to a remarkable degree, they often exclude users from essential decision making processes, causing many to effectively “turn off their brains” and over-rely on solutions provided by these systems. These behaviors can have severe consequences in critical domains, like software security. We propose Human-in-the-Loop Decoding, a novel interaction technique that allows users to observe and directly influence LLM decisions during code generation, in order to align the model’s output with their personal requirements. We implement this technique in HILDE, a code completion assistant that highlights critical decisions made by the LLM and provides local alternatives for the user to explore. In a within-subjects study (N=18) on security-related tasks, we found that HILDE led participants to generate significantly fewer vulnerabilities and better align code generation with their goals compared to a traditional code completion assistant.more » « less
-
Parsons problems are drag-and-drop computer programming puzzles that require learners to place code blocks in the correct order and sometimes indentation. Introductory computer programming instructors use them to teach novice programmers how to code while optimizing problem-solving efficiency and cognitive load. While there is research on the design of Parsons problems for programmers without disabilities and programmers with visual or motor impairments, research regarding their accessibility for programmers with cognitive disabilities is scant. To identify the accessibility barriers and benefits of Parsons problems for neurodiverse programmers, an exploratory multiple-case study was conducted. Participants were asked to read eight chapters of an interactive eBook on Python and to solve Parsons problems. Within-case analyses of 15 retrospective think-aloud interviews with five novice programmers with disabilities led to four recommendations for improving the cognitive accessibility of Parsons problems. For example, programmers with seizure disorders may experience seizures when solving programming problems that require numeric calculations. Hence, creating a range of Parsons problems that do not require mental arithmetic could improve the learning experience for programmers with seizure disorders and those who struggle with mental calculations by lowering their cognitive load. Given this study's qualitative and exploratory approach, it does not offer conclusive, broadly generalizable results. Yet, it reveals detailed and promising avenues for exploration in computing education research that might elude many quantitative techniques.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3641822.3641864
