More Like this

1. Evidence About Programmers for Programming Language Design (Dagstuhl Seminar 18061)

   https://doi.org/10.4230/DagRep.8.2.1  

   Stefik. A., Sharif (January 2018, Dagstuhl reports)

   The report documents the program and outcomes of Dagstuhl Seminar 18061 "Evidence About Programmers for Programming Language Design". The seminar brought together a diverse group of researchers from the fields of computer science education, programming languages, software engineering, human-computer interaction, and data science. At the seminar, participants discussed methods for designing and evaluating programming languages that take the needs of programmers directly into account. The seminar included foundational talks to introduce the breadth of perspectives that were represented among the participants; then, groups formed to develop research agendas for several subtopics, including novice programmers, cognitive load, language features, and love of programming languages. The seminar concluded with a discussion of the current SIGPLAN artifact evaluation mechanism and the need for evidence standards in empirical studies of programming languages. 

   more » « less
2. Efficient Counter-factual Type Error Debugging

   https://doi.org/10.1109/TASE.2019.00-13  

   Chen, Sheng; Wu, Baijun (July 2019, 2019 International Symposium on Theoretical Aspects of Software Engineering)

   Type inference is an important part of functional programming languages and has been increasingly adopted to imperative programming. However, providing effective error messages in response to type inference failures (due to type errors in programs) continues to be a challenge. Type error messages generated by compilers and existing error debugging approaches often point to bogus error locations or lack sufficient information for removing the type error, making error debugging ineffective. Counter-factual typing (CFT) addressed this problem by generating comprehensive error messages with each message includes a rich set of information. However, CFT has a large response time, making it too slow for interactive use. In particular, our recent study shows that programmers usually have to go through multiple iterations of updating and recompiling programs to remove a type error. Interestingly, our study also reveals that program updates are minor in each iteration during type error debugging. We exploit this fact and develop eCFT, an efficient version of CFT, which doesn't recompute all error fixes from scratch for each updated program but only recomputes error fixes that are changed in response to the update. Our key observation is that minor program changes lead to minor error suggestion changes. eCFT is based on principal typing, a typing scheme more amenable to reuse previous typing results. We have evaluated our approach and found it is about 12.4× faster than CFT in updating error fixes. 

   more » « less
3. Report on the Dagstuhl Seminar on Frontiers of Information Access Experimentation for Research and Education

   https://doi.org/10.1145/3636341.3636351  

   Bauer, Christine; Carterette, Ben; Ferro, Nicola; Fuhr, Norbert; Beel, Joeran; Breuer, Timo; Clarke, Charles_L A; Crescenzi, Anita; Demartini, Gianluca; Di_Nunzio, Giorgio Maria; et al (June 2023, ACM SIGIR Forum)

   This report documents the program and the outcomes of Dagstuhl Seminar 23031 Frontiers of Information Access Experimentation for Research and Education, which brought together 38 participants from 12 countries. The seminar addressed technology-enhanced information access (information retrieval, recommender systems, natural language processing) and specifically focused on developing more responsible experimental practices leading to more valid results, both for research as well as for scientific education. The seminar featured a series of long and short talks delivered by participants, who helped in setting a common ground and in letting emerge topics of interest to be explored as the main output of the seminar. This led to the definition of five groups which investigated challenges, opportunities, and next steps in the following areas:reality check, i.e. conducting real-world studies, human-machine-collaborative relevance judgment frameworks, overcoming methodological challenges in information retrieval and recommender systems through awareness and education, results-blind reviewing, and guidance for authors. Date:15--20 January 2023. Website:https://www.dagstuhl.de/23031. 

   more » « less
4. How should compilers explain problems to developers?

   https://doi.org/10.1145/3236024.3236040  

   Barik, Titus; Ford, Denae; Murphy-Hill, Emerson; Parnin, Chris (November 2018, Proceedings of teh 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering)

   Compilers primarily give feedback about problems to developers through the use of error messages. Unfortunately, developers routinely find these messages to be confusing and unhelpful. In this paper, we postulate that because error messages present poor explanations, theories of explanation---such as Toulmin's model of argument---can be applied to improve their quality. To understand how compilers should present explanations to developers, we conducted a comparative evaluation with 68 professional software developers and an empirical study of compiler error messages found in Stack Overflow questions across seven different programming languages. Our findings suggest that, given a pair of error messages, developers significantly prefer the error message that employs proper argument structure over a deficient argument structure when neither offers a resolution---but will accept a deficient argument structure if it provides a resolution to the problem. Human-authored explanations on Stack Overflow converge to one of the three argument structures: those that provide a resolution to the error, simple arguments, and extended arguments that provide additional evidence for the problem. Finally, we contribute three practical design principles to inform the design and evaluation of compiler error messages. 

   more » « less
5. UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior

   https://doi.org/10.1145/3411764.3445538  

   Schoop, Eldon; Huang, Forrest; Hartmann, Bjoern (May 2021, CHI Conference on Human Factors in Computing Systems (CHI '21))

   Training deep neural networks can generate non-descriptive error messages or produce unusual output without any explicit errors at all. While experts rely on tacit knowledge to apply debugging strategies, non-experts lack the experience required to interpret model output and correct Deep Learning (DL) programs. In this work, we identify DL debugging heuristics and strategies used by experts, andIn this work, we categorize the types of errors novices run into when writing ML code, and map them onto opportunities where tools could help. We use them to guide the design of Umlaut. Umlaut checks DL program structure and model behavior against these heuristics; provides human-readable error messages to users; and annotates erroneous model output to facilitate error correction. Umlaut links code, model output, and tutorial-driven error messages in a single interface. We evaluated Umlaut in a study with 15 participants to determine its effectiveness in helping developers find and fix errors in their DL programs. Participants using Umlaut found and fixed significantly more bugs and were able to implement fixes for more bugs compared to a baseline condition. 

   more » « less