skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: What’s (not) Working in Programmer User Studies?
A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users — software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data we contribute: (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.  more » « less
Award ID(s):
2016600
PAR ID:
10414980
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
ACM Transactions on Software Engineering and Methodology
ISSN:
1049-331X
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; (, Journal of Mechanical Design)
    null (Ed.)
    Abstract Information organization and utilization are integral to the design and development of creative ideas. However, navigating this often complex information space can be challenging, even for experienced designers. Therefore, a deep analysis of how expert software designers utilize and organize information is needed to provide qualitative insights into their information organization strategies. To address this, four professionals in the software design and development field were recruited for individual 3-hour design sessions. They were asked to generate ideas for a design challenge (reducing distraction-based pedestrian accidents) using information sheets specifically developed to contain different types of information, as identified by prior work. Results reveal individual differences in information approach and categorization, although these were motivated by similar underlying patterns of evaluating the relevance of the information for its ability to inform the project constraints, resources, or (user) requirements. Designer experience and use of design processes and knowledge transfer tools enhanced their ability to turn information into insights. 
    more » « less
  2. ; ; ; ; (, Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22))
    Deaf and Hard-of-Hearing (DHH) users face accessibility challenges during in-person and remote meetings. While emerging use of applications incorporating automatic speech recognition (ASR) is promising, more user-interface and user-experience research is needed. While co-design methods could elucidate designs for such applications, COVID-19 has interrupted in-person research. This study describes a novel methodology for conducting online co-design workshops with 18 DHH and hearing participant pairs to investigate ASR-supported mobile and videoconferencing technologies along two design dimensions: Correcting errors in ASR output and implementing notification systems for influencing speaker behaviors. Our methodological findings include an analysis of communication modalities and strategies participants used, use of an online collaborative whiteboarding tool, and how participants reconciled differences in ideas. Finally, we present guidelines for researchers interested in online DHH co-design methodologies, enabling greater geographically diversity among study participants even beyond the current pandemic. 
    more » « less
  3. ; (, International Design Engineering Technical Conferences and Computers and Information in Engineering Conference)
    Abstract Information organization and utilization are integral to the design and development of creative ideas. However, navigating this often complex information space can be challenging, even for experienced designers. Therefore, deep analysis of how expert designers utilize and organize information is needed to provide qualitative insights into their information organization strategies. To address this, four professionals in the software design and development field were recruited for individual 3-hour design sessions. They were asked to generate ideas for a design challenge (reducing distraction-based pedestrian accidents) using information sheets specifically developed to contain different types of information, as identified by prior work. Results reveal individual differences in information approach and categorization, although these were motivated by similar underlying patterns of evaluating the relevance of information for its ability to inform the project constraints, resources or (user) requirements. Designer experience and use of design processes and knowledge transfer tools enhanced their ability to turn information into insights. 
    more » « less
  4. ; ; ; ; (, ACM)
    Background and Context. Research software in the Computing Education Research (CER) domain frequently encounters issues with scalability and sustained adoption, which limits its educational impact. Despite the development of numerous CER programming (CER-P) tools designed to enhance learning and instruction, many fail to see widespread use or remain relevant over time. Previous research has primarily examined the challenges educators face in adopting and reusing CER tools, with few focusing on understanding the barriers to scaling and adoption practices from the tool developers’ perspective. Objectives. To address this, we conducted semi-structured interviews with 16 tool developers within the computing education community, focusing on the challenges they encounter and the practices they employ in scaling their CER-P tools. Method. Our study employs thematic analysis of the semi-structured interviews conducted with developers of CER-P tools. Findings. Our analysis revealed several barriers to scaling highlighted by participants, including funding issues, maintenance burdens, and the challenge of ensuring tool interoperability for a broader user base. Despite these challenges, developers shared various practices and strategies that facilitated some degree of success in scaling their tools. These strategies include the development of teaching materials and units of curriculum, active marketing within the academic community, and the adoption of flexible design principles to facilitate easier adaptation and use by educators and students. Implications. Our findings lay the foundation for further discussion on potential community action initiatives, such as the repository of CS tools and the community of tool developers, to allow educators to discover and integrate tools more easily in their classrooms and support tool developers by exchanging design practices to build high-quality education tools. Furthermore, our study suggests the potential benefits of exploring alternative funding models. 
    more » « less
  5. ; ; ; ; ; ; ; ; ; ; et al (, IEEE Transactions on Software Engineering)
    Previous research has revealed that newcomer women are disproportionately affected by gender-biased barriers in open source software (OSS) projects. However, this research has focused mainly on social/cultural factors, neglecting the software tools and infrastructure. To shed light on how OSS tools and infrastructure might factor into OSS barriers to entry, we conducted two studies: (1) a field study with five teams of software professionals, who worked through five use cases to analyze the tools and infrastructure used in their OSS projects; and (2) a diary study with 22 newcomers (9 women and 13 men) to investigate whether the barriers matched the ones identified by the software professionals. The field study produced a bleak result: software professionals found gender biases in 73% of all the newcomer barriers they identified. Further, the diary study confirmed these results: Women newcomers encountered gender biases in 63% of barriers they faced. Fortunately, many the kinds of barriers and biases revealed in these studies could potentially be ameliorated through changes to the OSS software environments and tool. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email