skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Toward Scientific Evidence Standards in Empirical Computer Science
Many scientific fields of study use formally established evidence standards during the peer review and evaluation process, such as Consolidated Standards of Reporting Trials (CONSORT) in medical research, the What Works Clearinghouse (WWC) used in education in the United States, or the APA Journal Article Reporting Standards (JARS) in psychology. The basis for these standards is community agreement on what to report in empirical studies. Such standards achieve two key goals. First, they make it easier to compare studies, facilitating replications, through transparent reporting and sharing of data, which can provide confidence that multiple research teams can obtain the same results. Second, they establish community agreement on how to report on and evaluate studies using different methodologies. The discipline of computer science does not have formalized evidence standards, even for major conferences or journals. This Dagstuhl Seminar has three primary objectives: 1. To establish a process for creating or adopting an existing evidence standard for empirical research in computer science. 2. To build a community of scholars that can discuss what a general standard should include. 3. To kickstart the discussion with scholars from software engineering, human-computer interac- tion, and computer science education. In order to better discuss and understand the implications of such standards across several empirical subfields of computer science and to facilitate adoption, we brought together participants from a range of backgrounds; including academia and industry, software engineering, computer- human interaction and computer science education, as well as representatives from several prominent journals.  more » « less
Award ID(s):
2121993
PAR ID:
10446715
Author(s) / Creator(s):
Date Published:
Journal Name:
Dagstuhl reports
Volume:
12
Issue:
10
ISSN:
2192-5283
Page Range / eLocation ID:
225-240
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. (, Dagstuhl reports)
    The report documents the program and outcomes of Dagstuhl Seminar 18061 "Evidence About Programmers for Programming Language Design". The seminar brought together a diverse group of researchers from the fields of computer science education, programming languages, software engineering, human-computer interaction, and data science. At the seminar, participants discussed methods for designing and evaluating programming languages that take the needs of programmers directly into account. The seminar included foundational talks to introduce the breadth of perspectives that were represented among the participants; then, groups formed to develop research agendas for several subtopics, including novice programmers, cognitive load, language features, and love of programming languages. The seminar concluded with a discussion of the current SIGPLAN artifact evaluation mechanism and the need for evidence standards in empirical studies of programming languages. 
    more » « less
  2. ; ; ; ; (, 2022 ASEE Annual Conference & Exposition)
    Scholars and public figures have called for improved ethics and social responsibility education in computer science degree programs in order to better address consequential technological issues in society. Indeed, rising public concern about computing technologies arguably represents an existential threat to the credibility of the computing profession itself. Despite these increasing calls, relatively little is known about the ethical development and beliefs of computer science students, especially compared to other science and engineering students. Gaps in scholarly research make it difficult to effectively design and evaluate ethics education interventions in computer science. Therefore, there is a pressing need for additional empirical study regarding the development of ethical attitudes in computer science students. Influenced by the Professional Social Responsibility Development Model, this study explores personal and professional social responsibility attitudes among undergraduate computing students. Using survey results from a sample of 982 students (including 184 computing majors) who graduated from a large engineering institution between 2017 and 2021, we compare social responsibility attitudes cross-sectionally among computer science students, engineering students, other STEM students, and non-STEM students. Study findings indicate computer science students have statistically significantly lower social responsibility attitudes than their peers in other science and engineering disciplines. In light of growing ethical concerns about the computing profession, this study provides evidence about extant challenges in computing education and buttresses calls for more effective development of social responsibility in computing students. We discuss implications for undergraduate computing programs, ethics education, and opportunities for future research. 
    more » « less
  3. ; ; ; ; ; (, Empirical software engineering)
    null (Ed.)
    For several years, the software engineering research community used eye trackers to study program comprehension, bug localization, pair programming, and other software engineering tasks. Eye trackers provide researchers with insights on software engineers’ cognitive processes, data that can augment those acquired through other means, such as on-line surveys and questionnaires. While there are many ways to take advantage of eye trackers, advancing their use requires defining standards for experimental design, execution, and reporting. We begin by presenting the foundations of eye tracking to provide context and perspective. Based on previous surveys of eye tracking for programming and software engineering tasks and our collective, extensive experience with eye trackers, we discuss when and why researchers should use eye trackers as well as how they should use them. We compile a list of typical use cases—real and anticipated—of eye trackers, as well as metrics, visualizations, and statistical analyses to analyze and report eye-tracking data. We also discuss the pragmatics of eye tracking studies. Finally, we offer lessons learned about using eye trackers to study software engineering tasks. This paper is intended to be a one-stop resource for researchers interested in designing, executing, and reporting eye tracking studies of software engineering tasks. 
    more » « less
  4. ; ; ; ; ; (, ASEE Annual Conference proceedings)
    This paper describes the Engineering Education Research (EER) Peer Review Training (PERT) project, which is designed to develop EER scholars’ peer review skills through mentored reviewing experiences. Supported by the National Science Foundation, the overall programmatic goals of the PERT project are to establish and evaluate a mentored reviewer program for 1) EER journal manuscripts and 2) EER grant proposals. Concurrently, the project seeks to explore how EER scholars develop schema for evaluating EER scholarship, whether these schema are shared in the community, and how schema influence recommendations made to journal editors during the peer review process. To accomplish these goals, the PERT project leveraged the previously established Journal of Engineering Education (JEE) Mentored Reviewer Program, where two researchers with little reviewing experience are paired with an experienced mentor to complete three manuscript reviews collaboratively. In this paper we report on focus group and exit survey findings from the JEE Mentored Reviewer Program and discuss revisions to the program in response to those findings. 
    more » « less
  5. ; ; ; ; ; (, 2022 Annual American Society of Engineering Education (ASEE) Conference and Exposition)
    This paper describes the Engineering Education Research (EER) Peer Review Training (PERT) project, which is designed to develop EER scholars’ peer review skills through mentored reviewing experiences. Supported by the National Science Foundation, the overall programmatic goals of the PERT project are to establish and evaluate a mentored reviewer program for 1) EER journal manuscripts and 2) EER grant proposals. Concurrently, the project seeks to explore how EER scholars develop schema for evaluating EER scholarship, whether these schema are shared in the community, and how schema influence recommendations made to journal editors during the peer review process. To accomplish these goals, the PERT project leveraged the previously established Journal of Engineering Education (JEE) Mentored Reviewer Program, where two researchers with little reviewing experience are paired with an experienced mentor to complete three manuscript reviews collaboratively. In this paper we report on focus group and exit survey findings from the JEE Mentored Reviewer Program and discuss revisions to the program in response to those findings. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email