skip to main content


Title: The Engineering Design Process Portfolio Scoring Rubric (EDPPSR): Initial Validity and Reliability
Research prior to 2005 found that no single framework existed that could capture the engineering design process fully or well and benchmark each element of the process to a commonly accepted set of referenced artifacts. Compounding the construction of a stepwise, artifact driven framework is that engineering design is typically practiced over time as a complex and iterative process. For both novice and advanced students, learning and applying the design process is often cumulative, with many informal and formal programmatic opportunities to practice essential elements. The Engineering Design Process Portfolio Scoring Rubric (EDPPSR) was designed to apply to any portfolio that is intended to document an individual or team driven process leading to an original attempt to design a product, process, or method to provide the best and most optimal solution to a genuine and meaningful problem. In essence, the portfolio should be a detailed account or “biography” of a project and the thought processes that inform that project. Besides narrative and explanatory text, entries may include (but need not be limited to) drawings, schematics, photographs, notebook and journal entries, transcripts or summaries of conversations and interviews, and audio/video recordings. Such entries are likely to be necessary in order to convey accurately and completely the complex thought processes behind the planning, implementation, and self-evaluation of the project. The rubric is comprised of four main components, each in turn comprised of three elements. Each element has its own holistic rubric. The process by which the EDPPSR was created gives evidence of the relevance and representativeness of the rubric and helps to establish validity. The EDPPSR model as originally rendered has a strong theoretical foundation as it has been developed by reference to the literature on the steps of the design process through focus groups and through expert review by teachers, faculty and researchers in performance based, portfolio rubrics and assessments. Using the unified construct validity framework, the EDDPSR’s validity was further established through expert reviewers (experts in engineering design) providing evidence supporting the content relevance and representativeness of the EDPPSR in representing the basic process of engineering design. This manuscript offers empirical evidence that supports the use of the EDPPSR model to evaluate student design-based projects in a reliable and valid manner. Intra-class correlation coefficients (ICC) were calculated to determine the inter-rater reliability (IRR) of the rubric. Given the small sample size we also examined confidence intervals (95%) to provide a range of values in which the estimate of inter-reliability is likely contained.  more » « less
Award ID(s):
2120746 1849430
NSF-PAR ID:
10345718
Author(s) / Creator(s):
Date Published:
Journal Name:
American Society for Engineering Education (ASEE) Conference & Exposition
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Industry leaders emphasize that engineering students' technical communication and writing skills must be improved. Despite various institutional efforts, which include technical communication courses or engineering design projects aimed at enhancing students’ communication abilities, many believe there has been only slow improvement in this area. There has also been a dearth of longitudinal studies that examine the development of engineering students’ technical communication competencies from undergraduate to industry. This paper aims to contribute to this area through the creation of a rubric that specifically examines the writing competencies and technical communication ability of engineering students. This paper is part of a larger, NSF-funded research study that examines the quality of students’ written and oral communication skills and seeks to understand their relationship to the students’ spatial abilities. First-year engineering students in their second semester at a large R1 Midwestern university were examined. Students were tasked with creating a written report responding to a set of questions that asked about their team-based engineering design project completed in their first semester. As this occurred months prior, this non-graded report became a reflection on their experience and innate abilities. While low stakes, it mimicked a more authentic writing experience students encounter in industry. Students' responses were examined collaboratively by an interdisciplinary team which created a rubric through an iterative process. This rubric was distributed to the interdisciplinary team and outside evaluators composed of individuals in industry and engineering faculty. An inter-rater reliability analysis was conducted to examine levels of agreement between the interdisciplinary team and outside evaluators, and implications of this inter-rater reliability score and the process of rubric application were documented. Results of this paper include details on the development of a rubric that examine students’ technical communication and writing skills. Traditional rubrics utilized by engineering faculty usually address an entire project for engineering students, which includes students' content knowledge, writing capabilities, and the requirements of the project. Such rubrics are often used to provide feedback to students and evaluation in the form of grades. The narrower focus of the rubric being developed here can provide insights into communication and writing competencies of engineering students. Scores secured through the use of this rubric will aid in the research study’s goal of finding correlations between engineering students’ communication skills and spatial abilities (assessed outside of this current effort). Spatial ability has been well-documented as an effective indicator of success in STEM, and interventions have been developed to support development in students with weaker spatial skills. 23, 24This has prompted this research to explore links between spatial skills and communication abilities, as validated spatial interventions may help improve communication abilities. These current results may also provide unique insights into first-year engineering students’ writing competencies when reporting on a more authentic (non-graded) engineering task. Such information may be useful in eventually shaping guidance of students’ communication instruction in hopes of better preparing them for industry; this is the focus of a planned future research study. 
    more » « less
  2. null (Ed.)
    The purpose of this study is to re-examine the validity evidence of the engineering design self-efficacy (EDSE) scale scores by Carberry et al. (2010) within the context of secondary education. Self-efficacy refers to individuals’ belief in their capabilities to perform a domain-specific task. In engineering education, significant efforts have been made to understand the role of self-efficacy for students considering its positive impact on student outcomes such as performance and persistence. These studies have investigated and developed measures for different domains of engineering self-efficacy (e.g., general academic, domain-general, and task-specific self-efficacy). The EDSE scale is a frequently cited measure that examines task-specific self-efficacy within the domain of engineering design. The original scale contains nine items that are intended to represent the engineering design process. Initial score validity evidence was collected using a sample consisting of 202 respondents with varying degrees of engineering experience including undergraduate/graduate students and faculty members. This scale has been primarily used by researchers and practitioners with engineering undergraduate students to assess changes in their engineering design self-efficacy as a result of active learning interventions, such as project-based learning. Our work has begun to experiment using the scale in a secondary education context in conjunction with an increased introduction to engineering in K-12 education. Yet, there still is a need to examine score validity and reliability of this scale in non-undergraduate populations such as secondary school student populations. This study fills this important gap by testing construct validity of the original nine items of the EDSE scale, supporting proper use of the scale for researchers and practitioners. This study was conducted as part of a larger, e4usa project investigating the development and implementation of a yearlong project-based engineering design course for secondary school students. Evidence of construct validity and reliability was collected using a multi-step process. First, a survey that includes the EDSE scale was administered to the project participating students at nine associated secondary schools across the US at the beginning of Spring 2020. Analysis of collected data is in progress and includes Exploratory Factor Analysis (EFA) on the 137 responses. The evidence of score reliability will be obtained by computing the internal consistency of each resulting factor. The resulting factor structure and items will be analyzed by comparing it with the original EDSE scale. The full paper will provide details about the psychometric evaluation of the EDSE scale. The findings from this paper will provide insights on the future usage of the EDSE scale in the context of secondary engineering education. 
    more » « less
  3. In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes. 
    more » « less
  4. Abstract

    Integrating engineering into the K‐12 science curriculum continues to be a focus in national reform efforts in science education. Although there is an increasing interest in research in and practice of integrating engineering in K‐12 science education, to date only a few studies have focused on the development of an assessment tool to measure students’ understanding of engineering design. Most of the existing measures focus only on knowledge and understanding of engineering design concepts using multiple‐choice items with the exception of the mixed‐format Engineering Concept Assessment (ECA). Also, advanced measurement models are lacking application in the testing of such mixed‐format assessments in science education. This study applied many‐faceted Rasch measurement to the modified ECA for eighth‐grade (ECA/M8) and a newly constructed rubric applied by five judges across 497 eighth‐grade students’ responses after experiencing an integrated learning unit on the engineering design process. The results supported the fit of the items and rubric rating scales to the Rasch specifications. Recommendations are made for item wording, and further reliability and validity testing of the ECA/M8, and use of the ECA/M8 in science education and research.

     
    more » « less
  5. Chinn, C. ; Tan, E. ; Chan, C. ; Kali, Y. (Ed.)
    Collaboration is an important learning process. During collaborative learning, students engage in group activities where they converge on goals, solve problems and make joint decisions. To understand the process of collaboration, we focused on how behavior and interaction patterns contribute to the social-relational space of collaboration. We have designed a multilayered conceptual model for the collaboration process and an observation rubric that identifies behaviors and interactions during collaboration that serves as the foundation for machine learning models that can provide behavioral insight into the process of collaboration. This study reports results on several validation studies performed to establish a validation argument for our collaboration conceptual model and collaboration rubric. Through disconfirming evidence, interrater reliability testing, expert reviews, and focus group interviews, we found that our stratified architecture of collaboration and rubric provide valid accounts and descriptions of human behavior and interactions that can be used to substantiate the collaboration process. 
    more » « less