More Like this

1. Exploring the Validity of the Engineering Design Self-Efficacy Scale for Secondary School Students (Research To Practice)

   Lee, E.; Carberry, A. R.; Dalal, M.; Miller, M. J. (July 2021, 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference)

   null (Ed.)

   The purpose of this study is to re-examine the validity evidence of the engineering design self-efficacy (EDSE) scale scores by Carberry et al. (2010) within the context of secondary education. Self-efficacy refers to individuals’ belief in their capabilities to perform a domain-specific task. In engineering education, significant efforts have been made to understand the role of self-efficacy for students considering its positive impact on student outcomes such as performance and persistence. These studies have investigated and developed measures for different domains of engineering self-efficacy (e.g., general academic, domain-general, and task-specific self-efficacy). The EDSE scale is a frequently cited measure that examines task-specific self-efficacy within the domain of engineering design. The original scale contains nine items that are intended to represent the engineering design process. Initial score validity evidence was collected using a sample consisting of 202 respondents with varying degrees of engineering experience including undergraduate/graduate students and faculty members. This scale has been primarily used by researchers and practitioners with engineering undergraduate students to assess changes in their engineering design self-efficacy as a result of active learning interventions, such as project-based learning. Our work has begun to experiment using the scale in a secondary education context in conjunction with an increased introduction to engineering in K-12 education. Yet, there still is a need to examine score validity and reliability of this scale in non-undergraduate populations such as secondary school student populations. This study fills this important gap by testing construct validity of the original nine items of the EDSE scale, supporting proper use of the scale for researchers and practitioners. This study was conducted as part of a larger, e4usa project investigating the development and implementation of a yearlong project-based engineering design course for secondary school students. Evidence of construct validity and reliability was collected using a multi-step process. First, a survey that includes the EDSE scale was administered to the project participating students at nine associated secondary schools across the US at the beginning of Spring 2020. Analysis of collected data is in progress and includes Exploratory Factor Analysis (EFA) on the 137 responses. The evidence of score reliability will be obtained by computing the internal consistency of each resulting factor. The resulting factor structure and items will be analyzed by comparing it with the original EDSE scale. The full paper will provide details about the psychometric evaluation of the EDSE scale. The findings from this paper will provide insights on the future usage of the EDSE scale in the context of secondary engineering education. 

   more » « less
2. Developing a Writing Rubric to Answer Research Questions (not for Grading!)

   Lynch, J W; Aller, B M; Sorby, S A; Murphy, T J (April 2024, ASEE Peer)

   Industry leaders emphasize that engineering students' technical communication and writing skills must be improved. Despite various institutional efforts, which include technical communication courses or engineering design projects aimed at enhancing students’ communication abilities, many believe there has been only slow improvement in this area. There has also been a dearth of longitudinal studies that examine the development of engineering students’ technical communication competencies from undergraduate to industry. This paper aims to contribute to this area through the creation of a rubric that specifically examines the writing competencies and technical communication ability of engineering students. This paper is part of a larger, NSF-funded research study that examines the quality of students’ written and oral communication skills and seeks to understand their relationship to the students’ spatial abilities. First-year engineering students in their second semester at a large R1 Midwestern university were examined. Students were tasked with creating a written report responding to a set of questions that asked about their team-based engineering design project completed in their first semester. As this occurred months prior, this non-graded report became a reflection on their experience and innate abilities. While low stakes, it mimicked a more authentic writing experience students encounter in industry. Students' responses were examined collaboratively by an interdisciplinary team which created a rubric through an iterative process. This rubric was distributed to the interdisciplinary team and outside evaluators composed of individuals in industry and engineering faculty. An inter-rater reliability analysis was conducted to examine levels of agreement between the interdisciplinary team and outside evaluators, and implications of this inter-rater reliability score and the process of rubric application were documented. Results of this paper include details on the development of a rubric that examine students’ technical communication and writing skills. Traditional rubrics utilized by engineering faculty usually address an entire project for engineering students, which includes students' content knowledge, writing capabilities, and the requirements of the project. Such rubrics are often used to provide feedback to students and evaluation in the form of grades. The narrower focus of the rubric being developed here can provide insights into communication and writing competencies of engineering students. Scores secured through the use of this rubric will aid in the research study’s goal of finding correlations between engineering students’ communication skills and spatial abilities (assessed outside of this current effort). Spatial ability has been well-documented as an effective indicator of success in STEM, and interventions have been developed to support development in students with weaker spatial skills. 23, 24This has prompted this research to explore links between spatial skills and communication abilities, as validated spatial interventions may help improve communication abilities. These current results may also provide unique insights into first-year engineering students’ writing competencies when reporting on a more authentic (non-graded) engineering task. Such information may be useful in eventually shaping guidance of students’ communication instruction in hopes of better preparing them for industry; this is the focus of a planned future research study. 

   more » « less
3. An Inquiry into the Use of Intercoder Reliability Measures in Qualitative Research

   Wilson-Lopez, A.; Minichiello, A.; Green, T. (July 2019, ASEE Annual Conference proceedings)

   In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes. 

   more » « less
4. Work-in-Progress Paper: Creating and Validating the Conceptual Assessment for Sedimentology courses

   Stepanova, Anna; Anwar, Saira; Laya, Juan Carlos; Alvarez-Zarikian, Carlos; Hammond, Tracy (October 2024, IEEE)

   Contribution: In this work-in-progress paper we describe the process of creating and validating a conceptual assessment in the field of sedimentology for undergraduate geoscience courses. The mechanism can aid future geoscience educators and researchers in the process of academic assessment development aligned with learning objectives in these courses. Background: Prior literature review supports the benefits of using active learning tools in STEM (Science, Technology, Engineering, and Mathematics) courses. This paper is part of a larger project to develop and incorporate research-based active learning software in sedimentology and other geoscience courses to improve grade point average (GPA) and time to graduation for Hispanic students at Texas A&M University. To evaluate the novel tool, we designed and validated the conceptual assessment instrument presented in this work. Research Question: What is the process to develop and validate a conceptual assessment for sedimentology? Methodology: This paper follows quantitative analysis and the assessment triangle approach and focuses on cognition, observation, and interpretation to design and evaluate the conceptual assessment. In the cognition element of the triangle, we explain the mechanism for creating the assessment instrument using students' learning objectives. The observation element explains the mechanism of data collection and the instrument revision. The interpretation element explains the results of the validation process using item response theory and reliability measures. We collected the conceptual assessment data from 17 participants enrolled in two courses where sedimentology topics are taught. Participants were geology majors in one of the courses and engineering majors in the other. Findings: The team developed a conceptual assessment that included eight multiple-choice (MCQ) and four open-ended response questions. The results of the design process described the conceptualization of questions and their validation. Also, the validity of created rubrics was established using inter-rater reliability measures, which showed good agreement between raters. Additionally, the results of the validation process indicated that the conceptual assessment was designed for students with average abilities. 

   more » « less
5. Rubric Development for Technical Reports in Chemical Engineering Unit Operations Laboratory Courses

   Brown, J. R.; Wettstein, S. G.; Hacker, D. J. (June 2023, ASEE Annual Conference proceedings)

   The purpose of this work was to test the inter-rater reliability (IRR) of a rubric used to grade technical reports in a senior-level chemical engineering laboratory course that has multiple instructors that grade deliverables. The rubric consisted of fifteen constructs that provided students detailed guidance on instructor expectations with respect to the report sections, formatting and technical writing aspects such as audience, context and purpose. Four student reports from previous years were scored using the rubric, and IRR was assessed using a two-way mixed, consistency, average-measures intraclass correlation (ICC) for each construct. Then, the instructors met as a group to discuss their scoring and reasoning. Multiple revisions were made to the rubric based on instructor feedback and constructs rated by ICC as poor. When fair or poor constructs were combined, the ICCs improved. In addition, the overall score construct continued to be rated as excellent, indicating that while different instructors may have variation at the individual construct level, they evaluate the overall quality of the report consistently. A key learning from this process was the importance of the instructor discussion around their reasoning for the scores and the importance of an ‘instructor orientation’ involving discussion and practice using the rubrics in the case of multiple instructors or a change in instructors. The developed rubric has the potential for broad applicability to engineering laboratory courses with technical writing components and could be adapted for alternative styles of technical writing genre. 

   more » « less