skip to main content


Title: An Engineering Computational Thinking Diagnostic: A Psychometric Analysis
This research-track work-in-progress paper contributes to engineering education by documenting progress in developing a new standard Engineering Computational Thinking Diagnostic to measure engineering student success in five factors of computational thinking. Over the past year, results from an initial validation attempt were used to refine diagnostic questions. A second statistical validation attempt was then completed in Spring 2021 with 191 student participants at three universities. Statistics show that all diagnostic questions had statistically significant factor loadings onto one general computational thinking factor that incorporates the five original factors of (a) Abstraction, (b) Algorithmic Thinking, (c) Decomposition, (d) Data Representation and Organization, and (e) Impact of Computing. This result was unexpected as our goal was a diagnostic that could discriminate among the five factors. A small population size caused by the virtual delivery of courses during the COVID-19 pandemic may be the explanation and a third round of validation in Fall 2021 is expected to result in a larger population given the return to face-to-face instruction. When statistical validation is completed, the diagnostic will help institutions identify students with strong entry level skills in computational thinking as well as students that require academic support. The diagnostic will inform curriculum design by demonstrating which factors are more accessible to engineering students and which factors need more time and focus in the classroom. The long-term impact of a successfully validated computational thinking diagnostic will be introductory engineering courses that better serve engineering students coming from many backgrounds. This can increase student self- efficacy, improve student retention, and improve student enculturation into the engineering profession. Currently, the diagnostic identifies general computational thinking skill  more » « less
Award ID(s):
1917352
NSF-PAR ID:
10348473
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
2021 IEEE Frontiers in Education Conference (FIE)
Page Range / eLocation ID:
1 to 5
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. This project has been dedicated to advance the way computational thinking is taught to engineering undergraduate students with a multitude of social identities. It is an expectation that with the understanding of the multiple factors that affect computational thinking skills development, students succeed in enculturating to the engineering professional practice. During the third year of this project, the first major result is the conclusion of the validation process of the Engineering Computational Thinking Diagnostic (ECTD) making use of exploratory and confirmatory factor analyses (EFA-CFA). Our validation showed that the ECTD questions cluster in one factor, what we call the computational thinking factor for engineers. Other validation statistical processes (i.e. correlations, regressions, ANOVA and t-tests) proved the predictability potential use of this tool in determining how well prepared students arrive to the engineering classroom and how their prior coding experience can determine their success in introductory coding engineering courses. The second major result is the revelation that the inequities caused by the many forms of privilege that some engineering students benefit from are being exacerbated by the integration of computational thinking into introductory engineering classes. Due to pandemic-related challenges in recruiting a representative sample of participants, the majority of the self-selected participants in our research identify with groups with disproportionately large participation in engineering (specifically White and Asian) and are academically successful in engineering. To respond to this challenge we are seeking to broaden our perspective by seeking participants with failing grades for a final round of data collection, although we are well aware that students in this group are often reluctant to participate in research. The fourth and last major result is related to the position of stress versus Artificial Intelligence (AI) perceptions, both part of the ECTD instrument. The position of stress questions involved perceived difficulty and confidence level after taking the ECTD. The artificial intelligence question asked the perceived impact of AI in students’ future career prospects. Preliminary analysis is suggesting that confidence level is correlated with AI positive perceptions. Although not part of the original NSF grant, we considered AI the natural evolution of computational thinking in the formation of engineers and plan to continue our work in this direction. 
    more » « less
  2. Computational thinking is an important skill in the formation of engineers. Many schools of engineering include a programming course during the first-year. In 2019, NSF funded the “Collaborative Research: Research in Improving Computational Thinking in the Formation of Engineers, a Multi-Institutional Initiative.” The project’s goal is to improve the way computational thinking is taught at the college engineering level via the understanding of the multiple factors that affect computational thinking development. The project’s research questions are: · Research Question 1: How does the integration of computing into the foundational engineering courses affect the formation of engineers? · Research Question 2: In what ways do social identities (e.g. gender, ethnicity, first generation college attending, socioeconomic status), choices (e.g. major, transfer status), and other factors impact the engineering student experience with computational thinking? · Research Question 3: In what ways do computational thinking skills develop over time in engineering students? In order to respond to these questions, the research team developed a Computational Thinking Hybrid Framework, an Engineering Computational Thinking Diagnostic (ECTD) and a Position of Stress Questionnaire. Amid COVID-19, the advances of the project include approximately 2000 participants responding to the diagnostic during the Fall of 2019 and the Fall of 2020. With this participation, two cycles of validation have taken place for the ECTD and results are presented in this poster session. The factors validated in this diagnostic are (1) Abstraction, (2) Algorithmic Thinking and Programming, (3) Data Representation, Organization, and Analysis, (4) Decomposition, and (5) Impact of Computing. The positions of stress have been collected for the Fall of 2020 and preliminary results are also presented in this session. 
    more » « less
  3. There is a critical need for more students with engineering and computer science majors to enter into, persist in, and graduate from four-year postsecondary institutions. Increasing the diversity of the workforce by inclusive practices in engineering and science is also a profound identified need. According to national statistics, the largest groups of underrepresented minority students in engineering and science attend U.S. public higher education institutions. Most often, a large proportion of these students come to colleges and universities with unique challenges and needs, and are more likely to be first in their family to attend college. In response to these needs, engineering education researchers and practitioners have developed, implemented and assessed interventions to provide support and help students succeed in college, particularly in their first year. These interventions typically target relatively small cohorts of students and can be managed by a small number of faculty and staff. In this paper, we report on “work in progress” research in a large-scale, first-year engineering and computer science intervention program at a public, comprehensive university using multivariate comparative statistical approaches. Large-scale intervention programs are especially relevant to minority serving institutions that prepare growing numbers of students who are first in their family to attend college and who are also under-resourced, financially. These students most often encounter academic difficulties and come to higher education with challenging experiences and backgrounds. Our studied first-year intervention program, first piloted in 2015, is now in its 5th year of implementation. Its intervention components include: (a) first-year block schedules, (b) project-based introductory engineering and computer science courses, (c) an introduction to mechanics course, which provides students with the foundation needed to succeed in a traditional physics sequence, and (d) peer-led supplemental instruction workshops for calculus, physics and chemistry courses. This intervention study responds to three research questions: (1) What role does the first-year intervention’s components play in students’ persistence in engineering and computer science majors across undergraduate program years? (2) What role do particular pedagogical and cocurricular support structures play in students’ successes? And (3) What role do various student socio-demographic and experiential factors play in the effectiveness of first-year interventions? To address these research questions and therefore determine the formative impact of the firstyear engineering and computer science program on which we are conducting research, we have collected diverse student data including grade point averages, concept inventory scores, and data from a multi-dimensional questionnaire that measures students’ use of support practices across their four to five years in their degree program, and diverse background information necessary to determine the impact of such factors on students’ persistence to degree. Background data includes students’ experiences prior to enrolling in college, their socio-demographic characteristics, and their college social capital throughout their higher education experience. For this research, we compared students who were enrolled in the first-year intervention program to those who were not enrolled in the first-year intervention. We have engaged in cross-sectional 2 data collection from students’ freshman through senior years and employed multivariate statistical analytical techniques on the collected student data. Results of these analyses were interesting and diverse. Generally, in terms of backgrounds, our research indicates that students’ parental education is positively related to their success in engineering and computer science across program years. Likewise, longitudinally (across program years), students’ college social capital predicted their academic success and persistence to degree. With regard to the study’s comparative research of the first-year intervention, our results indicate that students who were enrolled in the first-year intervention program as freshmen continued to use more support practices to assist them in academic success across their degree matriculation compared to students who were not in the first-year program. This suggests that the students continued to recognize the value of such supports as a consequence of having supports required as first-year students. In terms of students’ understanding of scientific or engineering-focused concepts, we found significant impact resulting from student support practices that were academically focused. We also found that enrolling in the first-year intervention was a significant predictor of the time that students spent preparing for classes and ultimately their grade point average, especially in STEM subjects across students’ years in college. In summary, we found that the studied first-year intervention program has longitudinal, positive impacts on students’ success as they navigate through their undergraduate experiences toward engineering and computer science degrees. 
    more » « less
  4. null (Ed.)
    Computational thinking is understood as the development of skills and knowledge in how to apply computers and technology to systematically solve problems. Computational thinking has been acknowledged as one key aspect in the taxonomy of engineering education and implied in multiple ABET student outcomes. Moreover, many introductory engineering courses worldwide have a component of programming or computational thinking. A preliminary study of enculturation to the engineering profession found that computational thinking was deemed a critical area of development at the early stages of instruction (Mendoza Diaz et al., 2018, 2019; Richard et al., 2016; Wickliff et al., 2018). No existing computational thinking framework was found to fully meet the needs of engineers, based on the expertise of researchers at three different institutions and the aid of a comprehensive literature review. As a result, a revised version of a computational thinking diagnostic was developed and renamed the engineering computational thinking diagnostic (ECTD). The five computational thinking factors of the ECTD are (1) Abstraction, (2) Algorithmic Thinking and Programming, (3) Data Representation, Organization, and Analysis, (4) Decomposition, and (5) Impact of Computing. This paper describes the development and revisions made to the ECTD using data collected from first-year engineering students at a Southwestern public university. The goal of the development of the ECTD is to capture the entry and exit skill levels of engineering students in an engineering program. 
    more » « less
  5. null (Ed.)
    Computational thinking is understood as the development of skills and knowledge in how to apply computers and technology to systematically solve problems. Computational thinking has been acknowledged as one key aspect in the taxonomy of engineering education and implied in multiple ABET student outcomes. Moreover, many introductory engineering courses worldwide have a component of programming or computational thinking. A preliminary study of enculturation to the engineering profession found that computational thinking was deemed a critical area of development at the early stages of instruction. No existing computational thinking framework was found to fully meet the needs of engineers, based on the expertise of researchers at three different institutions and the aid of a comprehensive literature review. As a result, a revised version of a computational thinking diagnostic was developed and renamed the engineering computational thinking diagnostic (ECTD). The five computational thinking factors of the ECTD are (1) Abstraction, (2) Algorithmic Thinking and Programming, (3) Data Representation, Organization, and Analysis, (4) Decomposition, and (5) Impact of Computing. This paper describes the development and revisions made to the ECTD using data collected from first-year engineering students at a Southwestern public university. The goal of the development of the ECTD is to capture the entry and exit skill levels of engineering students in an engineering program. 
    more » « less