skip to main content


Title: Exploring scenario‐based assessment of students' global engineering competency: Building evidence of validity of a China‐based situational judgment test
Abstract Background

Engineers operate in an increasingly global environment, making it important that engineering students develop global engineering competency to prepare them for success in the workplace. To understand this learning, we need assessment approaches that go beyond traditional self‐report surveys. A previous study (Jesiek et al.,Journal of Engineering Education2020; 109(3):1–21) began this process by developing a situational judgment test (SJT) to assess global engineering competency based in the Chinese context and administering it to practicing engineers.

Purpose

We built on this previous study by administering the SJT to engineering students to explore what prior experiences related to their SJT performance and how their SJT performance compared with practicing engineers' performance on the SJT.

Method

Engineering students completed a survey including the SJT and related self‐report survey instruments. We collected data from three groups of students: those who had studied abroad in China; those who had studied abroad elsewhere; and those who had not studied abroad.

Results

We found that students' SJT performance did not relate to their scores on the self‐report instruments, but did relate to their participation in study abroad programs. The students also performed better on the SJT when compared to the practicing engineers.

Conclusions

Our results highlight the need to use multiple forms of assessment for global engineering competence. Although building evidence for the validity of the Global Engineering Competency China SJT is an ongoing process, this data collection technique may provide new insights on global engineering competency compared to traditionally used assessments.

 
more » « less
NSF-PAR ID:
10448449
Author(s) / Creator(s):
 ;  ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
Journal of Engineering Education
Volume:
112
Issue:
4
ISSN:
1069-4730
Format(s):
Medium: X Size: p. 1032-1055
Size(s):
p. 1032-1055
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Background

    As globalization continues to impact the engineering profession, many programs aim to prepare current and future engineers to work across national and cultural boundaries. Yet there remains a lack of quality tools for assessing global competency among engineers and other technical professionals, including their behavioral tendencies in global work situations.

    Purpose

    We introduce development of a situational judgment test (SJT) covering three dimensions of global engineering competency (GEC) in Chinese national/cultural context. The main aim of this article is to describe how the SJT was developed through a systematic multistep process. Secondarily, we explore relationships between SJT performance and other theoretically relevant variables.

    Methods

    After generating a large initial pool of SJT scenarios and behavioral response items, we used ratings from subject matter experts (SMEs) to select six SJT scenarios and create scoring keys for 26 response items. To further explore the instrument's validity, we deployed the SJT items, other relevant measures, and a demographic survey to a sample of practicing engineers (n= 400).

    Results

    SME ratings provide strong evidence for the content relevance of the GEC–SJT tool. Survey results also suggest positive relationships between SJT performance and Chinese cultural knowledge, age, and years of work experience. However, more validity and reliability evidence is needed before recommending wider use of the instrument.

    Conclusions

    Our findings suggest the SJT format as a promising behavior‐based approach to measuring global competency and other professional attributes in engineering. We also discuss directions for future research and training efforts related to assessing and developing global competency.

     
    more » « less
  2. Abstract Background

    Conceptual understanding is critical to both engineering education and practice. In fact, many undergraduate courses focus on developing students' knowledge and understanding of core engineering concepts. At the same time, a growing body of literature points to substantial gaps across educational and professional practice contexts, including how problems are embodied and solved.

    Purpose

    The purpose of this study was to explore potential differences in conceptual understanding across engineering students and professional engineers. To do so, we compared the responses of civil engineering practitioners to the Statics Concept Inventory (SCI) to those of engineering students enrolled in statics courses.

    Design/Method

    We administered the SCI to 95 practicing civil engineers and compared their responses to an existing dataset from 1,372 engineering students. We conducted three comparisons: overall SCI score, concept subscores, and item‐by‐item.

    Results

    Students generally outperformed engineers on the SCI in terms of overall performance. However, on closer inspection, students' superior performance appears to be driven by differences in knowledge or understanding of specific statics concepts rather than a stronger understanding in general.

    Conclusions

    We caution against interpretations that imply students have a better understanding of statics concepts. Instead, our results suggest that differences in the way concepts are situated and applied across school and workplace contexts might account for the differences in the performance observed. These findings raise critical questions regarding the nature of concepts and the immutability of common academic representations, and point to the need for further qualitative and exploratory work investigating concepts in practice.

     
    more » « less
  3. Abstract Background

    Research points to family talk and interactions involving STEM concepts as one of the most influential informal learning experiences that shape an individual's STEM identity development and encourage their pursuit of a STEM career. However, a recent literature review uncovers limited research regarding the development of engineering identity in young children.

    Purpose

    The purpose of this study was to add to this scant literature by exploring how children position themselves as engineers and how children are positioned as engineers through interactions with parents and other adults within a program focused on family engagement within an engineering design process.

    Methods

    This study includes two parent–child dyads. We collected and analyzed approximately 19.5 h of video data of the two child–parent dyads interacting with one another throughout an engineering design process as part of an out‐of‐school program.

    Results

    Results highlight three ways in which the two children enacted various engineering identities through their positioning, negotiation, and acceptance and/or rejection of positionalities as they engaged in an engineering design process with a parent. These identity enactments included (a) possessing knowledge and authority to make decisions regarding the development of their self‐identified engineering problem and prototype; (b) questioning and challenging adult ideas, solutions, and construction of prototypes; and (c) documenting and communicating their thinking regarding the engineering design through sketches and notes.

    Conclusions

    The significance of this study lies in its potential to change the landscape of those who pursue an engineering career and to contribute to the limited research and ongoing conversations about how to foster environments that support families in creative and collaborative learning specific to the engineering discipline.

     
    more » « less
  4. Abstract Background

    Identifying as an engineer is essential for belonging and student success, yet the social context and professional norms make it more difficult for some students to establish an identity as an engineer.

    Purpose/Hypothesis

    This study investigated whether first‐generation college students' funds of knowledge supported their engineering role identity.

    Design/Methods

    Data came from a survey administered across the United States western, southern, and mountain regions in the fall semester of 2018. Only the sample of students who indicated they were the first in their families to attend college was used in the analysis (n = 378). Structural equation modeling was used to understand how first‐generation college students' funds of knowledge supported their engineering role identity; measurement invariance was examined to ensure that the model was valid for women and men alike.

    Results

    First‐generation college students' funds of knowledge individually supported the components of the engineering role identity development process. Tinkering knowledge from home and perspective‐taking helped inform interest and performance/competence beliefs. First‐generation college students' bids for external recognition were supported through their mediational skills, their connecting experiences, and their local network of college friends. The bundle of advice, resources, and emotional support from family members was the only fund of knowledge that directly supported students' perceptions of themselves as engineers.

    Conclusions

    The relationships we established between first‐generation college students' funds of knowledge and emerging engineering role identities call for engineering educators to integrate students' funds of knowledge into engineering learning and to broaden disciplinary norms of what counts as engineering‐relevant knowledge.

     
    more » « less
  5. Abstract Background

    COVID‐19 has spurred a global crisis that has disrupted everyday lives and impacted the traditional methods, experiences, and abilities of higher education institutions' students, faculty, and staff, especially at Historically Black Colleges and Universities (HBCUs).

    Purpose/Hypothesis

    Given the pressing need demonstrated by the National Academies to advance the utilization of science, technology, engineering, and mathematics (STEM) education at HBCUs, this study aimed to explore the abrupt transition to remote teaching and learning at HBCUs guided by the following research question: How has COVID‐19 impacted the success and persistence of engineering students, faculty, and staff at HBCUs?

    Design/Methods

    Three surveys were developed, tested, piloted, and sent to HBCU stakeholders using a snowball sampling approach via email and social media outreach.

    Results

    Of the 171 student respondents (126 engineering majors), 79% agreed that not being able to access faculty in person affected their academic performance. Additionally, across all HBCU stakeholders' surveys, students had a statistically significant higher response when asked if the transition to virtual learning increased their overall levels of stress and anxiety.

    Conclusions

    During a global pandemic, HBCUs continue to provide a culture of support and inclusion for students, faculty, and staff in engineering. Increased stress levels experienced by students indicate that a safe and adequate transition back to campus is essential for their social and academic persistence. Due to the well‐documented inequities HBCUs faced before the pandemic, the impact of this unprecedented on their continued contributions toward broadening participation in engineering for students should be further explored.

     
    more » « less