skip to main content


Title: Assessing learning processes rather than outcomes: using critical incidents to explore student learning abroad
Abstract

There is an increasing emphasis on assessing student learning outcomes from study abroad experiences, but this assessment often focuses on a limited range of outcomes and assessment methods. We argue for shifting to assessing student learningprocessesin study abroad and present the critical incident technique as one approach to achieve this goal. We demonstrate this approach in interviews with 79 students across a range of global engineering programs, through which we identified 173 incidents which were analyzed to identify common themes. This analysis revealed that students described a wide range of experiences and outcomes from their time abroad. Students’ experiences were messy and complex, making them challenging to understand through typical assessment approaches. Our findings emphasize the importance of using a range of assessment approaches and suggest that exploring students’ learning processes in addition to learning outcomes could provide new insights to inform the design of study abroad programs.

 
more » « less
Award ID(s):
1658604
NSF-PAR ID:
10363876
Author(s) / Creator(s):
;
Publisher / Repository:
Springer Science + Business Media
Date Published:
Journal Name:
Higher Education
Volume:
85
Issue:
2
ISSN:
0018-1560
Page Range / eLocation ID:
p. 341-357
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Student engagement, especially among Engineering and Computer science majors (E/CS), has been a priority for researchers. Although considerable efforts have been made to improve college students' engagement and interest, underrepresented minority groups and first-generation students are still at risk of dropping out of engineering majors due to lack of inclusiveness, motivation, and other related factors. According to Kuh (2008), student participation in High-Impact Educational Practices (HIEP) is correlated with student outcomes such as persistence, performance, achievement, and intent to complete their current major. The present study reviews the existing National Survey of Student Engagement (NSSE, 2012, 2017) data from two western land-grant universities to fully capture participation through the survey of first-year students and seniors (N = 674). The HIEP considered include service-learning, learning communities, research with faculty, internship or field experience, study abroad, and culminating senior experience. These practices are designed to encourage meaningful interactions between faculty and students, foster collaboration with students within different demographics groups, and facilitate learning outside the classroom. Insights were gleaned from how the students interacted with HIEP based on special characteristics such as sex, race, age, enrollment status, and residence. The purpose of the present study is to examine the extent to which E/CS students participate in HIEP and its effects on student outcomes. This study also offers comparisons or possible relationships between student demographics, student success, and HIEP involvement. For example, the participation rates of HIEP on different engineering and computer science majors, including civil, chemical, electrical, mechanical, and materials engineering, etc., are analyzed to examine the practices that work for a particular E/CS major. The present study reports findings from NSSE 2012 and 2017 surveys. Results show that among the E/CS seniors, service-learning, learning community, and study abroad program are the HIEP with the lowest participation rate with 41% (service-learning), 59% (learning community), and 68% (study abroad program), indicating that they do not plan to engage in these practices in their senior year. Conversely, internships and culminating senior experiences had the most participation among E/CS seniors with 52% (internships) and 68% (culminating senior experiences. Interestingly, first-year students showed a significant interest to participate in the following HIEP: internships, study abroad programs, and culminating senior experiences – with 76% (internships), 47% (study abroad program), and 68% (culminating senior experiences) indicating plans to engage in these practices. Finally, findings show that participation or engagement in HIEP is a significant predictor of student learning outcomes. Findings of this review may serve as a guide for future research in E/CS student participation in HIEP. The paper concludes with theoretical and practical implications of the findings on student engagement and learning. Key words: NSSE, high impact educational practices, Engagement 
    more » « less
  2. Previous studies have convincingly shown that traditional, content-centered, and didactic teaching methods are not effective for developing a deep understanding and knowledge transfer. Nor does it adequately address the development of critical problem-solving skills. Active and collaborative instruction, coupled with effective means to encourage student engagement, invariably leads to better student learning outcomes irrespective of academic discipline. Despite these findings, the existing construction engineering programs, for the most part, consist of a series of fragmented courses that mainly focus on procedural skills rather than on the fundamental and conceptual knowledge that helps students become innovative problem-solvers. In addition, these courses are heavily dependent on traditional lecture-based teaching methods focused on well-structured and closed-ended problems that prepare students to plug variables into equations to get the answer. Existing programs rarely offer a systematic approach to allow students to develop a deep understanding of the engineering core concepts and discover systematic solutions for fundamental problems. Without properly understanding these core concepts, contextualized in domain-specific settings, students are not able to develop a holistic view that will help them to recognize the big picture and think outside the box to come up with creative solutions for arising problems. The long history of empirical learning in the field of construction engineering shows the significant potential of cognitive development through direct experience and reflection on what works in particular situations. Of course, the complex nature of the construction industry in the twenty-first century cannot afford an education through trial and error in the real environment. However, recent advances in computer science can help educators develop virtual environments and gamification platforms that allow students to explore various scenarios and learn from their experiences. This study aims to address this need by assessing the effectiveness of guided active exploration in a digital game environment on students’ ability to discover systematic solutions for fundamental problems in construction engineering. To address this objective, through a research project funded by the NSF Division of Engineering Education and Centers (EEC), we designed and developed a scenario-based interactive digital game, called Zebel, to guide students solve fundamental problems in construction scheduling. The proposed gamified pedagogical approach was designed based on the Constructivism learning theory and a framework that consists of six essential elements: (1) modeling; (2) reflection; (3) strategy formation; (4) scaffolded exploration; (5) debriefing; and (6) articulation. We also designed a series of pre- and post-assessment instruments for empirical data collection to assess the effectiveness of the proposed approach. The proposed gamified method was implemented in a graduate-level construction planning and scheduling course. The outcomes indicated that students with no prior knowledge of construction scheduling methods were able to discover systematic solutions for fundamental scheduling problems through their experience with the proposed gamified learning method. 
    more » « less
  3. Abstract Background

    Engineers operate in an increasingly global environment, making it important that engineering students develop global engineering competency to prepare them for success in the workplace. To understand this learning, we need assessment approaches that go beyond traditional self‐report surveys. A previous study (Jesiek et al.,Journal of Engineering Education2020; 109(3):1–21) began this process by developing a situational judgment test (SJT) to assess global engineering competency based in the Chinese context and administering it to practicing engineers.

    Purpose

    We built on this previous study by administering the SJT to engineering students to explore what prior experiences related to their SJT performance and how their SJT performance compared with practicing engineers' performance on the SJT.

    Method

    Engineering students completed a survey including the SJT and related self‐report survey instruments. We collected data from three groups of students: those who had studied abroad in China; those who had studied abroad elsewhere; and those who had not studied abroad.

    Results

    We found that students' SJT performance did not relate to their scores on the self‐report instruments, but did relate to their participation in study abroad programs. The students also performed better on the SJT when compared to the practicing engineers.

    Conclusions

    Our results highlight the need to use multiple forms of assessment for global engineering competence. Although building evidence for the validity of the Global Engineering Competency China SJT is an ongoing process, this data collection technique may provide new insights on global engineering competency compared to traditionally used assessments.

     
    more » « less
  4. An increasingly global environment expects graduating Engineering students to perform, live and work across cultures. Most intercultural competence research and associated global engineering education is focused on developing the global engineering skill set through long-term travel experiences such as study abroad programs. These programs can be expensive from both a time and money standpoint, limiting the participation to more privileged members of a community, and are not scalable to support broader participation. This work-in-progress addresses this research gap by focusing on the development of the students’ global learner mindset without requiring extensive travel. The project will investigate four different global engagement interventions, including the use of engineering case studies, the intentional formation of multi-national student teams, a Collaborative Online International Learning (COIL) research project, and a community engaged project within a short course. These interventions can be used to develop a holistic global learner mindset and global engineering education approach to foster global competence in undergraduate engineering students. The four global engagement interventions will be grounded in the global engineering competency (GEC) theoretical framework and assessed for their ability to foster a global learner mindset in engineering students. A mixed-methods approach will be used to assess students’ global learner mindset and skill set. This research will use the Global Engagement Survey (GES), the Global Engineering Competency Scale (GECS) and specific questions developed by the researchers to evaluate improvements in the participating students’ global engineering skill set and answer specific research questions including: 1) To what extent can global competence be developed in engineering students through the use of the proposed global engagement interventions; and 2) what are the relative strengths of each of the proposed global engagement interventions in developing global engineering competence? Combined, these research measures will provide both an accurate picture of how each global engagement intervention impacts the formation of a global learner mindset in engineering education, and also its associated ability to develop and/or improve global engineering skills. The outcomes of this study will generate valuable knowledge to understand how each global engagement intervention impacts the formation of global engineering competence. In this work-in-progress study, the authors discuss the four global engagement interventions with specific learning objectives that have been mapped to the overall student outcomes for the project. These objectives have also been mapped to the GES and GECS instruments. Finally the faculty members have developed qualitative tools to augment the GES and GECS to identify the global engineering skill sets each intervention is generating. This paper lays the foundation before implementing the interventions and performing their associated assessments over the several subsequent semesters. 
    more » « less
  5. An increasingly global environment expects graduating Engineering students to perform, live and work across cultures. Most intercultural competence research and associated global engineering education is focused on developing the global engineering skill set through long-term travel experiences such as study abroad programs. These programs can be expensive from both a time and money standpoint, limiting the participation to more privileged members of a community, and are not scalable to support broader participation. This work-in-progress addresses this research gap by focusing on the development of the students’ global learner mindset without requiring extensive travel. The project will investigate four different global engagement interventions, including the use of engineering case studies, the intentional formation of multi-national student teams, a Collaborative Online International Learning (COIL) research project, and a community engaged project within a short course. These interventions can be used to develop a holistic global learner mindset and global engineering education approach to foster global competence in undergraduate engineering students. The four global engagement interventions will be grounded in the global engineering competency (GEC) theoretical framework and assessed for their ability to foster a global learner mindset in engineering students. A mixed-methods approach will be used to assess students’ global learner mindset and skill set. This research will use the Global Engagement Survey (GES), the Global Engineering Competency Scale (GECS) and specific questions developed by the researchers to evaluate improvements in the participating students’ global engineering skill set and answer specific research questions including: 1) To what extent can global competence be developed in engineering students through the use of the proposed global engagement interventions; and 2) what are the relative strengths of each of the proposed global engagement interventions in developing global engineering competence? Combined, these research measures will provide both an accurate picture of how each global engagement intervention impacts the formation of a global learner mindset in engineering education, and also its associated ability to develop and/or improve global engineering skills. The outcomes of this study will generate valuable knowledge to understand how each global engagement intervention impacts the formation of global engineering competence. In this work-in-progress study, the authors discuss the four global engagement interventions with specific learning objectives that have been mapped to the overall student outcomes for the project. These objectives have also been mapped to the GES and GECS instruments. Finally the faculty members have developed qualitative tools to augment the GES and GECS to identify the global engineering skill sets each intervention is generating. This paper lays the foundation before implementing the interventions and performing their associated assessments over the several subsequent semesters. 
    more » « less