The ever-increasing globalization entails ever-changing requirements for the education and qualification of new generations of engineering graduates. Both employers as well as educational institutions emphasize the need to maintain globally competitive, as a direct result from our globalized society and economy. Facilitating solutions to complex problems, while being innovative and knowledgeable, is made possible through international research collaborations (Lynn and Salzman 2006), which open the door to world-class equipment, laboratories, unique sources of ideas, concepts, and methods (Leak et al. 2018, Wagner andLeydesdorff 2005). Furthermore, as reported by the British Council (2012), international collaborations often result in high-quality research. Besides positive effects on conducted research, studies have shown tremendous impacts on the personal and professional development, intellectual growth, academic development, and intercultural competence of the participating students' development (Ingraham and Peterson 2004, Leak et al. 2018, Waldbaum 1996).
During their time abroad, students have to face and overcome several challenges. International experiences are associated with a lot of planning and making decisions. Students need to learn to interact in a foreign culture and in a foreign language, they encounter surprises, are confronted with problems, and have to deal with unfamiliar situations and environments. However, these challenges are important for the students' growth and development (Gmelch 1997). Thus, preparation and orientation activities are vital for increasing the impact of an intervention on students (Dwyer 2004, Sikkema andNiyekawa 1987).
While it is widely agreed upon that studying abroad offers great benefits, the state of the literature is far less clear when it comes to international research experiences (Black and Duhon 2006, Clarke et al. 2009, Doyle 2009). Unfortunately, the existing body of knowledge pertaining to international student experiences is dominated by contrast-lacking and subjective results (Kirk 2008, Royal Society 2011), which can be attributed to the lack of a common methodology.
Our research is aiming to design an international research experience capable of effectively, and accurately measuring the impact of such interventions. Based on extensive literature research, the current situation of study designs is critically assessed in order to design a study capable of capturing all aspects of an international research experience. In this context, a common terminology is developed to serve as a general basis for the disagreeing community to precisely describe and define measurable competencies.
Studies pertaining to this research were identified using a systematic review process. The databases Scopus and Education Resources Information Center (ERIC) were used to identify publications containing the search terms "international research experience", "research abroad", "study abroad", "international student experience", "student abroad", and "education abroad". Studies published in languages other than English were excluded, and only peer-reviewed journal articles as well as doctoral dissertations were considered. This resulted in 11,155 articles, published between 1996 and 2019, containing at least one of the keywords in the abstract or titles. This time frame was chosen to get a comprehensive overview of the literature to capture as many potential impacts as possible. Further selection criteria were applied to narrow down the number of applicable articles:
(1) college or university students;
(2) empirical studies on international student experiences, whereby these experiences comprise study abroad and international research experiences; and (3) analysis of competencies gained or improved during the intervention. Due to the amount of literature found, a pre-selection was made based on the titles. Thereby, 8.342 publications were sorted out, since they were not relevant to the topic (e.g., not related to international student or research experiences) resulting in 2.813 publications. The abstracts of the remaining articles were examined according to the abovementioned criteria. Further publications that did not meet these criteria could be excluded. For example, articles were excluded that analyzed post-graduate, teacher, or student advisor experiences, student participation motivation or choice of host university, effect of duration, integration of international experiences into the curricula, theoretical studies, or measures to increase the students´ interest in participating in an international experience. As a result, 40 empirical studies were selected based on these criteria and were analyzed using open and axial coding (Creswell and Creswell 2018), whereby three coding schemes were applied to categorize the data:
(1) competencies that are impacted by the intervention;
(2) research methods including measures, analysis, validation, data collection timing, and data sources; and (3) study characteristics such as duration, purpose, or number of participants. To identify the main competencies that are impacted by an international experience, competencies that students developed or gained through their intervention were extracted from the studies. Competencies pertaining to the same topic were grouped into subcompetencies and these subcompetencies into key competencies. In further rounds of coding, these competencies and subcompetencies were re-examined and re-defined resulting in 17 subcompetencies and five competencies include intercultural competence, personal development, intellectual growth, professional development, and academic development.
Lastly, additional literature regarding the design of social research was studied to identify applicable strategies capable of measuring the impact of international student experiences.
Effectively measuring the impact on students is as challenging as it is crucial, and generalizable findings can only be reported if the entire study design including the analysis and interpretation is done comprehensively. In order to improve on existing studies in the field, different and multiple methods, analysts, sources, and theories need to be combined to strengthen validity and minimize bias. Particularly, as defined by Patton (2002), triangulation is a versatile approach that should be considered. Four types of triangulation can be differentiated: (i) triangulation of methods, (ii) data sources, (iii) analysts, and (iv) theories. Especially in social studies, problems are typically too complex and comprehensive to be investigated by just one data collection method (Creswell and Creswell 2018). The triangulation of data sources can be ensured by collecting data multiple times at multiple occasions by varying means. Data analysis is improved through a multi-researcher reviewing of findings, while different theoretical perspectives should be applied. Systematic bias and distortion is significantly reduced by applying all four methods of triangulation (Patton 2002).
A well-designed study is reflected in both internal and external validity, where internal validity describes the credibility of the reported findings, while external validity characterizes the generalizability and applicability of the findings to a group or population. Additionally, measurement errors have to be accounted for, which means every measure has to reliably measure the unit it is supposed to while also be independent of the data collection timing (de Vaus 2001).
Following the idea of triangulating methods, both quantitative and qualitative methods need to be considered and implemented in a well-designed research plan. Quantitative methods aim to generalize and test relationships between variables purely through means of numerical data and statistics, which is an important base upon which generally applicable conclusions can be drawn with a high level of confidence. Qualitative research, on the other hand, tries to grasp the meaning behind a problem, allowing to study experiences and analyzing words and meanings (Creswell and Creswell 2018). This kind of method prefers an inductive strategy style by investigating a few individuals. As mentioned earlier, the complex nature of social studies essentially requires utilizing a holistic approach by mixing and combining multiple methods, enabling the complete understanding and general application of the researched intervention (Creswell and Creswell 2018).
The data collection method has to match the research purpose, for example, qualitative methods such as interviews and observations are well suited for inductive, subjective, and contextual research purposes (Morgan 2014). Contrasting purposes, such as deductive, objective, and general, pair well with quantitative strategies (Kumar 2014, Morgan 2014). In any case, the participants' response quality is highly depended on the data collection design (Kumar 2014), which at the same time emphasizes the importance of relying on multiple kinds of data (Pike 1995, Podsakoff andOrgan 1986). Further objectivity can be introduced by consulting sources close to the participant, i.e. domestic and host advisors, or domestic and international peer students. Additional control groups can be included to strengthen the validity. This strategy needs to be correctly implemented into the data collection schedule to maximize the usefulness. Generally, the data collection schedule needs to be rather wide, and comprises at least preintervention, right after the intervention (post), and months after (post-post) the intervention. This schedule facilitates the assessment of long-term impacts, where the pre-collection serves as the baseline (Kumar 2014).
As mentioned in the methodology section, five competencies have been identified and selected for measuring the impact of international student experiences (intercultural competence, personal development, intellectual growth, professional development, and academic development). This choice allows the focus of the study to be broad and general.
Lastly, specifically regarding international student experiences, the length of the intervention needs consideration, as Koester (1985) reported optimal lengths between six and twelve months. However, shorter interventions' impacts can be strengthened by providing solid preparation and post-reflection activities (Dwyer 2004, Sikkema andNiyekawa 1987).
Based on the 40 studies identified, as outlined in the methodology section, a critical analysis and assessment of past studies is discussed in the following, with a particular focus on the study design. Besides the design itself, student competencies reported are analyzed on the basis outlined in the previous section.
Student competencies. Researchers reported about several competencies that students could gain or improve during their time abroad. These competencies include:
-Intercultural Competence: interacting effectively in cross-cultural environments; recognizing own and other cultures -Personal Development: developing and recognizing own potential and abilities; knowing and building own identity -Intellectual Growth: developing the ability to think and reason; learning a language -Professional Development: maintaining and gaining professional skills and knowledge;
getting prepared for the professional life; advancing a career -Academic Development: developing skills and knowledge related to own research area;
producing research outcomes Scholars especially reported impacts on the students' intercultural development (80%), whereas academic development was only considered by 15% of the reviewed studies. Table 1 presents an overview of the competencies and subcompetencies that were measured by scholars. The distribution of subcompetencies also demonstrates clearly how the current body of knowledge is not comprehensive and hence not capable of reflecting general implications on students' developments. Increased knowledge and appreciation for other cultures (Chieffo and Griffiths 2004, Dwyer 2004, Norris and Gillespie 2009); extended worldviews (Hutchins 1996, Orahood et al. 2008, Witkowsky and Mendez 2018); increased understanding of own culture (Dwyer 2004, Ingraham and Peterson 2004, King and Ruiz-Gelices 2003); and increased interest in world affairs (Braskamp et al. 2009, Hadis 2005, Zhai 2000) Intercultural Sensitivity (43%)
Respect for other cultures (Doyle 2009, Kitsantas and Meyers 2001, Zhai 2000); increased cross-cultural tolerance (Black andDuhon 2006, Chieffo andGriffiths 2004); and better understanding of other cultures (Braskamp et al. 2009, Doyle 2009, Zhai 2000) Openness to Diversity (38%) Greater appreciation of diversity (Clarke et al. 2009, Wilson 2014); and increased diversity of friends and contacts (Dwyer 2004, Jesiek et Increased interest in learning a foreign language (Ingraham and Peterson 2004); and improved oral communication skills (Berg 2009, Dwyer 2004) Study Progress (30%)
Gaining course-related knowledge (Bender et al. 2009, Chieffo andGriffiths 2004); and increased interest in further studies (Dwyer 2004, King andRuiz-Gelices 2003) Creative Growth (8%)
Higher appreciation of arts (Chieffo andGriffiths 2004, Ingraham andPeterson 2004); and becoming more creative (Leak et al. 2018) Intellectual/Cognitive Skills (15%) Improved problem-solving skills (Gmelch 1997, Orahood et al. 2004, Wilson et al. 2019); and increased abilities in dealing with the unexpected (Ingraham and Peterson 2004) Professional Development (50%) Networking (10%)
Expanded networks (Chang et al. 2009, Leak et al. 2018, Norris and Gillespie 2009) Professional Skills (43%) Improved leadership skills (Chang et al. 2009); verbal and non-verbal communication skills (Chang et al. 2009, Wilson et al. 2019, Zhai 2000); presentation skills (Bender et al. 2017); interpersonal skills (Orahood et. al. 2008, Wilson et al. 2019); and self-efficiency (Jesiek et al. 2014) Career Prospects (40%)
Changing career directions (Leak et al. 2018, Norris andGillespie 2009); increased interest in working for an internationally oriented company (Ingraham and Peterson 2004;Orahood et al. 2004 (Leak et al. 2018) Research Outcome (8%) Increased research productivity (Leak et al. 2018, Wilson et al. 2019); increased number of patents and publications; and changes in research direction (Leak et al. 2018) The competencies previously presented are especially relevant since they have a huge impact on the students' professional or academic career (Clarke et al. 2009, Franklin 2010, Jesiek et al. 2014). Spending time in a foreign country is not only accompanied by valuable experiences and social opportunities, but also by challenges the students have to face (Gmelch 1997). Dealing with the unexpected and overcoming these challenges prepares students for their future careers. Nevertheless, students need to reflect on their achievements to effectively communicate them during their application process (Orahood et al. 2004).
In the following sections, the design methods are discussed including data analysis, validation, and characteristics.
Competencies. Most published studies failed to measure the impact of international interventions, particularly conspicuous when looking at the average of only 2.6 competencies considered in the reviewed literature. As a matter of fact, 33% of all studies only measured one competency, whereas 10% of the studies considered all five. A similar situation becomes apparent regarding sub-competencies, where just 4.9 out of 17 were considered on average.
Data collection tools. About half of all studies (48%) only employed one data collection tool, and nearly all relied exclusively on student surveys. Two methods were used by 28%, three methods were employed by just 20%, while only isolated cases of four and five employed methods could be found. This heterogeneity prevents a comprehensive understanding of impacts on competencies.
Data collection characteristics. Although it is commonly agreed upon using a mixedmethod approach is beneficial and vital for social research (Creswell and Creswell 2018), only 40% of the reviewed publications actually followed this ideal. Similarly, a significant portion relied solely on quantitative or qualitative data, with 35% and 25%, respectively. Furthermore, collection of data has to be timed adequately and conducted at different time points to assess the entirety of the observations correctly, however, only 5% collected data (including baseline data) pre, post, and post-post intervention. Control groups in particular were only utilized by four studies. Generally, no sustained impacts could be measured with the selected data collection times and frequencies.
About 85% did rely exclusively on the participants' responses, and did not account for bias introduced by this approach. Including data from advisors and peers dramatically helps reduce biased results and hence, should be part of the data collection process.
Study validation. Close to half (45%) of the studies neglected their study's validity and reliability, which is crucial when holistic conclusions are to be made. Furthermore, most methodology sections reviewed did not include sufficient information regarding the research design, and hence, in many cases, no sound assessment was possible. An easy way to establish validity is to use control data in order to create a baseline against which any findings can be compared, unfortunately, 35% did not include any kind of control data in their study design.
International experiences. Even though six to twelve months interventions are recommended, 35% of the studies assessed interventions with a duration between 3 and 5 months, 28% examined interventions under 3 months. Only 10% assessed international experiences with durations over 5 months. The remainder (27%) did not indicate the length of the international experiences. Only four studies (10%) provided orientation activities for their students. Two of them also offered post participation activities to provide students with the opportunity to reflect on their experiences.
The globalization of the economy and, to certain extends, society, requires the workforce to become increasingly skilled, particularly soft skills are in demand. Cross-cultural work settings are to be navigated with ease; adaptability, flexibility, and communication skills are among skills characterized most valuable by employers.
International student experiences offer valuable potentials for individual growth and impact the individual on many levels and competencies. Many studies in the past attempted to scientifically measure and assess these impacts, but for the most part, these studies were poorly designed and do not allow for a comprehensive understanding. Particularly a holistic research design would be of great value in order to assess potential impacts on student competencies, which subsequently would allow for optimized international student experiences.
Applying the concept of triangulation is crucial in designing a study with high validity and reliability, whilst reducing bias as much as possible. Careful attention has to be put into selecting measures, which are validated and output data reliably. Further careful selection is needed when it comes to data collection schedules, especially longitudinal and intermediate effects require frequent data collection. Ideally, a control group is used to establish a baseline, and data is collected at least pre, post, and post-post to guarantee a comprehensive understanding.
Furthermore, data needs to be collected from multiple sources, not only the participants. In case of international interventions, domestic and host advisors as well as peer students can be leveraged to create thorough data sets. Data acquisition methods need to be designed with care to avoid invaluable outcomes, for example when formulating survey questions and interviews.
Based on these principles, a study has been designed that is capable of capturing all facets of an international research experience. A mixed methods approach is used to allow a holistic understanding of the intervention. Therefore, data is collected through a mix of surveys and interviews. To measure the impact on the students' intercultural development, the Intercultural Development Inventory (IDI) was chosen, a reliable and valid survey that has been psychometrically tested (Berg 2009). In addition, the Student Progress Survey has been developed to measure the impact on the students' academic, professional, personal, and intellectual development. These two surveys are administered before, right after, and 1 year and 2 years after the intervention to capture not only short-term but also sustained impacts. To also capture qualitative aspects of the international experience, interviews with the students are conducted after the intervention. To not only rely on self-reported data, home and host mentors of the students are asked to evaluate the performance and development of their students. In order to optimize the impact of the international experience, a length of one semester was chosen for the intervention. In addition, preparation activities before the international experience will be provided, including intercultural awareness, leadership, and ethics in engineering courses. Moreover, after the intervention, students are encouraged to publish their results together with their host mentors and to engage in professional development activities.
The field is everything but scarce of international student experience studies, however, for the most part no generalizable findings were reported. Our research puts forth a set of competencies and subcompetencies to be used as a common basis to evaluate and report on international student experiences. Furthermore, we leverage the concept of triangulation in the design of a broad, comprehensive international student experience, in order to assess impacts on students' competencies properly. This study will radically change the state of literature and allow for the first time to generalize findings and holistically draw conclusions regarding impacts on students' skills. Furthermore, these findings will allow the community to re-assess current international student experiences and allow for optimized interventions that maximize benefits. Especially short(er) interventions can profit immensely, for example through pre-and postengagements of the students to raise their awareness and understanding of newly gained skills.
© ASCEConstruction Research Congress 2022 Downloaded from ascelibrary.org by Arizona State Univ on 06/21/22. Copyright ASCE. For personal use only; all rights reserved.