skip to main content


Title: WIP: Think-aloud interviews for assessment of engineering students' opportunities to practice professional skills
As the need for interdisciplinary collaboration increases, industry needs engineers who are not only affluent in technical engineering skills but also efficient in skills such as communication, problem-solving, engineering ethics, and business management. As a result, engineering programs are tasked with providing students with sufficient opportunities to develop non-technical professional skills to better prepare them for the workforce. Previous research has focused on exploring how and where students tend to develop profession skills and assessments have been established to measure the level of professional skills. However, without a means to measure whether students are getting sufficient opportunities for development, it is hard for educators and engineering programs to determine whether or where scaffolding are needed. We developed an instrument to assess undergraduate engineering students’ opportunities for professional skill development. To increase content validity, we conducted 20 think-aloud interviews with students from a large Midwestern university. The aim of this WIP is two-fold. We present the preliminary results of the think-aloud interview to determine what changes need to be made to existing items and what emerging themes appear regarding to participants’ professional skill development opportunities. After thematic analysis of the interview transcripts, we revised 10 items by simplifying the grammar or altering certain words that tend to confuse participants or carry negative connotations. We found that, compared to students who have only been involved in class projects, those with co-curricular experiences tend to report more opportunities in skills related to business management principles and problem-solving skills. Co-curricular activities were also the most referenced in building communication skills. Our next step will be piloting the instrument across multiple institutions and conducting validation analysis.  more » « less
Award ID(s):
2129282
NSF-PAR ID:
10343868
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
ASEE annual conference exposition proceedings
ISSN:
2153-5868
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. There is little research or understanding of curricular differences between two- and four-year programs, career development of engineering technology (ET) students, and professional preparation for ET early career professionals [1]. Yet, ET credentials (including certificates, two-, and four-year degrees) represent over half of all engineering credentials awarded in the U.S [2]. ET professionals are important hands-on members of engineering teams who have specialized knowledge of components and engineering systems. This research study focuses on how career orientations affect engineering formation of ET students educated at two-year colleges. The theoretical framework guiding this study is Social Cognitive Career Theory (SCCT). SCCT is a theory which situates attitudes, interests, and experiences and links self-efficacy beliefs, outcome expectations, and personal goals to educational and career decisions and outcomes [3]. Student knowledge of attitudes toward and motivation to pursue STEM and engineering education can impact academic performance and indicate future career interest and participation in the STEM workforce [4]. This knowledge may be measured through career orientations or career anchors. A career anchor is a combination of self-concept characteristics which includes talents, skills, abilities, motives, needs, attitudes, and values. Career anchors can develop over time and aid in shaping personal and career identity [6]. The purpose of this quantitative research study is to identify dimensions of career orientations and anchors at various educational stages to map to ET career pathways. The research question this study aims to answer is: For students educated in two-year college ET programs, how do the different dimensions of career orientations, at various phases of professional preparation, impact experiences and development of professional profiles and pathways? The participants (n=308) in this study represent three different groups: (1) students in engineering technology related programs from a medium rural-serving technical college (n=136), (2) students in engineering technology related programs from a large urban-serving technical college (n=52), and (3) engineering students at a medium Research 1 university who have transferred from a two-year college (n=120). All participants completed Schein’s Career Anchor Inventory [5]. This instrument contains 40 six-point Likert-scale items with eight subscales which correlate to the eight different career anchors. Additional demographic questions were also included. The data analysis includes graphical displays for data visualization and exploration, descriptive statistics for summarizing trends in the sample data, and then inferential statistics for determining statistical significance. This analysis examines career anchor results across groups by institution, major, demographics, types of educational experiences, types of work experiences, and career influences. This cross-group analysis aids in the development of profiles of values, talents, abilities, and motives to support customized career development tailored specifically for ET students. These findings contribute research to a gap in ET and two-year college engineering education research. Practical implications include use of findings to create career pathways mapped to career anchors, integration of career development tools into two-year college curricula and programs, greater support for career counselors, and creation of alternate and more diverse pathways into engineering. Words: 489 References [1] National Academy of Engineering. (2016). Engineering technology education in the United States. Washington, DC: The National Academies Press. [2] The Integrated Postsecondary Education Data System, (IPEDS). (2014). Data on engineering technology degrees. [3] Lent, R.W., & Brown, S.B. (1996). Social cognitive approach to career development: An overivew. Career Development Quarterly, 44, 310-321. [4] Unfried, A., Faber, M., Stanhope, D.S., Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineeirng, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7), 622-639. [5] Schein, E. (1996). Career anchors revisited: Implications for career development in the 21st century. Academy of Management Executive, 10(4), 80-88. [6] Schein, E.H., & Van Maanen, J. (2013). Career Anchors, 4th ed. San Francisco: Wiley. 
    more » « less
  2. POSTER. Presented at the Symposium (9/12/2019) Abstract: The Academy of Engineering Success (AcES) employs literature-based, best practices to support and retain underrepresented students in engineering through graduation with the ultimate goal of diversifying the engineering workforce. AcES was established in 2012 and has been supported via NSF S-STEM award number 1644119 since 2016. The 2016, 2017, and 2018 cohorts consist of 12, 20, and 22 students, respectively. Five S-STEM supported scholarships were awarded to the 2016 cohort, seven scholarships were awarded to students from the 2017 cohort, and six scholarships were awarded to students from the 2018 cohort. AcES students participate in a one-week summer bridge experience, a common fall semester course focused on professional development, and a common spring semester course emphasizing the role of engineers in societal development. Starting with the summer bridge experience, and continuing until graduation, students are immersed in curricular and co-curricular activities with the goals of fostering feelings of institutional inclusion and belonging in engineering, providing academic support and student success skills, and professional development. The aforementioned goals are achieved by providing (1) opportunities for faculty-student, student-student, and industry mentor-student interaction, (2) academic support, and student success education in areas such as time management and study skills, and (3) facilitated career and major exploration. Four research questions are being examined, (1) What is the relationship between participation in the AcES program and participants’ academic success?, (2) What aspects of the AcES program most significantly impact participants’ success in engineering, (3) How do AcES students seek to overcome challenges in studying engineering, and (4) What is the longitudinal impact of the AcES program in terms of motivation, perceptions, feelings of inclusion, outcome expectations of the participants and retention? Students enrolled in the AcES program participate in the GRIT, LAESE, and MSLQ surveys, focus groups, and one-on-one interviews at the start and end of each fall semester and at the end of the spring semester. The surveys provide a measure of students’ GRIT, general self-efficacy, engineering self-efficacy, test anxiety, math outcome efficacy, intrinsic value of learning, inclusion, career expectations, and coping efficacy. Focus group and interview responses are analyzed in order to answer research questions 2, 3, and 4. Survey responses are analyzed to answer research question 4, and institutional data such as GPA is used to answer research question 1. An analysis of the 2017 AcES cohort survey responses produced a surprising result. When the responses of AcES students who retained were compared to the responses of AcES students who left engineering, those who left engineering had higher baseline values of GRIT, career expectations, engineering self-efficacy, and math outcome efficacy than those students who retained. A preliminary analysis of the 2016, 2017, and 2018 focus group and one-on-one interview responses indicates that the Engineering Learning Center, tutors, organized out of class experiences, first-year seminar, the AcES cohort, the AcES summer bridge, the AcES program, AcES Faculty/Staff, AcES guest lecturers, and FEP faculty/Staff are viewed as valuable by students and cited with contributing to their success in engineering. It is also evident that AcES students seek help from peers, seek help from tutors, use online resources, and attend office hours to overcome their challenges in studying engineering. 
    more » « less
  3. CONTEXT - Judging the feasibility of solutions has become an increasingly important engineering skill as engineering problem solving has become more complex and technology-dependent. Engineering education must take care to foster engineering judgement in our students to produce robust problem solvers primed to critically evaluate and interpret output. Our work uses expertise development and dual-cognition processing theories (Dreyfus & Dreyfus, 1980; Smith, 2009; Simon, 1987) to frame such engineering judgement as engineering intuition or the ability to assess the outcome of an engineering solution and predict outcomes within an engineering scenario (Miskioğlu and Martin, 2019). PURPOSE OR GOAL - Our overarching goal is to create classroom interventions that explicitly recognize and enhance the development of engineering intuition. Accomplishing this goal requires a means of measuring engineering intuition before and after such interventions. This paper discusses our process to develop the Predicting and Evaluating Engineering Problem Solving (PEEPS) tool for measuring engineering intuition. APPROACH OR METHODOLOGY/METHODS - PEEPS is built directly on our prior qualitative work with practicing engineers, which revealed the construct of engineering intuition (Aaron et al., 2020). The emergent findings were combined with questions adapted from the Concept Assessment Tool for Statics (Steif & Dantzler, 2005) to create a preliminary survey assessing intuition. Additional items asked participants to assess their level of confidence in their answers. The survey was designed such that the statics problems could be switched out for other forms of engineering problems. Think-aloud sessions were used to check face validity and usability prior to full deployment in Spring 2021. ACTUAL OR ANTICIPATED OUTCOMES - This study details the process used to create PEEPS. Modifications were made following 19 think aloud sessions. The initial deployment in Spring 2021 resulted in 88 completed responses with responses primarily coming from white, male, aerospace engineering students who had previously performed well in their statics courses. CONCLUSIONS/RECOMMENDATIONS/SUMMARY - This work showcases a new survey designed to assess the engineering intuition of engineering students. Next steps include expanding the work to a more diverse sample of engineering students, further validity checks of the instrument, and pairing the instrument with newly created educational interventions designed to better foster engineering intuition development in students. KEYWORDS - engineering judgement, problem solving, survey development 
    more » « less
  4. CONTEXT - Judging the feasibility of solutions has become an increasingly important engineering skill as engineering problem solving has become more complex and technology-dependent. Engineering education must take care to foster engineering judgement in our students to produce robust problem solvers primed to critically evaluate and interpret output. Our work uses expertise development and dual-cognition processing theories (Dreyfus & Dreyfus, 1980; Smith, 2009; Simon, 1987) to frame such engineering judgement as engineering intuition or the ability to assess the outcome of an engineering solution and predict outcomes within an engineering scenario (Miskioğlu and Martin, 2019). PURPOSE OR GOAL - Our overarching goal is to create classroom interventions that explicitly recognize and enhance the development of engineering intuition. Accomplishing this goal requires a means of measuring engineering intuition before and after such interventions. This paper discusses our process to develop the Predicting and Evaluating Engineering Problem Solving (PEEPS) tool for measuring engineering intuition. APPROACH OR METHODOLOGY/METHODS - PEEPS is built directly on our prior qualitative work with practicing engineers, which revealed the construct of engineering intuition (Aaron et al., 2020). The emergent findings were combined with questions adapted from the Concept Assessment Tool for Statics (Steif & Dantzler, 2005) to create a preliminary survey assessing intuition. Additional items asked participants to assess their level of confidence in their answers. The survey was designed such that the statics problems could be switched out for other forms of engineering problems. Think-aloud sessions were used to check face validity and usability prior to full deployment in Spring 2021. ACTUAL OR ANTICIPATED OUTCOMES - This study details the process used to create PEEPS. Modifications were made following 19 think aloud sessions. The initial deployment in Spring 2021 resulted in 88 completed responses with responses primarily coming from white, male, aerospace engineering students who had previously performed well in their statics courses. CONCLUSIONS/RECOMMENDATIONS/SUMMARY - This work showcases a new survey designed to assess the engineering intuition of engineering students. Next steps include expanding the work to a more diverse sample of engineering students, further validity checks of the instrument, and pairing the instrument with newly created educational interventions designed to better foster engineering intuition development in students. KEYWORDS - engineering judgement, problem solving, survey development 
    more » « less
  5. This full research paper presents the exploratory factor analysis (EFA) results for the Professional Skill Opportunities survey (PSO) we designed to measure undergraduate engineering students’ opportunities to develop and practice important nontechnical professional skills. We use Dall’alba’s “ways of being” as the theoretical framework for the survey development and generated construct definitions based on past literature, expert review, and cognitive think-aloud interviews. We administered the survey in an engineering class at the beginning of the Spring 2022 semester. After comparing the three EFA models based on goodness-of-fit indices and model interpretability aligned to the theoretical model, the researchers selected a five-factor model. The EFA result and literature on leadership and teamwork showed these two skills are highly interrelated and could be combined into one construct to stress the “sharedness” of leadership responsibilities in teams. The result allowed our team to refine our item pool, revise construct definitions, and generate new items. In future work, we will administer the revised PSO survey to the same population at the end of the same semester as further validation. We also plan to explore the relationship between professional skill development opportunities and students’ social support. We hope the PSO survey can provide educators and institutions a means to offer scaffoldings and more opportunities for professional skill development and better prepare students for the engineering workforce. 
    more » « less