skip to main content


Title: Professional Skill Opportunities Survey: Development and Exploratory Factor Analysis
This full research paper presents the exploratory factor analysis (EFA) results for the Professional Skill Opportunities survey (PSO) we designed to measure undergraduate engineering students’ opportunities to develop and practice important nontechnical professional skills. We use Dall’alba’s “ways of being” as the theoretical framework for the survey development and generated construct definitions based on past literature, expert review, and cognitive think-aloud interviews. We administered the survey in an engineering class at the beginning of the Spring 2022 semester. After comparing the three EFA models based on goodness-of-fit indices and model interpretability aligned to the theoretical model, the researchers selected a five-factor model. The EFA result and literature on leadership and teamwork showed these two skills are highly interrelated and could be combined into one construct to stress the “sharedness” of leadership responsibilities in teams. The result allowed our team to refine our item pool, revise construct definitions, and generate new items. In future work, we will administer the revised PSO survey to the same population at the end of the same semester as further validation. We also plan to explore the relationship between professional skill development opportunities and students’ social support. We hope the PSO survey can provide educators and institutions a means to offer scaffoldings and more opportunities for professional skill development and better prepare students for the engineering workforce.  more » « less
Award ID(s):
2129308
NSF-PAR ID:
10430299
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Professional skill opportunities survey: Development and exploratory factor analysis
Page Range / eLocation ID:
1 to 8
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Identity, or how people choose to define themselves, is emerging as an attractive explanation for who persists in engineering. Many studies of engineering identity build off of prior work in math and science identity, emphasizing the academic aspects of engineering. However, affect towards professional practice is also central to engineering identity development. This paper describes the methods used to create a new survey measure of individuals’ affect toward elements of engineering practice. We followed the item generation, refinement, and instrument validation steps required for psychometric validation of a new survey measure. We generated items deductively using the literature on engineering professional skills and practice and inductively based on interviews with practicing engineers, engineering graduate students, and engineering undergraduate students. We blended the inductively and deductively derived item lists to create a list of initial items for the measure. We circulated this list of items to a set of engineering and professional identity experts to establish face validity and made modifications based on their feedback. The final list included 34 items. These 34 items were administered in a questionnaire survey in the fall of 2016 to 1465 engineering undergraduates in three majors at two institutions. We conducted an exploratory factor analysis (EFA) and established internal consistency using Cronbach’s alpha on a subset of the analytical sample data (n=384). The resulting factors fit our a priori assumption of the factors theorized to characterize affect towards engineering professional practice. Using the remaining data (n=904), we conducted a confirmatory factor analysis on the reduced set of items resulting from EFA. The results indicate an emergent factor structure for affect towards elements of engineering practice. 
    more » « less
  2. POSTER. Presented at the Symposium (9/12/2019) Abstract: The Academy of Engineering Success (AcES) employs literature-based, best practices to support and retain underrepresented students in engineering through graduation with the ultimate goal of diversifying the engineering workforce. AcES was established in 2012 and has been supported via NSF S-STEM award number 1644119 since 2016. The 2016, 2017, and 2018 cohorts consist of 12, 20, and 22 students, respectively. Five S-STEM supported scholarships were awarded to the 2016 cohort, seven scholarships were awarded to students from the 2017 cohort, and six scholarships were awarded to students from the 2018 cohort. AcES students participate in a one-week summer bridge experience, a common fall semester course focused on professional development, and a common spring semester course emphasizing the role of engineers in societal development. Starting with the summer bridge experience, and continuing until graduation, students are immersed in curricular and co-curricular activities with the goals of fostering feelings of institutional inclusion and belonging in engineering, providing academic support and student success skills, and professional development. The aforementioned goals are achieved by providing (1) opportunities for faculty-student, student-student, and industry mentor-student interaction, (2) academic support, and student success education in areas such as time management and study skills, and (3) facilitated career and major exploration. Four research questions are being examined, (1) What is the relationship between participation in the AcES program and participants’ academic success?, (2) What aspects of the AcES program most significantly impact participants’ success in engineering, (3) How do AcES students seek to overcome challenges in studying engineering, and (4) What is the longitudinal impact of the AcES program in terms of motivation, perceptions, feelings of inclusion, outcome expectations of the participants and retention? Students enrolled in the AcES program participate in the GRIT, LAESE, and MSLQ surveys, focus groups, and one-on-one interviews at the start and end of each fall semester and at the end of the spring semester. The surveys provide a measure of students’ GRIT, general self-efficacy, engineering self-efficacy, test anxiety, math outcome efficacy, intrinsic value of learning, inclusion, career expectations, and coping efficacy. Focus group and interview responses are analyzed in order to answer research questions 2, 3, and 4. Survey responses are analyzed to answer research question 4, and institutional data such as GPA is used to answer research question 1. An analysis of the 2017 AcES cohort survey responses produced a surprising result. When the responses of AcES students who retained were compared to the responses of AcES students who left engineering, those who left engineering had higher baseline values of GRIT, career expectations, engineering self-efficacy, and math outcome efficacy than those students who retained. A preliminary analysis of the 2016, 2017, and 2018 focus group and one-on-one interview responses indicates that the Engineering Learning Center, tutors, organized out of class experiences, first-year seminar, the AcES cohort, the AcES summer bridge, the AcES program, AcES Faculty/Staff, AcES guest lecturers, and FEP faculty/Staff are viewed as valuable by students and cited with contributing to their success in engineering. It is also evident that AcES students seek help from peers, seek help from tutors, use online resources, and attend office hours to overcome their challenges in studying engineering. 
    more » « less
  3. Measures of subject-related role identities in physics and math have been developed from research on the underlying constructs of identity in science education. The items for these measures capture three constructs of identity: students’ interest in the subject, students’ feeling of recognition by others, and students’ beliefs about their performance/competence in the subject area. In prior studies with late secondary and early post-secondary students, participants did not distinguish between performance beliefs (e.g., believing that they can do well in a particular subject) and competence beliefs (e.g., believing that they can understand a particular subject); therefore, performance/competence beliefs are measured as a single construct. These validated measures have been successful in predicting STEM career choices including physics, math, and engineering. Based on these measures of identity, literature on engineering identity, and my prior work on understanding engineering choice and belongingness through students’ science and math identities at the transition from high school to college, I developed a set of new engineering identity measures that capture and overall identification as an engineer, future engineering career identification, and students’ engineering-related interest, recognition, and performance/competence beliefs. I conducted a pilot survey of 371 first-year engineering students at three institutions within the U.S. during the spring semester of 2015. An exploratory factor analysis (EFA) was performed to examine the underlying structure of the piloted questions about students’ engineering identity. The measures loaded on three separate constructs that were consistent with the hypothesized constructs of interest, performance/competence and recognition. The developed items were used in a subsequent study deployed in the fall semester of 2015 that measured more than 2500 first-year engineering students’ attitudes and beliefs at four institutions within the U.S. The data on engineering identity measures from this second survey were analyzed using confirmatory factor analysis (CFA). The results indicated that the developed measures do extract a significant portion of the average variance in the latent constructs and the internal consistency of the measures (Cronbach’s α) falls within the acceptable and better range. The development of these items provides ways for engineering education researchers to more deeply explore the underlying self-beliefs in students’ engineering identity formation through quantitative measures with strong evidence for validity. 
    more » « less
  4. This S-STEM Project responds to a growing disparity among technology firms and the number of under-represented people in managerial and executive positions. Of particular interest is developing mentorship relationships and intrapreneurial competencies (i.e., entrepreneurship within established firms). Mentorship and increased skills preemptively aid in the retention and promotability of engineering undergraduates (upon entering the workforce). Specifically, the project was designed to produce electrical and computer engineering graduates with intrapreneurial knowledge and skills, which are characteristic of managers and innovators. Using the Intrepreneurial Competencies literature, the authors develop and test a multi-phased project among a diverse group of engineering undergraduates. The literature suggests enhancing intrapreneurial skills of students in engineering can be achieved through a combination of curricular and real-world experiences. Thus, this project incorporates faculty and industry mentorship, workforce development seminars, an industrial internship, entrepreneurship programs, and scholarships. Cohort 1 is comprised of a diverse group of 16 students (8 men, 8 women, 8 ethnic minorities). Students attended lectures by prominent engineering entrepreneurs, participated in a 3-day start-up weekend, attended engineering job fairs and two semesters of project-focused seminars, and read entrepreneurial and/or leadership-related books. Two primary data sets were collected utilizing a repeated measures design. Data were collected in the form of student reflections about being a mentee in the mentor relationships and interview data from mentors (i.e., engineering professionals). Students documented their mentoring sessions, which were reviewed by the project team. A primary theme that emerged from mentor reports was the effects of COVID-19, mostly how students felt about their coursework and how their industry mentors felt about their jobs. Although there was deep concern about the impacts of COVID-19, the students expressed a sense of growth and learning in spite of the virus. Students self-reported that the S-STEM experience was still highly beneficial, even as much of the coursework and mentoring for the latter half of the Spring semester had to be moved online. The students responded well, with the average semester GPA rising from 3.483 in the Fall to 3.774 in the Spring. Second, data were collected by survey pre- and post-semester to measure improvements in Intrapreneurial Competencies. The “Intrapreneurial Competencies Measurement Scale”(ICMS) by Vargas-Halabi et al. was used to measure and evaluate the development of intrapreneurial competencies, which include: (1) Opportunity promoter, (2) Proactivity, (3) Flexibility, (4) Drive, and (5) Risk taking. Each of the six categories of the ICMS is divided into 3-9 sub-categories to assess skill and mindset in the six general categories. In answering the questions on the ICMS test, students evaluated their proficiency in each of the areas. Growth was evident for almost all the categories and sub-categories across each of the three data-gathering points. 
    more » « less
  5. null (Ed.)
    The Academy of Engineering Success (AcES) program, established in 2012 and supported by NSF S-STEM award number 1644119 throughout 2016-2021, employs literature-based, best practices to support and retain underprepared and underrepresented students in engineering through graduation with the ultimate goal of diversifying the engineering workforce. A total of 71 students, including 21 students supported by S-STEM scholarships, participated in the AcES program between 2016-2019 at a large R1 institution in the mid-Atlantic region. All AcES students participate in a common program during their first year, comprised of: a one-week summer bridge experience, a common fall professional development course and spring “Engineering in History” course, and a common academic advisor. These students also have opportunities for: (1) faculty-student, student-student, and industry mentor-student interaction, (2) academic support and student success education, and (3) major and career exploration – all designed to help students develop feelings of institutional inclusion, engineering self-efficacy and identity, and academic and professional success skills. They also participate in the GRIT, Longitudinal Assessment of Engineering Self-Efficacy (LAESE), and the Motivated Strategies for Learning Questionnaire (MSLQ) surveys plus individual and focus group interviews at the start, midpoint, and end of each fall semester and at the end of the spring semester. The surveys provide a measure of students’ GRIT, their beliefs related to the intrinsic value of engineering and learning, their feelings of inclusion and test anxiety, and their self-efficacy related to engineering, math, and coping skills. The interviews provide information related to the student experience, feelings of inclusion, and program impact. Institutional data, combined with the survey and interview responses, are used to examine four research questions designed to examine the relationship of the elements of the AcES program to participants’ academic success and retention in engineering. Early analyses of the student retention data and survey responses from the 2017 and 2018 cohorts indicated students who ultimately left engineering before the start of their second year initially scored higher in areas of engineering self-efficacy and test anxiety, than those who stayed in engineering, while those who retained to the second year began their engineering education with lower self-efficacy scores, but higher scores related to the belief in the intrinsic value of engineering, learning strategy use, and coping self-efficacy. These results suggest that students who start with unrealistically high expectations of their performance leave engineering at higher rates than students who start with lower personal performance expectations, but have stronger value of the field and strategies for meeting challenges. These data appear to support the Kruger-Dunning effect in which students with limited knowledge of a specific field overestimate their abilities to perform in that area or underestimate the level of effort success may require. This paper will add an analysis of the academic success and retention data from 2019 cohort to this research, discuss the impact of COVID-19 to this program and research, as well as illuminate the quantitative results with the qualitative data from individual and focus group interviews regarding the aspects of the AcES program that impact student success, their expectations and methods for overcoming academic challenges, and their feelings of motivation and inclusion. 
    more » « less