Engineering undergraduates have significant career opportunities and potential for social mobility, but economically disadvantaged yet academically gifted students often lack adequate support. Many low-income students juggle part-time jobs and family responsibilities, limiting their focus on academics and impacting their social integration and goal clarity. To address this, the SSTEM project provides financial assistance, fosters a peer support network for collaborative learning, and promotes inclusivity through diversity, equity, and inclusion initiatives. The impact of income inequality in college students has been a topic highly researched in the past and can be rooted back to a lack of enrollment from low-income students [2].
Funded by the NSF, the SSTEM project implements four key interventions: scholarships, an Engineering Learning Community (ELC) promoting cohort-based learning and living, mentorship, and personal and professional development seminars. The NSF-SSTEM grant provides low-income, high-achieving students with opportunities to pursue engineering degrees by alleviating financial burdens through scholarships. The ELC program fosters peer connections by housing first-year engineering students together and ensuring they share at least two classes with roommates. By providing both financial support and social networks, the program helps reduce external pressures, allowing students to focus on academic success. According to Tinto [3], student persistence improves when institutions foster academic and social integration through four key elements: intensity and clarity of goal to graduate, selfefficacy, sense of belonging, and perceived curriculum relevance. The SSTEM project reinforces these elements through faculty advising, peer course enrollment, the ELC experience, and DEI workshops. These workshops address the lack of diversity in STEM fields, equipping students with the knowledge and tools to become leaders in inclusive academic, professional, and social settings [4,5].
This paper examines the robustness of validity evidence for an instrument assessing Tinto's four key elements through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Building on research from the 2023-2024 academic year, which included a preliminary EFA with 200 first-year engineering students, this study expands the dataset, conducting an EFA with 400 responses and a CFA with 350 responses.
The overall research is based on Tinto's conceptual model of student motivation and persistence. This model looks at persistence through the students' perspective [3] and suggests that student persistence is influenced by intensity and clarity of goal to graduate and motivation, with motivation being further broken down into self-efficacy, sense of belonging, and perception of the curriculum. When combined, these four elements-intensity and clarity of goal to graduate, self-efficacy, sense of belonging, and perception of curriculum-provide a comprehensive framework for increasing persistence as seen in Figure 1. Like other past models, this model defines persistence as an individual staying in an engineering major or completing an engineering degree [6]. A student who switches from one engineering major to another but still earns an engineering degree is still considered to have persisted. The exploratory factor analysis based on the 2023-2024 preliminary data revealed the following six factors: comfort with the professor, academic engineering confidence (selfefficacy), sense of belonging, engineering identity (sense of belonging), advanced engineering degree (intensity and clarity of goal to graduate), and DEI components (perception of curriculum). The factor of comfort with the professor reflects students' ease in asking questions and interacting with faculty. The factor of sense of belonging focuses on students' feelings of acceptance among peers and at the university. The factor of engineering identity is associated with whether students see themselves as engineers and think like one. The factor of an advanced engineering degree is focused on students' intentions to pursue graduate studies in engineering. The factor of academic engineering confidence relates to students' performance in coursework, exams, and comprehension. The factor of DEI components is associated with discussions on diversity, equity, and inclusion among instructors and students. Later in this paper we will examine how strong each factor is and how well the items measure the factors. The six factors align with Tinto's four key elements of student motivation and persistence, serving as subcomponents of intensity and clarity of goal to graduate, self-efficacy, sense of belonging, and curriculum perception, as illustrated in Figure 2. The two factors of sense of belonging and engineering identity are associated with the element described by Tinto as sense of belonging. Tinto describes a sense of belonging as the broader campus climate and daily interactions with other students, faculty, staff, and administrators on campus [3]. The factor of sense of belonging is associated with how a student fits in with the institution and peers, while the factor of engineering is associated with a student's personal belonging and feeling of being an engineer.
The factors of comfort with professor and academic engineering confidence are both associated with the element of self-efficacy. Self-efficacy is described as an individual's perceived ability to succeed in a certain situation. Comfort with professor is associated with a student's ability to engage and communicate with a professor, while academic engineering is associated with a student's ability to succeed within their classes.
The factor of an advanced engineering degree is associated with Tinto's element of intensity and clarity of goals to graduate. The factor of advanced engineering degree is associated with the likelihood of a student pursuing a graduate program after undergraduate school. This factor correlates to Tinto's element of intensity and clarity of goals to graduate, because while not all students who aim to graduate plan to continue their education, those intending to pursue a graduate program demonstrate a clear commitment to furthering their academic goals. The survey included items related to starting an engineering career postgraduation, expected to align with the element of intensity and clarity of goals to graduate, but these items failed to form a factor during the EFA process.
The factor of DEI components, though less clearly defined, aligns with Tinto's element of curriculum perception. It reflects how diversity, equity, and inclusion are integrated into the curriculum. DEI components are a big aspect of the SSTEM project and students within the project are expected to take DEI workshops, learn about complex topics and are encouraged to get help from different clubs within the institution. The survey included items on curriculum content and future courses, expected to align with the element of curriculum perception, but they did not form a factor during the EFA process.
Our approach to this research differs from past research by focusing on persistence rather than just motivation. While persistence includes motivation, it also encompasses intensity and clarity of goals to graduate. Most instruments based on Tinto's model measure only one predictor, such as self-efficacy or sense of belonging. In contrast, our instrument tries to assess all the predictors of Tinto's proposed elements through a survey designed for first-year engineering students.
We surveyed all first-year engineering students during their required introduction to engineering courses, complementing data from the 2023-2024 academic year. With no survey incentives, we achieved an 87% response rate. Using RStudio for factor analysis allowed us to include responses even with skipped questions or missing data. The 2023-2024 dataset includes both pre-and post-surveys, while the 2024-2025 dataset currently consists of pre-survey responses. As noted, the EFA used 400 responses from 2023-2024, while the CFA analyzed 350 responses from the 2024-2025 pre-survey.
As previously stated above, the survey was designed to assess student motivation (including self-efficacy, sense of belonging, and perceptions of the curriculum), the student's intensity and clarity of goals to graduate and their perceptions on the implementation and conversations of diversity, equity, and inclusion (DEI) within the institution. The survey consisted of thirty-nine items asked on a Likert scale and consisted of academic and professional development, perceived self-efficacy, and retention [1]. The survey was primarily developed by the project's external evaluator, Quality Measures LLC, using two previously validated instruments detailed in papers by Ahn et al. [8], Hoffman et al. [7], and others. The survey consisted of thirty-nine items, with ten items having a loading score below the threshold of 0.3. This paper focuses on the twenty-nine items with a loading score above 0.3, while the excluded items are discussed in our previous paper [1].
We conducted an EFA to reduce data complexity and identify underlying themes. During the EFA process, items with correlations above 0.9 were removed. A Kaiser-Meyer-Olkin (KMO) test confirmed the data's suitability for factor analysis (KMO > 0.5). Parallel analysis and a Scree Plot suggested starting with six factors, which produced the strongest correlations compared to five, four, or three-factor models. The EFA was conducted using ProMax rotation. Items that cross-loaded or failed to load were removed first, followed by those with communality scores (h²) < 0.3. While the standard cutoff is 0.4, it was lowered to 0.3 to retain more items [9]. The EFA was rerun after each removal, improving communality scores. This process resulted in the deletion of ten out of thirty-nine survey items.
For this paper, we conducted a CFA using the 2024-2025 pre-survey data, modeling the six factors identified in the EFA. The model quantifies the strength of relationships between items and latent variables (factors) by standardizing all variables for easier comparison. During the CFA process, the software also generated covariance estimates to assess relationships and mutual influence among the six factors. Additionally, we assessed reliability, ensuring consistency and stability in measuring latent variables. Reliability indicates how much variance in observed items is attributed to the latent factor rather than random error [10].
Table 1 presents the EFA results for the sense of belonging factor, which emerged as the strongest factor with ten associated items. Standard loading measures the strength of the relationship between each item and the factor, with higher values indicating stronger associations. All items in this factor had acceptable standard loadings of 0.4 or higher. Bolded items and variables in the table highlight key elements discussed in the paper. I am respected by the university staff (non-instructors) at (blinded for review). 0.822 0.628 0.564 3.) I am respected by other students at (blinded for review). 0.750 0.583 0.474 4.) I have found (blinded for review) to be welcoming. 0.695 0.470 0.272 5.) I can talk to my instructor if I have a problem. 0.692 0.530 0.228 6.) I can talk to a fellow student if I have a problem. 0.644 0.435 0.526 7.) I believe that I matter to others at (blinded for review). 0.599 0.531 0.790 8.) I believe the people at (blinded for review) understand me as a person. 0.593 0.464 0.970 9.) I am happy with my choice to be a student at (blinded for review). 0.584 0.355 0.624 10.) I can talk to my advisor if I have a problem. 0.567 0.416 0.714
Table 2 presents academic engineering confidence, our second strongest factor. While one item had a high standard loading score, another had a notably low score. Item 8, initially expected to be grouped with a factor associated with the element of perception of curriculum, instead grouped with this factor, contributing to Tinto's element of self-efficacy. Item 4, which assesses students' confidence during tests, exhibited high variance. I am familiar with the curriculum I have chosen within the engineering program at (blinded for review). 0.360 0.377 0.478
Table 3 presents the EFA results for engineering identity, which consists of three items. Item 1 showed the strongest association with the factor, while the weakest item still had a better standard loading than the lowest-scoring item in academic engineering confidence. While this remains a relatively strong factor, adding more items could further enhance its reliability. 4 presents the items associated with the DEI components factor, which was included in the survey due to the SSTEM project's focus on supporting underrepresented and economically disadvantaged yet academically gifted students. The items that were loaded with this factor were expected, as they were similarly structured and addressed related themes. While the factor of DEI components is a relatively strong factor, its role in Tinto's theory is less clearly defined compared to other factors. Table 5 presents the EFA results for advanced engineering degree, one of the weaker factors. Several expected items failed to load properly and had low communality scores (h²). Without adjusting the h² cutoff from 0.4 to 0.3, this factor would have contained only one item. Item 2 had the highest variance score among all tested items, with a low communality score, indicating that most of its variance is unexplained by the factors. Table 6 presents the comfort with professor factor. While both items have relatively high loadings, analysis is limited due to the small number of items. With only two items, this is one of the weaker factors, as fewer measures increase the potential for error. Although the standard loadings are acceptable, the factor's limited scope may hinder future testing. During the CFA process, the software generated covariance estimates among the six factors, measuring their relationships and mutual influence. The covariance scores of the factors can be seen in Figure 3. Each factor exhibited covariance with the others, with varying strengths. The strongest covariance was between engineering identity and academic engineering confidence (0.855), followed by comfort with professor and academic engineering (0.570). The weakest covariance score was unexpectedly between academic engineering confidence and DEI components (0.147). Table 7 presents the internal consistency reliability of the six factors. Alpha measures internal consistency, omega assesses total reliability, and AVE evaluates convergent validity. Acceptable thresholds are ≥0.7 for alpha and omega and ≥0.5 for AVE. If all three metrics are deemed acceptable then the factor has high reliability. Comfort with professor was a factor that showed high reliability and had overall great alpha, omega, and AVE scores. Engineering identity also showed very promising and strong results. Meanwhile the other four factors each lacked a bit when it came to internal consistency reliability.
In this section, we analyze the research results and examine the tables in greater depth. Due to the CFA results indicating poor reliability among the four out of six of the factors, it is difficult to determine which elements are effectively measured. While Tinto's four elementsintensity and clarity of goal to graduate, sense of belonging, self-efficacy, and curriculum perception-were expected to align with these factors, the findings are inconclusive. This is largely due to lower-than-expected factor reliability. Before delving into the CFA results, we will first summarize the EFA findings from Tables 123456.
Table 5 (Advanced Engineering Degree Table) shows the highest variance, likely because many first-year engineering students have not yet considered graduate studies. Their primary focus is on completing their current semester, and some may be uncertain about what graduate studies entail. Conversely, Items 6 and 7 in Table 2 (Academic Engineering Confidence Table ) exhibit very little variance. This may be because students feel confident in understanding course content, or, alternatively, they may already be struggling. Additionally, assignment-related confidence could be due to early-semester coursework being relatively simple.
The covariance results indicate a strong relationship between engineering identity and academic engineering confidence, suggesting that students who identify as engineers tend to be more confident in their studies. The connection between engineering identity and academic engineering confidence has also been studied and researched in past papers [6]. Additionally, academic engineering confidence correlates strongly with comfort with professor, implying that academic confidence may be linked to a student's engagement with faculty.
The reliability analysis of the six factors yielded mixed results. Alpha and Omega are two ways to assess how well items correlate and measure the same construct, with four out of six factors showing acceptable values for both metrics. Average Variance Extracted (AVE) measures convergent validity, assessing whether items strongly correlate with their intended construct [11]. Only one factor met the AVE threshold of 0.5, with another factor being extremely close to meeting the threshold. Even with acceptable alpha (α > 0.7) or omega (ω > 0.7), a low AVE suggests potential item convergence issues [11]. This indicates that many of the factors are not acceptable when it comes to their reliability, suggesting a need to rework our approach and review the items within each factor.
Although the CFA showed overall weak reliability for the factors identified in the EFA, further research and additional data collection over the next year may yield stronger results. We may need to reassess some of the factors, review survey items to ensure they measure the intended elements and address potential formatting or presentation issues. Before conducting future surveys, we plan to consult peer experts to refine the instrument. As we gather more data, some factors may merge, or new factors may emerge. Re-wording existing items and adding new ones could potentially strengthen these factors. We also aim to identify additional factors that enhance connections to Tinto's four elements. For example, intensity and clarity of goals to graduate could include a factor related to securing an engineering job post-graduation, while curriculum perception could incorporate a factor addressing students' understanding of their academic pathway and required coursework.
Before we can move on to the next stage of our project and compare students within the SSTEM program with standard engineering experience, we must refine survey items and improve factor reliability to strengthen our instrument. Once reliability improves, we can advance our research and develop a more robust persistence model. This study has identified the factors and items that need the most revision to enhance our model moving forward.
This study is limited by its single institution focus, reducing generalizability. If implemented elsewhere, survey items would need adjustments to align with different program structures. Another limitation is the time constraint in survey administration. Pre-surveys, conducted in the first two weeks of the academic year, may capture uninformed responses as students are still adjusting to college. Post-surveys, administered in the last two weeks, risk rushed or less accurate responses as students focus on finishing the semester. A mid-semester survey could yield more accurate results, but logistical challenges make scheduling difficult. A final limitation is the inability to determine how many participants answered the survey seriously versus those who provided random responses, given the study's sample size and nature.
To move on to our next stage of the project and answer our big future research question of whether the SSTEM project is improving students' persistence, further testing is needed. We plan to eventually compare the results of students in the SSTEM project to the first-year engineering students that were not in any special engineering program. One thing we must take into consideration is that the SSTEM project may vary across cohorts, which may cause the effectiveness of this instrument to vary as well. Another plan is to interview first year students after the post survey. Future research will include post-survey interviews to gain deeper insight into students' experiences and improve data accuracy.
Another plan for the future of the project is to rework our persistence model by incorporating Alexander Astin's theory of involvement and John Bean's theory of student attrition [12]. This approach has been done in the past on research about student retention. We also plan to conduct regression analysis on persistence, explore correlations with other variables, and check for measurement invariance across subpopulations. We also plan to calculate standard fit indices such as RMSEA (Root Mean Square Error of Approximation), CFI (Comparative Fit Index), and TLI (Tucker-Lewis Index) in the future [13]. Before advancing, we must reassess our EFA and CFA approaches to develop a stronger persistence model, likely in collaboration with an expert in factor analysis.