skip to main content

Title: Using Participant Voices to Inform Validity Evidence in the Survey Development
This study presents qualitative findings from a larger instrument validation study. Undergraduates and subject matter experts (SMEs) were pivotal in early-stage development of a survey focusing on the four domains of Validation Theory (academic-in-class, academic-out-of-class, interpersonal-in-class, interpersonal-out-of-class). An iterative approach allowed for a more rigorously constructed survey refined through multiple phases. The research team met regularly to determine how feedback from undergraduates and SMEs could improve items and if certain populations were potentially being excluded. To date, the research team has expanded on the original 47 items up to 51 to address feedback provided by SMEs and undergraduate participants. Numerous item wording revisions have been made. Support for content, response process, and consequential validity evidence is strong.  more » « less
Award ID(s):
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Annual meeting program American Educational Research Association
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Identity, or how people choose to define themselves, is emerging as an attractive explanation for who persists in engineering. Many studies of engineering identity build off of prior work in math and science identity, emphasizing the academic aspects of engineering. However, affect towards professional practice is also central to engineering identity development. This paper describes the methods used to create a new survey measure of individuals’ affect toward elements of engineering practice. We followed the item generation, refinement, and instrument validation steps required for psychometric validation of a new survey measure. We generated items deductively using the literature on engineering professional skills and practice and inductively based on interviews with practicing engineers, engineering graduate students, and engineering undergraduate students. We blended the inductively and deductively derived item lists to create a list of initial items for the measure. We circulated this list of items to a set of engineering and professional identity experts to establish face validity and made modifications based on their feedback. The final list included 34 items. These 34 items were administered in a questionnaire survey in the fall of 2016 to 1465 engineering undergraduates in three majors at two institutions. We conducted an exploratory factor analysis (EFA) and established internal consistency using Cronbach’s alpha on a subset of the analytical sample data (n=384). The resulting factors fit our a priori assumption of the factors theorized to characterize affect towards engineering professional practice. Using the remaining data (n=904), we conducted a confirmatory factor analysis on the reduced set of items resulting from EFA. The results indicate an emergent factor structure for affect towards elements of engineering practice. 
    more » « less
  2. The research and evaluation team of an S-STEM project at a large, research-intensive Southeastern public university conducted a cross-sectional survey as a first step to compare factors which may influence undergraduate student persistence in engineering and computing. All engineering and computing students were invited to participate in the survey, and 282 (10.4%) provided responses. The respondents included 15 high financial need students who were participating in the S-STEM program, of which 7 were first-year students and 8 were sophomores. The remaining 267 respondents were undergraduates ranging from first-year to seniors. Survey questions were adapted from previously developed instruments on self-efficacy, sense-of-belonging, identity, community involvement, and overall college experience. Additional questions related to stress levels, academic life, use and effectiveness of academic supports, and the impacts of COVID-19 on their college experiences. The team compared responses by level of academic progression, declared major, gender, and race/ethnicity. Student responses showed a variety of similarities and differences between subgroups. Overall, the students said that they often attended lectures (in-person or online) and came to class prepared. At the same time, students rated these activities as the least effective academic supports. On the other hand, the students rated working assigned or extra homework problems and studying for exams as their most effective activities. Consistently among the subgroups, the students said their community involvement and identity as developing engineers were relatively low while self-efficacy and team self-efficacy were seen as stronger personal skills. The students said they were highly stressed about their grades and academic success in general, and about finances and future careers. They reported feeling less stress about aspects such as living away from home and negotiating the university social scene. Students reported spending the most time preparing for class in their first year compared to students in later years. Female students (104 responses) reported higher levels of community involvement, engineering identity, and engagement in college life compared to male students (142 responses) while there was little gender-related difference in self-efficacy and sense of belonging. Levels of self-efficacy and team self-efficacy did not show large differences based on year in college. Interestingly, first-year students expressed the highest levels of engineering identity while senior students the lowest. Senior students reported the lowest community involvement, sense of belonging, and engineering identity compared to other students. Overall, students from different races self-reported the same levels of self-efficacy. Black/African American students reported the highest levels of community involvement, college life, and identity. There were no substantial differences in self-efficacy among the different engineering and computing majors. This study is a first step in analysis of the students’ input. In addition to surveying the students, the team also conducted interviews of the participating S-STEM students, and analysis of these interviews will provide greater depth to interpretation of the survey results. Overall, the research and evaluation team’s intention is to provide insight to the project’s leadership in how best to support the success of first-year engineering and computing students. 
    more » « less
  3. Identity, or how people choose to define themselves, is emerging as an explanation for who pursues and persists in engineering. Recent developments in the study of engineering identity, including studies of math and science identity, tend to emphasize the academic aspects of engineering without considering aspects of professional practice central to the development of an engineering identity. This paper outlines the methods used to create a new survey measure: affect toward elements of engineering practice. We followed the item generation, refinement, and instrument validation steps required for psychometric validation of a new survey measure. Through this process a final list of 34 items was administered in a survey in the fall of 2016 to engineering undergraduates. We conducted an exploratory factor analysis and established internal consistency using Cronbach’s alpha on a subset of the data sample (n=384). The resulting factors reflect key elements of affect towards engineering professional practice. 
    more » « less
  4. Abstract. The Circumplex Team Scan (CTS) assesses the degree to which a team’s interaction/communication norms reflect each segment (16th) of the interpersonal circle/circumplex. We developed and evaluated an abbreviated 16-item CTS-16 that uses one CTS item to measure each segment. Undergraduates ( n = 446) completing engineering course projects in 139 teams completed the CTS-16. CTS-16 items showed a good fit to confirmatory structural models (e.g., that expect greater positive covariation between items theoretically closer to the circumplex). Individuals’ ratings sufficiently reflected team-level norms to justify averaging team members’ ratings. However, individual items’ marginal reliabilities suggest using the CTS-16 to assess general circumplex-wide patterns rather than specific segments. CTS-16 ratings correlated with respondents’ and their teammates’ ratings of team climate (inclusion, justice, psychological safety). Teams with more extraverted (introverted) members were perceived as having more confident/engaged (timid/hesitant) cultures. Members predisposed to social alienation perceived their team’s culture as relatively disrespectful/unengaged, but their teammates did not corroborate those perceptions. The results overall support the validity and utility of the CTS-16 and of an interpersonal circumplex model of team culture more generally. 
    more » « less
  5. Effects of High Impact Educational Practices on Engineering and Computer Science Student Participation, Persistence, and Success at Land Grant Universities: Award# RIEF-1927218 – Year 2 Abstract Funded by the National Science Foundation (NSF), this project aims to investigate and identify associations (if any) that exist between student participation in High Impact Educational Practices (HIP) and their educational outcomes in undergraduate engineering and computer science (E/CS) programs. To understand the effects of HIP participation among E/CS students from groups historically underrepresented and underserved in E/CS, this study takes place within the rural, public university context at two western land grant institutions (one of which is an Hispanic-serving institution). Conceptualizing diversity broadly, this study considers gender, race and ethnicity, and first-generation, transfer, and nontraditional student status to be facets of identity that contribute to the diversity of academic programs and the technical workforce. This sequential, explanatory, mixed-methods study is guided by the following research questions: 1. To what extent do E/CS students participate in HIP? 2. What relationships (if any) exist between E/CS student participation in HIP and their educational outcomes (i.e., persistence in major, academic performance, and graduation)? 3. How do contextual factors (e.g., institutional, programmatic, personal, social, financial, etc.) affect E/CS student awareness of, interest in, and participation in HIP? During Project Year 1, a survey driven quantitative study was conducted. A survey informed by results of the National Survey of Student Engagement (NSSE) from each institution was developed and deployed. Survey respondents (N = 531) were students enrolled in undergraduate E/CS programs at either institution. Frequency distribution analyses were conducted to assess the respondents’ level of participation in extracurricular HIPs (i.e., global learning and study aboard, internships, learning communities, service and community-based learning, and undergraduate research) that have been shown in the literature to positively impact undergraduate student success. Further statistical analysis was conducted to understand the effects of HIP participation, coursework enjoyability, and confidence at completing a degree on the academic success of underrepresented and nontraditional E/CS students. Exploratory factor analysis was used to derive an "academic success" variable from five items that sought to measure how students persevere to attain academic goals. Results showed that a linear relationship in the target population exists and that the resultant multiple regression model is a good fit for the data. During the Project Year 2, survey results were used to develop focus group interview protocols and guide the purposive selection of focus group participants. Focus group interviews were conducted with a total of 27 undergraduates (12 males, 15 females, 16 engineering students, 11 computer science students) across both institutions via video conferencing (i.e., ZOOM) during the spring and fall 2021 semesters. Currently, verified focus group transcripts are being systematically analyzed and coded by a team of four trained coders to identify themes and answer the research questions. This paper will provide an overview of the preliminary themes so far identified. Future project activities during Project Year 3 will focus on refining themes identified during the focus group transcript analysis. Survey and focus group data will then be combined to develop deeper understandings of why and how E/CS students participate in the HIP at their university, taking into account the institutional and programmatic contexts at each institution. Ultimately, the project will develop and disseminate recommendations for improving diverse E/CS student awareness of, interest in, and participation in HIP, at similar land grant institutions nationally. 
    more » « less