skip to main content

Title: Using Participant Voices to Inform Validity Evidence in the Survey Development
This study presents qualitative findings from a larger instrument validation study. Undergraduates and subject matter experts (SMEs) were pivotal in early-stage development of a survey focusing on the four domains of Validation Theory (academic-in-class, academic-out-of-class, interpersonal-in-class, interpersonal-out-of-class). An iterative approach allowed for a more rigorously constructed survey refined through multiple phases. The research team met regularly to determine how feedback from undergraduates and SMEs could improve items and if certain populations were potentially being excluded. To date, the research team has expanded on the original 47 items up to 51 to address feedback provided by SMEs and undergraduate participants. Numerous item wording revisions have been made. Support for content, response process, and consequential validity evidence is strong.
; ; ; ;
Award ID(s):
Publication Date:
Journal Name:
Annual meeting program American Educational Research Association
Sponsoring Org:
National Science Foundation
More Like this
  1. Effects of High Impact Educational Practices on Engineering and Computer Science Student Participation, Persistence, and Success at Land Grant Universities: Award# RIEF-1927218 – Year 2 Abstract Funded by the National Science Foundation (NSF), this project aims to investigate and identify associations (if any) that exist between student participation in High Impact Educational Practices (HIP) and their educational outcomes in undergraduate engineering and computer science (E/CS) programs. To understand the effects of HIP participation among E/CS students from groups historically underrepresented and underserved in E/CS, this study takes place within the rural, public university context at two western land grant institutions (one of which is an Hispanic-serving institution). Conceptualizing diversity broadly, this study considers gender, race and ethnicity, and first-generation, transfer, and nontraditional student status to be facets of identity that contribute to the diversity of academic programs and the technical workforce. This sequential, explanatory, mixed-methods study is guided by the following research questions: 1. To what extent do E/CS students participate in HIP? 2. What relationships (if any) exist between E/CS student participation in HIP and their educational outcomes (i.e., persistence in major, academic performance, and graduation)? 3. How do contextual factors (e.g., institutional, programmatic, personal, social, financial, etc.) affectmore »E/CS student awareness of, interest in, and participation in HIP? During Project Year 1, a survey driven quantitative study was conducted. A survey informed by results of the National Survey of Student Engagement (NSSE) from each institution was developed and deployed. Survey respondents (N = 531) were students enrolled in undergraduate E/CS programs at either institution. Frequency distribution analyses were conducted to assess the respondents’ level of participation in extracurricular HIPs (i.e., global learning and study aboard, internships, learning communities, service and community-based learning, and undergraduate research) that have been shown in the literature to positively impact undergraduate student success. Further statistical analysis was conducted to understand the effects of HIP participation, coursework enjoyability, and confidence at completing a degree on the academic success of underrepresented and nontraditional E/CS students. Exploratory factor analysis was used to derive an "academic success" variable from five items that sought to measure how students persevere to attain academic goals. Results showed that a linear relationship in the target population exists and that the resultant multiple regression model is a good fit for the data. During the Project Year 2, survey results were used to develop focus group interview protocols and guide the purposive selection of focus group participants. Focus group interviews were conducted with a total of 27 undergraduates (12 males, 15 females, 16 engineering students, 11 computer science students) across both institutions via video conferencing (i.e., ZOOM) during the spring and fall 2021 semesters. Currently, verified focus group transcripts are being systematically analyzed and coded by a team of four trained coders to identify themes and answer the research questions. This paper will provide an overview of the preliminary themes so far identified. Future project activities during Project Year 3 will focus on refining themes identified during the focus group transcript analysis. Survey and focus group data will then be combined to develop deeper understandings of why and how E/CS students participate in the HIP at their university, taking into account the institutional and programmatic contexts at each institution. Ultimately, the project will develop and disseminate recommendations for improving diverse E/CS student awareness of, interest in, and participation in HIP, at similar land grant institutions nationally.« less
  2. Identity, or how people choose to define themselves, is emerging as an attractive explanation for who persists in engineering. Many studies of engineering identity build off of prior work in math and science identity, emphasizing the academic aspects of engineering. However, affect towards professional practice is also central to engineering identity development. This paper describes the methods used to create a new survey measure of individuals’ affect toward elements of engineering practice. We followed the item generation, refinement, and instrument validation steps required for psychometric validation of a new survey measure. We generated items deductively using the literature on engineering professional skills and practice and inductively based on interviews with practicing engineers, engineering graduate students, and engineering undergraduate students. We blended the inductively and deductively derived item lists to create a list of initial items for the measure. We circulated this list of items to a set of engineering and professional identity experts to establish face validity and made modifications based on their feedback. The final list included 34 items. These 34 items were administered in a questionnaire survey in the fall of 2016 to 1465 engineering undergraduates in three majors at two institutions. We conducted an exploratory factor analysismore »(EFA) and established internal consistency using Cronbach’s alpha on a subset of the analytical sample data (n=384). The resulting factors fit our a priori assumption of the factors theorized to characterize affect towards engineering professional practice. Using the remaining data (n=904), we conducted a confirmatory factor analysis on the reduced set of items resulting from EFA. The results indicate an emergent factor structure for affect towards elements of engineering practice.« less
  3. Over the years, researchers have found that student engagement facilitates desired academic success outcomes for college undergraduate students. Much research on student engagement has focused on academic tasks and classroom context. High impact engagement practices (HIEP) have been shown to be effective for undergraduate student academic success. However, less is known about the effects of HIEP specifically on engineering and computer science (E/CS) student outcomes. Given the high attrition rates for E/CS students, student involvement in HIEP could be effective in improving student outcomes for E/CS students, including those from various underrepresented groups. More generally, student participation in specific HIEP activities has been shown to shape their everyday experiences in school, both academically and socially. Hence, the primary goal of this study is to examine the factors that predict academic success in E/CS using multiple regression analysis. Specifically, this study seeks to understand the effects of high impact engagement practices (HIEP), coursework enjoyability, confidence at completing a degree on academic success of the underrepresented and nontraditional E/CS students. We used exploratory factor analyses to derive “academic success” variable from five items that sought to measure how students persevere to attain academic goals. A secondary goal of the present study ismore »to address the gap in research literature concerning how participation in HIEP affects student persistence and success in E/CS degree programs. Our research team developed and administered an online survey to investigate and identify factors that affect participation in HIEP among underrepresented and nontraditional E/CS students. Respondents (N = 531) were students enrolled in two land grant universities in the Western U.S. Multiple regression analyses were conducted to examine the proportion of the variation in the dependent variable (academic success) explained by the independent variables (i.e., high impact engagement practice (HIEP), coursework enjoyability, and confidence at completing a degree). We hypothesized that (1) high impact engagement practices will predict academic success; (2) coursework enjoyability will predict academic success; and (3) confidence at completing a degree will predict academic success. Results showed that the multiple regression model statistically predicted academic success , F(3, 270) = 33.064, p = .001, adjusted R2 = .27. This results indicate that there is a linear relationship in the population and the multiple regression model is a good fit for the data. Further, findings show that confidence at completing a degree is significantly predictive of academic success. In addition, coursework enjoyability is a strong predictor of academic success. Specifically, the result shows that an increase in high impact engagement activity is associated with an increase in students’ academic success. In sum, these findings suggest that student participation in High Impact Engagement Practices might improve academic success and course retention. Theoretical and practical implications are discussed.« less
  4. Abstract. The Circumplex Team Scan (CTS) assesses the degree to which a team’s interaction/communication norms reflect each segment (16th) of the interpersonal circle/circumplex. We developed and evaluated an abbreviated 16-item CTS-16 that uses one CTS item to measure each segment. Undergraduates ( n = 446) completing engineering course projects in 139 teams completed the CTS-16. CTS-16 items showed a good fit to confirmatory structural models (e.g., that expect greater positive covariation between items theoretically closer to the circumplex). Individuals’ ratings sufficiently reflected team-level norms to justify averaging team members’ ratings. However, individual items’ marginal reliabilities suggest using the CTS-16 to assess general circumplex-wide patterns rather than specific segments. CTS-16 ratings correlated with respondents’ and their teammates’ ratings of team climate (inclusion, justice, psychological safety). Teams with more extraverted (introverted) members were perceived as having more confident/engaged (timid/hesitant) cultures. Members predisposed to social alienation perceived their team’s culture as relatively disrespectful/unengaged, but their teammates did not corroborate those perceptions. The results overall support the validity and utility of the CTS-16 and of an interpersonal circumplex model of team culture more generally.
  5. This work-in-progress research paper introduces the Educational Ecosystem Health Survey (EEHS), an educational survey instrument designed by the Eco-STEM team at California State University, Los Angeles, a federally designated Hispanic Serving Institution. The Eco-STEM project applies a framework of Community Cultural Wealth and explores the metaphor of a healthy ecosystem to envision systemic change that responds to the needs and values the assets of diverse actors, who learn together for both their individual and collective good, within the educational “ecosystem.” As part of the project, the Eco-STEM team has developed the EEHS survey instrument to measure the “health” of the educational ecosystem. The results will provide valuable insight into the perceptions and experiences of students from socially and structurally oppressed groups. The Eco-STEM EEHS is comprised of constructs from several survey instruments that have already undergone statistical validation within educational contexts, many of them within higher education. The items peruse issues of social climate, belonging, thriving and wellbeing, interest, mindfulness, stress, and perceptions of the future. Given the Community Cultural Wealth framework and the fact that two-thirds of the student body at California State University, Los Angeles identifies as Hispanic, the EEHS is offered in both Spanish and English. Studentsmore »are asked to provide a multitude of institutionally relevant demographic information, such that results may be disaggregated along many categories. The EEHS is also administered to faculty, staff, and administration / management in addition to students. By including these essential actors in the analysis of the state of the educational ecosystem, we intend to also measure perceptions of experience serving the STEM educational community, rather than solely receiving it. We will pilot the EEHS during the Spring 2022 semester. Over the next four years of the Eco- STEM project, semesterly administrations will quantify the progress of the project’s initiatives to implement effective systemic change. Our analyses will investigate the perspectives of those with oppressed social identities – individuals who actually hold majority representation within the unique demographic composition of California State University, Los Angeles. The results will offer critically important feedback to Hispanic-Serving Institutions and all institutions who strive to serve students from communities who have been left behind and even exploited by the existing systems and structures of higher education. Keywords: Educational Ecosystems, Community Cultural Wealth, Surveys« less