Search for: All records

Abstract—This WIP research paper presents validity evidence for a survey instrument designed to assess student learning in makerspaces. We report findings from expert reviews of item content and student interpretations of survey questions. The instrument was developed using a theory-driven approach to define constructs, followed by the development of questions aligned with those constructs. We solicited written feedback from 30 experts in instrument development and/or makerspaces, who rated the alignment of items with our constructs. Based on this input, we revised our items for clarity and consistency. We then conducted 25 cognitive interviews with a diverse group of students who use makerspaces, asking them to explain their understanding of each item and the reasoning behind their responses. Our recruitment ensured diversity in terms of race, gender, ethnicity, and academic background, extending beyond engineering majors. From our initial 45 items, we removed 6, modified 36, and added 1 based on expert feedback. During cognitive interviews, we began with 40 items, deleted one, and revised 23, resulting in 39 items for the pilot survey. Key findings included the value of examples in clarifying broad terms and improved student engagement with a revised rating scale—shifting from a 7-point Likert agreement scale to a self-description format encouraged fuller use of the scale. Our study contributes to the growing body of research on makerspaces by offering insights into how students describe their learning experiences and by providing initial validation evidence for a tool to assess those experiences, ultimately strengthening the credibility of the instrument. 

It is well established that access to social supports is essential for engineering students’ persistence and yet access to supports varies across groups. Understanding the differential supports inherent in students’ social networks and then working to provide additional needed supports can help the field of engineering education become more inclusive of all students. Our work contributes to this effort by examing the reliability and fairness of a social capital instrument, the Undergraduate Supports Survey (USS). We examined the extent to which two scales were reliable across ability levels (level of social capital), gender groups and year-in-school. We conducted two item response theory (IRT) models using a graded response model and performed differential item functioning (DIF) tests to detect item differences in gender and year-in-school. Our results indicate that most items have acceptable to good item discrimination and difficulty. DIF analysis shows that multiple items report DIF across gender groups in the Expressive Support scale in favor of women and nonbinary engineering students. DIF analysis shows that year-in-school has little to no effect on items, with only one DIF item. Therefore, engineering educators can use the USS confidently to examine expressive and instrumental social capital in undergraduates across year-in-school. Our work can be used by the engineering education research community to identify and address differences in students’ access to support. We recommend that the engineering education community works to be explicit in their expressive and instrumental support. Future work will explore the measurement invariance in Expressive Support items across gender.