An official website of the United States government
Abstract—This WIP research paper presents validity evidence for a survey instrument designed to assess student learning in makerspaces. We report findings from expert reviews of item content and student interpretations of survey questions. The instrument was developed using a theory-driven approach to define constructs, followed by the development of questions aligned with those constructs. We solicited written feedback from 30 experts in instrument development and/or makerspaces, who rated the alignment of items with our constructs. Based on this input, we revised our items for clarity and consistency. We then conducted 25 cognitive interviews with a diverse group of students who use makerspaces, asking them to explain their understanding of each item and the reasoning behind their responses. Our recruitment ensured diversity in terms of race, gender, ethnicity, and academic background, extending beyond engineering majors. From our initial 45 items, we removed 6, modified 36, and added 1 based on expert feedback. During cognitive interviews, we began with 40 items, deleted one, and revised 23, resulting in 39 items for the pilot survey. Key findings included the value of examples in clarifying broad terms and improved student engagement with a revised rating scale—shifting from a 7-point Likert agreement scale to a self-description format encouraged fuller use of the scale. Our study contributes to the growing body of research on makerspaces by offering insights into how students describe their learning experiences and by providing initial validation evidence for a tool to assess those experiences, ultimately strengthening the credibility of the instrument.
more »
« less
Developing a Measure of Engineering Students’ Makerspace Learning, Perceptions, and Interactions
Makerspaces have become a rather common structure within engineering education programs. The spaces are used in a wide range of configurations but are typically intended to facilitate student collaboration, communication, creativity, and critical thinking, essentially giving students the opportunity to learn 21st century skills and develop deeper understanding of the processes of engineering. Makerspace structure, layout, and use has been fairly well researched, yet the impact of makerspaces on student learning is understudied, somewhat per a lack of tools to measure student learning in these spaces. We developed a survey tool to assess undergraduate engineering students’ perceptions and learning in makerspaces, considering levels of students’ motivation, professional identity, engineering knowledge, and belongingness in the context of makerspaces. Our survey consists of multiple positively-phrased (supporting a condition) and some negatively-phrased (refuting a condition) survey items correlated to each of our four constructs. Our final survey contained 60 selected response items including demographic data. We vetted the instrument with an advisory panel for an additional level of validation and piloted the survey with undergraduate engineering students at two universities collecting completed responses from 196 participants. Our reliability analysis and additional statistical calculations revealed our tool was statistically sound and was effectively gathering the data we designed the instrument to measure.
more »
« less
- Award ID(s):
- 1664274
- PAR ID:
- 10087318
- Date Published:
- Journal Name:
- American Society of Engineering Education
- Volume:
- 22089
- Page Range / eLocation ID:
- 1-12
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Building upon our two years of research on the use of makerspaces in undergraduate engineering programs, we engaged in a large-scale data collection from students enrolled in undergraduate engineering preparation programs with affiliated makerspaces established for a minimum of three years. Using web searches, and other sources of information (e.g. references from other researchers or faculty members), we have identified 28 institutions that met our criteria. Working with a third party, we gathered over 574 responses from undergraduate engineering students with makerspace experiences spread across the 28 institutions. To gather our data, we created and validated an online survey with a combination of quantitative and qualitative items. We constructed a survey with subscales aligned with motivation to learn, growth mindset, learning goal orientation, knowledge of engineering as a profession, and belongingness and inclusion, as associated with work within makerspaces. We found significant positive correlations among the variables, positive levels of motivation, growth mindset, knowledge of engineering as a profession, and belongingness. We found differences in levels for gender, engineering majors, and student class standing. We discuss the implications for our findings in the context of undergraduate engineering student learning in makerspaces.more » « less
-
Building upon our two years of research on the use of makerspaces in undergraduate engineering programs, we engaged in a large-scale data collection from students enrolled in undergraduate engineering preparation programs with affiliated makerspaces established for a minimum of three years. Using web searches, and other sources of information (e.g. references from other researchers or faculty members), we have identified 28 institutions that met our criteria. Working with a third party, we gathered over 574 responses from undergraduate engineering students with makerspace experiences spread across the 28 institutions. To gather our data, we created and validated an online survey with a combination of quantitative and qualitative items. We constructed a survey with subscales aligned with motivation to learn, growth mindset, learning goal orientation, knowledge of engineering as a profession, and belongingness and inclusion, as associated with work within makerspaces. We found significant positive correlations among the variables, positive levels of motivation, growth mindset, knowledge of engineering as a profession, and belongingness. We found differences in levels for gender, engineering majors, and student class standing. We discuss the implications for our findings in the context of undergraduate engineering student learning in makerspaces.more » « less
-
The purpose of this study is to develop an instrument to measure student perceptions about the learning experiences in their online undergraduate engineering courses. Online education continues to grow broadly in higher education, but the movement toward acceptance and comprehensive utilization of online learning has generally been slower in engineering. Recently, however, there have been indicators that this could be changing. For example, ABET has accredited online undergraduate engineering degrees at Stony Brook University and Arizona State University (ASU), and an increasing number of other undergraduate engineering programs also offer online courses. During this period of transition in engineering education, further investigation about the online modality in the context of engineering education is needed, and survey instrumentation can support such investigations. The instrument presented in this paper is grounded in a Model for Online Course-level Persistence in Engineering (MOCPE), which was developed by our research team by combining two motivational frameworks used to study student persistence: the Expectancy x Value Theory of Achievement Motivation (EVT), and the ARCS model of motivational design. The initial MOCPE instrument contained 79 items related to students’ perceptions about the characteristics of their courses (i.e., the online learning management system, instructor practices, and peer support), expectancies of course success, course task values, perceived course difficulties, and intention to persist in the course. Evidence of validity and reliability was collected using a three-step process. First, we tested face and content validity of the instrument with experts in online engineering education and online undergraduate engineering students. Next, the survey was administered to the online undergraduate engineering student population at a large, Southwestern public university, and an exploratory factor analysis (EFA) was conducted on the responses. Lastly, evidence of reliability was obtained by computing the internal consistency of each resulting scale. The final instrument has seven scales with 67 items across 10 factors. The Cronbach alpha values for these scales range from 0.85 to 0.97. The full paper will provide complete details about the development and psychometric evaluation of the instrument, including evidence of and reliability. The instrument described in this paper will ultimately be used as part of a larger, National Science Foundation-funded project investigating the factors influencing online undergraduate engineering student persistence. It is currently being used in the context of this project to conduct a longitudinal study intended to understand the relationships between the experiences of online undergraduate engineering students in their courses and their intentions to persist in the course. We anticipate that the instrument will be of interest and use to other engineering education researchers who are also interested in studying the population of online students.more » « less
-
Abstract As the popularity of makerspaces and maker culture has skyrocketed over the past two decades, numerous studies have been conducted to investigate the benefits of makerspaces for university students and how to best establish an inclusive, welcoming environment in these spaces on college campuses. However, unprecedented disruptions, such as the COVID-19 pandemic, have the potential to greatly affect the way that students interact with makerspaces and the benefits that result. In this study, a survey asking about prior makerspace involvement, tool usage, and student demographics was administered to students who use academic makerspaces at two large public universities. Survey data was collected for three semesters (Fall 2020, Spring 2021, and Spring 2022) and spanned both during and after the height of the COVID-19 pandemic. To quantify the differences between the semesters, nestedness and connectance metrics inspired by ecological plant-pollinator networks were utilized. These ecological metrics allow for the structure of the interactions of a network to be measured, with nestedness highlighting how students interact with tools and connectance with the quantity of student-to-tool interaction. The network analysis was used to better gauge the health of the makerspace and the type and frequency of interactions between tools. The raw survey data combined with the ecological metrics provided unique insight into the struggles the makerspaces encountered throughout the pandemic. It was found that nestedness, a measure of system stability, decreases with a decrease in tool usage. Additionally, the higher the connectance the more students interacted with the space. Utilizing metrics such as these and better understanding student tool interactions can aid makerspaces in monitoring their success and maintaining a healthy and welcoming space, as well as tracking the current health of the space. In combination with the survey results, a deep understanding of what challenges the space is facing can be captured.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
