skip to main content


Title: Design and testing of a quantitative instrument to evaluate engineering research center participation
This research paper reports the in-progress validation of a quantitative instrument designed to assess the perceived impact of participating in a National Science Foundation (NSF)-funded Engineering Research Center (ERC). A multi-institutional consortium composed of ERC education directors, researchers, and evaluators from six NSF-funded ERCs designed easily accessible evaluation instruments and tools that specifically help measure anticipated outcomes for ERC participants for all ERCs. The total effort underway by the consortium includes creating a suite of qualitative and quantitative instruments, an evaluator toolkit, and a user-friendly online platform to host the inventory materials. This paper focuses on the quantitative instrument created to evaluate the experiences of those who engage with a center. It consists of Likert-type questions assessing the impact of the ERC on participants' self-reported: 1) understanding of the ERC, 2) research and communication skills, 3) climate of inclusion, 4) mentorship experiences, and 5) program satisfaction. The instrument also included additional demographic questions and questions to capture STEM-related future plans. The instrument was designed using multiple rounds of design iterations and pilot tests. Separate surveys used by individual ERCs were compiled and categorized to ensure all requirements from the National Science Foundation were met. The web-based survey was administered to six ERCs during the Summer of 2021, Fall of 2021, and Spring of 2022. A total of 549 responses were collected; 535 were used following data cleaning procedures. Sample sizes for each component of the survey varied because some ERCs chose to only use some parts of the new instrument. Exploratory Factor Analyses (EFA) were performed to identify latent factors and items that needed further revision. The following factors emerged from our analyses: 1) ERC general understanding; 2) development of research skills; 3) development of professional skills; 4) experience in the ERC; 5) feelings toward the ERC; 6) Beliefs about the ERC, 7) mentors performance; and 8) mentorship experience. The results provide preliminary evidence that the survey can be used across ERCs. This effort is the first that has been undertaken to develop a shared ERC instrument. The data collected was used to continue in-progress validation. The collaborative nature of this effort can provide ways for ERCs to benchmark impacts of their efforts and share effective practices across ERCs and other similarly structured STEM centers going forward.  more » « less
Award ID(s):
2023275
NSF-PAR ID:
10436302
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Design and Testing of a Quantitative Instrument to Evaluate Engineering Research Center Participation
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. The Engineering Research Centers (ERCs), funded by the National Science Foundation (NSF), play an important role in improving engineering education, bridging engineering academia and broad communities, and promoting a culture of diversity and inclusion. Each ERC must partner with an independent evaluation team to annually assess their performance and impact on progressing education, connecting community, and building diversified culture. This evaluation is currently performed independently (and in isolation), which leads to inconsistent evaluations and a redundant investment of ERCs’ resources into such tasks (e.g. developing evaluation instruments). These isolated efforts by ERCs to quantitatively evaluate their education programs also typically lack adequate sample size within a single center, which limits the validity and reliability of the quantitative analyses. Three ERCs, all associated with a large southwest university in the United States, worked collaboratively to overcome sample size and measure inconsistency concerns by developing a common quantitative instrument that is capable of evaluating any ERC’s education and diversity impacts. The instrument is the result of a systematic process with comparing and contrasting each ERC’s existing evaluation tools, including surveys and interview protocols. This new, streamlined tool captures participants’ overall experience as part of the ERC by measuring various constructs including skillset development, perception of diversity and inclusion, future plans after participating in the ERC, and mentorship received from the ERC. Scales and embedded items were designed broadly for possible use with both yearlong (e.g. graduate and undergraduate student, and postdoctoral scholars) and summer program (Research Experience for Undergraduates, Research Experience for Teachers, and Young Scholar Program) participants. The instrument was distributed and tested during Summer 2019 with participants in the summer programs from all three ERCs. The forthcoming paper will present the new common cross-ERC evaluation instrument, demonstrate the effort of collecting data across all three ERCs, present preliminary findings, and discuss collaborative processes and challenges. The preliminary implication for this work is the ability to directly compare educational programs across ERCs. The authors also believe that this tool can provide a fast start for new ERCs on how to evaluate their educational programs. 
    more » « less
  2. National Science Foundation (NSF) funded Engineering Research Centers (ERC) must complement their technical research with various education and outreach opportunities to: 1) improve and promote engineering education, both within the center and to the local community; 2) encourage and include the underrepresented populations to participate in Engineering activities; and 3) advocate communication and collaboration between industry and academia. ERCs ought to perform an adequate evaluation of their educational and outreach programs to ensure that beneficial goals are met. Each ERC has complete autonomy in conducting and reporting such evaluation. Evaluation tools used by individual ERCs are quite similar, but each ERC has designed their evaluation processes in isolation, including evaluation tools such as survey instruments, interview protocols, focus group protocols, and/or observation protocols. These isolated efforts resulted in redundant resources spent and lacking outcome comparability across ERCs. Leaders from three different ERCs led and initiated a collaborative effort to address the above issue by building a suite of common evaluation instruments that all current and future ERCs can use. This leading group consists of education directors and external evaluators from all three partners ERCs and engineering education researchers, who have worked together for two years. The project intends to address the four ERC program clusters: Broadening Participation in Engineering, Centers and Networks, Engineering Education, and Engineering Workforce Development. The instruments developed will pay attention to culture of inclusion, outreach activities, mentoring experience, and sustained interest in engineering. The project will deliver best practices in education program evaluation, which will not only support existing ERCs, but will also serve as immediate tools for brand new ERCs and similar large-scale research centers. Expanding the research beyond TEEC and sharing the developed instruments with NSF as well as other ERCs will also promote and encourage continual cross-ERC collaboration and research. Further, the joint evaluation will increase the evaluation consistency across all ERC education programs. Embedded instrumental feedback loops will lead to continual improvement to ERC education performance and support the growth of an inclusive and innovative engineering workforce. Four major deliveries are planned. First, develop a common quantitative assessment instrument, named Multi-ERC Instrument Inventory (MERCII). Second, develop a set of qualitative instruments to complement MERCII. Third, create a web-based evaluation platform for MERCII. Fourth, update the NSF ERC education program evaluation best practice manual. These deliveries together will become part of and supplemented by an ERC evaluator toolbox. This project strives to significantly impact how ERCs evaluate their educational and outreach programs. Single ERC based studies lack the sample size to truly test the validity of any evaluation instruments or measures. A common suite of instruments across ERCs would provide an opportunity for a large scale assessment study. The online platform will further provide an easy-to-use tool for all ERCs to facilitate evaluation, share data, and reporting impacts. 
    more » « less
  3. National Science Foundation (NSF) funded Engineering Research Centers (ERC) are required to develop and implement education and outreach opportunities related to their core technical research topics to broaden participation in engineering and create partnerships between industry and academia. Additionally, ERCs must include an independent evaluation of their education and outreach programming to assess their performance and impacts. To date, each ERC’s evaluation team designs its instruments/tools and protocols for evaluation, resulting in idiosyncratic and redundant efforts. Nonetheless, there is much overlap among the evaluation topics, concepts, and practices, suggesting that the ERC evaluation and assessment community might benefit from having a common set of instruments and protocols. ERCs’ efforts could then be better spent developing more specific, sophisticated, and time-intensive evaluation tools to deepen and enrich the overall ERC evaluation efforts. The implementation of such a suite of instruments would further allow each ERC to compare its efforts to those across other ERCs as one data point for assessing its effectiveness and informing its improvement efforts. Members of a multi-ERC collaborative team, funded by the NSF, have been leading a project developing a suite of common instruments and protocols which contains both quantitative and qualitative tools. This paper reports on the development of a set of qualitative instruments that, to date, includes the following: (a) a set of interview/focus group protocols intended for various groups of ERC personnel, centered around five common topics/areas, and (b) rubrics for summer program participants' verbal poster/presentations and their written poster/slide deck presentation artifacts. The development process is described sequentially, beginning with a review of relevant literature and existing instruments, followed by the creation of an initial set of interview questions and rubric criteria. The initial versions of the tools were then pilot-tested with multiple ERCs. Feedback sessions with education/evaluation leaders of those piloting ERCs were then conducted, through which further revision efforts were made. 
    more » « less
  4. Due to the COVID-19 crisis preventing face-to-face interaction, three National Science Foundation (NSF)-funded centers employed a virtual/remote format for their summer Research Experiences for Teachers (RET) Programs, reaching K-12 STEM teachers across the country. Teachers participated virtually from four different states by joining engineering research teams from four different universities in three different RET programs. Lab experiences depended on the nature of the research and institution-specific guidelines for in-lab efforts, resulting in some teachers conducting lab experiments with materials sent directly to their homes, some completing their experience fully online, and some completing portions of lab work in person on campus. Each teacher developed an engineering lesson plan based on the corresponding center’s research to be implemented either in person or virtually during the 2020-2021 academic school year. Research posters, created with support from graduate student and faculty mentors, were presented to industry partners, education partners, center members, and the NSF. Support for the teachers as they implement lessons, present posters, and disseminate their developed curricula, has continued throughout the year. Common survey and interview/focus group protocols, previously designed specifically for measuring the impact of engineering education programs, were adapted and used to separately evaluate each of the three virtual programs. Strengths and suggested areas of improvement will be explored and discussed to inform future use of the common evaluation instruments. Additionally, preliminary results, highlighting general successes and challenges of shifting RET programming to a virtual/remote format across the three centers, will be discussed. 
    more » « less
  5. Millions of dollars each year are invested in intervention programs to broaden participation and improve bachelor degree graduation rates of students enrolled in science, technology, engineering, and mathematics (STEM) disciplines. The Virginia–North Carolina Louis Stokes Alliance for Minority Participation (VA-NC Alliance), a consortium of 11 higher education institutions and one federal laboratory funded by the National Science Foundation (NSF), is one such investment., The VA-NC Alliance partners implement evidence-based STEM intervention programs (SIPs) informed by research and specifically designed to increase student retention and graduation rates in STEM majors. The VA-NC Alliance is conducting an Alliance-wide longitudinal research project based in Social Cognitive Career Theory (SCCT) titled “What’s Your STEMspiration?” The goal of the research project is to assess the differentiated impacts and effectiveness of the Alliance’s broadening participation efforts and identify emergent patterns, adding to the field of knowledge about culturally responsive SIPs. In other words, “What’s Your STEMspiration?” explores what influences and inspires undergraduates to pursue a STEM degree and career; and how does the development of a STEM identity support students in achieving their goals. In order to complete this research, the research team developed a survey instrument to conduct the quantitative portion of the study. Two preliminary studies, statistical analysis, and cognitive interviews were used to develop and validate the survey instrument. This paper discusses the theoretical and conceptual frameworks and preliminary studies upon which the survey is built, the methodology used to validate the instrument, and the resulting final survey tool.

     
    more » « less