skip to main content

Title: Design and Development: NSF Engineering Research Centers Unite: Developing and Testing a Suite of Instruments to Enhance Overall Education Program Evaluation
National Science Foundation (NSF) funded Engineering Research Centers (ERC) must complement their technical research with various education and outreach opportunities to: 1) improve and promote engineering education, both within the center and to the local community; 2) encourage and include the underrepresented populations to participate in Engineering activities; and 3) advocate communication and collaboration between industry and academia. ERCs ought to perform an adequate evaluation of their educational and outreach programs to ensure that beneficial goals are met. Each ERC has complete autonomy in conducting and reporting such evaluation. Evaluation tools used by individual ERCs are quite similar, but each ERC has designed their evaluation processes in isolation, including evaluation tools such as survey instruments, interview protocols, focus group protocols, and/or observation protocols. These isolated efforts resulted in redundant resources spent and lacking outcome comparability across ERCs. Leaders from three different ERCs led and initiated a collaborative effort to address the above issue by building a suite of common evaluation instruments that all current and future ERCs can use. This leading group consists of education directors and external evaluators from all three partners ERCs and engineering education researchers, who have worked together for two years. The project intends to address more » the four ERC program clusters: Broadening Participation in Engineering, Centers and Networks, Engineering Education, and Engineering Workforce Development. The instruments developed will pay attention to culture of inclusion, outreach activities, mentoring experience, and sustained interest in engineering. The project will deliver best practices in education program evaluation, which will not only support existing ERCs, but will also serve as immediate tools for brand new ERCs and similar large-scale research centers. Expanding the research beyond TEEC and sharing the developed instruments with NSF as well as other ERCs will also promote and encourage continual cross-ERC collaboration and research. Further, the joint evaluation will increase the evaluation consistency across all ERC education programs. Embedded instrumental feedback loops will lead to continual improvement to ERC education performance and support the growth of an inclusive and innovative engineering workforce. Four major deliveries are planned. First, develop a common quantitative assessment instrument, named Multi-ERC Instrument Inventory (MERCII). Second, develop a set of qualitative instruments to complement MERCII. Third, create a web-based evaluation platform for MERCII. Fourth, update the NSF ERC education program evaluation best practice manual. These deliveries together will become part of and supplemented by an ERC evaluator toolbox. This project strives to significantly impact how ERCs evaluate their educational and outreach programs. Single ERC based studies lack the sample size to truly test the validity of any evaluation instruments or measures. A common suite of instruments across ERCs would provide an opportunity for a large scale assessment study. The online platform will further provide an easy-to-use tool for all ERCs to facilitate evaluation, share data, and reporting impacts. « less
Authors:
; ; ; ; ; ; ; ; ;
Award ID(s):
2023275
Publication Date:
NSF-PAR ID:
10343960
Journal Name:
American Society for Engineering Education
Sponsoring Org:
National Science Foundation
More Like this
  1. National Science Foundation (NSF) funded Engineering Research Centers (ERC) are required to develop and implement education and outreach opportunities related to their core technical research topics to broaden participation in engineering and create partnerships between industry and academia. Additionally, ERCs must include an independent evaluation of their education and outreach programming to assess their performance and impacts. To date, each ERC’s evaluation team designs its instruments/tools and protocols for evaluation, resulting in idiosyncratic and redundant efforts. Nonetheless, there is much overlap among the evaluation topics, concepts, and practices, suggesting that the ERC evaluation and assessment community might benefit from havingmore »a common set of instruments and protocols. ERCs’ efforts could then be better spent developing more specific, sophisticated, and time-intensive evaluation tools to deepen and enrich the overall ERC evaluation efforts. The implementation of such a suite of instruments would further allow each ERC to compare its efforts to those across other ERCs as one data point for assessing its effectiveness and informing its improvement efforts. Members of a multi-ERC collaborative team, funded by the NSF, have been leading a project developing a suite of common instruments and protocols which contains both quantitative and qualitative tools. This paper reports on the development of a set of qualitative instruments that, to date, includes the following: (a) a set of interview/focus group protocols intended for various groups of ERC personnel, centered around five common topics/areas, and (b) rubrics for summer program participants' verbal poster/presentations and their written poster/slide deck presentation artifacts. The development process is described sequentially, beginning with a review of relevant literature and existing instruments, followed by the creation of an initial set of interview questions and rubric criteria. The initial versions of the tools were then pilot-tested with multiple ERCs. Feedback sessions with education/evaluation leaders of those piloting ERCs were then conducted, through which further revision efforts were made.« less
  2. The Engineering Research Centers (ERCs), funded by the National Science Foundation (NSF), play an important role in improving engineering education, bridging engineering academia and broad communities, and promoting a culture of diversity and inclusion. Each ERC must partner with an independent evaluation team to annually assess their performance and impact on progressing education, connecting community, and building diversified culture. This evaluation is currently performed independently (and in isolation), which leads to inconsistent evaluations and a redundant investment of ERCs’ resources into such tasks (e.g. developing evaluation instruments). These isolated efforts by ERCs to quantitatively evaluate their education programs also typicallymore »lack adequate sample size within a single center, which limits the validity and reliability of the quantitative analyses. Three ERCs, all associated with a large southwest university in the United States, worked collaboratively to overcome sample size and measure inconsistency concerns by developing a common quantitative instrument that is capable of evaluating any ERC’s education and diversity impacts. The instrument is the result of a systematic process with comparing and contrasting each ERC’s existing evaluation tools, including surveys and interview protocols. This new, streamlined tool captures participants’ overall experience as part of the ERC by measuring various constructs including skillset development, perception of diversity and inclusion, future plans after participating in the ERC, and mentorship received from the ERC. Scales and embedded items were designed broadly for possible use with both yearlong (e.g. graduate and undergraduate student, and postdoctoral scholars) and summer program (Research Experience for Undergraduates, Research Experience for Teachers, and Young Scholar Program) participants. The instrument was distributed and tested during Summer 2019 with participants in the summer programs from all three ERCs. The forthcoming paper will present the new common cross-ERC evaluation instrument, demonstrate the effort of collecting data across all three ERCs, present preliminary findings, and discuss collaborative processes and challenges. The preliminary implication for this work is the ability to directly compare educational programs across ERCs. The authors also believe that this tool can provide a fast start for new ERCs on how to evaluate their educational programs.« less
  3. This Innovative Practice Work in Progress paper presents the collaborative efforts made by three NSF-funded Engineering Research Centers (ERCs) to synthesize common tools for educational program evaluation. The aim of the NSF ERCs is to achieve transformative changes by integrating engineering research and education with technological innovation within areas at the frontiers of science and engineering (e.g., NSF's 10 Big Ideas). Such centers across the nation study and innovate within their technical area using similar structures and implementation strategies, including the coordination of educational endeavors. Independent partners are enlisted as part of these centers to evaluate education and diversity impactsmore »annually. Each center typically performs this task in isolation from other such centers. The effort required to create resources for such evaluation outcome can result in redundancy and an inability for psychometric analysis due to small available populations within a single center. This paper elaborates on the ongoing efforts of this collaborative research aimed at addressing these issues by creating a streamlined, customizable, and standardized set of evaluation instruments that can be applied to any ERC evaluation.« less
  4. In August 2016, the authors, faculty members at Lafayette College, were awarded a National Science Foundation (NSF) grant (Grant No. CMMI-1632963) based on an unsolicited proposal to the NSF’s CMMI Division. Like many faculty at strictly undergraduate institutions, we routinely provide opportunities for students to work on research projects and fund this research in some situations through external grants. An innovation in this particular grant was the creation of a research collaboration between faculty and students at Lafayette and an NSF-funded Engineering Research Center (ERC). As stated on the NSF website, “The goal of the ERC Program is to integratemore »engineering research and education with technological innovation to transform national prosperity, health, and security.” To accomplish this goal, collaborations between ERCs and other institutions are inherent in the work of an ERC; however, research collaborations between ERCs and small liberal arts colleges are rare and we know of no other collaboration of this type. In our most recent research project, we have developed and implemented a model that successfully provides our students and ourselves with opportunities to collaborate on an interdisciplinary research project with faculty, researchers, and graduate students at the NSF-funded Center for Bio-mediated and Bio-inspired Geotechnics (CBBG). This paper provides a brief overview of the goals of the research project and describes our motivation for establishing the collaboration, the structure of the collaboration, the anticipated broader impacts associated with the work, and the results from the first 18 months of the partnership. A logic model is included to illustrate the connections between the resources, strategies, outcomes, and long-term impacts associated with the collaboration. The goal of this paper is to describe the collaboration between Lafayette College and the ERC from the point of view of the faculty members at Lafayette, to describe the positive outcomes that have resulted from this collaboration, and to encourage faculty members at other small colleges to consider developing similar collaborations.« less
  5. In August 2016, the authors, faculty members at Lafayette College, were awarded a National Science Foundation (NSF) grant (Grant No. CMMI-1632963) based on an unsolicited proposal to the NSF’s CMMI Division. Like many faculty at strictly undergraduate institutions, we routinely provide opportunities for students to work on research projects and fund this research in some situations through external grants. An innovation in this particular grant was the creation of a research collaboration between faculty and students at Lafayette and an NSF-funded Engineering Research Center (ERC). As stated on the NSF website, “The goal of the ERC Program is to integratemore »engineering research and education with technological innovation to transform national prosperity, health, and security.” To accomplish this goal, collaborations between ERCs and other institutions are inherent in the work of an ERC; however, research collaborations between ERCs and small liberal arts colleges are rare and we know of no other collaboration of this type. In our most recent research project, we have developed and implemented a model that successfully provides our students and ourselves with opportunities to collaborate on an interdisciplinary research project with faculty, researchers, and graduate students at the NSF-funded Center for Bio-mediated and Bio-inspired Geotechnics (CBBG). This paper provides a brief overview of the goals of the research project and describes our motivation for establishing the collaboration, the structure of the collaboration, the anticipated broader impacts associated with the work, and the results from the first 18 months of the partnership. A logic model is included to illustrate the connections between the resources, strategies, outcomes, and long-term impacts associated with the collaboration. The goal of this paper is to describe the collaboration between Lafayette College and the ERC from the point of view of the faculty members at Lafayette, to describe the positive outcomes that have resulted from this collaboration, and to encourage faculty members at other small colleges to consider developing similar collaborations.« less