skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Leveraging the Kirkpatrick four‐level model to evaluate evaluation capacity building work
Abstract In this article, we reflect on a decade of using the Kirkpatrick four‐level model to evaluate a multifaceted evaluation capacity building (ECB) initiative. Traditionally used to assess business training efforts, the Kirkpatrick model encourages evidence to be gathered at four levels: reaction, learning, behavior, and results. We adapted these levels to fit the context and information needs of the EvaluATE project, an ECB initiative funded by the National Science Foundation. As members of the external evaluation and project teams, throughout the article we describe how each level was modified and translated into evaluation questions. Our adapted Kirkpatrick levels are implementation and reach, satisfaction, learning, application, and impact. Using these adapted Kirkpatrick levels to ground our evaluation challenged us to integrate multiple data sources to tell a comprehensive story that served the information needs of the project team and the funder. Overall, we found the Kirkpatrick model to be practical, accessible, and flexible, allowing us to capture the multidimensional aspects of the ECB initiative. However, there are opportunities to enhance the utility of the Kirkpatrick framework by integrating other evaluation approaches, such as culturally responsive and equitable evaluation and principles‐focused evaluation.  more » « less
Award ID(s):
1841783
PAR ID:
10561188
Author(s) / Creator(s):
; ; ; ; ;
Publisher / Repository:
Wiley
Date Published:
Journal Name:
New Directions for Evaluation
Volume:
2024
Issue:
183
ISSN:
1097-6736
Page Range / eLocation ID:
81 to 94
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, BMC Medical Education)
    Abstract Background Traditionally, doctoral student education in the biomedical sciences relies on didactic coursework to build a foundation of scientific knowledge and an apprenticeship model of training in the laboratory of an established investigator. Recent recommendations for revision of graduate training include the utilization of graduate student competencies to assess progress and the introduction of novel curricula focused on development of skills, rather than accumulation of facts. Evidence demonstrates that active learning approaches are effective. Several facets of active learning are components of problem-based learning (PBL), which is a teaching modality where student learning is self-directed toward solving problems in a relevant context. These concepts were combined and incorporated in creating a new introductory graduate course designed to develop scientific skills (student competencies) in matriculating doctoral students using a PBL format. Methods Evaluation of course effectiveness was measured using the principals of the Kirkpatrick Four Level Model of Evaluation. At the end of each course offering, students completed evaluation surveys on the course and instructors to assess their perceptions of training effectiveness. Pre- and post-tests assessing students’ proficiency in experimental design were used to measure student learning. Results The analysis of the outcomes of the course suggests the training is effective in improving experimental design. The course was well received by the students as measured by student evaluations (Kirkpatrick Model Level 1). Improved scores on post-tests indicate that the students learned from the experience (Kirkpatrick Model Level 2). A template is provided for the implementation of similar courses at other institutions. Conclusions This problem-based learning course appears effective in training newly matriculated graduate students in the required skills for designing experiments to test specific hypotheses, enhancing student preparation prior to initiation of their dissertation research. 
    more » « less
  2. ; ; ; ; (, New Directions for Evaluation)
    Mason, Sarah; Montrosse-Moorhead, Bianca (Ed.)
    This article seeks to bring practitioner experience to bear on existing models of ECB with a particular focus on the models through the lens of our own ECB practice. We reflect on how our ECB practices align with or challenge these models, and how the insights that come from those reflections can inform future ECB research and frameworks for evaluating ECB initiatives. As is often the case when theory collides with practice, current models may not always reflect and serve the work at hand, and the value and usefulness, as well as accuracy and relevance, of existing models is worth investigating. With this in mind, we offer input to inform future models of ECB that are more inclusive of and relevant for the broad spectrum of current ECB practice and, subsequently, its evaluation. 
    more » « less
  3. ; ; ; ; ; ; ; ; ; (, Frontiers in Psychology)
    What if the environment could be transformed in culturally-responsive and inclusive ways to foster high-quality interactions and spark conversations that drive learning? In this article, we describe a new initiative accomplishing this, called Playful Learning Landscapes (PLL). PLL is an evidence-based initiative that blends findings from the science of learning with community-based participatory research to transform physical public spaces and educational settings into playful learning hubs. Here, we describe our model for conducting this research, which is mindful of three key components: community input, how children learn best, and what children need to learn to be successful in the 21st century economy. We describe how this model was implemented in two PLL case studies: one in a predominantly Latine community and the second in early childhood education classrooms. Furthermore, we describe how research employing our model can be rigorously and reliably evaluated using observational and methodological tools that respond to diverse cultural settings and learning outcomes. For example, our work evaluates how PLL impacts adult–child interaction quality and language use, attitudes about play and learning, and community civic engagement. Taken together, this article highlights new ways to involve community voices in developmental and educational research and provides a model of how science can be translated into practice and evaluated in culturally responsive ways. This synthesis of our process and evaluation can be used by researchers, policymakers, and educators to reimagine early educational experiences with an eye toward the built environment that children inhabit in everyday life, creating opportunities that foster lifelong learning. 
    more » « less
  4. ; ; ; ; ; (, Journal of Curriculum and Teaching)
    Ella Smith (Ed.)
    The Billion Oyster Project and Curriculum and Community Enterprise for the Restoration of New York Harbor with New York City Public Schools (BOP-CCERS) program is a National Science Foundation (NSF) supported initiative that involves multiple stakeholder collaborations and is led by Pace University. Within Pace, the initiative crosses over three different schools and colleges and across multiple departments. Pace University’s NSF project of focus in this article is the Innovative Technology Experiences for Students and Teachers (ITEST) program. The purpose is to increase student motivation and preparation for pursuing STEM careers. This article presents results of programming to increase student scientific identity. Findings revealed that students in the 11th grade had the highest level of scientific identity compared to other high school grades. These findings indicate that 11th grade may be an ideal timeframe for interventions to improve scientific identity. Moreover, project participants had higher levels of engagement in STEM-related activities, were more likely to watch videos made by scientists, and read articles written by scientists than those in the control group. 
    more » « less
  5. ; ; ; ; (, Oceanography)
    Look at a graph of oceanographic data. How would you convey the information only using sound? This is the challenge addressed by the US National Science Foundation-funded “Accessible Oceans” project, which seeks to increase access to ocean data in informal learning environments like museums, science centers, and aquariums. Quantitative information in these settings is almost always conveyed with visual displays. For audience members who are blind or have low vision, or those with data illiteracy or innumeracy, these displays remain inaccessible. In this article, we discuss our interdisciplinary project using an inclusive human-centered design process to develop auditory displays that support learning and understanding of ocean science in informal learning environments. We share information about our project and takeaways for inclusive auditory display design, hoping to inspire others to examine the accessibility of their own work. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email