skip to main content

Title: The Systems Evaluation Protocol for evaluation planning

In this paper, we present a comprehensive integrative approach to evaluation planning based on Relational Systems Evaluation (RSE). We describe the major implementation approaches and the tools and resources that support evaluation planning in RSE embodied in the Systems Evaluation Protocol (SEP), a step‐by‐step guide to the tasks that every evaluation should address. The SEP provides both a conceptual framework for thinking about evaluation and a set of specific methods and tools that expand awareness of the influences of multiple parts of complex systems. The SEP is implemented through a partnership approach, joining evaluators and program professionals in an iterative, phased approach to evaluation planning, which also often emphasizes evaluation capacity building (ECB) as an intentional goal. This paper provides a practice‐oriented overview of the specific steps comprising the three stages outlined in the SEP: preparation, model development, and evaluation plan development.

more » « less
Author(s) / Creator(s):
 ;  ;  ;  ;  ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
New Directions for Evaluation
Page Range / eLocation ID:
p. 31-45
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    In this paper, we present six case studies that illustrate the application of the Systems Evaluation Protocol (SEP) in different real‐world implementation conditions. The SEP is a step‐by‐step guide for how to implement Relational Systems Evaluation (RSE), accounting for the complex factors inherent in the larger systems within which a given program is embedded. We discuss the specific SEP steps used and products developed in each case study project, including an emphasis on how decisions were made to include particular SEP elements in different contexts. The six cases differ in terms of the scale of the programs involved, the nature of the SEP delivery or facilitation process, and the balance among multiple desired outcomes reflecting differential emphases on evaluation capacity building. As such, the cases reported here demonstrate that RSE, as implemented using the SEP, is a widely applicable approach to evaluation and ECB.

    more » « less
  2. Abstract

    In this paper, we introduce Relational Systems Evaluation (RSE), the focal topic of this volume. RSE is a framework for program planning and evaluation that is theoretically grounded, empirically tested and focused on building evaluation capacity. Theoretically, RSE is rooted in an evolutionary approach to program development and evaluation informed by systems thinking. The application of RSE involves a collaborative partnership approach pairing evaluators or researchers with program practitioners, the use of collaborative modeling and planning tools, and an emphasis on nurturing an Evaluative Thinking mindset. The goal of RSE is to produce more thoughtful and useful evaluations and, ultimately, better programs.

    more » « less
  3. To reshape energy systems towards renewable energy resources, decision makers need to decide today on how to make the transition. Energy scenarios are widely used to guide decision making in this context. While considerable effort has been put into developing energy scenarios, researchers have pointed out three requirements for energy scenarios that are not fulfilled satisfactorily yet: The development and evaluation of energy scenarios should (1) incorporate the concept of sustainability, (2) provide decision support in a transparent way and (3) be replicable for other researchers. To meet these requirements, we combine different methodological approaches: story-and-simulation (SAS) scenarios, multi-criteria decision-making (MCDM), information modeling and co-simulation. We show in this paper how the combination of these methods can lead to an integrated approach for sustainability evaluation of energy scenarios with automated information exchange. Our approach consists of a sustainability evaluation process (SEP) and an information model for modeling dependencies. The objectives are to guide decisions towards sustainable development of the energy sector and to make the scenario and decision support processes more transparent for both decision makers and researchers. 
    more » « less
  4. Abstract

    Telementoring platforms can help transfer surgical expertise remotely. However, most telementoring platforms are not designed to assist in austere, pre-hospital settings. This paper evaluates the system for telementoring with augmented reality (STAR), a portable and self-contained telementoring platform based on an augmented reality head-mounted display (ARHMD). The system is designed to assist in austere scenarios: a stabilized first-person view of the operating field is sent to a remote expert, who creates surgical instructions that a local first responder wearing the ARHMD can visualize as three-dimensional models projected onto the patient’s body. Our hypothesis evaluated whether remote guidance with STAR could lead to performing a surgical procedure better, as opposed to remote audio-only guidance. Remote expert surgeons guided first responders through training cricothyroidotomies in a simulated austere scenario, and on-site surgeons evaluated the participants using standardized evaluation tools. The evaluation comprehended completion time and technique performance of specific cricothyroidotomy steps. The analyses were also performed considering the participants’ years of experience as first responders, and their experience performing cricothyroidotomies. A linear mixed model analysis showed that using STAR was associated with higher procedural and non-procedural scores, and overall better performance. Additionally, a binary logistic regression analysis showed that using STAR was associated to safer and more successful executions of cricothyroidotomies. This work demonstrates that remote mentors can use STAR to provide first responders with guidance and surgical knowledge, and represents a first step towards the adoption of ARHMDs to convey clinical expertise remotely in austere scenarios.

    more » « less
  5. Abstract

    The use of machine learning techniques to expedite the discovery and development of new materials is an essential step towards the acceleration of a new generation of domain-specific highly functional material systems. In this paper, we use the test case of bulk metallic glasses to highlight the key issues in the field of high throughput predictions and propose a new probabilistic analysis of rules for glass forming ability using rough set theory. This approach has been applied to a broad range of binary alloy compositions in order to predict new metallic glass compositions. Our data driven approach takes into account not only a broad variety of thermodynamic, structural and kinetic based criteria, but also incorporates qualitative and descriptive attributes associated with eutectic points in phase diagrams. For the latter, we demonstrate the use of automated machine learning methods that go far beyond text recognition approaches by also being able to interpret phase diagrams. When combined with structural descriptors, this approach provides the foundations to develop a hierarchical probabilistic predication tool that can rank the feasibility of glass formation.

    more » « less