skip to main content


Title: From Vision to Evaluation: A Metrics Framework for the ACCESS Allocations Service
Abstract

The Allocations Service for the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program is charged with accepting, reviewing, and processing researchers’ requests to use resources that are integrated into the ACCESS ecosystem. We present as a case study the metrics framework used to evaluate the Allocations Service project, a metrics framework that aligns with the project’s goals and identifies key performance indicators (KPIs). Several of our top-level KPIs reflect complex concepts and are composite measures built from suites of metrics compiled from two primary sources: a well-instrumented allocations and accounting system and an annual survey of the ACCESS researcher community. This approach allows us to describe and measure complex concepts such as “democratization” and “ecosystem access time” in a quantitative manner and to target improvements to project activities. The metrics framework is augmented by metrics to measure the performance of the project team, to describe general ecosystem and allocations activity, and to capture publications from the researcher community. We used this framework to gather and present data as part of the ACCESS Allocations Service first annual NSF panel review. The metrics were largely successful at communicating our progress, but we also encountered a few unexpected technical issues with the data and calculations themselves, which we are continuing to refine. Presented here as a case study, this approach to a metrics framework for the Allocations Service has proved valuable in complementing more subjective descriptions of the project, its accomplishments, and progress toward our goals.

 
more » « less
NSF-PAR ID:
10502410
Author(s) / Creator(s):
; ;
Publisher / Repository:
Springer Science + Business Media
Date Published:
Journal Name:
SN Computer Science
Volume:
5
Issue:
5
ISSN:
2661-8907
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Urban water system managers face a set of interrelated water security challenges as they pursue the goals of sustainable sources of water, mitigating flood hazards, and improving water quality. These challenges are often subject to change (and hence highly uncertain) due to the coupled effects of hydro-climatic variability, socio-economic trends, and regulatory reforms. To meet these intersecting goals, we present a mechanistic framework with illustrative examples that evaluates an urban water system’s resilience under future uncertainty. By employing principles from engineering design, ecosystem science, and social equity studies, our resilient urban water systems (ReUWS) framework explores the potential of effectively combining green and gray infrastructure (GGI) in an urban watershed while prioritizing stakeholder and community engagement throughout the lifecycle of water system projects. A nested set of hydrology, ecosystem, and hydraulic models are developed with data flow among them defining the boundary and initial conditions for each other. An example is shown with the Baltimore water system on an approach to evaluate the effects of GGI hybrids on major water security metrics. The corresponding engineering designs, ecosystem service potentials, and measures of equitable access to services are also analyzed using the framework. The results evaluate performance of the existing systems under future conditions and also compare different GGI-based strategies for improving resilience in urban water systems. The findings of the study help to evaluate the potential for using GGI strategies to cope with changing climate extremes and other environmental factors as well as social change. Trade-offs derived from the case studies also can be used to adjust local/regional policies and regulations. 
    more » « less
  2. Facilitating the development of a common framework for monitoring progress in K-12 computer science (CS) education and advocacy with an emphasis on broadening participation is the key to constructing strong CS education policy. Based on a project that brought together leadership teams from six states, a framework for measuring broadening participation in computing (BPC) and setting the foundation for national scaling was developed. Built around a collaboration of leaders representing experience in data gathering, data analysis, data reporting, and data utilization, this project applied the tenets of collective impact to address the challenge of consistently measuring progress toward BPC across state contexts. By establishing a common agenda, including mutually agreed upon definitions of computer science education and broadening participation, these leaders guided the selection of metrics. This led to the development of shared measurement systems and built a deeper understanding of state data systems across the participating states. This phase resulted in common goals and a monitoring system to measure BPC efforts that could inform state policy efforts. Mutually reinforcing activities included the development and sharing of tools, allowing stakeholders to quickly and accurately analyze and disseminate data that drives BPC measurement and policy work. Guided by backbone support to coordinate the work and continuous communication, meaningful participation of all stakeholders was central to the project. Making the case for CS education policy via common metrics and measuring progress across a region stands to impact BPC policy efforts across the United States. The common framework developed in this project serves as a call to action, especially for state and local education agencies committed to increasing diversity in computer science pathways.

     
    more » « less
  3. The researchers conducted a qualitative case study to describe the experiences (over the course of a semester) of an inter-disciplinary team of three special education and three psychology undergraduates who participated in a relational learning community and a graduate student who designed and facilitated this learning community. An associate professor and special education researcher oversaw and co-facilitated the project. The design of the learning community promoted the building of rapport and trust among the group members and the progress of the group toward a common goal of incorporating principles from psychology to develop teaching strategies for students who are struggling in math and experiencing severe math anxiety. Gathering more frequent and individualized feedback would have helped the learning community facilitator make some key adjustments earlier in the project, but the incorporation of rapport building activities that supported trust and collaboration among the group was supportive of group progress toward a common goal. We learned key lessons about how to design and implement a learning community that can be applied to the field of education, interdisciplinary collaboration, and other contexts. 
    more » « less
  4. Data Management Plans (DMP) are now a routine part of research proposals but are generally not referred to after funding is granted. The Belmont Forum requires an extensive document, a ‘Data and Digital Object Management Plan’ (D(DO)MP) for its awarded projects that is expected to be kept current over the life of the project. The D(DO)MP is intended to record team decisions about major tools and practices to be used over the life of the project for data and software stewardship, and for preservation of data and software products, aligned with the desired Open Science outcomes relevant to the project. Here we present one of the first instances of the use of Belmont’s D(DO)MP through a case study of the PARSEC project, a multinational and multidisciplinary investigation of the socioeconomic impacts of protected areas. We describe the development and revision of our interpretation of the D(DO)MP and discuss its adoption and acceptance by our research group. We periodically assessed the data management sophistication of team members and their use of the various nominated tools and practices. As a result, for example, we included summaries to enable the key components of the D(DO)MP to be readily viewed by the researcher. To meet the Open Science outcomes in a complex project like PARSEC, a comprehensive and appropriately structured D(DO)MP helps project leaders (a) ensure that team members are committed to the collaboration goals of the project, (b) that there is regular and effective feedback within the team, (c) training in new tools is provided as and when needed, and (d) there is easy access to a short reference to the tools and descriptions of the nominated practices. 
    more » « less
  5. Seagroves, Scott ; Barnes, Austin ; Metevier, Anne ; Porter, Jason ; Hunter, Lisa (Ed.)
    Data literacy and the ability to synthesize and communicate complex concepts are core components of modern scientific practice. Here we present the design and implementation of an inquiry activity about climate variability that was taught as a part of the University of California, Santa Cruz (UCSC) Workshops for Engineering & Science Transfers (ClimateWEST) in 2019. The two-day activity introduced interdisciplinary undergraduate and community college transfer students pursuing graduate school to the field of climate science through a series of inquiry activities. Climate science is a complex topic, and research shows that there are certain concepts that are particularly difficult to grasp. Our climate activity focused on disentangling some of those misconceptions, by emphasizing the following themes or core dimensions of climate variability: (1) Climate varies on both shorter timescales (e.g. seasonal or annual cycle) and on longer timescales (e.g. climate change); (2) Both climate and climate trends vary spatially/geographically and are different from global climate; and (3) Climate is complex and includes not only temperature but also other key variables such as precipitation, ice, wind, ocean circulation, etc. We discuss the inquiry components, assessment-driven tools, facilitation and equity and inclusion design, as well as summarize students' progress toward our goals in the activity. 
    more » « less