skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: From Vision to Evaluation: A Metrics Framework for the ACCESS Allocations Service
Abstract The Allocations Service for the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program is charged with accepting, reviewing, and processing researchers’ requests to use resources that are integrated into the ACCESS ecosystem. We present as a case study the metrics framework used to evaluate the Allocations Service project, a metrics framework that aligns with the project’s goals and identifies key performance indicators (KPIs). Several of our top-level KPIs reflect complex concepts and are composite measures built from suites of metrics compiled from two primary sources: a well-instrumented allocations and accounting system and an annual survey of the ACCESS researcher community. This approach allows us to describe and measure complex concepts such as “democratization” and “ecosystem access time” in a quantitative manner and to target improvements to project activities. The metrics framework is augmented by metrics to measure the performance of the project team, to describe general ecosystem and allocations activity, and to capture publications from the researcher community. We used this framework to gather and present data as part of the ACCESS Allocations Service first annual NSF panel review. The metrics were largely successful at communicating our progress, but we also encountered a few unexpected technical issues with the data and calculations themselves, which we are continuing to refine. Presented here as a case study, this approach to a metrics framework for the Allocations Service has proved valuable in complementing more subjective descriptions of the project, its accomplishments, and progress toward our goals.  more » « less
Award ID(s):
2138259
PAR ID:
10502410
Author(s) / Creator(s):
; ;
Publisher / Repository:
Springer Science + Business Media
Date Published:
Journal Name:
SN Computer Science
Volume:
5
Issue:
5
ISSN:
2661-8907
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, World Environmental and Water Resources Congress 2021)
    null (Ed.)
    Urban water system managers face a set of interrelated water security challenges as they pursue the goals of sustainable sources of water, mitigating flood hazards, and improving water quality. These challenges are often subject to change (and hence highly uncertain) due to the coupled effects of hydro-climatic variability, socio-economic trends, and regulatory reforms. To meet these intersecting goals, we present a mechanistic framework with illustrative examples that evaluates an urban water system’s resilience under future uncertainty. By employing principles from engineering design, ecosystem science, and social equity studies, our resilient urban water systems (ReUWS) framework explores the potential of effectively combining green and gray infrastructure (GGI) in an urban watershed while prioritizing stakeholder and community engagement throughout the lifecycle of water system projects. A nested set of hydrology, ecosystem, and hydraulic models are developed with data flow among them defining the boundary and initial conditions for each other. An example is shown with the Baltimore water system on an approach to evaluate the effects of GGI hybrids on major water security metrics. The corresponding engineering designs, ecosystem service potentials, and measures of equitable access to services are also analyzed using the framework. The results evaluate performance of the existing systems under future conditions and also compare different GGI-based strategies for improving resilience in urban water systems. The findings of the study help to evaluate the potential for using GGI strategies to cope with changing climate extremes and other environmental factors as well as social change. Trade-offs derived from the case studies also can be used to adjust local/regional policies and regulations. 
    more » « less
  2. ; ; ; ; ; ; ; ; ; ; et al (, Journal of Agriculture, Food Systems, and Community Development)
    Food system sustainability, and ways of measuring it, are widely explored and discussed in academic literature. Measurement efforts are challenging because food systems are inherently complex and multifaceted, spanning diverse components, indus­tries, sectors, and scales. Several systems of indica­tors and metrics have been proposed to measure sustainability; however, most existing research focuses either on narrow scales (e.g., farm level or within a single supply chain), expansive scales that can gloss over complexity (e.g., national or global assessments), or limited scopes (e.g., only consider­ing environmental factors). A gap in the literature is a holistic local or regional approach to food sys­tem sustainability that integrates components across the system at a regional scale. In this reflec­tive essay, we describe our development of a framework to measure and track sustainability in such systems. We use a tiered framework that includes five sustainability dimensions and a system of indices, indicators, and metrics that allows for the measurement of important food system charac­teristics in a feasible and reproducible way. We employ a collaborative, transdisciplinary, facilitated team science process to first propose, and then refine, a sustainability assessment framework, using the U.S. state of Vermont as a case study. This paper details our process and progress, as well as reflections on challenges and recommendations for other team scientists. We further propose a plan to implement the framework, collect data, and engage with community members. The experiences and findings described here serve as a foundation for our own team’s continued work, as well as a springboard for other similar research efforts. 
    more » « less
  3. ; ; ; ; (, Policy Futures in Education)
    Facilitating the development of a common framework for monitoring progress in K-12 computer science (CS) education and advocacy with an emphasis on broadening participation is the key to constructing strong CS education policy. Based on a project that brought together leadership teams from six states, a framework for measuring broadening participation in computing (BPC) and setting the foundation for national scaling was developed. Built around a collaboration of leaders representing experience in data gathering, data analysis, data reporting, and data utilization, this project applied the tenets of collective impact to address the challenge of consistently measuring progress toward BPC across state contexts. By establishing a common agenda, including mutually agreed upon definitions of computer science education and broadening participation, these leaders guided the selection of metrics. This led to the development of shared measurement systems and built a deeper understanding of state data systems across the participating states. This phase resulted in common goals and a monitoring system to measure BPC efforts that could inform state policy efforts. Mutually reinforcing activities included the development and sharing of tools, allowing stakeholders to quickly and accurately analyze and disseminate data that drives BPC measurement and policy work. Guided by backbone support to coordinate the work and continuous communication, meaningful participation of all stakeholders was central to the project. Making the case for CS education policy via common metrics and measuring progress across a region stands to impact BPC policy efforts across the United States. The common framework developed in this project serves as a call to action, especially for state and local education agencies committed to increasing diversity in computer science pathways. 
    more » « less
  4. (, Education sciences)
    The researchers conducted a qualitative case study to describe the experiences (over the course of a semester) of an inter-disciplinary team of three special education and three psychology undergraduates who participated in a relational learning community and a graduate student who designed and facilitated this learning community. An associate professor and special education researcher oversaw and co-facilitated the project. The design of the learning community promoted the building of rapport and trust among the group members and the progress of the group toward a common goal of incorporating principles from psychology to develop teaching strategies for students who are struggling in math and experiencing severe math anxiety. Gathering more frequent and individualized feedback would have helped the learning community facilitator make some key adjustments earlier in the project, but the incorporation of rapport building activities that supported trust and collaboration among the group was supportive of group progress toward a common goal. We learned key lessons about how to design and implement a learning community that can be applied to the field of education, interdisciplinary collaboration, and other contexts. 
    more » « less
  5. ; ; ; (, IEEE)
    This innovative practice work in progress paper presents a systematic approach for screening and aligning service-learning projects that maximize student learning outcomes. We introduce a feasibility assessment model with criteria evaluated through a standardized rubric that guides instructors to critically assess the project fit to help in proactively identifying risks to student outcomes. The rubric serves a dual purpose: guiding the assessment process and prompting discussions with potential project partners. These discussions elicit crucial details about the project scope, potential challenges, and other critical factors. This not only facilitates effective project selection but also allows for necessary adjustments to project parameters, significantly improving the chances of successful student completion. This work builds on the experience accumulated by CCSU's Software Engineering Studio which connects community project partners with teams of 4–5 seniors working on software development projects spanning one or several semesters. Since 2014, the Software Engineering Studio has facilitated over 65 distinct projects and engaged over 550 students. By capturing the lessons learned across a wide range of successful service-learning projects, we show the value of using a feasibility assessment model to evaluate potential projects based on criteria including alignment with course goals, student skill sets, workload manage-ability, educational engagement, and other considerations. The application of this model is illustrated with a case study, which demonstrates how this model helps instructors align projects with academic goals while considering scope, risks, and other critical elements. This example demonstrates how the model facilitates communication with project partners, identifies potential risks, and guides project adjustments to ensure a successful learning experience for students. The approach is transferable to other disciplines with adaptations for project types and student skills. This work contributes to the field of service learning by offering a practical framework for integrating valuable real-world projects into the curriculum while prioritizing student learning outcomes. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email