skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Water Data Explorer: An Open-Source Web Application and Python Library for Water Resources Data Discovery
We present the design and development of an open-source web application called Water Data Explorer (WDE), designed to retrieve water resources observation and model data from data catalogs that follow the WaterOneFlow and WaterML Service-Oriented Architecture standards. WDE is a fully customizable web application built using the Tethys Platform development environment. As it is open source, it can be deployed on the web servers of international government agencies, non-governmental organizations, research teams, and others. Water Data Explorer provides uniform access to international data catalogs, such as the Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI) Hydrologic Information System (HIS) and the World Meteorological Organization (WMO) Hydrological Observing System (WHOS), as well as to local data catalogs that support the WaterOneFlow and WaterML standards. WDE supports data discovery, visualization, downloading, and basic data interpolation. It can be customized for different regions by modifying the user interface (i.e., localization), as well as by including pre-defined data catalogs and data sources. Access to WDE functionality is provided by a new open-source Python package called “Pywaterml” which provides programmable access to WDE methods to discover, visualize, download, and interpolate data. We present two case studies that access the CUAHSI HIS and WHOS catalogs and demonstrate regional customization, data discovery from WaterOneFlow web services, data visualization of time series observations, and data downloading.  more » « less
Award ID(s):
1664061 1664018
PAR ID:
10296842
Author(s) / Creator(s):
; ; ; ; ; ; ;
Date Published:
Journal Name:
Water
Volume:
13
Issue:
13
ISSN:
2073-4441
Page Range / eLocation ID:
1850
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. . Granting agencies invest millions of dollars on the generation and analysis of data, making these products extremely valuable. However, without sufficient annotation of the methods used to collect and analyze the data, the ability to reproduce and reuse those products suffers. This lack of assurance of the quality and credibility of the data at the different stages in the research process essentially wastes much of the investment of time and funding and fails to drive research forward to the level of potential possible if everything was effectively annotated and disseminated to the wider research community. In order to address this issue for the Hawai'i Established Program to Stimulate Competitive Research (EPSCoR) project, a water science gateway was developed at the University of Hawai‘i (UH), called the ‘Ike Wai Gateway. In Hawaiian, ‘Ike means knowledge and Wai means water. The gateway supports research in hydrology and water management by providing tools to address questions of water sustainability in Hawai‘i. The gateway provides a framework for data acquisition, analysis, model integration, and display of data products. The gateway is intended to complement and integrate with the capabilities of the Consortium of Universities for the Advancement of Hydrologic Science's (CUAHSI) Hydroshare by providing sound data and metadata management capabilities for multi-domain field observations, analytical lab actions, and modeling outputs. Functionality provided by the gateway is supported by a subset of the CUAHSI’s Observations Data Model (ODM) delivered as centralized web based user interfaces and APIs supporting multi-domain data management, computation, analysis, and visualization tools to support reproducible science, modeling, data discovery, and decision support for the Hawai'i EPSCoR ‘Ike Wai research team and wider Hawai‘i hydrology community. By leveraging the Tapis platform, UH has constructed a gateway that ties data and advanced computing resources together to support diverse research domains including microbiology, geochemistry, geophysics, economics, and humanities, coupled with computational and modeling workflows delivered in a user friendly web interface with workflows for effectively annotating the project data and products. Disseminating results for the ‘Ike Wai project through the ‘Ike Wai data gateway and Hydroshare makes the research products accessible and reusable. 
    more » « less
  2. Abstract The era of ‘big data’ promises to provide new hydrologic insights, and open web‐based platforms are being developed and adopted by the hydrologic science community to harness these datasets and data services. This shift accompanies advances in hydrology education and the growth of web‐based hydrology learning modules, but their capacity to utilize emerging open platforms and data services to enhance student learning through data‐driven activities remains largely untapped. Given that generic equations may not easily translate into local or regional solutions, teaching students to explore how well models or equations work in particular settings or to answer specific problems using real data is essential. This article introduces an open web‐based module developed to advance data‐driven hydrologic process learning, targeting upper level undergraduate and early graduate students in hydrology and engineering. The module was developed and deployed on the HydroLearn open educational platform, which provides a formal pedagogical structure for developing effective problem‐based learning activities. We found that data‐driven learning activities utilizing collaborative open web platforms like CUAHSI HydroShare and JupyterHub to store and run computational notebooks allowed students to access and work with datasets for systems of personal interest and promoted critical evaluation of results and assumptions. Initial student feedback was generally positive, but also highlighted challenges including trouble‐shooting and future‐proofing difficulties and some resistance to programming and new software. Opportunities to further enhance hydrology learning include better articulating the benefits of coding and open web platforms upfront, incorporating additional user‐support tools, and focusing methods and questions on implementing and adapting notebooks to explore fundamental processes rather than tools and syntax. The profound shift in the field of hydrology toward big data, open data services and reproducible research practices requires hydrology instructors to rethink traditional content delivery and focus instruction on harnessing these datasets and practices in the preparation of future hydrologists and engineers. 
    more » « less
  3. Abstract Throughout the past decade, many studies have reported adverse effects in biota following microplastic exposure. Yet, the field is still emerging as the current understanding of microplastic toxicity is limited. At the same time, recent legislative mandates have required environmental regulators to devise strategies to mitigate microplastic pollution and develop health-based thresholds for the protection of human and ecosystem health. The current publication rate also presents a unique challenge as scientists, environmental managers, and other communities may find it difficult to keep up with microplastic research as it rapidly evolves. At present, there is no tool that compiles and synthesizes the data from these studies to allow for visualization, interpretation, or analysis. Here, we present the Toxicity of Microplastics Explorer (ToMEx), an open access database and open source accompanying R Shiny web application that enables users to upload, search, visualize, and analyze microplastic toxicity data. Though ToMEx was originally created to facilitate the development of health-based thresholds to support California legislations, maintaining the database by the greater scientific community will be invaluable to furthering research and informing policies globally. The database and web applications may be accessed at https://microplastics.sccwrp.org/ . Graphical Abstract 
    more » « less
  4. Abstract Many have argued that datasets resulting from scientific research should be part of the scholarly record as first class research products. Data sharing mandates from funding agencies and scientific journal publishers along with calls from the scientific community to better support transparency and reproducibility of scientific research have increased demand for tools and support for publishing datasets. Hydrology domain‐specific data publication services have been developed alongside more general purpose and even commercial data repositories. Prominent among these are the Hydrologic Information System (HIS) and HydroShare repositories developed by the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI). More broadly, however, multiple organizations have been involved in the practice of data publication in the hydrology domain, each having different roles that have shaped data publication and reuse. Bibliographic and archival approaches to data publication have been advanced, but both have limitations with respect to hydrologic data. Specific recommendations for improving data publication infrastructure, support, and practices to move beyond existing limitations and enable more effective data publication in support of scientific research in the hydrology domain include: improving support for journal article‐based data access and data citation, considering the workflow for data publication, enhancing support for reproducible science, encouraging publication of curated reference data collections, advancing interoperability standards for sharing data and metadata among repositories, developing partnerships with university libraries offering data services, and developing more specific data management plans. While presented in the context of CUAHSI's data repositories and experience, these recommendations are broadly applicable to other domains. This article is categorized under:Science of Water > Methods 
    more » « less
  5. A vast proportion of scientific data remains locked behind dynamic web interfaces, often called the deep web—inaccessible to conventional search engines and standard crawlers. This gap between data availability and machine usability hampers the goals of open science and automation. While registries like FAIRsharing offer structured metadata describing data standards, repositories, and policies aligned with the FAIR (Findable, Accessible, Interoperable, and Reusable) principles, they do not enable seamless, programmatic access to the underlying datasets. We present FAIRFind, a system designed to bridge this accessibility gap. FAIRFind autonomously discovers, interprets, and operationalizes access paths to biological databases on the deep web, regardless of their FAIR compliance. Central to our approach is the Deep Web Communication Protocol (DWCP), a resource description language that represents web forms, HyperText Markup Language (HTML) tables, and file-based data interfaces in a machine-actionable format. Leveraging large language models (LLMs), FAIRFind combines a specialized deep web crawler and web-form comprehension engine to transform passive web metadata into executable workflows. By indexing and embedding these workflows, FAIRFind enables natural language querying over diverse biological data sources and returns structured, source-resolved results. Evaluation across multiple open-source LLMs and database types demonstrates over 90% success in structured data extraction and high semantic retrieval accuracy. FAIRFind advances existing registries by turning linked resources from static references into actionable endpoints, laying a foundation for intelligent, autonomous data discovery across scientific domains. 
    more » « less