More Like this

1. Enabling Fast, Effective Visualization of Voluminous Gridded Spatial Datasets

   https://doi.org/10.1109/CCGrid57682.2023.00061  

   Khandelwal, Paahuni; Warushavithana, Menuka; Pallickara, Sangmi Lee; Pallickara, Shrideep (May 2023, 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid))

   Gridded spatial datasets arise naturally in environmental, climatic, meteorological, and ecological settings. Each grid point encapsulates a vector of variables representing different measures of interest. Gridded datasets tend to be voluminous since they encapsulate observations for long timescales. Visualizing such datasets poses significant challenges stemming from the need to preserve interactivity, manage I/O overheads, and cope with data volumes. Here we present our methodology to significantly alleviate I/O requirements by leveraging deep neural network-based models and a distributed, in-memory cache to facilitate interactive visualizations. Our benchmarks demonstrate that deploying our lightweight models coupled with back-end caching and prefetching schemes can reduce the client's query response time by 92.3% while maintaining a high perceptual quality with a PSNR (peak signal-to-noise ratio) of 38.7 dB. 

   more » « less
2. Griddle: Effective Query Support over Voluminous Gridded Spatial Datasets

   https://doi.org/10.1109/BigData55660.2022.10020796  

   Smith, Pierce; Pallickara, Sangmi Lee; Pallickara, Shrideep (December 2022, 2022 IEEE International Conference on Big Data (Big Data))

   Gridded datasets occur in several domains. These datasets comprise (un)structured grid points, where each grid point is characterized by XY(Z) coordinates in a spatial referencing system. The data available at individual grid points are high-dimensional encapsulating multiple variables of interest. This study has two thrusts. The first targets supporting effective management of voluminous gridded datasets while reconciling challenges relating to colocation and dispersion. The second thrust is to support sliding (temporal) window queries over the gridded dataset. Such queries involve sliding a temporal window over the data to identify spatial locations and chronological time points where the specified predicate evaluates to true. Our methodology includes support for a space-efficient data structure for organizing information within the data, query decomposition based on dyadic intervals, support for temporal anchoring, query transformations, and effective evaluation of query predicates. Our empirical benchmarks are conducted on representative voluminous high dimensional datasets such as gridMET (historical meteorological data) and MACA (future climate datasets based on the RCP 8.5 greenhouse gas trajectory). In our benchmarks, our system can handle throughputs of over 3000 multi-predicate sliding window queries per second. 

   more » « less
3. From rain to data: A review of the creation of monthly and daily station‐based gridded precipitation datasets

   https://doi.org/10.1002/wat2.1555  

   Serrano‐Notivoli, Roberto; Tejedor, Ernesto (August 2021, WIREs Water)

   Abstract Monthly and daily gridded precipitation datasets are one of the most demanded products in climatology and hydrology. These datasets describe the high spatial and temporal variability of precipitation as a continuous surface and for defined periods. However, due to the complex characteristics of precipitation, it is difficult to obtain accurate estimations. Thus, the creation of a gridded dataset from observations requires the comprehensive and precise application of quality control, reconstruction, and gridding procedures. Yet, despite multiple advances, most of the gridded datasets created and published since the mid‐1990s to the present use a wide variety of techniques, methods, and outputs, which can completely change the final representativity of the data. It is, therefore, critical to provide general guidelines for the development of future and more robust gridded datasets based on the data characteristics, geographical factors, and advanced statistical techniques. We identified gaps and challenges for near‐future perspectives and provide guidelines for implementing improved approaches based on the performance of 48 products. Finally, we concluded that, despite better spatial and temporal resolutions, data access, and data processing capabilities, observational coverage remains a challenge. Moreover, scientists should adopt tailored strategies to improve the representativity and uncertainty of the estimates. This article is categorized under:Science of Water > Hydrological ProcessesScience of Water > Water ExtremesScience of Water > Methods 

   more » « less
4. Henosis: workload-driven small array consolidation and placement for HDF5 applications on heterogeneous data stores

   https://doi.org/10.1145/3330345.3330380  

   Kang, Donghe; Patel, Vedang; Nair, Ashwati; Blanas, Spyros; Wang, Yang; Parthasarathy, Srinivasan (June 2019, Proceedings of the 33rd ACM International Conference on Supercomputing - ICS '19)

   Scientific data analysis pipelines face scalability bottlenecks when processing massive datasets that consist of millions of small files. Such datasets commonly arise in domains as diverse as detecting supernovae and post-processing computational fluid dynamics simulations. Furthermore, applications often use inference frameworks such as TensorFlow and PyTorch whose naive I/O methods exacerbate I/O bottlenecks. One solution is to use scientific file formats, such as HDF5 and FITS, to organize small arrays in one big file. However, storing everything in one file does not fully leverage the heterogeneous data storage capabilities of modern clusters. This paper presents Henosis, a system that intercepts data accesses inside the HDF5 library and transparently redirects I/O to the in-memory Redis object store or the disk-based TileDB array store. During this process, Henosis consolidates small arrays into bigger chunks and intelligently places them in data stores. A critical research aspect of Henosis is that it formulates object consolidation and data placement as a single optimization problem. Henosis carefully constructs a graph to capture the I/O activity of a workload and produces an initial solution to the optimization problem using graph partitioning. Henosis then refines the solution using a hill-climbing algorithm which migrates arrays between data stores to minimize I/O cost. The evaluation on two real scientific data analysis pipelines shows that consolidation with Henosis makes I/O 300× faster than directly reading small arrays from TileDB and 3.5× faster than workload-oblivious consolidation methods. Moreover, jointly optimizing consolidation and placement in Henosis makes I/O 1.7× faster than strategies that perform consolidation and placement independently. 

   more » « less
5. MPI-Vector-IO: Parallel I/O and Partitioning for Geospatial Vector Data

   https://doi.org/10.1145/3225058.3225105  

   Puri, Satish; Paudel, Anmol; Prasad, Sushil K. (August 2018, Proceedings of the 47th International Conference on Parallel Processing (ICPP 2018).)

   In recent times, geospatial datasets are growing in terms of size, complexity and heterogeneity. High performance systems are needed to analyze such data to produce actionable insights in an efficient manner. For polygonal a.k.a vector datasets, operations such as I/O, data partitioning, communication, and load balancing becomes challenging in a cluster environment. In this work, we present MPI-Vector-IO, a parallel I/O library that we have designed using MPI-IO specifically for partitioning and reading irregular vector data formats such as Well Known Text. It makes MPI aware of spatial data, spatial primitives and provides support for spatial data types embedded within collective computation and communication using MPI message-passing library. These abstractions along with parallel I/O support are useful for parallel Geographic Information System (GIS) application development on HPC platforms. Performance evaluation is done on Lustre and GPFS filesystems. MPI-Vector-IO scales well with MPI processes and file size and achieves bandwidth up to 22 GB/s for common spatial data access patterns. We observed that independent file read functions performed better than collective functions in MPI-IO for contiguous access pattern on Lustre. In general, the I/O is improved by one to two orders of magnitude over real-world datasets using up to 1152 CPU cores. Spatial Join query is used as an exemplar to demonstrate an end-to-end application using MPI-Vector-IO. 

   more » « less