More Like this

1. Stimulus: Accelerate Data Management for Scientific AI applications in HPC

   https://doi.org/10.1109/CCGrid54584.2022.00020  

   Devarajan, Hariharan; Kougkas, Anthony; Zheng, Huihuo; Vishwanath, Venkatram; Sun, Xian-He (May 2022, 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid))

   Modern scientific workflows couple simulations with AI-powered analytics by frequently exchanging data to accelerate time-to-science to reduce the complexity of the simulation planes. However, this data exchange is limited in performance and portability due to a lack of support for scientific data formats in AI frameworks. We need a cohesive mechanism to effectively integrate at scale complex scientific data formats such as HDF5, PnetCDF, ADIOS2, GNCF, and Silo into popular AI frameworks such as TensorFlow, PyTorch, and Caffe. To this end, we designed Stimulus, a data management library for ingesting scientific data effectively into the popular AI frameworks. We utilize the StimOps functions along with StimPack abstraction to enable the integration of scientific data formats with any AI framework. The evaluations show that Stimulus outperforms several large-scale applications with different use-cases such as Cosmic Tagger (consuming HDF5 dataset in PyTorch), Distributed FFN (consuming HDF5 dataset in TensorFlow), and CosmoFlow (converting HDF5 into TFRecord and then consuming that in TensorFlow) by 5.3 x, 2.9 x, and 1.9 x respectively with ideal I/O scalability up to 768 GPUs on the Summit supercomputer. Through Stimulus, we can portably extend existing popular AI frameworks to cohesively support any complex scientific data format and efficiently scale the applications on large-scale supercomputers. 

   more » « less
2. SEGUL: Ultrafast, memory‐efficient and mobile‐friendly software for manipulating and summarizing phylogenomic datasets

   https://doi.org/10.1111/1755-0998.13964  

   Handika, Heru; Esselstyn, Jacob A (April 2024, Molecular Ecology Resources)

   Abstract Phylogenetic studies now routinely require manipulating and summarizing thousands of data files. For most of these tasks, currently available software requires considerable computing resources and substantial knowledge of command‐line applications. We develop an ultrafast and memory‐efficient software, SEGUL, that performs common phylogenomic dataset manipulations and calculates statistics summarizing essential data features. Our software is available as standalone command‐line interface (CLI) and graphical user interface (GUI) applications, and as a library for Rust, R and Python, with possible support of other languages. The CLI and library versions run native on Windows, Linux and macOS, including Apple ARM Macs. The GUI version extends support to include mobile iOS, iPadOS and Android operating systems. SEGUL leverages the high performance of the Rust programming language to offer fast execution times and low memory footprints regardless of dataset size and platform choice. The inclusion of a GUI minimizes bioinformatics barriers to phylogenomics while SEGUL's efficiency reduces economic barriers by allowing analysis on inexpensive hardware. Our support for mobile operating systems further enables teaching phylogenomics where access to computing power is limited. 

   more » « less
3. River Plumes and Idealized Coastal Corners: ROMS and MATLAB files

   https://doi.org/10.17632/m4ndry2dw3.2  

   Whitney, Michael (January 2024, Mendeley Data)

   This dataset is associated with a manuscript on river plumes and idealized coastal corners with first author Michael M. Whitney. The dataset includes source code, compilation files, and routines to generate input files for the Regional Ocean Modeling System (ROMS) runs used in this study. ROMS output files in NetCDF format are generated by executing the compiled ROMS code with the input files. The dataset also includes MATLAB routines and datafiles for the analysis of model results and generation of figures in the manuscript. The following zip files are included: ROMS_v783_Yan_code.zip [ROMS source code branch used in this study] coastalcorner_ROMS_compilation.zip [files to compile ROMS source code and run-specific Fortran-90 built code] coastalcorner_ROMS_input_generate_MATLAB.zip [ROMS ASCII input file and MATLAB routines to generate ROMS NetCDF input files for runs] coastalcorner_MATLAB_output_analysis.zip [MATLAB data files with selected ROMS output fields and custom analysis routines and datafiles in MATLAB formats used in this study] coastalcorner_MATLAB_figures.zip [custom MATLAB routine for manuscript figure generation and MATLAB data files with all data fields included in figures] coastalcorner_tif_figures.zip [TIF image files of each figure in manuscript] 

   more » « less
4. Open Government Data and File Formats: Constraints on Collaboration

   https://doi.org/10.1145/3085228.3085232  

   Washington, Anne L.; Morar, David (June 2017, Proceedings of the 18th Annual International Conference on Digital Government Research dg.o 2017)

   This exploratory interpretive case study investigated the collaborative potential of open government data available through data.gov, the US federal open data catalog. Open data is a central aspect of open government collaboration because it fosters exchange and communication between governments and the public. Government organizations that release open data make choices about file formats that have a substantial impact on the potential for collaboration. A file format, such as a document or a spreadsheet, is a constraint on which programs can read the file and what actions a user can do with the file. Overall, we found data.gov formats with limited collaboration potential but files that could be accessed by people with a wide range of skills. The findings are incorporated into suggestions for future iterations of open data policy. The advantages and limitations of using file formats for open data research are considered. The exploratory findings raise questions about future user-centric open data evaluations. 

   more » « less
5. Recorder 2.0: Efficient Parallel I/O Tracing and Analysis

   Wang, C.; Sun, J.; Snir, M.; Mohror, K.; Gonsiorowski, E. (May 2020, The IEEE International Workshop on High-Performance Storage)

   Recorder is a multi-level I/O tracing tool that captures HDF5, MPI-I/O, and POSIX I/O calls. In this paper, we present a new version of Recorder that adds support for most metadata POSIX calls such as stat, link, and rename. We also introduce a compressed tracing format to reduce trace file size and run time overhead incurred from collecting the trace data. Moreover, we add a set of post-mortem and visualization routines to our new version of Recorder that manage the compressed trace data for users. Our experiments with four HPC applications show a file size reduction of over 2× and reduced post-processing time by 20% when using our new compressed trace file format. 

   more » « less