More Like this

1. Notebooks for combining the National Water Model results/inputs with observations from SNOTEL and MODIS at SNOTEL sites

   https://doi.org/10.4211/hs.493e0ad05c2d45199427cc41a6c76de0  

   Garousi-Nejad, Irene; Tarboton, David (July 2022, HydroShare)

   This resource includes Jupyter Notebooks that combine (merge) model results with observations. There are four folders: - NWM_SnowAssessment: This folder includes codes required for combining model results with observations. It also has an output folder that contains outputs of running five Jupyter Notebooks within the code folder. The order to run the Jupyter Notebooks is as follows. First run Combine_obs_mod_[*].ipynb where [*] is P (precipitation), SWE (snow water equivalent), TAir (air temperature), and FSNO (snow covered area fraction). This combines the model outputs and observations for each variable. Then, run Combine_obs_mod_P_SWE_TAir_FSNO.ipynb. - NWM_Reanalysis: This folder contains the National Water Model version 2 retrospective simulations that were retrieved and pre-processed at SNOTEL sites using https://doi.org/10.4211/hs.3d4976bf6eb84dfbbe11446ab0e31a0a and https://doi.org/10.4211/hs.1b66a752b0cc467eb0f46bda5fdc4b34. - SNOTEL: This folder contains preprocessed SNOTEL observations that were created using https://doi.org/10.4211/hs.d1fe0668734e4892b066f198c4015b06. - GEE: This folder contains MODIS observations that we downloaded using https://doi.org/10.4211/hs.d287f010b2dd48edb0573415a56d47f8. Note that the existing CSV file is the merged file of the downloaded CSV files. 

   more » « less
2. Example data for 'lacunr: Efficient 3-D lacunarity for voxelized LiDAR data from forested ecosystems'

   https://doi.org/10.5281/zenodo.16116069  

   Smeds, Elliott A; Cooper, Zane; Bentley, Lisa Patrick (January 2025, Zenodo)

   {"Abstract":["This repository contains data and code for reproducing the results of the worked example in Smeds et al. 2025 "lacunr: Efficient 3-D lacunarity for voxelized LiDAR data from forested ecosystems", published in Methods in Ecology and Evolution (https://doi.org/10.1111/2041-210X.70126).\n\nThe worked example analyzes terrestrial LiDAR scans of two forest stands at the Saddle Mountain Open Space Preserve in Sonoma County, California, which were burned in the 2020 Glass Fire. The first of these scans, Plot 1, in included in the 'lacunr' software package as an example dataset, while the second plot is archived here. The included R script allows users to reproduce the lacunarity curves displayed in Figure 2 and Figure S1 of Smeds et al.\n\nThe deposited zip archive, lacunr_example.zip, can be decompressed into a corresponding folder with the following contents:\n\n\n\ndata/ - a data folder which contains two height-normalized terrestrial LiDAR point clouds in .laz format:\n\n\n\nc6_tls_p6_prefire.laz - 24*24m forest stand before wildfire\n\nc10_tls_p6_postfire.laz - the same forest stand after wildfire\n\n\n\nlacunr_example_data.R - R script for replicating the lacunarity curves presented in the publication\n\nlacunr_example.Rproj - the R project file which allows users to open the R script in a self-contained environment. This file should be opened in RStudio prior to executing the R script\n\noutput/ - the folder where external image files are exported by the R script\n\nREADME.txt - a text file containing instructions for opening the R project and running the associated code"]} 

   more » « less
3. Extreme precipitation datasets for the contiguous US, from gridded analyses and a convection-permitting model simulation

   https://doi.org/10.5061/dryad.jq2bvq8q2  

   Schumacher, Russ; Hill, Aaron; Schumacher, Russ; Hill, Aaron (January 2026, Dryad)

   Abstract: This is a collection of datasets that show where and when precipitation exceeded average recurrence interval (ARI) thresholds at durations from 24-72 hours over the contiguous US (CONUS). Specifically, included are 100- and 1000-yr exceedances (i.e., precipitation accumulations with annual exceedance probabilities of ≤1% and ≤0.1%, respectively) from these precipitation datasets: NOAA's Stage IV analysis, 2002-2024 Parameter-Elevation Regressions on Independent Slopes Model (PRISM), 1981-2024 NOAA Multi-Radar Multi-Sensor (MRMS) system, 2018-2024 CONUS404 simulation, 1981-2024 Methods: The Stage IV, MRMS, and CONUS404 datasets were regridded to the 4-km latitude/longitude grid used by PRISM, using bilinear interpolation with the xESMF package, version 0.8.2 (Zhuang et al. 2023). This regridding was performed to allow for direct comparisons of spatial coverage of rainfall between datasets. Gridded precipitation estimates were compared to precipitation frequency estimates from NOAA Atlas 14, and all points exceeding the threshold for a given duration and ARI were recorded. Points were clustered into events using Density-Based Spatial Clustering of Applications with Noise (DBSCAN).  An extensive set of automated and manual quality-control procedures were then applied to the exceedance points and events; details of these QC procedures are described in the Schumacher and Hill (2025) manuscript.  TechnicalInfo: # Extreme precipitation datasets for the contiguous US, from gridded analyses and a convection-permitting model simulation Dataset DOI: [10.5061/dryad.jq2bvq8q2](https://doi.org/10.5061/dryad.jq2bvq8q2) ## Description of the data and file structure Lists of locations and times of points exceeding average recurrence interval thresholds (defined by NOAA Atlas 14) for three gridded precipitation datasets and the output of a regional climate simulation (CONUS404).  ### Files and variables The naming convention for the included files is as follows: ``` _-__yh ``` `dataset` is one of: * stage4 (NOAA Stage IV analysis) * prism (PRISM analysis) * mrms (MRMS system) * conus404 (CONUS404 simulation). `startyear` is the first year included in the analysis (either 1981, 2002, or 2018), and `endyear` is the last year included in the analysis (either 2022 or 2024).  `datatype` describes what is included in the file: * "removed_points" are the exceedance points that were removed by either automated or manual QC methods * "events" are the points retained and included in the analysis `ARI` is the average recurrence interval, either 100 or 1000 years. `duration` is 24, 48, or 72 hours.  For durations of 48 and 72 hours, a second set of files is included with the suffix "nooverlaps". This is the dataset with overlapping time periods removed: any grid points in a dataset that had multiple exceedances within 3 days of each other at a given ARI and duration were identified. The date with the highest precipitation amount among these overlaps was retained in the dataset, and the others were removed. So, for example, the file named "stage4_2002-2024_events_1000y72h_edit_nooverlaps.csv" has the exceedances of the 1000-yr, 72-h ARI in the Stage IV dataset from 2002-2024. Overlapping time periods have been removed in this file. [In the file "stage4_2002-2024_events_1000y72h_edit.csv", points from overlapping time periods have not been removed.] The structure of each file is otherwise the same, as described below. Each row in a file represents one point on the 4-km grid that exceeded the ARI threshold for that duration.  ##### Variables * time: Time in UTC, with the format "YYYY-mm-dd HH:MM:SS" * lat: Latitude in degrees * lon: Longitude in degrees * tp: Total precipitation (mm) * tp_minus_ari: Total precipitation minus the ARI threshold (mm) * tp_pct_of_ari: Total precipitation divided by the ARI threshold * event_num: The event number for this point, obtained from DBSCAN clustering. The event numbers are arbitrary (i.e., that an event was assigned number 1 or 5 has no special meaning.)  ## Code/software Files are in comma-separated value (csv) format, which can be read by standard software packages such as Microsoft Excel, R, and the pandas python package.  Software used to create these files and perform the QC procedures is in python, available online at [https://doi.org/10.5281/zenodo.22709265](https://doi.org/10.5281/zenodo.22709265) ## Access information Other publicly accessible locations of the data: * Interactive maps showing the data provided here can be found at: [https://schumacher.atmos.colostate.edu/precip_monitor/interactive.php](https://schumacher.atmos.colostate.edu/precip_monitor/interactive.php) Data was derived from the following sources: * Stage IV analyses were obtained from Du (2011). * PRISM analyses were obtained from [https://data.prism.oregonstate.edu](https://data.prism.oregonstate.edu). * MRMS data were obtained from a combination of sources, including a local archive maintained at Colorado State University, the Registry of Open Data on Amazon Web Services (AWS) ([https://registry.opendata.aws/noaa-mrms-pds](https://registry.opendata.aws/noaa-mrms-pds)), and the archive at the Iowa Environmental Mesonet ([https://mesonet.agron.iastate.edu/archive/mrms.php](https://mesonet.agron.iastate.edu/archive/mrms.php)). CONUS404 was accessed via Rasmussen et al. (2023) on the NSF NCAR high-performance computing systems. * NOAA Atlas 14 was obtained from [https://hdsc.nws.noaa.gov/pfds/pfds_gis.html](https://hdsc.nws.noaa.gov/pfds/pfds_gis.html) **References** Daly, C., and Coauthors, 2021: Challenges in observation-based mapping of daily precipitation across the conterminous United States. *J. Atmos. Oceanic Technol.,* **38** (11), 1979–1992, [https://doi.org/10.1175/JTECH-D-21-0054.1.](https://doi.org/10.1175/JTECH-D-21-0054.1.) Du, J., 2011: NCEP/EMC 4KM Gridded Data (GRIB) Stage IV Data. Version 1.0. NSF NCAR Earth Observing Laboratory, Accessed 25 January 2025, [https://doi.org/10.5065/D6PG1QDD.281](https://doi.org/10.5065/D6PG1QDD.281) Lin, Y., and K. E. Mitchell, 2005: The NCEP Stage II/IV hourly precipitation analyses: Development and applications. Preprints, 19th Conference on Hydrology, Amer. Meteor. Soc., San Diego, CA, [Available online at [http://ams.confex.com/ams/pdfpapers/83847.pdf](http://ams.confex.com/ams/pdfpapers/83847.pdf)]. Rasmussen, R. M., and Coauthors, 2023: Four-kilometer long-term regional hydroclimate reanalysis over the conterminous United States (CONUS). NSF National Center for Atmospheric Research, Boulder, CO, [https://doi.org/10.5065/ZYY0-Y036.888](https://doi.org/10.5065/ZYY0-Y036.888) Zhang, J., and Coauthors, 2016: Multi-Radar Multi-Sensor (MRMS) quantitative precipitation estimation: Initial operating capabilities. *Bull. Amer. Meteor. Soc.*, **97** (4), 621–638, [https://doi.org/10.1175/BAMS-D-14-00174.1](https://doi.org/10.1175/BAMS-D-14-00174.1). Zhuang, J., and Coauthors, 2023: Pangeo-data/xESMF: V0.8.2. Zenodo, [https://doi.org/10.5281/zenodo.8356796.364](https://doi.org/10.5281/zenodo.8356796.364) 

   more » « less
4. DeepCaImX: An end-to-end recurrent compressed sensing method to denoise, detect and demix calcium imaging data

   https://doi.org/10.5281/zenodo.12650420  

   Zhang, Kangning; Tang, Sean; Zhu, Vivian; Barchini, Majd; Yang, Weijian (September 2024, Zenodo)

   {"Abstract":["# DeepCaImX## Introduction#### Two-photon calcium imaging provides large-scale recordings of neuronal activities at cellular resolution. A robust, automated and high-speed pipeline to simultaneously segment the spatial footprints of neurons and extract their temporal activity traces while decontaminating them from background, noise and overlapping neurons is highly desirable to analyze calcium imaging data. In this paper, we demonstrate DeepCaImX, an end-to-end deep learning method based on an iterative shrinkage-thresholding algorithm and a long-short-term-memory neural network to achieve the above goals altogether at a very high speed and without any manually tuned hyper-parameters. DeepCaImX is a multi-task, multi-class and multi-label segmentation method composed of a compressed-sensing-inspired neural network with a recurrent layer and fully connected layers. It represents the first neural network that can simultaneously generate accurate neuronal footprints and extract clean neuronal activity traces from calcium imaging data. We trained the neural network with simulated datasets and benchmarked it against existing state-of-the-art methods with in vivo experimental data. DeepCaImX outperforms existing methods in the quality of segmentation and temporal trace extraction as well as processing speed. DeepCaImX is highly scalable and will benefit the analysis of mesoscale calcium imaging. \n\n## System and Environment Requirements#### 1. Both CPU and GPU are supported to run the code of DeepCaImX. A CUDA compatible GPU is preferred. * In our demo of full-version, we use a GPU of Quadro RTX8000 48GB to accelerate the training speed.* In our demo of mini-version, at least 6 GB momory of GPU/CPU is required.#### 2. Python 3.9 and Tensorflow 2.10.0#### 3. Virtual environment: Anaconda Navigator 2.2.0#### 4. Matlab 2023a\n\n## Demo and installation#### 1 (_Optional_) GPU environment setup. We need a Nvidia parallel computing platform and programming model called _CUDA Toolkit_ and a GPU-accelerated library of primitives for deep neural networks called _CUDA Deep Neural Network library (cuDNN)_ to build up a GPU supported environment for training and testing our model. The link of CUDA installation guide is https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html and the link of cuDNN installation guide is https://docs.nvidia.com/deeplearning/cudnn/installation/overview.html. #### 2 Install Anaconda. Link of installation guide: https://docs.anaconda.com/free/anaconda/install/index.html#### 3 Launch Anaconda prompt and install Python 3.x and Tensorflow 2.9.0 as the virtual environment.#### 4 Open the virtual environment, and then  pip install mat73, opencv-python, python-time and scipy.#### 5 Download the "DeepCaImX_training_demo.ipynb" in folder "Demo (full-version)" for a full version and the simulated dataset via the google drive link. Then, create and put the training dataset in the path "./Training Dataset/". If there is a limitation on your computing resource or a quick test on our code, we highly recommand download the demo from the folder "Mini-version", which only requires around 6.3 GB momory in training. #### 6 Run: Use Anaconda to launch the virtual environment and open "DeepCaImX_training_demo.ipynb" or "DeepCaImX_testing_demo.ipynb". Then, please check and follow the guide of "DeepCaImX_training_demo.ipynb" or or "DeepCaImX_testing_demo.ipynb" for training or testing respectively.#### Note: Every package can be installed in a few minutes.\n\n## Run DeepCaImX#### 1. Mini-version demo* Download all the documents in the folder of "Demo (mini-version)".* Adding training and testing dataset in the sub-folder of "Training Dataset" and "Testing Dataset" separately.* (Optional) Put pretrained model in the the sub-folder of "Pretrained Model"* Using Anaconda Navigator to launch the virtual environment and opening "DeepCaImX_training_demo.ipynb" for training or "DeepCaImX_testing_demo.ipynb" for predicting.\n\n#### 2. Full-version demo* Download all the documents in the folder of "Demo (full-version)".* Adding training and testing dataset in the sub-folder of "Training Dataset" and "Testing Dataset" separately.* (Optional) Put pretrained model in the the sub-folder of "Pretrained Model"* Using Anaconda Navigator to launch the virtual environment and opening "DeepCaImX_training_demo.ipynb" for training or "DeepCaImX_testing_demo.ipynb" for predicting.\n\n## Data Tailor#### A data tailor developed by Matlab is provided to support a basic data tiling processing. In the folder of "Data Tailor", we can find a "tailor.m" script and an example "test.tiff". After running "tailor.m" by matlab, user is able to choose a "tiff" file from a GUI as loading the sample to be tiled. Settings include size of FOV, overlapping area, normalization option, name of output file and output data format. The output files can be found at local folder, which is at the same folder as the "tailor.m".\n\n## Simulated Dataset#### 1. Dataset generator (FISSA Version): The algorithm for generating simulated dataset is based on the paper of FISSA (_Keemink, S.W., Lowe, S.C., Pakan, J.M.P. et al. FISSA: A neuropil decontamination toolbox for calcium imaging signals. Sci Rep 8, 3493 (2018)_) and SimCalc repository (https://github.com/rochefort-lab/SimCalc/). For the code used to generate the simulated data, please download the documents in the folder "Simulated Dataset Generator". #### Training dataset: https://drive.google.com/file/d/1WZkIE_WA7Qw133t2KtqTESDmxMwsEkjJ/view?usp=share_link#### Testing Dataset: https://drive.google.com/file/d/1zsLH8OQ4kTV7LaqQfbPDuMDuWBcHGWcO/view?usp=share_link\n\n#### 2. Dataset generator (NAOMi Version): The algorithm for generating simulated dataset is based on the paper of NAOMi (_Song, A., Gauthier, J. L., Pillow, J. W., Tank, D. W. & Charles, A. S. Neural anatomy and optical microscopy (NAOMi) simulation for evaluating calcium imaging methods. Journal of neuroscience methods 358, 109173 (2021)_). For the code use to generate the simulated data, please go to this link: https://bitbucket.org/adamshch/naomi_sim/src/master/code/## Experimental Dataset#### We used the samples from ABO dataset:https://github.com/AllenInstitute/AllenSDK/wiki/Use-the-Allen-Brain-Observatory-%E2%80%93-Visual-Coding-on-AWS.#### The segmentation ground truth can be found in the folder "Manually Labelled ROIs". #### The segmentation ground truth of depth 175, 275, 375, 550 and 625 um are manually labeled by us. #### The code for creating ground truth of extracted traces can be found in "Prepro_Exp_Sample.ipynb" in the folder "Preprocessing of Experimental Sample"."]} 

   more » « less
5. Streamflow datasets from the high-resolution, multiscale, differentiable HBV hydrologic models

   https://doi.org/10.5281/zenodo.13774373  

   Song, Yalan; Shen, Chaopeng; Lonzarich, Leo; Ji, Haoyu; Bindas, Tadd; Song, Yalan; Shen, Chaopeng; Bindas, Tadd; Ji, Haoyu; Lonzarich, Leo (January 2026, Zenodo)

   Abstract: NEW: This dataset has now been moved from Microsoft OneDrive to an AWS S3 bucket for your convenience in downloading. Please use the download code we provided to pull data from the AWS S3 bucket without login credentials. This dataset is from the high-resolution, multiscale, differentiable HBV hydrologic models, provided by the Multi-scale Hydrology, Processes, and Intelligence (MHPI) team from The Pennsylvania State University, led by Dr. Chaopeng Shen's group in Hydrologic Deep Learning and Modeling.  Please cite: Song, Y., Bindas, T., Shen, C., Ji, H., Knoben, W. J. M., Lonzarich, L., et al. (2025). High‐resolution national‐scale water modeling is enhanced by multiscale differentiable physics‐informed machine learning. Water Resources Research, 61, e2024WR038928. https://doi.org/10.1029/2024WR038928 The dataset is generated by High-resolution, multiscale, differentiable HBV hydrologic models, dHBV2.0UH and dHBV2.0dMC. dHBV2.0UH is a high-resolution, multiscale model that uses unit hydrograph routing. dHBV2.0dMC is a high-resolution, multiscale model that uses external Muskingum-Cunge routing. Please use the link in the Readme.docx file to access the dataset. A link to the code for loading the dataset is also provided in Readme.docx. The dHBV_streamflow_simulation_gages folder includes 40 years (1980–2020) of streamflow simulations at over 7,000 gage stations from GAGES-II, using both dHBV2.0UH and dHBV2.0dMC models. This data is useful for comparison with observations.  The MERIT_flux_states folder includes 40 years (1980–2020) of spatially seamless simulations of hydrologic variables over 180 thousand MERIT unit basins on CONUS from dHBV2.0UH, including baseflow, evapotranspiration (ET), soil moisture, snow water equivalent, and runoff.  The dHBV2.0_MERIT_river_network_simulation includes 40 years (1980-2020) of streamflow simulation on seamless MERIT river network by dHBV2.0dMC (New updates!). The dHBV2.0UH code is available at mhpi/generic_deltaModel: High-resolution differentiable model, 𝛿HBV2.0. https://doi.org/10.5281/zenodo.14827983 

   more » « less