Attention:The NSF Public Access Repository (PAR) system and access will be unavailable from 5:00 PM ET until 8:00 PM ET on Friday, September 11 due to maintenance. We apologize for the inconvenience.


Title: DeepInMiniscope: Deep-learning-powered physics-informed integrated miniscope
Mask-based integrated fluorescence microscopy is a compact imaging technique for biomedical research. It can perform snapshot 3D imaging through a thin optical mask with a scalable field of view (FOV) and a thin device thickness. Integrated microscopy uses computational algorithms for object reconstruction, but efficient reconstruction algorithms for large-scale data have been lacking. Here, we developed DeepInMiniscope, a miniaturized integrated microscope featuring a custom-designed optical mask and a multi-stage physics-informed deep learning model. This reduces the computational resource demands by orders of magnitude and facilitates fast reconstruction. Our deep learning algorithm can reconstruct object volumes over 4×6×0.6 mm3. We demonstrated substantial improvement in both reconstruction quality and speed compared to traditional methods for large-scale data. Notably, we imaged neuronal activity with near-cellular resolution in awake mouse cortex, representing a substantial leap over existing integrated microscopes. DeepInMiniscope holds great promise for scalable, large-FOV, high-speed, 3D imaging applications with compact device footprint. # DeepInMiniscope: Deep-learning-powered physics-informed integrated miniscope [https://doi.org/10.5061/dryad.6t1g1jx83](https://doi.org/10.5061/dryad.6t1g1jx83) ## Description of the data and file structure ### DeepInMiniscope: Learned Integrated Miniscope ### Datasets, models and codes for 2D and 3D sample reconstructions. Dataset for 2D reconstruction includes test data for green stained lens tissue. Input: measured images of green fluorescent stained lens tissue, dissembled into sub-FOV patches. Output: the slide containing green lens tissue features. Dataset for 3D sample reconstructions includes test data for 3D reconstruction of in-vivo mouse brain video recording. Input: Time-series standard-derivation of difference-to-local-mean weighted raw video. Output: reconstructed 4-D volumetric video containing a 3-dimensional distribution of neural activities. ## Files and variables ### Download data, code, and sample results 1. Download data `data.zip`, code `code.zip`, results `results.zip`. 2. Unzip the downloaded files and place them in the same main folder. 3. Confirm that the main folder contains three subfolders: `data`, `code`, and `results`. Inside the `data` and `code` folder, there should be subfolders for each test case. ## Data 2D_lenstissue **data_2d_lenstissue.mat:** Measured images of green fluorescent stained lens tissue, disassembled into sub-FOV patches. * **Xt:** stacked 108 FOVs of measured image, each centered at one microlens unit with 720 x 720 pixels. Data dimension in order of (batch, height, width, FOV). * **Yt:** placeholder variable for reconstructed object, each centered at corresponding microlens unit, with 180 x 180 voxels. Data dimension in order of (batch, height, width, FOV). **reconM_0308:** Trained Multi-FOV ADMM-Net model for 2D lens tissue reconstruction. **gen_lenstissue.mat:** Generated lens tissue reconstruction by running the model with code **2D_lenstissue.py** * **generated_images:** stacked 108 reconstructed FOVs of lens tissue sample by multi-FOV ADMM-Net, the assembled full sample reconstruction is shown in results/2D_lenstissue_reconstruction.png 3D_mouse **reconM_g704_z5_v4:** Trained 3D Multi-FOV ADMM-Net model for 3D sample reconstructions **t_img_recd_video0003 24-04-04 18-31-11_abetterrecordlong_03560_1_290.mat:** Time-series standard-deviation of difference-to-local-mean weighted raw video. * **Xts:** test video with 290 frames and each frame 6 FOVs, with 1408 x 1408 pixels per FOV. Data dimension in order of (frames, height, width, FOV). **gen_img_recd_video0003 24-04-04 18-31-11_abetterrecordlong_03560_1_290_v4.mat:** Generated 4D volumetric video containing 3-dimensional distribution of neural activities. * **generated_images_fu:** frame-by-frame 3D reconstruction of recorded video in uint8 format. Data dimension in order of (batch, FOV, height, width, depth). Each frame contains 6 FOVs, and each FOV has 13 reconstruction depths with 416 x 416 voxels per depth. Variables inside saved model subfolders (reconM_0308 and reconM_g704_z5_v4): * **saved_model.pb:** model computation graph including architecture and input/output definitions. * **keras_metadata.pb:** Keras metadata for the saved model, including model class, training configuration, and custom objects. * **assets:** external files for custom assets loaded during model training/inference. This folder is empty, as the model does not use custom assets. * **variables.data-00000-of-00001:** numerical values of model weights and parameters. * **variables.index:** index file that maps variable names to weight locations in .data. ## Code/software ### Set up the Python environment 1. Download and install the [Anaconda distribution](https://www.anaconda.com/download). 2. The code was tested with the following packages * python=3.9.7 * tensorflow=2.7.0 * keras=2.7.0 * matplotlib=3.4.3 * scipy=1.7.1 ## Code **2D_lenstissue.py:** Python code for Multi-FOV ADMM-Net model to generate reconstruction results. The function of each script section is described at the beginning of each section. **lenstissue_2D.m:** Matlab code to display the generated image and reassemble sub-FOV patches. **sup_psf.m:** Matlab script to load microlens coordinates data and to generate PSF pattern. **lenscoordinates.xls:** Microlens units coordinates table. **3D mouse.py:** Python code for Multi-FOV ADMM-Net model to generate reconstruction results. The function of each script section is described at the beginning of each section. **mouse_3D.m:** Matlab code to display the reconstructed neural activity video and to calculate temporal correlation. ## Access information Other publicly accessible locations of the data: * [https://github.com/Yang-Research-Laboratory/DeepInMiniscope-Learned-Integrated-Miniscope](https://github.com/Yang-Research-Laboratory/DeepInMiniscope-Learned-Integrated-Miniscope)  more » « less
Award ID(s):
1847141
PAR ID:
10644633
Author(s) / Creator(s):
; ;
Publisher / Repository:
Dryad
Date Published:
Edition / Version:
7
Subject(s) / Keyword(s):
FOS: Electrical engineering, electronic engineering, information engineering FOS: Electrical engineering, electronic engineering, information engineering Lensless miniscope Calcium imaging ADMM Fluorescence microscopy Miniaturized microscope physics-informed Integrated microscope Deep learning
Format(s):
Medium: X Size: 4269376866 bytes
Size(s):
4269376866 bytes
Sponsoring Org:
National Science Foundation
More Like this
  1. Mask-based integrated fluorescence microscopy is a compact imaging technique for biomedical research. It can perform snapshot 3D imaging through a thin optical mask with a scalable field of view (FOV). Integrated microscopy uses computational algorithms for object reconstruction, but efficient reconstruction algorithms for large-scale data have been lacking. Here, we developed DeepInMiniscope, a miniaturized integrated microscope featuring a custom-designed optical mask and an efficient physics-informed deep learning model that markedly reduces computational demand. Parts of the 3D object can be individually reconstructed and combined. Our deep learning algorithm can reconstruct object volumes over 4 millimeters by 6 millimeters by 0.6 millimeters. We demonstrated substantial improvement in both reconstruction quality and speed compared to traditional methods for large-scale data. Notably, we imaged neuronal activity with near-cellular resolution in awake mouse cortex, representing a substantial leap over existing integrated microscopes. DeepInMiniscope holds great promise for scalable, large-FOV, high-speed, 3D imaging applications with compact device footprint. 
    more » « less
  2. {"Abstract":[" \n\nThis submission provides all the data and R code necessary to 1) generate the species distribution estimates and phenological predictions underlying all analyses, and 2) all results presented in the main text of the manuscript titled "Shifts in phenology and species ranges synergistically alter the timing and species composition of the flowering season" (Ramirez-Parada, Park, Peng, Nishino, Kartesz, Record, Davis, and Mazer, 2025). A preprint of this work can be accessed at: https://doi.org/10.32942/X28W5Z . This work was published in Global Change Biology (DOI: https://doi.org/10.1002/ecs2.70070).\n\nIn order for the code to work, users need to preserve the folder structure of the code and data, as the R Markdown files in the "Code" folder have relative file paths that read and write files within the "Data" folder and subfolders therein. These two folders are placed at the same level (i.e., they should both be stored within the same folder). Moving either will require re-writing the filepaths across Rmds.\n\nFor detailed descriptions of all materials (code and data) and instructions for using them, please refer to the "READ ME.md" file within "Project folder" and the main text of the manuscript.Note: This version corrects an error in  the .csv files containing the performance metrics for the three types of SDM fit for each species in the data (described in the main text). The previous version erroneously contained the performance metrics for the same model type across all three .csv files.\n\n "]} 
    more » « less
  3. Head-mounted miniaturized two-photon microscopes are powerful tools to record neural activity with cellular resolution deep in the mouse brain during unrestrained, free-moving behavior. Two-photon microscopy, however, is traditionally limited in imaging frame rate due to the necessity of raster scanning the laser excitation spot over a large field-of-view (FOV). Here, we present two multiplexed miniature two-photon microscopes (M-MINI2Ps) to increase the imaging frame rate while preserving the spatial resolution. Two different FOVs are imaged simultaneously and then demixed temporally or computationally. We demonstrate large-scale (500×500 µm2 FOV) multiplane calcium imaging in visual cortex and prefrontal cortex in freely moving mice during spontaneous exploration, social behavior, and auditory stimulus. Furthermore, the increased speed of M-MINI2Ps also enables two-photon voltage imaging at 400 Hz over a 380×150 µm2 FOV in freely moving mice. M-MINI2Ps have compact footprints and are compatible with the open-source MINI2P. M-MINI2Ps, together with their design principles, allow the capture of faster physiological dynamics and population recordings over a greater volume than currently possible in freely moving mice, and will be a powerful tool in systems neuroscience. # Data for: Multiplexed miniaturized two-photon microscopy (M-MINI2Ps) Dataset DOI: [10.5061/dryad.kd51c5bkp](10.5061/dryad.kd51c5bkp) ## Description of the data and file structure Calcium and Voltage imaging datasets from Multiplexed Miniaturized Two-Photon Microscopy (M-MINI2P) ### Files and variables #### File: TM_MINI2P_Voltage_Cranial_VisualCortex.zip **Description:** Voltage imaging dataset acquired in mouse primary visual cortex (V1) using the TM-MINI2P system through a cranial window preparation. This .zip file contains two Tif files, corresponding to the top field of view (FOV) and the bottom field of view (FOV) of the demultiplexed recordings. #### File: TM_MINI2P_Calcium_GRIN_PFC_Auditory_Free_vs_Headfix.zip **Description:** Volumetric calcium imaging dataset from mouse prefrontal cortex (PFC) using the TM-MINI2P system with a GRIN lens implant, comparing neural responses during sound stimulation versus quiet periods, under both freely moving and head-fixed conditions. This .zip file contains 12 Tif files: top and bottom fields of view (FOVs) of the multiplexed recordings at three imaging depths (100 μm, 155 μm, and 240 μm from the end of the implanted GRIN lens), with six files from freely moving conditions and six files from head-fixed conditions. #### File: CM_MINI2P_Calcium_Cranial_VisualCortex_SocialBehavior.zip **Description:** Calcium imaging dataset from mouse primary visual cortex (V1) using the CM-MINI2P system through a cranial window, recorded during social interaction and isolated conditions. This .zip file contains 6 Tif files: multiplexed recordings from the top and bottom fields of view (FOVs), and single-FOV recordings at two imaging depths (170 µm and 250 µm). #### File: TM_MINI2P_Calcium_Cranial_VisualCortex.zip **Description:** Multi-depth calcium imaging dataset from mouse primary visual cortex (V1) using the TM-MINI2P system through a cranial window during spontaneous exploration. This .zip file contains 6 Tif files: demultiplexed recordings from two fields of view (FOV1 and FOV2) at three imaging depths (110 µm, 170 µm, and 230 µm). ## Code/software All datasets are in .tiff format and ImageJ can be used for visualization. Analysis of calcium imaging data and voltage imaging data were analyzed using CaImAn and Volpy, respectively, which are open-source packages available at [https://github.com/flatironinstitute/CaImAn](https://github.com/flatironinstitute/CaImAn). 
    more » « less
  4. {"Abstract":["Why rivers confine flow to a single channel (single-thread planform) or\n divide flow into multiple sub-channels (multi-thread planform) forms a\n longstanding fundamental question in river science, which to date remains\n poorly understood. In the associated manuscript, we probe planform origins\n using a novel dataset of 11+ million riverbank migration vectors mapped\n from 36 years of global satellite imagery along 84 river systems. Results\n show single-thread rivers originate from a balance between bank erosion\n and opposing-bar deposition, which maintains an equilibrium width as\n channels migrate. In contrast, multi-thread rivers originate from\n imbalance: bank erosion outpaces opposing-bar deposition, causing\n sub-channels to repeatedly widen and split. This width instability\n challenges equilibrium paradigms in river science, endangers riverside\n communities, and lowers the potential costs of nature-based river\n restoration projects along multi-thread rivers. Here, we provide the data\n and codes that form the foundation of this manuscript. "],"TechnicalInfo":["# Data and code for: River planforms originate from (im)balance between\n bank erosion and bar accretion --- author: Austin Chadwick contact:\n [achadwick@ldeo.columbia.edu](mailto:achadwick@ldeo.columbia.edu),\n [austin.chadwick23@gmail.com](mailto:austin.chadwick23@gmail.com) These\n materials are organized into two folders, Codes and Data. The Data folder\n contains spreadsheets, GIS geopackages, remote sensing images, and MATLAB\n data files for the associated manuscript "River planforms originate\n from (im)balance between bank erosion and bar accretion". The Codes\n folder provides MATLAB scripts and functions to generate the analysis and\n figures of the main manuscript. The last folder, temp, is empty; it acts\n as a temporary destination for output files that can be generated by the\n codes. ## Description of the data and file structure All data are found in\n the folder "Data", which can be accessted by opening the zip\n file "Data.zip". The following is a breakdown of each file and\n subfolder in the Data folder. * DataLog_082224_1.xlsx This excel document\n records the files used to apply PIV to remote sensing imagery, organized\n on a site-by-site basis. All files referenced therein are found in the\n Data folder. See the first sheet therein "Readme" for details. *\n Polygons subfolder This subfolder contains geopackage files (.gpkg) for\n each study reach in the main manuscript. Each file contains geopackage\n data for a single polygon that outlines and defines the study reach. These\n data were used to extract GEOTIFFs using google earth engine, and were\n obtained by manual selection in QGIS. See main manuscript for details on\n the selection process and sizing of each polygon. * Geotiffs subfolder\n This folder contains DSWE-derived river-mask GEOTIFFs derived from Landsat\n median annual composites, used for the analysis and figures of the main\n manuscript "River planforms originate from (im)balance between bank\n erosion and bar accretion". The DSWE-derived river masks were created\n in Google Earth Engine using the "GEE_watermasks" code found on\n the GitHub Repository\n ([https://github.com/evan-greenbrg/GEE_watermasks](https://github.com/evan-greenbrg/GEE_watermasks)). Each subfolder corresponds to a study reach in the main manuscript (see Table 1 in main manuscript), with nested subfolders containing river masks derived using DSWE confidence levels 1 through 4 (mask1, mask2, mask3, mask4). * PreparedImagery subfolder This folder contains true-color images and DSWE-derived river masks (.tif) for each study reach. These data were used as input data for PIV. For each site, there is a subfolder containing truecolor images ("Color") and sixteen subfolders containing DSWE-derived river masks for the four different confidence levels and four different tilt angles ("MaskX_TiltYY", where X is the confidence level and YY is the tilt angle). In each of these subfolders, there are two nested subfolders "RemovedBlanks" and "RemovedForSubsampling" which contain additional .tif's that were removed because they were blank or because optimal PIV analysis required subsampling, respectively. For details of this process, see: [https://doi.org/10.1029/2023JF007177](https://doi.org/10.1029/2023JF007177). * OutputFromPIVlab subfolder This folder contains the Raw PIV data output from PIVlab software for each site. Each site-specific subfolder contains sixteen .mat files, each of which corresponds to a specific confidence level and tilt angle in the prepared imagery (see DataLog_082224_1.xlsx for details on which file corresponds to which confidence level and tilt angle). For additional instructions on how to operate PIVlab , please see the following links: [http://sead-published.ncsa.illinois.edu/seadrepository/api/researchobjects/urn:uuid:6154f24ae4b0312e761fb761](http://sead-published.ncsa.illinois.edu/seadrepository/api/researchobjects/urn:uuid:6154f24ae4b0312e761fb761) [https://www.mathworks.com/matlabcentral/fileexchange/27659-pivlab-particle-image-velocimetry-piv-tool-with-gui](https://www.mathworks.com/matlabcentral/fileexchange/27659-pivlab-particle-image-velocimetry-piv-tool-with-gui) * PostprocessedPIV subfolder This folder contains the postprocessed PIV data output. Each site-specific subfolder contains four .mat files. These files contain the postprocessed data before filtering (Unfiltered.mat), the postprocessed data after filtering ("Filtered.mat"), the postprocessed data after filtering and setting all NaN values along high-confidence banks to zero ("Filtered_NanToZeroOnBank"). * BankNormalVectors subfolder This folder contains the bank-normal vector fields for each image. The data structure is identical to the PostprocssedPIV data, with a single .mat file for each river "BankNormalVectors.mat". All vectors have a length of 1, and are oriented perpendicular to the nearest channel bank, pointing away from the wetted area and towards the dry area. For details on this calculation, see: [https://doi.org/10.1029/2021WR031236](https://doi.org/10.1029/2021WR031236). * CrossSections subfolder This folder contains the data for the thread cross sections randomly sampled in each reach to calculate the differential migration rate (delv). Each site-specific subfolder contains a single mat file, "CrossSections.mat". This mat contains the positions (x,y) and the bank migration vectors (u,v) for each cross section. Spatial units are in pixels, and temporal units are in frames, such that velocities are in units of pixels/frame. * WettedAreas subfolder This folder contains the data for the wetted channel areas, used to investigate apparent changes in water discharge and plot supplemental figures. Each site-specific subfolder contains a single mat file, "WettedAreas.mat". This mat contains the positions (x,y) and the bank migration vectors (u,v) for each cross section. ## Sharing/Access information Some data referenced in the Datalog_082224.xlsx spreadsheet was derived from previous work: * Sylvester et al., 2019: [https://doi.org/10.1130/G45608.1](https://doi.org/10.1130/G45608.1) * Rowland et al., 2019: [https://doi.org/10.15485/1571527](https://doi.org/10.15485/1571527) * Galeazzi et al., 2021: [https://doi.org/10.1130/G49121.1](https://doi.org/10.1130/G49121.1) * Ielpi et al., 2023: [https://doi.org/10.1029/2022GL101285](https://doi.org/10.1029/2022GL101285) ## Code/Software All codes are found in the folder "Codes", which can be accessed by opening the zip folder "Codes.zip". To run any codes, it is necessary to add the "Codes" and "Data" folders (and their subfolders) to your current MATLAB path. This can be easily achieved by right-clicking the folder in the MATLAB GUI and selecting "Add To Path-->SelectedFolders and Subfolders", and will ensure that all input and output files are identified properly. The following is a breakdown of each file and subfolder in the Codes folder: * AnalsysisAndFigures1_Worldmap_082924_1.m This MATLAB script reproduces Figure 1A of the manuscript and its associated analysis. * AnalysisAndFigures2_PIVmethods_082924_1.m This MATLAB script reproduces Figure 1E of the manuscript and its associated analysis. * AnalysisAndFigures3_DifferentialMigration_082924_1.m This MATLAB script reproduces Figure 2A and Figure 4 of the manuscript, Text S1 Figures 1–4 of the supplement, and their associated analysis. It also reproduces the tabular data in Data S1, performs the statistical tests on delv* reported in the text, and counts the number of vector measurements in our database as reported in the text. * AnalysisAndFigures4_TimeSlices_082924_1.m This MATLAB script reproduces Figure 2B-G and Figure 4 of the manuscript and Figures S1–S3 of the supplement and their associated analysis. In the first code block (L23–36), the user designates the reach and cross section to plot by uncommenting the corresponding line. See code's comments on L23–36 for details. * AnalysisAndFigures5_MaterialsAndMethods_082924_1 This MATLAB script reproduces the supplement's Materials & Methods Figures 1–6 and their associated analysis. In the first code block (L19-23), the user designates the reach to plot by uncommenting the corresponding line. See code's comments on L19-23 for details. * AnalysisAndFigures6_SuppMovieFrames_082924_1.m This MATLAB script reproduces the frames of supplementary Movies S1–S3. In the first code block (L18-23), the user designates the reach to plot by uncommenting the corresponding line. See code's comments on L18-23 for details. * LatLonBySite_081924_1.xlsx This excel document records the approximate latitude and longitude coordinates of each studied reach in the main manuscript. These coordinates were used to generate Fig. 1 in the main manuscript, and were obtained by manually inspecting the coordinates of each .gpkg file in QGIS. The precise geographic coordinates for each reach can be found in the .gpkg files, in the Polygons subfolder described above. ### Change Log Version 2: minor changes in README."]} 
    more » « less
  5. This dataset contains maps of water yield and nitrogen (N) yield each year, covering the Mississippi/Atchafalaya River Basin (MARB) spanning from 1980 to 2017. These maps were reconstructed by aggregating from a daily model (Dynamic Land Ecosystem Model, DLEM) estimates and are at 5-min×5-min (0.08333° Lat × 0.08333° Lon) resolution. There are two subfolders, "TT" and "DT", within this folder. "TT" and "DT" respectively indicate "traditional timing" and "dynamic timing" of nitrogen fertilizer applications in regards to the model experiments in the main text. The "TT" folder contains the gridded model estimates of water yield (named by "Runoff") and nitrogen yield (named by "Nleach") at annual bases. TT reflects our best estimate of water and N fluxes within the context of multi-factor environmental changes including climate, atmospheric CO2 concentration, N deposition, land use, and human management practices (such as fertilizer use, tillage, tile drainage, etc.). The "DT" folder only contains the model estimates of nitrogen yield (“Nleach”) under an alternative N management practice. More details can be found in the linked publication. 
    more » « less