More Like this

1. Deep Denoising for Scientific Discovery: A Case Study in Electron Microscopy

   Mohan, Sreyas; Manzorro, Ramon; Vincent, Joshua L.; Tang, Binh; Sheth, Dev Y.; Simoncelli, Eero P.; Matteson, David S.; Crozier, Peter A.; Fernandez-Granda, Carlos (March 2021, ArXivorg)

   null (Ed.)

   Denoising is a fundamental challenge in scientific imaging. Deep convolutional neural networks (CNNs) provide the current state of the art in denoising natural images, where they produce impressive results. However, their potential has been inadequately explored in the context of scientific imaging. Denoising CNNs are typically trained on real natural images artificially corrupted with simulated noise. In contrast, in scientific applications, noiseless ground-truth images are usually not available. To address this issue, we propose a simulation-based denoising (SBD) framework, in which CNNs are trained on simulated images. We test the framework on data obtained from transmission electron microscopy (TEM), an imaging technique with widespread applications in material science, biology, and medicine. SBD outperforms existing techniques by a wide margin on a simulated benchmark dataset, as well as on real data. We analyze the generalization capability of SBD, demonstrating that the trained networks are robust to variations of imaging parameters and of the underlying signal structure. Our results reveal that state-of-the-art architectures for denoising photographic images may not be well adapted to scientific-imaging data. For instance, substantially increasing their field-of-view dramatically improves their performance on TEM images acquired at low signal-to-noise ratios. We also demonstrate that standard performance metrics for photographs (such as PSNR and SSIM) may fail to produce scientifically meaningful evaluation. We propose several metrics to remedy this issue for the case of atomic resolution electron microscope images. In addition, we propose a technique, based on likelihood computations, to visualize the agreement between the structure of the denoised images and the observed data. Finally, we release a publicly available benchmark dataset of TEM images, containing 18,000 examples. 

   more » « less
2. Model-based system matrix for iterative reconstruction in sub-diffuse angular-domain fluorescence optical projection tomography

   https://doi.org/10.1364/BOE.414404  

   Torres, Veronica_C; Li, Chengyue; Brankov, Jovan_G; Tichauer, Kenneth_M (February 2021, Biomedical Optics Express)

   This work concerns a fluorescence optical projection tomography system for low scattering tissue, like lymph nodes, with angular-domain rejection of highly scattered photons. In this regime, filtered backprojection (FBP) image reconstruction has been shown to provide reasonable quality images, yet here a comparison of image quality between images obtained by FBP and iterative image reconstruction with a Monte Carlo generated system matrix, demonstrate measurable improvements with the iterative method. Through simulated and experimental phantoms, iterative algorithms consistently outperformed FBP in terms of contrast and spatial resolution. Moreover, when projection number was reduced, in order to reduce total imaging time, iterative reconstruction suppressed artifacts that hampered the performance of FBP reconstruction (structural similarity of the reconstructed images with “truth” was improved from 0.15 ± 1.2 × 10⁻³to 0.66 ± 0.02); and although the system matrix was generated for homogenous optical properties, when heterogeneity (62.98 cm^-1variance inµ_s) was introduced to simulated phantoms, the results were still comparable (structural similarity homo: 0.67 ± 0.02 vs hetero: 0.66 ± 0.02). 

   more » « less
3. An end-to-end recurrent compressed sensing method to denoise, detect and demix calcium imaging data

   https://doi.org/10.1038/s42256-024-00892-w  

   Zhang, Kangning; Tang, Sean; Zhu, Vivian; Barchini, Majd; Yang, Weijian (September 2024, Nature Machine Intelligence)

   Two-photon calcium imaging provides large-scale recordings of neuronal activities at cellular resolution. A robust, automated and high-speed pipeline to simultaneously segment the spatial footprints of neurons and extract their temporal activity traces while decontaminating them from background, noise and overlapping neurons is highly desirable to analyse calcium imaging data. Here we demonstrate DeepCaImX, an end-to-end deep learning method based on an iterative shrinkage-thresholding algorithm and a long short-term memory neural network to achieve the above goals altogether at a very high speed and without any manually tuned hyperparameter. DeepCaImX is a multi-task, multi-class and multi-label segmentation method composed of a compressed sensing-inspired neural network with a recurrent layer and fully connected layers. The neural network can simultaneously generate accurate neuronal footprints and extract clean neuronal activity traces from calcium imaging data. We trained the neural network with simulated datasets and benchmarked it against existing state-of-the-art methods with in vivo experimental data. DeepCaImX outperforms existing methods in the quality of segmentation and temporal trace extraction as well as processing speed. DeepCaImX is highly scalable and will benefit the analysis of mesoscale calcium imaging. 

   more » « less
4. MAGIC: Multitask Automated Generation of Inter-modal CT Perfusion Maps via Generative Adversarial Network

   Fullerton, Garrett; Kato, Simon; Fang, Ruogu (October 2021, Biomedical Engineering Society Annual Meeting)

   Introduction: Computed tomography perfusion (CTP) imaging requires injection of an intravenous contrast agent and increased exposure to ionizing radiation. This process can be lengthy, costly, and potentially dangerous to patients, especially in emergency settings. We propose MAGIC, a multitask, generative adversarial network-based deep learning model to synthesize an entire CTP series from only a non-contrasted CT (NCCT) input. Materials and Methods: NCCT and CTP series were retrospectively retrieved from 493 patients at UF Health with IRB approval. The data were deidentified and all images were resized to 256x256 pixels. The collected perfusion data were analyzed using the RapidAI CT Perfusion analysis software (iSchemaView, Inc. CA) to generate each CTP map. For each subject, 10 CTP slices were selected. Each slice was paired with one NCCT slice at the same location and two NCCT slices at a predefined vertical offset, resulting in 4.3K CTP images and 12.9K NCCT images used for training. The incorporation of a spatial offset into the NCCT input allows MAGIC to more accurately synthesize cerebral perfusive structures, increasing the quality of the generated images. The studies included a variety of indications, including healthy tissue, mild infarction, and severe infarction. The proposed MAGIC model incorporates a novel multitask architecture, allowing for the simultaneous synthesis of four CTP modalities: mean transit time (MTT), cerebral blood flow (CBF), cerebral blood volume (CBV), and time to peak (TTP). We propose a novel Physicians-in-the-loop module in the model's architecture, acting as a tunable layer that allows physicians to manually adjust the amount of anatomic detail present in the synthesized CTP series. Additionally, we propose two novel loss terms: multi-modal connectivity loss and extrema loss. The multi-modal connectivity loss leverages the multi-task nature to assert that the mathematical relationship between MTT, CBF, and CBV is satisfied. The extrema loss aids in learning regions of elevated and decreased activity in each modality, allowing for MAGIC to accurately learn the characteristics of diagnostic regions of interest. Corresponding NCCT and CTP slices were paired along the vertical axis. The model was trained for 100 epochs on a NVIDIA TITAN X GPU. Results and Discussion: The MAGIC model’s performance was evaluated on a sample of 40 patients from the UF Health dataset. Across all CTP modalities, MAGIC was able to accurately produce images with high structural agreement between the entire synthesized and clinical perfusion images (SSIMmean=0.801 , UQImean=0.926). MAGIC was able to synthesize CTP images to accurately characterize cerebral circulatory structures and identify regions of infarct tissue, as shown in Figure 1. A blind binary evaluation was conducted to assess the presence of cerebral infarction in both the synthesized and clinical perfusion images, resulting in the synthesized images correctly predicting the presence of cerebral infarction with 87.5% accuracy. Conclusions: We proposed a MAGIC model whose novel deep learning structures and loss terms enable high-quality synthesis of CTP maps and characterization of circulatory structures solely from NCCT images, potentially eliminating the requirement for the injection of an intravenous contrast agent and elevated radiation exposure during perfusion imaging. This makes MAGIC a beneficial tool in a clinical scenario increasing the overall safety, accessibility, and efficiency of cerebral perfusion and facilitating better patient outcomes. Acknowledgements: This work was partially supported by the National Science Foundation, IIS-1908299 III: Small: Modeling Multi-Level Connectivity of Brain Dynamics + REU Supplement, to the University of Florida. 

   more » « less
5. Joint Commonsense and Relation Reasoning for Image and Video Captioning

   Hou, J; Wu, X; Jia, Y; Luo, J. (February 2020, Proceedings of the AAAI Conference on Artificial Intelligence)

   Exploiting relationships between objects for image and video captioning has received increasing attention. Most existing methods depend heavily on pre-trained detectors of objects and their relationships, and thus may not work well when facing detection challenges such as heavy occlusion, tiny-size objects, and long-tail classes. In this paper, we propose a joint commonsense and relation reasoning method that exploits prior knowledge for image and video captioning without relying on any detectors. The prior knowledge provides semantic correlations and constraints between objects, serving as guidance to build semantic graphs that summarize object relationships, some of which cannot be directly perceived from images or videos. Particularly, our method is implemented by an iterative learning algorithm that alternates between 1) commonsense reasoning for embedding visual regions into the semantic space to build a semantic graph and 2) relation reasoning for encoding semantic graphs to generate sentences. Experiments on several benchmark datasets validate the effectiveness of our prior knowledge-based approach. 

   more » « less