Existing applications of deep learning in computational imaging and microscopy mostly depend on supervised learning, requiring large-scale, diverse and labelled training data. The acquisition and preparation of such training image datasets is often laborious and costly, leading to limited generalization to new sample types. Here we report a self-supervised learning model, termed GedankenNet, that eliminates the need for labelled or experimental training data, and demonstrate its effectiveness and superior generalization on hologram reconstruction tasks. Without prior knowledge about the sample types, the self-supervised learning model was trained using a physics-consistency loss and artificial random images synthetically generated without any experiments or resemblance to real-world samples. After its self-supervised training, GedankenNet successfully generalized to experimental holograms of unseen biological samples, reconstructing the phase and amplitude images of different types of object using experimentally acquired holograms. Without access to experimental data, knowledge of real samples or their spatial features, GedankenNet achieved complex-valued image reconstructions consistent with the wave equation in free space. The GedankenNet framework also shows resilience to random, unknown perturbations in the physical forward model, including changes in the hologram distances, pixel size and illumination wavelength. This self-supervised learning of image reconstruction creates new opportunities for solving inverse problems in holography, microscopy and computational imaging.
Deep learning-based image reconstruction methods have achieved remarkable success in phase recovery and holographic imaging. However, the generalization of their image reconstruction performance to new types of samples never seen by the network remains a challenge. Here we introduce a deep learning framework, termed Fourier Imager Network (FIN), that can perform end-to-end phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting unprecedented success in external generalization. FIN architecture is based on spatial Fourier transform modules that process the spatial frequencies of its inputs using learnable filters and a global receptive field. Compared with existing convolutional deep neural networks used for hologram reconstruction, FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ~0.04 s per 1 mm2of the sample area. We experimentally validated the performance of FIN by training it using human lung tissue samples and blindly testing it on human prostate, salivary gland tissue and Pap smear samples, proving its superior external generalization and image reconstruction speed. Beyond holographic microscopy and quantitative phase imaging, FIN and the underlying neural network architecture might open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.
more » « less- NSF-PAR ID:
- 10369691
- Publisher / Repository:
- Nature Publishing Group
- Date Published:
- Journal Name:
- Light: Science & Applications
- Volume:
- 11
- Issue:
- 1
- ISSN:
- 2047-7538
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
more » « less
Abstract Deep learning has been broadly applied to imaging in scattering applications. A common framework is to train a descattering network for image recovery by removing scattering artifacts. To achieve the best results on a broad spectrum of scattering conditions, individual “expert” networks need to be trained for each condition. However, the expert’s performance sharply degrades when the testing condition differs from the training. An alternative brute-force approach is to train a “generalist” network using data from diverse scattering conditions. It generally requires a larger network to encapsulate the diversity in the data and a sufficiently large training set to avoid overfitting. Here, we propose an
adaptive learning framework, termed dynamic synthesis network (DSN), whichdynamically adjusts the model weights andadapts to different scattering conditions. The adaptability is achieved by a novel “mixture of experts” architecture that enables dynamically synthesizing a network by blending multiple experts using a gating network. We demonstrate the DSN in holographic 3D particle imaging for a variety of scattering conditions. We show in simulation that our DSN provides generalization across acontinuum of scattering conditions. In addition, we show that by training the DSN entirely on simulated data, the network can generalize to experiments and achieve robust 3D descattering. We expect the same concept can find many other applications, such as denoising and imaging in scattering media. Broadly, our dynamic synthesis framework opens up a new paradigm for designing highlyadaptive deep learning and computational imaging techniques. -
more » « less
Abstract Digital holographic microscopy enables the 3D reconstruction of volumetric samples from a single-snapshot hologram. However, unlike a conventional bright-field microscopy image, the quality of holographic reconstructions is compromised by interference fringes as a result of twin images and out-of-plane objects. Here, we demonstrate that cross-modality deep learning using a generative adversarial network (GAN) can endow holographic images of a sample volume with bright-field microscopy contrast, combining the volumetric imaging capability of holography with the speckle- and artifact-free image contrast of incoherent bright-field microscopy. We illustrate the performance of this “bright-field holography” method through the snapshot imaging of bioaerosols distributed in 3D, matching the artifact-free image contrast and axial sectioning performance of a high-NA bright-field microscope. This data-driven deep-learning-based imaging method bridges the contrast gap between coherent and incoherent imaging, and enables the snapshot 3D imaging of objects with bright-field contrast from a single hologram, benefiting from the wave-propagation framework of holography.
-
Holographic microscopy has emerged as a tool for in situ imaging of microscopic organisms and other particles in the marine environment: appealing because of the relatively larger sampling volume and simpler optical configuration compared to other imaging systems. However, its quantitative capabilities have so far remained uncertain, in part because hologram reconstruction and image recognition have required manual operation. Here, we assess the quantitative skill of our automated hologram processing pipeline (CCV Pipeline), to evaluate the size and concentration measurements of environmental and cultured assemblages of marine plankton particles, and microspheres. Over 1 million particles, ranging from 10 to 200 μm in equivalent spherical diameter, imaged by the 4-Deep HoloSea digital inline holographic microscope (DIHM) are analyzed. These measurements were collected in parallel with a FlowCam (FC), Imaging FlowCytobot (IFCB), and manual microscope identification. Once corrections for particle location and nonuniform illumination were developed and applied, the DIHM showed an underestimate in ESD of about 3% to 10%, but successfully reproduced the size spectral slope from environmental samples, and the size distribution of cultures (Dunaliella tertiolecta, Heterosigma akashiwo, and Prorocentrum micans) and microspheres. DIHM concentrations (order 1 to 1000 particles ml−1) showed a linear agreement (r2 = 0.73) with the other instruments, but individual comparisons at times had large uncertainty.Overall, we found the DIHM and the CCV Pipeline required extensive manual correction, but once corrected,provided concentration and size estimates comparable to the other imaging systems assessed in this study. Holographic cameras are mechanically simple, autonomous, can operate at very high pressures, and provide a larger sampling volume than comparable lens-based tools. Thus, we anticipate that these characterization efforts will be rewarded with novel discovery in new oceanic environments.more » « less
-
more » « less
Abstract Holographic microscopy has emerged as a tool for in situ imaging of microscopic organisms and other particles in the marine environment: appealing because of the relatively larger sampling volume and simpler optical configuration compared to other imaging systems. However, its quantitative capabilities have so far remained uncertain, in part because hologram reconstruction and image recognition have required manual operation. Here, we assess the quantitative skill of our automated hologram processing pipeline (CCV Pipeline), to evaluate the size and concentration measurements of environmental and cultured assemblages of marine plankton particles, and microspheres. Over 1 million particles, ranging from 10 to 200
μ m in equivalent spherical diameter, imaged by the 4‐Deep HoloSea digital inline holographic microscope (DIHM) are analyzed. These measurements were collected in parallel with a FlowCam (FC), Imaging FlowCytobot (IFCB), and manual microscope identification. Once corrections for particle location and nonuniform illumination were developed and applied, the DIHM showed an underestimate in ESD of about 3% to 10%, but successfully reproduced the size spectral slope from environmental samples, and the size distribution of cultures (Dunaliella tertiolecta ,Heterosigma akashiwo , andProrocentrum micans ) and microspheres. DIHM concentrations (order 1 to 1000 particles ml−1) showed a linear agreement (r 2= 0.73) with the other instruments, but individual comparisons at times had large uncertainty. Overall, we found the DIHM and the CCV Pipeline required extensive manual correction, but once corrected, provided concentration and size estimates comparable to the other imaging systems assessed in this study. Holographic cameras are mechanically simple, autonomous, can operate at very high pressures, and provide a larger sampling volume than comparable lens‐based tools. Thus, we anticipate that these characterization efforts will be rewarded with novel discovery in new oceanic environments.
