Deep learning (DL) has revolutionized the field of computational imaging, 1,2 providing powerful solutions to enhance performance and address various challenges. [3][4][5][6][7][8][9][10][11] The effectiveness of DL in computational imaging lies in its ability to capture the underlying imaging model and exploit object priors, enabling robust solutions to ill-posed inverse problems. 1 However, the most widely used reconstruction methods in computational imaging rely on discrete pixels to represent the objects. For instance, a convolutional neural network (CNN) for computational imaging is typically trained on a fixed pixel or voxel grid. 1 This representation is inherently limited by the resolution and size of the grid and does not capture the continuous nature and multiscale details of the physical objects. Furthermore, the pixel grid representation poses challenges in scaling to process and store large-scale multi-dimensional computational imaging data. 5 The neural field (NF) framework 12 has recently gained significant interest for its ability to represent continuous objects. Unlike CNN structures, NF uses a coordinate-based neural representation, where spatial coordinates are mapped to physical values using a multi-layer perceptron (MLP). This allows the NF to encode objects in a continuous representation, decoupled from a discrete grid. It enables on-demand synthesis of any part of the object by simply querying relevant coordinates across arbitrary dimensions and resolutions. Several NF-based DL techniques have been recently introduced in computational imaging for solving inverse problems. 5,[13][14][15][16][17][18][19] However, these methods are limited by the high computational cost and limited generalization ability. They either require retraining a new NF network for each object 5,[13][14][15][17][18][19] or suffer from the limited representation power of the latent space learned only on the global scale, 16 restricting their ability to generalize to diverse objects.
To overcome these limitations, we propose a novel local conditional neural field (LCNF) framework for solving imaging inverse problems. We demonstrate LCNF's unique capabilities for solving the highly ill-posed phase retrieval problem in multiplexed Fourier ptychographic microscopy (FPM). 20,21 We term this LCNF-based phase retrieval method as neural phase retrieval, or NeuPh [Fig. 1(a)]. NeuPh utilizes a CNN-based encoder to learn measurement-specific information from a set of low-resolution measurements and encode them into a compact latent-space representation. This CNN-based encoder effectively extracts the diffraction information that spreads over an area in the measurement by utilizing its extended receptive field, condensing them into latent vectors. Subsequently, an MLP-based decoder is employed to reconstruct the phase values of the object at specific locations based on the corresponding latent information. This decoder is conditioned on a learned latent vector that incorporates information across a region of the input images. This conditioning enables adaptation to different objects since each set of measurements is projected onto a distinct latent space representation by the encoder. In addition, a crucial aspect of FPM phase retrieval is achieving "superresolution" reconstruction, surpassing the diffraction limit of the input low-resolution measurements. 22 To reach this goal, NeuPh's decoder further utilizes "super-resolved" latent information during training. By combining these components, NeuPh achieves scalable and generalizable DL-based continuousdomain high-resolution image reconstruction based on lowresolution measurements that is applicable to arbitrary objects with varying spatial scales and resolutions.
Our results highlight NeuPh's ability to apply continuous and smooth priors to the reconstruction and showcase more accurate reconstruction results compared to existing models. Using experimental datasets captured on Hela cells, we show that NeuPh can accurately reconstruct complex intricate subcellular structures. In addition, we highlight NeuPh reconstruction compared to state-of-the-art neural networks used in FPM phase retrieval problems.
Furthermore, we showcase NeuPh's strong generalization capabilities, which cannot be achieved by existing NF networks. First, we demonstrate that NeuPh can consistently perform highresolution reconstruction even when trained with very limited data or under different experimental conditions. Moreover, we demonstrate that NeuPh can be trained using exclusively physics-model-simulated datasets composed of natural images. We show that this physics simulator-trained NeuPh generalizes well to experimental biological samples. We further investigate a hybrid training strategy combining both experimental and simulated datasets. By collectively analyzing the results from the pure-experimental-based, hybrid, and pure-simulation-based training methods, we elucidate the effects of domain shift and underscore the importance of aligning the data distribution between the simulation and experiment to ensure effective network training with simulation. Finally, we show that all NeuPh networks, regardless of the training dataset, can reliably reconstruct high-resolution images across a wide field of view (FOV). The results are robust to unknown spatially varying aberrations, benefiting from the smooth object priors provided by NeuPh.
In summary, we introduce a novel LCNF framework as a scalable, robust, accurate, and generalizable approach for solving highly ill-posed large-scale imaging inverse problems. By leveraging a continuous neural representation, NeuPh effectively captures multiscale object information from lowresolution measurements and provides robust resolution enhancement in the reconstruction. Furthermore, our LCNF framework exhibits superior reconstruction performance compared to existing models, highlighting the advantages of the MLP-based decoder and coordinate-based training strategy. The framework's ability to generalize with very limited training data and its capacity to leverage physics-based simulation further enhance its potential for advancing DL-based computational imaging techniques.
We investigate different datasets for training NeuPh using both experimental and simulated datasets. The experimental data are taken on Hela cells fixed in ethanol [Hela(E)] or formalin [Hela(F)] from Ref. 23. Due to different fixation procedures, the Hela cells show different morphological features.
To briefly describe the experiment, our multiplexed measurements use five LED patterns (central wavelength of 630 nm), including two brightfield (BF) semi-circle and three darkfield (DF) 120 deg arc patterns [Fig. 1(b)], to efficiently encode high-resolution phase information across a wide FOV. Matching sequential FPM datasets are also captured by single-LED illumination with 185 LEDs. In both schemes, the maximum illumination numerical aperture (NA) is 0.41. Images are collected using a 4×, 0.1 NA objective (Nikon CFI Plan Achromat) and an sCMOS camera (PCO.edge 5.5) with 2560 pixel × 2160 pixel and a 6.5 μm pixel size. We collect 22 groups of Hela(E) measurements and 20 groups of Hela(F) measurements. The groundtruth images are computed using the model-based algorithm 21 on the sequential FPM measurements to process the central 250 pixel × 250 pixel regions and reconstruct the high-resolution phase images yielding 1500 pixel × 1500 pixel. Accordingly, the full-pitch resolution 24 is theoretically improved from 6.3 (corresponding to 0.1 NA) to 1.235 μm (corresponding to 0.51 NA).
For the simulated dataset, we collect images from DIV2K dataset. 25 The whole simulated dataset consists of 900 cropped high-resolution natural images (600 pixel × 600 pixel) and corresponding simulated low-resolution multiplexed illuminated intensity images (100 pixel × 100 pixel). The forward model for the simulation, model-based reconstruction algorithms, data acquisition, and preparation are detailed in the Supplementary Material.
NeuPh's network structure is shown in Fig. 1(a). The encoder E θ e takes six low-resolution images as input and projects the learned information into a latent space. The input consists of two BF and three DF images, along with a low-resolution phase estimate using the two BF images and the differential phase contrast (DPC) method 20 (see details in the Supplementary Material). To handle the distinct features of BF, DF, and DPC images, three separate encoders fe 1 ; e 2 ; e 3 g are employed to extract the latent information. Each encoder utilizes convolution layers and residual blocks to extract spatial features. The lateral dimensions of the feature maps match those of the input image, allowing for direct coordinate-based latent information retrieval during decoding. The features learned from the three encoders are concatenated to form the final latent space representation Φ ∈ R H×W×D , where H and W are the numbers of pixels in the lateral dimensions and D is the total number of concatenated feature maps in the latent space.
To enable high-resolution reconstruction independent of a fixed grid, a coordinate-based decoder is employed using an MLP f θ . For local conditioning, a latent vector ϕ ∈ R 1×1×D is concatenated with the corresponding coordinate c before being input to f θ . This ensures that the learned mapping by the MLP is informed by the input, allowing for model generalization. The decoder's output is a predicted phase value at location c. NeuPh is trained end-to-end by minimizing a loss function L½f θ ðc; ϕÞ; ψðcÞ that measures the difference between the predicted [f θ ðc; ϕÞ] and the ground-truth phase ψðcÞ at c. To accelerate the training, we randomly select N coordinates from small training batches at each step. Correspondingly, we perform the minimization:
where θ e and θ are the weights of the encoder and decoder, respectively, c n is the n'th selected coordinate, ϕ n ¼ E θ e ðm; c n Þ is the latent vector encoded from input m for the queried coordinate c n , ψðc n Þ is the ground-truth phase value at position c n , and k • k 1 denotes the L 1 norm. The ground-truth images are obtained from the paired sequential FPM measurements and reconstructed by a model-based algorithm (see Supplementary Material).
A key aspect of FPM reconstruction is "super resolution." To facilitate learning high-resolution information beyond the low-resolution H × W grid, NeuPh is also trained on "offthe-grid" high-resolution data queried from a denser grid H 0 × W 0 . However, the corresponding off-the-grid latent vector is not readily available from the encoded latent space. To solve this issue, the nearest latent vector is used for the decoder.
In addition, to inform the decoder about the relative position of the queried off-the-grid location with respect to the nearest latent vector location, the implementation of Eq. ( 1) utilizes their relative coordinate Δc instead of the absolute coordinate c, following the LIIF method. 26 Furthermore, to utilize the information provided by the neighboring latent vectors and improve the continuity of the reconstruction, enhancement techniques including feature unfolding, local ensemble, and cell decoding 26 are applied.
After training, NeuPh allows for querying arbitrary points on the object by inputting the corresponding measurements and the queried coordinates. This eliminates the need for a fixed input grid in traditional model-based and CNN methods. The highresolution phase reconstruction can be achieved on any desired grid. This feature is demonstrated in Fig. 1(c), where reconstructions are queried at three distinct pixel densities, showcasing smooth transitions across these diverse spatial scales without any artifacts. More details about NeuPh's structure, reconstruction enhancement techniques, and ablation studies of its architecture can be found in Secs. 2, 3, 6, Fig. S6, and Table S2 in the Supplementary Material.
To comprehensively evaluate NeuPh's generalization capability and investigate the impact of domain shift between the simulated and experimental datasets on reconstruction result, in total, we explored four different training strategies, including training with the full experimental dataset (NeuPh Eð18Þ , NeuPh Fð16Þ ), training with a single pair of experimental datasets (NeuPh Eð1Þ , NeuPh Fð1Þ ), training with a purely simulated natural image dataset (NeuPh Simð18Þ , NeuPh Simð16Þ , NeuPh Sim ), and training with mixed experimental and simulated images (NeuPh E∶Simð9∶9Þ , NeuPh E∶Simð1∶17Þ , NeuPh F∶Simð8∶8Þ , and NeuPh F∶Simð1∶15Þ ).
For all four scenarios, the training typically converges at 500 epochs. Training the network with a single pair of experimental datasets takes ∼3 h, while training with the full experimental dataset, a blend of an experimental and simulated datasets, or the simulated dataset takes ∼24 to 48 h to converge on an NVIDIA Tesla P100 GPU. More details about the implementation of network training and datasets used for training different networks can be found in the Sec. 5 in the Supplementary Material.
To perform inference, we input the preprocessed measurements of the desired FOV and configure the pixel resolution for the reconstruction. During the inference, NeuPh predicts the phase value for each queried position. In our main results, we perform inference with a 6× denser grid compared to the raw measurement. We first divide the measurement into small patches with 250 pixel × 250 pixel. Next, we reconstruct each patch individually, producing high-resolution images with 1500 pixel × 1500 pixel. This process takes ∼25 s, resulting in an average rate of ∼1 × 10 -5 s∕pixel on an NVIDIA Quadro RTX8000 GPU. In comparison, the model-based FPM reconstruction with sequential illumination takes around 8 min. In addition, the model-based multiplexed FPM reconstruction algorithm fails to generate high-resolution phase images from multiplexed illumination with only five intensity images. 23 To create the final wide-FOV reconstruction, we employed alpha blending 23 to stitch together the individual reconstructions, forming a high-resolution image with a diameter of 12,960 pixel. It is worth noting that NeuPh is inherently capable of directly inferring the entire FOV image without any stitching. However, due to the GPU memory (48 GB) limit on our GPU, we apply this patch-wise inference method.
We first demonstrate NeuPh's capability on experimentally imaged Hela cells fixed with ethanol or formalin. Here, we show results using the networks that are trained separately for these two data types (NeuPh Eð18Þ and NeuPh Fð16Þ ) to showcase NeuPh's best performance since different cell fixations induce distinctive morphological features within single cells and thus a domain shift in the data (which we explore in more detail in Sec. 3.5). In Fig. 1(c), we present the raw low-resolution BF intensity image, the model-based FPM reconstruction, and the NeuPh reconstruction of ethanol-fixed Hela cells. In Figs. 2(a) and 2(b), we show additional reconstruction results for Hela cells fixed with ethanol or formalin, respectively, with a few zoomed-in regions. It is evident that NeuPh successfully reconstructs high-resolution phase images from low-resolution measurements, accurately recovering intricate subcellular structures and achieving much better results compared with the linear DPC estimate.
To showcase NeuPh's continuous object representation capability, we conduct inference on arbitrary coordinates. As shown in Fig. 1(c) and Fig. S5 in the Supplementary Material, we perform queries for subarea 1 at pixel densities of 6×, 21×, 49.8×, 73.5×, and 105.9× compared to the input low-resolution intensity image. NeuPh successfully reconstructs the phase at these density grids. For comparison, we also include the model-based reconstruction of the same area. Due to predefined grids, the reconstruction exhibits discrete grid artifacts in the enlarged image. Furthermore, it may suffer from phase unwrapping artifacts (see Sec. 3.2 for more details). In contrast, NeuPh provides continuous object reconstruction without any discrete or other phase artifacts.
Next, we highlight NeuPh's robustness to various phase artifacts that arise from practical FPM experiments, including measurement noise, phase unwrapping errors, and artifacts resulting from an imperfect imaging model. As shown in Figs. 1(c) and 3(a), model-based FPM reconstruction may exhibit discontinuous artifacts due to imperfect phase unwrapping when dealing with samples with a phase range exceeding 2π. Moreover, Fig. 3(a) illustrates rippling artifacts in the background region of the model-based reconstruction, possibly resulting from the model-mismatch in the reconstruction. 27 In contrast, NeuPh (NeuPh Eð18Þ ) effectively eliminates these artifacts and achieves accurate, smooth, and continuous reconstructions without any explicit regularization term. We quantitatively evaluate the artifact-suppression capability of NeuPh by measuring the residual random fluctuations in the background regions. 24 Our analysis shows that NeuPh can reduce the background artifacts by several folds compared with the modelbased FPM reconstruction, as shown in Fig. 3(a). Additional results on phase artifact removal using NeuPh are shown in Fig. S7(a) in the Supplementary Material.
This result is particularly intriguing because the NeuPh network was trained using this type of 'noisy' images from real experiments, and the network shows remarkable robustness in both the training and testing stages. We attribute this robustness to the continuity of the learned latent space and the continuous representation imposed by the coordinate-based decoder, both of which implicitly regularize the ill-posed inverse problem. First, NeuPh encodes the input images into a continuous latent space representation, effectively filtering out noisy information. Second, NeuPh decodes the phase value by a coordinate-based network, which implicitly learns a continuous neural representation of the object.
We conduct a thorough ablation study to illustrate NeuPh's superior performance compared to CNN-based models. The detailed procedure of the ablation study is provided in the Supplementary Material, and the reconstruction results of ethanol-fixed Hela cells are shown in Fig. 3(b). Specific subareas of interest are zoomed in, with subcellular details highlighted by red circles. In addition, spatial spectra are included at the bottom left of each figure, along with quantitative metrics, including the mean square error (MSE) and frequency measurement (FM) metric noted at the bottom. The FM quantifies the recovery of frequency components 28,29 -a metric for measuring the resolving power of the reconstruction algorithm-where higher FM values represent the recovery of more frequency components. We also compute the structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR), as detailed in Table S3 in the Supplementary Material. Additional comparisons are shown in Figs. S7(b) and S10 in the Supplementary Material. The metrics for an additional 100 testing image patches outside the training FOV region are provided in Table S4 in the Supplementary Material. Our results show that the CNN-based network produces more reconstruction artifacts and fails to recover as many high-frequency components as NeuPh, as evident in the red circled regions and the reconstruction spectra in Fig. 3(b), as well as the FM scores. FPM unwrapped phase NeuPh reconstruction ) E ( a l e H raw FPM phase 50 5 rad r d rad 0 7 0 0.2 0.4 0.6 0.8 1.0 0 2 1 3 4 5 6 × 10 3 phase (rad) pixel number Model-based FPM LCNF NeuPh = . = . (a) a.u. 0 1 Brightfield measurement Model-based FPM 50 10 ) E ( a l e H NeuPhE CNNE rad 0 9
To further evaluate our network's phase retrieval performance, we compared NeuPh with existing state-of-the-art networks used in FPM reconstructions. Specifically, we compared our network's performance with phase retrieval results provided by a GAN-based network 23 and a traditional NF network. 19 We trained the GAN-based network using the same singlepaired experimental dataset as the CNN-based network and NeuPh in Sec.
A notable advantage of our LCNF framework is its superior generalization capability, overcoming the limitations of existing NF frameworks 5 that require retraining for different objects. To thoroughly evaluate NeuPh's generalizability, we perform phase reconstruction for ethanol-fixed Hela cells with different dataset-trained networks. The reconstruction results are presented in Fig. 4 with the corresponding MSE and FM scores provided in the figures. We also compute the PSNR and SSIM scores in Table S5 in the Supplementary Material. In addition, to gain insights into NeuPh's robustness against realistic spatially varying aberrations in our experiment, we evaluate the reconstructions on an additional 100 image patches outside the training region, whose performance metrics are provided in Table S6 in the Supplementary Material. In Fig. S8 and Tables S5 and S6 in the Supplementary Material, we repeat the same evaluation on formalin-fixed Hela cells.
Our NeuPh successfully reconstructs high-resolution phase images regardless of the training dataset, showcasing its strong generalization. The MSE generally increases, while the PSNR, SSIM, and FM decrease when training NeuPh with a very limited dataset or a different type of data compared to the network trained with the same cell type and the full experimental dataset. This indicates that the network's generalization performance generally degrades when it is trained on a smaller training dataset or the distribution of the testing data is shifted from that of the training data, which is expected. However, the changes in the metric scores are small and hardly noticeable in the visualizations in Fig. 4, even for the network trained with a single paired training dataset. This highlights NeuPh's robust generalization. When NeuPh is trained with ethanol-fixed Hela cells (NeuPh Eð18Þ ) and applied to formalin-fixed Hela cells, the FM is slightly higher than those of the network trained with formalinfixed Hela cells (NeuPh Fð16Þ ), as indicated in Fig. S8 and Tables S5 and S6 in the Supplementary Material. We attribute this unusual result to the fact that ethanol-fixed Hela cells contain more structural details and provide a broader spectrum compared to formalin-fixed Hela cells [see the spectra shown in Furthermore, we demonstrate NeuPh's robust generalization by employing pure simulator-based training (NeuPh Simð18Þ , NeuPh Simð16Þ ) to reconstruct the experimental Hela cell dataset, showcasing the straightforward application of simulator-based training for generating subcellular phase images from low-resolution intensity images. Our approach overcomes the limitations of existing networks when obtaining paired experimental training data is challenging, as they either require retraining 5,[13][14][15]17 or are hindered by the limitations of generative adversarial networks (GANs). 23 Although the quantitative results indicate that the simulator-trained NeuPh performs slightly worse than the experimental-data-trained networks, this difference is expected due to the significantly different image features between the natural images used in training and the cells in the experiment.
To better understand the influence of domain shift between simulation and experiment, Fig. 4 compares inference results in NeuPh Eð18Þ , NeuPh E∶Simð9∶9Þ , NeuPh E∶Simð1∶17Þ , and NeuPh Simð18Þ which are trained by different blends of experimental and simulated datasets. From the reconstructed images and the metrics in Table S5 in the Supplementary Material, it is evident that incorporating experimental datasets consistently leads to better performance compared to networks trained solely on simulated datasets. Generally speaking, an increase in the proportion of the simulated data in the training set results in slightly degraded reconstruction performance on experimental measurements, indicating the adverse impact of the domain shifts between the simulation and experimental datasets. However, this performance degradation is minimal by implementing our distribution matching procedure, as detailed in the Supplementary Material and Fig. S3 in the Supplementary Material. In addition, we show that the NeuPh trained solely on the experimentally captured biological images can successfully reconstruct simulated measurements from natural images in Fig. S9 in the Supplementary Material.
We Overall, the NeuPh networks achieve high-quality reconstructions, with intricate subcellular details distinctly visible and minimal artifacts. However, some distortions are noticeable at the FOV's extreme periphery, as seen in the subarea (iv). These distortions can be attributed to the unaddressed spatially varying aberrations in our setup, which become more pronounced at the periphery of the FOV. 30
We have introduced LCNF, a scalable and generalizable DL framework for solving imaging inverse problems. Unlike traditional model-based or CNN frameworks, LCNF employs a continuous neural representation, facilitating flexible reconstruction of multiscale information. A novel local conditioning technique combined with a coordinate-based training strategy is incorporated, improving its accuracy compared with existing state-ofthe-art deep learning models and significantly enhancing its generalization capability compared to existing NF frameworks.
By applying LCNF to solve the multiplexed FPM phase retrieval problem, we demonstrate a novel deep neural network, NeuPh, for continuous-domain large-scale super-resolution phase reconstruction from a sparse set of low-resolution intensity measurements. Notably, NeuPh exhibits robustness against noisy training data. Reconstructions by NeuPh are free from typical artifacts, such as residual phase unwrapping errors, noise, and background ripples, which contaminate the training data obtained by traditional FPM reconstructions. In addition, we observe the superior reconstruction performance of NeuPh, which recovers more frequency components with fewer artifacts compared to the CNN-based model and shows better reconstruction performance on multiplex FPM than existing state-of-the-art networks. This observation suggests that NeuPh has the potential to mitigate the inherent spectral bias associated with CNN-based models.
Furthermore, NeuPh demonstrates remarkable generalization across different object types and experimental conditions, surpassing existing NF frameworks. We demonstrate that NeuPh trained on various datasets including purely simulated ones can reliably generalize to biological samples and reconstruct subcellular details with super-resolution without requiring network retraining or transfer learning. This highlights an effective strategy to circumvent the collection of experimental training data altogether, leveraging the knowledge of imaging physics to simulate training datasets. In addition, we emphasize the importance of aligning dataset distributions between the experimental and simulated datasets to optimize reconstruction results.
As indicated by our wide-FOV reconstruction results in Sec. 3.6, the unknown spatially varying aberrations in the imaging system present a main limitation that needs to be addressed to further improve NeuPh's performance. Both advanced physical modeling methods 7,8 and novel network designs 8,31 have been effective in addressing this issue in fluorescence microscopy. Adapting these ideas to phase microscopy offers a promising direction for future research. In addition, NF has demonstrated its unique capability in effectively modeling multi-dimensional spatiotemporal information in several dynamic imaging applications. 14,17,18 By integrating these spatiotemporal NF frameworks with our local conditioning technique, one may achieve broad generalization and significantly reduce the computational cost in dynamic image reconstruction, making it another promising direction for future advancements.
In conclusion, we present LCNF as a scalable, robust, accurate, and generalizable DL-based continuous-domain image reconstruction framework. While our main focus in this work centers on addressing large-scale, super-resolution phase retrieval based on the multiplexed FPM technique, we envision that the LCNF framework holds potential for broader applications. It can be adapted to various computational imaging techniques for solving the underlying ill-posed inverse problems in areas, such as holographic imaging, imaging through complex media, and light-field microscopy.
Advanced Photonics Nexus 056005-2 Sep∕Oct 2024 • Vol. 3(5)
Advanced Photonics Nexus 056005-3 Sep∕Oct 2024 • Vol. 3(5)
Advanced Photonics Nexus 056005-4 Sep∕Oct 2024 • Vol. 3(5)
Advanced Photonics Nexus 056005-6 Sep∕Oct 2024 • Vol. 3(5)
Advanced Photonics Nexus 056005-7Sep∕Oct 2024 • Vol. 3(5)
Advanced Photonics Nexus 056005-8 Sep∕Oct 2024 • Vol. 3(5)
Advanced Photonics Nexus 056005-10 Sep∕Oct 2024 • Vol. 3(5)