Phase-contrast imaging is a well-established modality of label-free widefield microscopy for the visualization of transparent biological cells. Several different techniques exist [1], ranging from Zernike phase-contrast microscopy to methods based on interferometry or oblique illumination, all of which involve rather complex and bulky systems. Recently we have introduced a new type of image sensors that can resolve the phase gradient of the incident light directly with a particularly simple measurement setup [2]. The key innovation of these devices is a specially designed plasmonic metasurface integrated on a photodetector illumination window to create a strong asymmetric dependence of responsivity on illumination angle near normal incidence [Figs. 1(a) and 1(b)]. A pixel array of such phase-imaging meta-sensors (PIMSs) in a standard microscope configuration can then visualize transparent phase objects via "oblique detection", which essentially involves a first-order differentiation of the object phase distribution along an in-plane direction determined by the metasurface orientation. Correspondingly, the edges of the object perpendicular to the same direction are enhanced relative to all other features in the resulting image. A pixel array of PIMSs oriented along multiple directions can therefore produce multiple filtered images, each highlighting different features of the same object [Fig. 1(c)]. This behavior is analogous to the first stage of the visual recognition process in the human cortex. The same strategy is also used in convolutional neural networks (CNNs) for image classification, where the first layer generates multiple feature maps by convolving the full image of the object with different filtering kernels. These considerations suggest that a suitably designed PIMS array could be used to facilitate the classification of transparent phase objects (a functionality with important biomedical applications) by replacing the first CNN layer. Specifically, the filtered images recorded by PIMS pixels oriented along different directions could be fed directly into the subsequent layers of a digital CNN trained for a specific image recognition task [Fig. 1(c)]. Compared to a fully digital implementation, this hybrid optoelectronic approach could potentially produce substantial savings in computational time and power consumption, determined by the number of multiplyaccumulate operations (MACs) that are normally performed in the first CNN layer to generate the feature maps.
The PIMS metasurface design [Fig. 1(a)] is a variation of a plasmonic structure previously used to demonstrate lensless compound-eye vision with ultrawide field of view [3]. This structure relies on a metagrating to selectively couple light incident at a target detection angle into surface plasmons on a supporting metal film, which are then scattered by a nearby set of slits into the underlying photodetector active layer. Its ability to provide quantitative phase imaging has been reported in ref. 2. The goal of the present work is to establish the suitability of these devices for realistic image classification tasks, and to evaluate the expected performance gains in terms of computational complexity [4]. For that purpose, we consider a set of transparent cancer cells from 7 different lines, belonging to 3 different subtypes of lung cancer (SCLC, SCC, and ADC). A representative image from each cell line is shown in the top row of Figs. 2(a)-(g). For the automated classification of these cells, we investigate the use of a 36×36 PIMS array partitioned into blocks of 2×2 adjacent devices, each coated with the metasurface of Fig. 1(b) oriented along four different directions. This arrangement is illustrated in Fig. 1(c), where the four devices in each block are rotated by -45, 0, 45, or 90, respectively. An optical microscope with this sensor array can therefore produce, in a single shot, four 18×18 filtered images, each highlighting the edges of the same object along four different in-plane orientations in steps of 45. These images are evaluated with computational imaging techniques based on the measured PIMS angular characteristics of Fig. 1(b), which account for the practical capabilities and limitations of our experimental devices. The bottom row of Figs. 2(a)-(g) shows the resulting images for the 7 cells in the top row, computed for pixels oriented at 0. Accurate image classification within this dataset can be achieved using a shallow digital CNN (labeled FDN-1 in the following) consisting of a convolutional layer with 4 output channels followed by max-pooling, batch normalization, nonlinear activation (ReLu), and a fully connected layer. We also investigated an even simpler fully digital network (FDN-2) comprising two fully connected layers without any convolutional stage. For the hybrid optoelectronic networks under study, we remove the convolutional layer of FDN-1 and feed the PIMS-filtered images directly into the max-pooling layer. Within this configuration, we consider three variations of the PIMS array, where the four pixels in each block are rotated relative to one another by 45 [as in Fig. 1(c)], 90, or 30. The resulting networks are labeled OEN(45), OEN(90), and OEN(30), respectively. Each network was trained with 14,000 images from the dataset using the Adam optimizer (a variant of stochastic gradient descent), and then tested on 2,100 images.
The table in Fig. 2(h) summarizes the performance of the studied networks in terms of correct classification rate (CCR) and number of MACs. The baseline digital network FDN-1 reaches a large accuracy of 92.7 % with 67,068 MACs for identification of the 7 cell lines. A very similar classification accuracy is achieved with all three hybrid optoelectronic networks, but with a much smaller number of MACs (by a factor of 19×). A commensurate reduction in computational time and electronic power consumption can therefore be expected. At the same time, the other fully digital network FDN-2 requires a comparable number of MACs as the hybrid optoelectronic networks, but with a significantly smaller CCR of 84.9 % (8.7 % lower than the best performing OEN). The comparison with both FDN-1 and FDN-2 therefore indicates that the use of PIMS arrays for cell classification can strongly reduce computational complexity without sacrificing accuracy in a way that would not be possible with standard digital solutions.
These results establish the promise of PIMS-based optoelectronic CNNs for the classification of transparent phase objects with reduced power consumption and higher speed compared to fully digital systems. Combined with its single-shot measurement capability in a simple microscope setup, this technology is therefore quite attractive for biomedical applications such as in vivo microscopy requiring fast automated image analysis and high miniaturization.
This work was supported by NSF under Grant ECCS-2139451.