Optics, [1] which comprises a variety of subfields such as nonlinear optics, [2] quantum optics, [3] nanophotonics, [4] biophotonics, [5] and optical engineering, [6] has grown rapidly. In particular, the development and deployment of various optical spectroscopy and imaging techniques have impacted scientific research and engineering applications in a broad range of fields. [7] One generates large amounts of optical data when applying spectroscopy and imaging in medicine, [8] biology, [9] informatics, [10] physics, [11] and materials. [12] It has become increasingly challenging to decode the ever-expanding number of complex spectra and images from different optical measurements and applications to accurately reveal the relevant information. For example, Raman spectroscopy captures the vibrational information of molecular bonds in order to identify the compositions of the molecules. [13] When applying Raman spectroscopy to investigate multiple analytes in complex biological environments, one often faces difficulty in interpreting Raman spectra due to the large spectral overlap arising from the common bonds in the analytes. [14] To retrieve the accurate information in fluorescence imaging, one needs a thorough understanding of functional fluorophores, optical imaging system, and light-fluorophore interaction. [15] To read rich information about complex nano-resonators from their optical scattering spectra, one requires a good understanding of various electric and magnetic modes. [16] These requirements and challenges arise from the fact that conventional analysis of optical data is DOI: 10.1002/lpor.202000422 a physics-based and experience-driven task. Moreover, human errors can hardly be avoided when relevant information is encoded in increasingly complex optical spectra and images.
Machine learning (ML), a subdomain of artificial intelligence (AI), has provided an alternative way toward gaining insights into complex data. [17] ML comprises a set of algorithms that learn through gained experience. [18] Therefore, ML can find the relationships among complex data which are not discernable by conventional analytical methods. An ML algorithm builds an internal mathematical model for new data processing based on the training data fed into it and searches for hidden connections within the data. The model tunes its internal parameters with multiple data input cycles until it converges to a certain optimal goal. Depending on the goals, ML algorithms can be broadly classified as supervised and unsupervised. [19] Supervised ML algorithms have the training data labeled with a specific target value to accurately predict a target when given new input data. Unsupervised algorithms are fed unlabeled inputs and aim to cluster the data into distinct categories. Both types of algorithms are robust in handling high-dimensional input data and finding complex and unintuitive relations. More importantly, they can output new predictions near-instantly once trained. Overall, the rapid analysis of multiple parameters grants these algorithms the capacity for accurate prediction and classification of complex data.
Intelligent optics, an emerging field that interfaces ML and optics, is developing rapidly. Major research sub-fields of intelligent optics include inverse design of optical structures and materials with ML, [20] all-optical neural networks (NNs), [20a,21] and decoding of optical data with ML. [22] Recent years have witnessed a large number of research articles reporting progress in MLassisted optical data analysis for a wide range of applications (Figure 1). For example, to benefit the optical data analysis, ML algorithms have been developed to search for the characteristic parameters among data space and establish a statistical relationship with the target information. New ML algorithms also seek to reveal the intrinsic physical mechanism hidden behind the complex optical data, in order to build a bridge between the raw data and the final target. In other words, when a common database is available for pre-feeding, this statistical scheme can shorten the data analysis time by skipping the intermediate segments and provide an insight into the unknown physics. However, a comprehensive review article that covers these new developments of decoding optical data with ML is not available yet. We believe that it is appropriate to write a review article that presents an overview of this topic with a focus on new developments and applications. In Section 2, we introduce various types of optical data and ML algorithms applicable to data decoding. In Section 3, we discuss recent progress in ML-assisted optical data decoding with a focus on applications in disease diagnosis, noninvasive biological study, information technology, fundamental studies in optics, and materials science and engineering. The massive boost in the efficiency, accuracy, and new information of optical data analysis make these ML-based approaches extremely attractive. We also acknowledge the great strides taken in ML-assisted optical methods in agriculture. There have been quite a few excellent review articles in this area. [23] In Section 4, we conclude with opportunities and future directions of this exciting field that combines optics and ML. [24]
A conventional analysis of optical spectra and images is often done by researchers with strong experiences in the related field. Such an analysis of complex optical data can be time-consuming and error-prone. ML has been demonstrated to improve the efficiency and accuracy in classifying optical signals, [25] revealing object properties, [26] and predicting optical field distributions. [27] The emergence of high-volume optical data and powerful ML algorithms has led to the rapid development of ML-based decoding of optical data as a data-driven analytical technique. [22a] Various optical spectra can be recorded using optical transmission, reflection, absorption, scattering, Raman, and fluorescence spectroscopies. Optical imaging data can be collected from holographic, fluorescence, and tomographic microscopies. All these spectra and images contain different types of information depending on the objects being measured. A better retrieval, understanding and prediction of crucial information from these optical data is the major goal of the ML-assisted data decoding.
A few examples of optical data are described below, followed by a brief introduction to ML.
Scattering spectra of metal nanoparticles contain information about optical oscillations [28] and can be measured by dark-field spectroscopy at a high spatial resolution. Dark-field spectroscopy benefits from the high contrast between the background and scattering objects and offers a significant advantage in single-particle scattering measurements. It is commonly used to investigate nanoscale light-matter interactions. [27a,29] Optical transmission, reflection, and absorption spectra have been frequently measured from periodic nanostructures and optically uniform materials. [30] When an incident light is circularly polarized, chiral response of materials can be measured. [31] Raman spectroscopy, which is based on a nonlinear scattering process, can measure molecular vibrational modes, [13] and has been frequently applied for biological studies. [25a,26a,32] Fourier-transform infrared spectroscopy (FTIR) can also detect the molecular vibrational modes. [33,34] Its sensitivity to certain groups of chemical bonds can make up for the lacunae of Raman spectroscopy.
Conventional optical microscopy has been applied extensively to various fields. Digital holographic microscopy (DHM) involves the digital creation of a holograph from microscopic images. The data generated by DHM includes both amplitude and phase information, giving the comprehensive topological information of a sample. DHM can be a versatile 4D imaging technique, which monitors the dynamics of samples. With the minimum optical aberration, DHM can record high-quality images. [35] In fluorescence microscopy, [15] multiple fluorophores show different degrees of fluorescence, and hence can be used to distinguish different sub-micron-scale organelles in biology. Optical coherence tomography (OCT) is an imaging method that is conceptually similar to the diagnostic ultrasound method. [36] OCT has been preferred over methods such as magnetic resonance imaging and ultrasonography in biological imaging due to its higher resolution. [37] These optical imaging techniques can gain tremendously from ML methods, leading to higher-resolution images [38] and automatic detection of useful information from the images. [39] Artificial NNs, random forests (RFs), and support vector machines (SVMs) have been popularly utilized to process optical data. These are typically supervised learning algorithms. NNs work by a large series of interconnected processing nodes, which connect the inputs to the outputs through a series of intermediate layers (Figure 2a). [40] Deep NNs (DNN), and the associated technique of deep learning, encompass NNs with a large number of layers. [20b,41] The large number of internal parameters that can be optimized during the training offers extremely high plasticity to learn complex and non-linear relationships within the data. One distinct advantage of NNs is that a wide variety of types of layers and architectures can be employed to model the specific data more efficiently and accurately. For example, fully connected NNs have every neuron in one layer connected to every neuron in the next. This allows for an extraordinary capability to learn complex global high-and low-level relations within data at the tradeoff of a high computational cost. [42] Convolutional NNs (CNNs) pass a series of filters over the data that can learn from the relations Diagram of a random forest comprised of a large series of decisions trees. Each tree learns optimal branching points from the data, guiding the input to the output in a flowchart manner. Each tree will see a randomized portion of the data so that the branching points may differ between trees. The predictions of all trees (green) are aggregated to make a final prediction. c) Diagram of a support vector machine (SVM), which uses the data to learn an optimal boundary to separate two classes of data (labeled here as blue and red). An SVM attempts to maximize the margin between the boundary, and the lines hitting the nearest points of each class, as denoted by the dotted lines.
of each input to its neighboring inputs, enabling itself to learn from imaging and spectral data more efficiently. [43] RFs are another type of algorithms built as a collection of decision trees. [44] Each decision tree operates like a flowchart, making a series of binary splits of the input data into different branches till arriving at a final prediction or classification. RF is then built from a large collection of trees, each with access to different random subsets of the training data (Figure 2b). Despite the simplicity of the model, RF has shown to be effective in modeling complex data. RF is also highly resistant to overfitting, a problem in ML algorithms that memorize the training data but become unable to generalize to the new unseen data. [45] In addition, such decision-tree-based models can also learn and rank the relative importance of the different input variables in making the final prediction. This lends a degree of physical interpretability as compared to NNs, which typically operate more like black boxes. Both NNs and RFs are typically used for classification tasks, where the output being predicted is a binary or categorical variable, and for regression tasks, where the output is a continuous variable. SVM is an algorithm typically reserved for binary classification tasks, where the data falls into one of two categories (Figure 2c). SVMs operate by using the training data to calculate an optimal hyperplane, a boundary that most efficiently splits the data into the two classes. [46] While SVMs typically have a more limited range of data they are applicable to, they can often outperform NNs and RFs in these specific tasks and include computation-ally cheaper training processes. [47] SVMs can also be extended to multiple-category classification problems, typically by reframing the multiple-classification task as a series of binary classification tasks and training multiple SVMs. [48] As the number of categories rises, other ML algorithms for classification such as NNs and RFs tend to be more suitable. Whereas many other ML algorithms such as hierarchical clustering [49] and t-distributed stochastic neighbor embedding exist, [50] NN, RF, and SVM have seen the widest uses in decoding optical data.
The algorithms listed above are typically supervised algorithms, meaning they work with labeled data. Unsupervised algorithms will not have any specific target value that the model is attempting to predict. Instead, they seek to extract useful information out of the dataset by grouping similar points together for purposes such as dimensionality reduction or clustering data. Some of such algorithms are rate distortion theory (RDT), agglomerative hierarchical clustering (AHC), and Gaussian mixture models (GMMs). AHC works in a recursive bottom-up approach. It combines data into clusters, and then combines these clusters into larger clusters, building up a tree diagram as it proceeds higher. RDT is based on data compression, finding the most compact representations of data. Therefore, it enables more efficient clustering when there is a high overlap between the possible clusters. [51] GMMs use a series of Gaussian distributions with varying weighting parameters to cluster data. In addition, some algorithms that are typically used for supervised learning have unsupervised variations, including the self-organizing map, which is a type of NNs used for dimensionality reduction.
As with supervised algorithms, unsupervised algorithms derive their power from finding complex relations within the data that may not be easily accessible through human intuition or conventional data analysis. However, since the algorithm has no way of knowing what the clustered or reduced data correlates with because the data is not labeled, unsupervised learning will typically need to be combined with other data analysis methods. For example, one could apply an unsupervised algorithm to split a dataset into a number of clusters, and then manually investigate if those clusters correlate with any useful insights. The choice of supervised versus unsupervised algorithms, as with the choice of algorithms among those categories, depends on the availability of the data and the goals of the researchers. Unlabeled data is typically far easier and cheaper to obtain than labeled data, but labeled data allows for more specifically targeted applications, and more accurate and useful clustering.
With many optical data and ML algorithms available, it is critical to choose the most suitable algorithms for targeted tasks. For example, a two-cluster classification task can be easily achieved by an SVM with computationally cheaper training processes, whereas the solution to complex relations hidden behind multiple parameters usually requires powerful and consequently computationally expensive NNs. However, the propensity for overfitting may render NNs unsuitable in some highly specific tasks. Overfitting can be addressed by techniques like regularization, a process of imposing additional harsh penalties on the objective function during training, but this in turn can cause the model to underfit. This tradeoff is at the core of finding an optimal ML model for a given task. Once a task is determined, physical interpretability is another important issue to be considered. Ranking the importance of input variables is thus a significant advantage of RF. A few recent works have taken the advantage of emerging explainable ML algorithms [52] and dug into the concept of AI-assisted knowledge discovery in optics and photonics research. [53] One common theme in ML-assisted optical data decoding is that many ML algorithms still rely on inputs from experienced researchers despite their goal of achieving fully autonomous operation. Researchers need to compare different ML algorithms for a specific physical problem or manually set constraints and modify the algorithms based on their experience. Moreover, as a data-driven process, ML-assisted optical data decoding always relies on good training input. For instance, the quality of the training data directly limits the best performance of a supervised model. [54] More informative optical data can also lead to a higher accuracy in an unsupervised model. [55] Therefore, the generation and selection of high-quality data is just as important, if not more so, than the sophisticated design of advanced ML algorithms. It is a common problem in optics community that simulated data can be generated rather easily, but experimental data is expensive. To overcome the major challenge of training ML algorithm on only few data, transfer learning can be a good solution, wherein a model pre-trained for one task (e.g., simulations) can be retooled for another similar task (e.g., experiments). Some initial tries have been demonstrated in optics and photonics. [56] Better compatibility and more efficient combination are expected in this exciting field that merges ML and optics.
There has been an explosion in the field of AI-assisted disease diagnosis. [57] ML algorithms have been applied to interpret optical data in the context of spectroscopic analysis and image recognition. In this section, we will highlight some representative works. We will examine the versatility of ML algorithms in their applicability to different categories of diseases such as neoplastic, [26b,55,58] infectious, [25a,39,59] inflammatory, [60] and miscellaneous. [61] In the area of neoplastic diseases, ML algorithms are implemented to distinguish malignant tumor tissue from benign cells based on microscopic images. The conventional approach to cancer diagnosis involves human interpretation of the images, which is inefficient and susceptible to human error. To overcome such challenges, various ML models have been used in combination with optical techniques for intelligent diagnosis. The algorithms include supervised models such as SVMs, [62] DNNs, [63] and RFs, as well as unsupervised models like RDT and AHC, as mentioned in Section 2.
Taylor et al. carried out diagnosis of follicular thyroid cancer by ML-assisted analysis of Raman microscopic images. [55] FTC-133 cells (malignant) and Nthy-ori-3-1 cells (benign) were chosen as specimens, and two sets of data were introduced at cellular and subcellular resolutions. Subcellular data provide more spectral information. As an example, Figure 3a shows the subcellularresolution Raman images with differential localized signals at the typical responsive energies of cytochromes (750 cm -1 ), proteins (1670 cm -1 ), and lipids (2850 cm -1 ). These are overlaid in the frame to the right of Figure 3a, indicating the distribution of these species within the cells. During the analysis, an AHC was applied with the cellular-resolution data, while RDT was chosen at the subcellular level. This is because the outliers and detection noises are much more significant in the subcellular data and AHC is susceptible to these factors. Moreover, RDT can correlate the spectral class importance to the type of cells. RDT was able to perform better than AHC, with an accuracy of around 90% in comparison to 77% with the latter. In addition, as depicted in Figure 3b, the introduction of more spectral classes in the subcellular classification can lead to increasingly better predictions, from 33 out of 49 correctly identified with 2 classes to 44 out of 49 for 8 classes. The analysis concluded that the FTC-133 cells tend to have higher amounts of lipid molecules, which is in accordance with previous research. [64] This study reveals the advantage of ML methods in decoding the more informative subcellular Raman data, which provides a better understanding of the chemical nature of cancer.
In another study by Kingston et al., 3D microscopy data was efficiently segmented and used to predict the delivery of gold nanoparticles to micrometastases in mice with the help of ML. [26b] Micrometastases play a crucial role in the spread of cancer, and their small size, heterogeneity, and distribution throughout the body make tracking them a monumental task. In this study, the authors mapped the distribution of nanoparticles delivered to these micrometastases. The gold nanoparticles were detected by optical scattering microscopy, whereas the vessels and micrometastases were observed and labelled by fluorescent , with an overlay of these to the extreme right. b) RDT segmentation, with increasing spectral classes from left to right. The corresponding accuracy of prediction is listed at the bottom. Reproduced with permission. [55] Copyright 2019, American Chemical Society. c) A 3D image of the sample and the extracted details, including micrometastases, vessels, nuclei, nanoparticle (NP) intensity, and the distance of NP to the vessel. d) Prediction of nanoparticle delivery based on listed physiological parameters. 80% of the dataset was used for training and the remaining was used for prediction. Multiple SVM models were developed to predict different delivery parameters such as mean nanoparticle intensity and nanoparticle density Reproduced with permission. [26b] Copyright 2019, United States National Academy of Sciences.
microscopy, combined into a 3D image. An SVM-based tool was used for the segmentation of the 3D images, identifying micrometastases, vessels, nuclei, and nanoparticles as shown in Figure 3c. The segmentation provides essential information such as the distance of the nanoparticles from the vessels, which is useful in determining the efficacy of nanoparticle-based cancer therapy. Another SVM model was developed by the authors for the prediction of nanoparticle delivery to micrometastases. As illustrated in Figure 3d, this model took in parameters such as distance from the vessels and cellular density to predict the nanoparticle delivery output. Three SVM models were tested on the data, namely linear, quadratic, and cubic. Their distinction is based on the nature of the optimal hyperplane used for classification. It was observed that the quadratic SVM model had the best performance. A predictable correlation between the pathophysiology of the micrometastases and delivery was established.
Multiple ML-based diagnostic studies have also been carried out on infectious diseases. For example, applying a CNN on Raman spectra, Ho et al. demonstrated the identification of 30 common pathogenic bacteria and the automatic assignment of appropriate antibiotic treatment. [25a] Figure 4a illustrates the average of 2000 spectra from 30 isolates. They are color-grouped according to the manually selected antibiotic treatment. As depicted in Figure 4b, a 1D residual network with 25 total convolutional layers was used to classify low-signal Raman spectra as one of 30 isolates (strains), and to assign the correct antibiotic treatment. For example, Vancomycin was assigned to both MRSA and MSSA (methicillin-resistant and methicillin-susceptible Staphylococcus aureus). With an accuracy of 82%, the model was able to beat an SVM in the identification of individual isolates by 8% on the same dataset. This is consistent with our assertion in Section 2 that NNs are superior in case of multiple outputs, whereas SVMs are computationally much cheaper. Additionally, the accuracy in predicting antibiotic treatment was close to 97%, which is more relevant in a clinical setting. Practically, the authors have also demonstrated that such a good performance could be achieved in only ≈10 clinical spectra on average, as illustrated in Figure 4c. The authors also showed that a fine-tuned model, which involves a small amount of data from clinical blood samples in the training process, was able to perform even better. Moreover, as a proof-of-concept, a binary classifier of antibiotic susceptibility was modeled on MRSA and MSSA. Figure 4d shows the high specificity (true negative rate) and sensitivity (true positive rate) for this demonstration. To sum up, this study utilized the ML approach to provide highly accurate diagnostic results from noisy Raman spectra. It also shows potential to be easily extended to other conventional clinical samples such as sputum and urine.
In the area of inflammatory diseases, Helal et al. have demonstrated an ML-assisted method to aid and inform human diagnoses. [60a] With the help of RDT and an AHC model, they analyzed Raman spectroscopic data for the identification of chemical factors that contribute to nonalcoholic fatty liver (NAFL) and nonalcoholic steatohepatitis (NASH) in rats. Of the two, the latter is deadlier, being linked to the development of liver cirrhosis and hepatocellular carcinoma (liver cancer). First, liver cells of rats on three different diets (standard (SD), high fat (HFD), high fat and high cholesterol diets (HFHC)) were characterized by Raman mapping. After the preprocessing by super-pixel segmentation, the Raman maps were clustered by RDT (Figure 4e) and the corresponding spectral feature importance was determined using an RF classifier. Then, these clusters were applied back to the cellular maps, illustrating the biochemical environment of the cell. Further, these cluster maps were grouped according to the similarity of their chemical distribution by an AHC model and a threshold was determined to relate molecular information to the diets and liver diseases. The AHC dendrogram along with the thresholds is illustrated in Figure 4f. Threshold 4 was determined to be optimal, dividing the spectra into 5 distinct groups, with a diet prediction accuracy of ≈91%. The AHC also reveals several qualitative trends. There is an increase in the cellular presence of lipid molecules in the rats with consumption of SD, HFD, and HFHC diets. There is also a sequential increase in liver disease associated with a progressive rise in lipid molecules presence, which shows the marked interdependence between diets and liver diseases. In conclusion, this study provided an effective method to analyze biomolecular information from Raman spectra and to act as a diagnostic aid for histopathologists.
An example of a major disease that does not fall clearly into either of the previous categories is Diabetes mellitus. Diabetes is primarily diagnosed by analyzing fasting blood glucose levels. Kühner et al. used surface-enhanced infrared absorption spectroscopy and ML to noninvasively evaluate the quantitative blood glucose levels. [61c] By implementing principal component analysis (PCA), which is a data analysis technique for dimensionality reduction, on the vibrational information, the authors achieved selective detection of glucose content from a mixture of sucrose and glucose, with a sensitivity down to 10 g L -1 . This study shows the potential of ML-assisted, noninvasive, and highly sensitive diagnosis.
It is worth noting that many of the optical tools in the studies we mentioned above are not portable. However, there is a major push recently for point-of-care testing with the use of portable diagnostic devices. We note that some of these portable devices have also combined ML methods for the better interpretation of optical data. Therefore, we would like to cite some of these studies for readers' reference, in the categories of neoplastic, [38,59a,65] infectious, [39,59b,59e,66] and inflammatory diseases. [67]
Besides diagnosis, ML has also been applied to investigate fundamental biological phenomena. Examples include ML-assisted detection of biological species and bio-imaging quality improvement. We will discuss these applications based on the type of optical methods in use.
Many studies have used FTIR to detect biological species. [59d,68] Ellis et al. combined FTIR and genetic algorithms (GAs) to quantify microbial activity in meat. [69] GA works analogously to natural selection in the real world. There is a population of possible solutions. In each generation, new solutions are generated from the prior ones with some "mutations," and the worst performing solutions are removed. This process repeats, slowly optimizing until reaching some optimal state. In this study, the mutations were stopped after they had crossed a certain threshold value of bacterial counts, set to 10 7 . Genetic program (GP), one application of GA, was also utilized to derive a mathematical equation or a rule for the problem. FTIR spectra and the vibrations corresponding to the various amide and amine bonds are represented in Figure 5a. All the spectra are overlapped, revealing the region of significant difference in the 1000-1500 cm -1 wavenumber range. Partial least squares (PLS), another statistical method, was also demonstrated as shown in Figure 5b. While good at prediction, it is unable to identify the underlying physical phenomena that contribute to the spectra. On the other hand, GP predicted the spoilage of meat and revealed the important regions of the FT-IR spectra. As illustrated in Figure 5c, the frequency of the wavenumbers in the GA analysis is highest in the range of 1088-1096 cm -1 . This region was associated with the C─N bonds in amines, which were expected to be present in the larger numbers in the spoiled meat.
Raman spectroscopy has also been frequently used in biological study. [70] For example, Rösch et al. used Raman microspectroscopy to detect single bacteria. [70c] By combining SVM with Raman imaging, they were able to identify single bacteria at a short integration time of ≈60 s. Different species of bacteria could be successfully distinguished, for example, vegetative and spores, or, colored and uncolored. Figure 5d shows a representative image of spore cells and vegetative cells from the typical contaminants found in cleanrooms. Figure 5e illustrates the bulk spectra for vegetative and spore cells. The spectral difference mainly comes from the layers of the spores. While a simple distinction is easily drawn between these two spectra, identification of cells is not as straightforward. Spore cells are especially difficult to be detected by bulk Raman spectra due to the presence of exterior layers, which leads to heterogeneous signals. Hence, single-cell spectra are more useful in detection of these species. The authors collected spectra from different depths of the cells as depicted in Figure 5f, which provided more informative training data and thus led to better prediction accuracy. The corresponding positional Raman maps are plotted. The ML-assisted identification method provided an accuracy rate of up to 93%. Moreover, different strains of bacteria arising from different culture conditions could also be identified with an accuracy rate of ≈89%. The performance could be improved if more factors were considered as inputs of the ML model in the training process, for example, the photobleaching effect for colored species.
A major success of ML in decoding images is on the efficient extraction of the hidden information. [26a,71] Here we would like to introduce an ML interface (named Aro) developed by Wu and Rifkin. Based on an RF model, Aro is capable of identifying single molecules in cells from images taken by fluorescence microscopy. [72] The Aro program allows the users to manually curate datasets for the model, serving as a generalizable and versatile tool for the detection of different biological and chemical species. As an example, they presented Aro-assisted noninvasive identification of messenger RNA in cells based on their fluorescent images. The authors observed that training sets with a few hundred positive and negative examples were sufficient for stable classification. As illustrated in the left panel of Figure 6a, the blue outlines on the image represent signal spots, whereas the yellow boxes are the noisy spots. All of them were manually identified for the training set. The right panel of Figure 6a is an example of Aro accurately identifying the signal spots after the training. For the images with low signal-to-noise ratio (SNR), Aro could still perform well by accurately estimating a confidence interval of the global maxima (true spots) from the local maxima (background speckle). As shown in Figure 6b, large areas under the receiver operating characteristic (ROC) curve are achieved despite low SNR. Figure 6c compares the performance of Aro with that of two other methods, that is, FISH-Quant and threshold picking. Aro performed best among the three, with a correlation coefficient value close to 0.99. Being demonstrably robust and versatile, Aro provides an ML platform for the facile analysis of fluorescent images.
The use of ML in decoding images has facilitated the development of portable devices and point-of-care diagnosis. [59a,59e,65a,71a] Feizi et al. developed a lens-free on-chip microscopy to recognize the viability of yeast cells and analyze their concentration. [71a] They used an LED source (coupled with an optical fiber and band-pass filter) and a CMOS image sensor chip to capture the holographic shadows of the samples, as illustrated in the left panel of Figure 6d. The holograms were used to reconstruct backpropagated images with SVM. The SVM algorithm accounted for ten spatial features, including the maximum and minimum pixel values on both phase images and amplitude images. A workflow is shown in the right panel of Figure 6d. Briefly, the sensor captured the hologram, which was converted to an image. Features were extracted from the image as inputs. After an autofocus operation, the stain status was determined. Trained on manually identified yeast cell images, the algorithm could give the concentration and the viability of the yeast cells. Figure 6e shows good compatibility of this method with samples of different dilutions. The setup was further integrated with a graphical user interface (GUI) for easy operation. This platform, named AYAP (automatic yeast analysis platform), overcomes the disadvantages of bulky components that are involved in conventional flow-cytometers, providing a good example of combining ML and optical imaging for practical applications.
One example of combining ML with optical spectroscopy and imaging for biological applications was demonstrated by Pavillon et al. [26a] In their work, Raman spectroscopy, autofluorescence (AF) imaging, and quantitative phase microscopy (QPM) were combined to study macrophage activation induced The ROC curve for a binary classifier of MRSA/MSSA with the x-axis representing sensitivity (true positive rate) and the y-axis specificity (true negative rate). The area under the curve (≈0. 95) indicates the accuracy of the model. Reproduced with permission. [25a] Copyright 2019, Springer Nature. e) Averaged Raman spectra of the seven clusters based on 48 cell Raman images. The numbering is in increasing intensity of lipid signals. f) The AHC dendrogram with thresholds identified and labelled. The dendrogram is used to group the cluster maps by the similarity in biochemical distribution. Reproduced with permission. [60a] Copyright 2019, Wiley. ), amide II (1550 cm -1 ), and amine (1240 and 1088 cm -1 ). b) The analysis by PLS method, showing good agreement between predicted and measured total viable count (TVC). Circle, calibration set. Triangle, cross-validation set. Square, test set. c) Summed frequency of the input in ten independent GPs at different wavenumbers. Two representative spectra are included. Two representative spectra are included. Reproduced with permission. [69] Copyright 2002, American Society for Microbiology. d) Image showing the vegetative and spore cells. e) The bulk spectra of vegetative and spore cells. f) The different depths of spores at which Raman spectroscopy was carried out, along with the maps obtained. Reproduced with permission. [70c] Copyright 2005, American Society for Microbiology.
by lipopolysaccharide (LPS) at the cellular level. The optical setup and workflow are illustrated in Figure 6f. A PCA algorithm was used to decode Raman spectra and an automated cell classifier was applied to segment the QPM and AF images. Individual statistical models were generated to predict the activation of the macrophages through morphological parameters and Raman spectra, respectively. These models yielded activation probabilities for the individual cells, indicating their state (Figure 6g). The results agreed with the measurements of the tumor necrosis factor-𝛼 (TNF-𝛼), a molecule primarily secreted by macrophages. The individual model accuracy was around 84-87%. The AF images of the intracellular levels of inducible nitric oxide (NO) synthase (iNOS) was also used to support the prediction by ML (the left panel of Figure 6h), as it is known to be involved in the immune response by promoting the production of NO. More importantly, the combination of the morphological and spectral models led to even better distinction, as depicted by the clearer separation in the right panel of Figure 6h. In addition, the researchers also implemented this combined model in a system with both LPS and progesterone (an activation inhibitor). Despite the complexity arising, the model was still able to predict the macrophage activation accurately.
Application of ML in optical information technology has attracted great attention over the past decades. [73] Optical technologies have potential for the higher information density and the faster processing speed when compared with the electronic techniques. [74] Rapid adoption of ML in optical information technology has been driven by an unprecedented growth in the Reproduced with permission. [72] Copyright 2015, BioMed Central. d) An on-chip lens-free microscope used to determine yeast viability. Left: A brief schematic of the set-up, including the bandpass filter, on-chip CMOS sensor and the LED illumination source. Right: The process of analyzing the obtained holograms. e) Comparison of the AYAP model with manual counting for samples with variable dilutions. The blue dots represent the counts of the device and the red dots represent manual counts. Reproduced with permission. [71a] Copyright 2016, Royal Society of Chemistry. f) An optical setup of an ML-assisted multimodal optical imaging and spectroscopy along with the workflow. The setup allows for the simultaneous acquisition of QPM images through an interferometric microscope and a set of flipping mirrors also enable the recording of AF images through a wide-field epifluorescence system. Raman spectra are collected with a scanning microscope. g) Activation probability plotted for the morphological (left) and spectral (right) models. h) Left: AF microscopy images showing iNOS in green and nuclei in red for the control and LPS cases. The scale bar is 10 𝜇m. Right: The relative scores of morphological and spectral parameters were plotted on x-and y-axes, respectively. Reproduced with permission. [26a] Copyright 2018, United States National Academy of Sciences.
amount and complexity of data. [75] ML is instrumental in extracting meaningful information from optical data, [27b,76] assisting the decision making [77] and overcoming the physical limits of practical performance. [78] We will examine ML-assisted decoding of optical signals in optical communications and decoding of optical data beyond the diffraction limit of light for higher-density optical storage.
Orbital angular momentum (OAM) of light can be controlled by a spatial light modulator, [79] a spiral phase plate, [80] or unique lenses. [81] An OAM beam can carry a heterogeneous optical signal, leading to enhanced channel capacity. The effective multiplexing and demultiplexing of OAM beams are promising for next-generation optical communications. [82] Doster et al. [27b] and Li et al. [76] have employed a CNN algorithm to decode the encoded information from a single OAM intensity pattern with the goal of improving the demodulation efficiency and accuracy without complex optical components, as shown in Figure 7a. CNN was selected due to its advantage in image recognition. A rectified linear unit (black squares in Figure 7b) was applied to further improve the CNN's learning process by introducing additional layers in the algorithm. Once trained, the CNN would produce a probability for each input intensity image as belonging to one of the output demodulated modes. A general communication process based on the training results is displayed in Figure 7c. The OAM intensity images could still be distinguished accurately even with environmental distortions. Such a CNNbased demodulating method offers a cost-effective and powerful approach to high-information-density optical communications.
Another type of applications of ML in optical communications aims at self-estimating network traffic and optimizing the quality of propagating modes. For example, Yao et al. studied the spatial crosstalk (XT) and mode coupling effects in multi-core fibers using an ML approach. They proposed a crosstalk-aware resource allocation scheme to instruct the optical network designs. [77a] A simplified fiber model with two modes (i.e., LP01 and LP11 where LP means linear polarization modes) in each core was input into an ML-assisted crosstalk evaluation model with the strategy shown in Figure 7d. NN was chosen to solve the nonlinear relationships within the data. The prediction targets were the influences of the different wavelengths and the crosstalk between two modes in the communication performance. Figure 7e illustrates an example of ML-assisted estimations on the LP01 mode performance. The authors also demonstrated a performance evaluation on the results of different NN models by conducting the related simulations, which can be used in other ML-assisted communication systems as well. For readers' reference, ML-assisted optical propagating mode analysis has also been reported in ref. [83] while ML-based self-estimation task in optical networks is an emerging topic with some initial exciting demonstrations. [77c,84] It is worth mentioning that, besides being an essential optical information transmission media, the optical fiber itself has also attracted great research interests in ML-assisted optical data decoding. Multimode optical fibers are typically complex scattering media and generate speckle patterns at the outputs. For the purpose of imaging [85] or object classification [86] through such a complex media, DNNs have been trained to provide accurate recognition with remarkable robustness against environmental instabilities. Based on this platform, people have also demonstrated ML-assisted laser speckle wavemeters, [87] hybrid scattering images, [85c,88] and specklegram sensors. [89] In addition, we will discuss the extreme events in optical fibers in Section 3.4, where ML helps researchers to study the nonlinear fiber optics. [56a] Optical data storage is another area of optical information technology where ML-assisted decoding of optical data is playing an important role. Optical methods have the advantages of longevity and low energy consumption in data storage. [90] However, the information storage density is inevitably limited by the diffraction limit of light. Engineering solutions to improving optical storage density include holographic memory [91] and near-field optical recording. [92] Wiecha et al. reported an ML approach to push the optical storage beyond the diffraction limit of light. [78] Specifically designed silicon nanostructures display complex and unique spectral features at visible frequencies, which allow information to be encoded in the nanostructures and be retrieved by far-field optical measurements. Figure 8a illustrates the structure encoding 9 bits, with each block/void sized 105 × 105 nm 2 representing "1/0" and an L-shaped sidewall to distinguish symmetric structures. The total size of this structure was less than one wavelength. A 1D CNN followed by a fully connected network was used to decode the scattering spectra from the silicon nanostructures (Figure 8b). The CNN algorithm learned the relationship between the polarized spectra as an input and the bit sequence as an output. NNs were chosen because of their strength in pattern and spectrum classification tasks. Once trained, the network functioned as a reader that operated as per the scheme in Figure 8c, where the output neurons indicated the encoded bit sequence based on their activations. The training was conducted on two types of data sets, as shown in Figure 8d. The full scattering spectra have proven effective for robust readout. Moreover, acceptable performance could also be achieved even when the training data contained polarized scattering intensity information at only 3 probed wavelengths, which showed great potential for fast readout with a simple optical instrument. The authors have made an initial demonstration on an RGB-color-based data readout using a modulated NN with only the fully connected part (Figure 8e). Such an image-based approach could dramatically increase the efficiency in both ML training and information readout process.
ML-assisted decoding of simulated and experimental data can enhance fundamental study in optics by categorizing optical data, revealing implicit information, and reconstructing desired field distributions. [27a,56a,93] For example, Barth and Becker utilized ML to study electromagnetic modes in a photonic crystal. [93a] Briefly, electric field distributions near a hexagonal nanohole array in a silicon slab were generated by a commercial finite element electromagnetic solver to serve as input data for the ML algorithm. [94] The fields on three selected symmetry planes were used in the training. The photonic modes to be categorized were dependent on four variables: the polarization of incidence (i.e., transverse electric (TE) or transverse magnetic (TM)), the inplane azimuthal angle that defines the high-symmetry direction (i.e., Γ-K or Γ-M for a hexagonal lattice), the light wavelength, [27b] Copyright 2017, The Optical Society. c) Numerical processing model of the OAM shift keying communication system (bottom), using a CNN similar to that in (a, b) as an adaptive demodulator (top). Reproduced with permission. [76] Copyright 2017, Institute of Electrical and Electronics Engineers. d) A crosstalk evaluation model based on NN and beam propagation method (top), where the slots of central frequency f c and starting frequency f 0 were used as guard bands (bottom). e) Crosstalk estimation with different wavelengths when the LP01 mode was used as the main transmission mode. A spatial crosstalk (XT) value was evaluated for both "LP01@core2 to LP01@core1" (top) and "LP11@core2 to LP01@core1" (bottom) cases. Reproduced with permission. [77a] Copyright 2018, Institute of Electrical and Electronics Engineers. Figure 8. ML decoding of scattering spectra from artificial nanostructures for optical data storage. a) A silicon nanostructure encoding nine bits with the nine silicon blocks (block: "1"; no block: "0"). An L-shaped sidewall was used to distinguish symmetric arrangements via polarized optical spectroscopy. b) A DNN for decoding scattering data of (a). The network comprises a 1D CNN, followed by a fully connected network, for pattern-recognition tasks. c) The training and readout schemes and d) the corresponding readout accuracy when using the full spectra (left) or scattering intensities at a few probed wavelengths (right). In both cases, X-and Y-polarized data were used simultaneously. e) A fully connected NN (right) for the data readout simply via RGB color values (left). X-and Y-polarized data were averaged in a 3 × 3 array. Reproduced with permission. [78] Copyright 2019, Springer Nature. and the angle of incidence. A representative example is shown in Figure 9a for the Γ-K, TE polarization configuration, which has the first two variables fixed but allows the wavelength and angle of incidence to vary. Flexible GMM clustering technique, an unsupervised algorithm, was used. This algorithm was chosen due to its competence in handling different cluster sizes and unknown cluster shapes. The results were further examined by inspecting the modal field distribution prototypes (Figure 9b) for the leaky modes, where the illumination conditions were determined using the silhouette coefficients for classification. [95] Photonic modes were successfully identified from their 3D field distribution data.
Närhi et al. demonstrated an NN (Figure 9c) that enabled transfer learning from simulated to experimental data to study the extreme events in optical fiber modulation instability (MI). [56a] It is straightforward to simulate both spectral and tem-poral properties of the instabilities that drive extreme events in nonlinear optics. However, experimental observations are often limited to spectral data. To bridge the data gap in ML training, the authors used stochastic numerical simulations of a generalized nonlinear Schrödinger equation (NLSE) model [96] to generate a large ensemble of training data (both temporal and spectral) associated with a chaotic MI field. The ML model was trained with 20 000 simulated data, and a comparison between ML and simulation results is shown in Figure 9d. A good agreement was achieved in the probability density function (PDF) along the maximum intensity of the temporal intensity profiles. The simulation-trained ML model was applied to experimental data with the results shown in Figure 9e. A good agreement between these two indicates the ability of this ML approach to transfer new "knowledge" into practical problems. It is applicable to many propagation problems in both linear and nonlinear optics. The patterns were color-coded using a heat map. Reproduced with permission. [93a] Copyright 2018, Springer Nature. c) Schematic of the NN used to correlate spectral and temporal characteristics of modulation instability (MI). Spectral intensity vectors X n are the input, and the maximum intensity in the time-domain intensity profile (i.e., the circled peak) is the output. Extreme events in optical fiber MI were analyzed using this NN. Probability density function (PDF) of the maximum intensity of the temporal intensity profiles predicted by ML based on d) simulation data and e) experimental data. A simulated standard curve is plotted for comparison. Reproduced with permission. [56a] Copyright 2018, Springer Nature.
In the field of materials science and engineering, ML-assisted decoding of optical data enables quicker and more accurate characterization or classification of material properties. Trained on spectroscopic and imaging data, ML algorithms have been applied to investigate various aspects of material properties. We will discuss a few examples that cover different dimensions of materials, including single particles, low-dimensional van der Waals materials, and bulk materials.
Hussain et al. demonstrated optical analysis of particle sizes with the assistance of ML. [97] As illustrated in Figure 1 a, the experimental setup was based on a lens-free CMOS image sensor combined with a small-factor angular spatial filter. The filtered light from glass microbeads of 13-125 𝜇m was collected on the CMOS sensor and the resulted image was used as the ML training data. Several challenges existed for conventional data analysis in the optical measurement of particle sizes. The collected light signal intensity was also sensitive to the concentration of the particles, which interfered with the size-dependent signal intensity. As shown in Figure 10b, the light intensity dropped dramatically for the particles at the higher concentrations. The measurement of the smaller particles even suffered from a low SNR. Two different RF models were proposed to overcome these challenges. In comparison to Model A, Model B excluded concentration as an input parameter, giving rise to a small increase in the error. Nevertheless, reasonable results were still reached as shown in Figure 10c. This ML-assisted optical method showed a better size resolution than dynamic light scattering [98] and was not affected by polydisperse samples, enabling reliable, fast, and cost-effective particle analysis.
Lin et al. developed an ML optical identification (MOI) method to interpret low-dimensional van der Waals materials from their optical images (Figure 10d). [25b] The MOI method was based on an SVM model, which could work with small datasets and low number of classification groups. It was trained to identify the available RGB makeup (Figure 10e) of the collected images and to recognize the structures and compositions of the materials. In the training process, material information from atomic force microscopy (AFM) and Raman spectroscopy were used as input parameters to inform the algorithm and to relate the RGB data to the thickness and nature of the sample. Implemented on MoS 2 and graphene samples, the model achieved pixel-to-pixel identification accuracy of 97% and 94%, respectively. The same model was used for the identification of a vertical heterojunction of MoS 2 and graphene with an accuracy of 90%.
Van der Waals materials such as 2D transition metal dichalcogenides also show interesting exciton valley polarization at low temperatures, which can be investigated spectrally under circularly polarized light excitation. As illustrated in Figure 10f, valley polarization at a temperature of 300 K (left panel) was observed to be the same for both left circularly polarized and right circularly polarized light excitations, whereas there was a distinct difference in the two excitations at a temperature of 15 K (right panel). Tanaka et al. applied an RF model to predict the exciton valley landscapes of monolayer WSe 2 without the low-temperature measurements. [53c] The model was trained on four factors of the room-temperature valley spectra, that is, 𝛼, 𝛽, 𝛾, and 𝛿, as indicated in the left panel of Figure 10f. As shown in Figure 10g, no single variable could provide complete information to predict the valley polarization. Interestingly, the ML algorithm could circumvent difficulties in identifying correlations among these factors (Figure 10h). The ranking of different parameters in the RF algorithm suggested that the PL intensity (𝛼) and the ratio of the trion-exciton intensities (𝛿) were the most important factors in the prediction. This, in fact, correlates well to the physical parameters in the equation describing exciton valley polarization. In the equation, the polarization value is inversely related to the effective lifetime of the bright excitons, which scales with 𝛼.
Another key factor is the valley relaxation rate of bright excitons. This is dependent on the local carrier density, which can be expressed in terms of 𝛿. Based on this, they were able to validate the predictions of the polarization maps of samples with the corresponding room-temperature polarization spectra as inputs.
As an example of ML-assisted study of bulk materials, Bulgarevich et al. demonstrated pattern recognition of optical micrographs of steels via RF. [99] An RF model was trained on manually segmented optical images of steels in order to efficiently identify and automatically segment different phase regions in the microstructure (Figure 10i). Each pixel of the data was assigned a certain class by probability, giving fine segmentation in complex microstructures. As illustrated in Figure 10j, the ML-based segmentation gave comparable results to the manual segmentation, with minimal difference in the smaller areas. Moreover, the model made finer distinctions between the Ferritic (F) and Bainitic (B) areas, which was not possible by manual segmentation. This generalized method is expected to work on electron micrographs as well. This approach is on par with human experts in terms of quality while being efficient enough to aid industrial productivity.
ML-assisted decoding of optical data has made rapid progress, continuously enabling exciting applications. [100] This review provides a glimpse into this interdisciplinary field by discussing a small portion of research examples. Despite tremendous accomplishments made so far, there are unaddressed problems and opportunities in both scientific research and applications of ML decoding of optical data. We envisage some exciting research directions, which have shown promising initial results, but not been discussed in detail yet.
Raman spectroscopy combined with ML techniques is becoming a powerful tool for cellular biology, providing insights into complex biological phenomena like cell division. With its improved capability of demodulating high-volume data without sacrificing the quality of analysis, ML will be instrumental in real-time tracking of cellular and sub-cellular dynamic processes through Raman spectroscopy and imaging to reveal the working mechanisms in greater details. [101] Future in vivo studies are expected to require highly efficient data processing and analysis, which can be provided by ML algorithms. [102] To further enhance the efficiency and robustness, automation of data collection and filtering for large-sample-set biological studies will be highly desired. [103] In the area of materials science and engineering, one will witness further development of ML-assisted decoding of optical images and spectra, which is expected to improve the choices [116] as circles. Model A made use of the filter hole intensities, circle areas, and the particle concentration as input parameters. Model B excluded the concentration from its parameter list for an easier data collection process. Reproduced with permission. [97] Copyright 2020, CIOMP, Chinese Optical Society and Springer Nature. d) Left: An optical image and (inset) an AFM image of a MoS 2 flake. Right: The same flake with identified RGB color intensity landscape. e) A map of the RGB values for the MoS 2 samples with different numbers of layers. Reproduced with permission. [25b] Copyright 2018, Springer Nature. f) Polarized photoluminescence (PL) spectra of monolayer WSe 2 at room temperature (Left) and at 15 K (Right). 𝛼 is the peak intensity, 𝛽 is the peak photon energy, 𝛾 is the full width at half maximum (FWHM), and 𝛿•𝛼 is the spectra intensity at (𝛽-30) meV. X, T, and L stand for the PL features associated with the exciton, trion, and localized states, respectively. g) The correlations between the factors (i.e., 𝛼, 𝛽, 𝛾, and 𝛿) and the polarization are plotted. h) The predicted polarization is plotted against the experimental values to show their correlation. Reproduced with permission. [53c] Copyright 2019, American Chemical Society. i) A schematic representation of the various possible microstructures in steel and the pattern recognition with ML. j) A side-by-side comparison of the manual segmentation method and the RF method. The different phases, that is, Ferrite (F), Pearlite (P), and Bainite (B), and their respective areal fractions are given. Reproduced with permission. [99] Copyright 2018, Springer Nature.
of optimum materials for targeted applications and to facilitate multi-scale investigations of material properties. [104] For example, ellipsometry is an optical technique for investigating the dielectric properties of thin films. [105] However, it is challenging to measure dielectric properties of colloidal particles [106] or materials of other types of structures using ellipsometry. [107] Existing theoretical methods such as effective medium theories give us a solution but are often limited to specific scenarios. [108] Thus, an ML-ellipsometer system with the input of optical spectra and the output of dielectric properties for arbitrary types of materials would be attractive. In addition, the establishment of relationships between near-and far-field optical responses of a given nanostructure can also benefit from this ML-based data analysis approach. [29,109] Other areas of interest include translation of traces of a micro-robot into kinetic analysis of forces, [110] and real-time monitoring for feedback control and automatic operation of devices. [111] As ML in optics expands, one of the most pressing needs is for large high-quality datasets for the algorithm training. To further bolster ML-assisted decoding, one needs to develop more effective approaches that can ease the data requirements. Advances in computer vision may inspire solutions. For example, data augmentation is one of the most powerful tools used in computer vision, which encompasses a range of techniques such as geometric transformations and mixing images to expand the size of an existing dataset. [112] Synthetic data, that is, fake data generated entirely from an algorithm to mirror real-world data, [113] has been utilized successfully in computer vision applications. [114] Transfer learning is another promising method to circumvent the dataset problem, wherein a model pre-trained for one application can be retooled for another similar application with the shorter training time and the less training data. [56b] Publicly available high-quality optical datasets and trained models can also be valuable tools to the broader community.
The concept of optical data decoding has also been integrated with photonic-platform-based ANNs. [115] As an example, Mennel et al. demonstrated an AI vision sensor. They constructed their ANN with a reconfigurable 2D semiconductor photodiode array, which was trained to classify and encode images that were directly projected onto the array. In addition, ML can also be applied to address forward problem in optics. For example, Wiecha and Muskens proposed to train DNNs as fast predictors of the optical responses of planar plasmonic and dielectric nanostructures. [27a] Both works show us the extended possibilities in the combination of AI and optics when different ML concepts are simultaneously utilized in the multiple steps of an optics problem.
While ML-assisted decoding of optical data has provided excellent results, it has shortcomings. The presence of hidden variables and the long-standing "black box" problem always exist in ML data decoding process. We are glad to see that initial tries in optics and photonics have been made with the emerging explainable ML algorithms targeting on such problems. [52,53b] In addition, ML is susceptible to overfitting and recognizing wrong features in data sets. [24] Thus, it is advised to carefully analyze problems to be solved, consider physical constraints, and apply ML methods with an adequate understanding of the algorithms in order to obtain the desired results. We hope this review can bridge the knowledge gap between the communities of AI and optics. As researchers from both communities gain a better understanding of the other field, we expect more significant advancements in this exciting field that merges ML and optics.
Laser Photonics Rev. 2021,15, 2000422 © 2020 Wiley-VCH GmbH