skip to main content

Title: Generative adversarial network enables rapid and robust fluorescence lifetime image analysis in live cells

Fluorescence lifetime imaging microscopy (FLIM) is a powerful tool to quantify molecular compositions and study molecular states in complex cellular environment as the lifetime readings are not biased by fluorophore concentration or excitation power. However, the current methods to generate FLIM images are either computationally intensive or unreliable when the number of photons acquired at each pixel is low. Here we introduce a new deep learning-based method termedflimGANE(fluorescencelifetimeimaging based onGenerativeAdversarialNetworkEstimation) that can rapidly generate accurate and high-quality FLIM images even in the photon-starved conditions. We demonstrated our model is up to 2,800 times faster than the gold standard time-domain maximum likelihood estimation (TD_MLE) and thatflimGANEprovides a more accurate analysis of low-photon-count histograms in barcode identification, cellular structure visualization, Förster resonance energy transfer characterization, and metabolic state analysis in live cells. With its advantages in speed and reliability,flimGANEis particularly useful in fundamental biological research and clinical applications, where high-speed analysis is critical.

; ; ; ; ; ; ; ; ; ; ;
Award ID(s):
Publication Date:
Journal Name:
Communications Biology
Nature Publishing Group
Sponsoring Org:
National Science Foundation
More Like this
  1. Traditional fluorescence microscopy is blind to molecular microenvironment information that is present in a fluorescence lifetime, which can be measured by fluorescence lifetime imaging microscopy (FLIM). However, most existing FLIM techniques are slow to acquire and process lifetime images, difficult to implement, and expensive. Here we present instant FLIM, an analog signal processing method that allows real-time streaming of fluorescence intensity, lifetime, and phasor imaging data through simultaneous image acquisition and instantaneous data processing. Instant FLIM can be easily implemented by upgrading an existing two-photon microscope using cost-effective components and our open-source software. We further improve the functionality, penetration depth, and resolution of instant FLIM using phasor segmentation, adaptive optics, and super-resolution techniques. We demonstrate through-skull intravital 3D FLIM of mouse brains to depths of 300 µm and present the firstin vivo4D FLIM of microglial dynamics in intact and injured zebrafish and mouse brains for up to 12 h.

  2. Abstract STUDY QUESTION

    Is the combined use of fluorescence lifetime imaging microscopy (FLIM)-based metabolic imaging and second harmonic generation (SHG) spindle imaging a feasible and safe approach for noninvasive embryo assessment?


    Metabolic imaging can sensitively detect meaningful metabolic changes in embryos, SHG produces high-quality images of spindles and the methods do not significantly impair embryo viability.


    Proper metabolism is essential for embryo viability. Metabolic imaging is a well-tested method for measuring metabolism of cells and tissues, but it is unclear if it is sensitive enough and safe enough for use in embryo assessment.


    This study consisted of time-course experiments and control versus treatment experiments. We monitored the metabolism of 25 mouse oocytes with a noninvasive metabolic imaging system while exposing them to oxamate (cytoplasmic lactate dehydrogenase inhibitor) and rotenone (mitochondrial oxidative phosphorylation inhibitor) in series. Mouse embryos (n = 39) were measured every 2 h from the one-cell stage to blastocyst in order to characterize metabolic changes occurring during pre-implantation development. To assess the safety of FLIM illumination, n = 144 illuminated embryos were implanted into n = 12 mice, and n = 108 nonilluminated embryos were implanted into n = 9 mice.


    Experiments were performed in mouse embryos and oocytes. Samples weremore »monitored with noninvasive, FLIM-based metabolic imaging of nicotinamide adenine dinucleotide (NADH) and flavin adenine dinucleotide (FAD) autofluorescence. Between NADH cytoplasm, NADH mitochondria and FAD mitochondria, a single metabolic measurement produces up to 12 quantitative parameters for characterizing the metabolic state of an embryo. For safety experiments, live birth rates and pup weights (mean ± SEM) were used as endpoints. For all test conditions, the level of significance was set at P < 0.05.


    Measured FLIM parameters were highly sensitive to metabolic changes due to both metabolic perturbations and embryo development. For oocytes, metabolic parameter values were compared before and after exposure to oxamate and rotenone. The metabolic measurements provided a basis for complete separation of the data sets. For embryos, metabolic parameter values were compared between the first division and morula stages, morula and blastocyst and first division and blastocyst. The metabolic measurements again completely separated the data sets. Exposure of embryos to excessive illumination dosages (24 measurements) had no significant effect on live birth rate (5.1 ± 0.94 pups/mouse for illuminated group; 5.7 ± 1.74 pups/mouse for control group) or pup weights (1.88 ± 0.10 g for illuminated group; 1.89 ± 0.11 g for control group).


    The study was performed using a mouse model, so conclusions concerning sensitivity and safety may not generalize to human embryos. A limitation of the live birth data is also that although cages were routinely monitored, we could not preclude that some runt pups may have been eaten.


    Promising proof-of-concept results demonstrate that FLIM with SHG provide detailed biological information that may be valuable for the assessment of embryo and oocyte quality. Live birth experiments support the method’s safety, arguing for further studies of the clinical utility of these techniques.


    Supported by the Blavatnik Biomedical Accelerator Grant at Harvard University and by the Harvard Catalyst/The Harvard Clinical and Translational Science Center (National Institutes of Health Award UL1 TR001102), by NSF grants DMR-0820484 and PFI-TT-1827309 and by NIH grant R01HD092550-01. T.S. was supported by a National Science Foundation Postdoctoral Research Fellowship in Biology grant (1308878). S.F. and S.A. were supported by NSF MRSEC DMR-1420382. Becker and Hickl GmbH sponsored the research with the loaning of equipment for FLIM. T.S. and D.N. are cofounders and shareholders of LuminOva, Inc., and co-hold patents (US20150346100A1 and US20170039415A1) for metabolic imaging methods. D.S. is on the scientific advisory board for Cooper Surgical and has stock options with LuminOva, Inc.

    « less
  3. Abstract

    Scanning two-photon (2P) fiberscopes (also termed endomicroscopes) have the potential to transform our understanding of how discrete neural activity patterns result in distinct behaviors, as they are capable of high resolution, sub cellular imaging yet small and light enough to allow free movement of mice. However, their acquisition speed is currently suboptimal, due to opto-mechanical size and weight constraints. Here we demonstrate significant advances in 2P fiberscopy that allow high resolution imaging at high speeds (26 fps) in freely-behaving mice. A high-speed scanner and a down-sampling scheme are developed to boost imaging speed, and a deep learning (DL) algorithm is introduced to recover image quality. For the DL algorithm, a two-stage learning transfer strategy is established to generate proper training datasets for enhancing the quality of in vivo images. Implementation enables video-rate imaging at ~26 fps, representing 10-fold improvement in imaging speed over the previous 2P fiberscopy technology while maintaining a high signal-to-noise ratio and imaging resolution. This DL-assisted 2P fiberscope is capable of imaging the arousal-induced activity changes in populations of layer2/3 pyramidal neurons in the primary motor cortex of freely-behaving mice, providing opportunities to define the neural basis of behavior.

  4. Abstract

    Label-free optical microscopy has matured as a noninvasive tool for biological imaging; yet, it is criticized for its lack of specificity, slow acquisition and processing times, and weak and noisy optical signals that lead to inaccuracies in quantification. We introduce FOCALS (Fast Optical Coherence, Autofluorescence Lifetime imaging, and Second harmonic generation) microscopy capable of generating NAD(P)H fluorescence lifetime, second harmonic generation (SHG), and polarization-sensitive optical coherence microscopy (OCM) images simultaneously. Multimodal imaging generates quantitative metabolic and morphological profiles of biological samples in vitro, ex vivo, and in vivo. Fast analog detection of fluorescence lifetime and real-time processing on a graphical processing unit enables longitudinal imaging of biological dynamics. We detail the effect of optical aberrations on the accuracy of FLIM beyond the context of undistorting image features. To compensate for the sample-induced aberrations, we implemented a closed-loop single-shot sensorless adaptive optics solution, which uses computational adaptive optics of OCM for wavefront estimation within 2 s and improves the quality of quantitative fluorescence imaging in thick tissues. Multimodal imaging with complementary contrasts improves the specificity and enables multidimensional quantification of the optical signatures in vitro, ex vivo, and in vivo, fast acquisition and real-time processing improve imaging speed by 4–40 × while maintainingmore »enough signal for quantitative nonlinear microscopy, and adaptive optics improves the overall versatility, which enable FOCALS microscopy to overcome the limits of traditional label-free imaging techniques.

    « less
  5. Two-photon fluorescence lifetime imaging microscopy (FLIM) is a widely used technique in biomedical optical imaging. Presently, many two-photon time-domain FLIM setups are limited by long acquisition and postprocessing times that decrease data throughput and inhibit the ability to image fast sub-second processes. Here, we present a versatile two-photon FLIM setup capable of video-rate (up to 25 fps) imaging with graphics processing unit (GPU)-accelerated pixelwise phasor analysis displayed and saved simultaneously with acquisition. The system uses an analog output photomultiplier tube in conjunction with 12-bit digitization at 3.2 GHz to overcome the limited maximum acceptable photon rate associated with the photon counting electronics in many FLIM systems. This allows for higher throughput FLIM acquisition and analysis, and additionally enables the user to assess sample fluorescence lifetime in real-time. We further explore the capabilities of the system to examine the kinetics of Rhodamine B uptake by human breast cancer cells and characterize the effect of pixel dwell time on the reduced nicotinamide adenine dinucleotide and reduced nicotinamide adenine dinucleotide phosphate (NAD(P)H) autofluorescence lifetime estimation accuracy.