skip to main content


Title: Phasetime: Deep Learning Approach to Detect Nuclei in Time Lapse Phase Images
Time lapse microscopy is essential for quantifying the dynamics of cells, subcellular organelles and biomolecules. Biologists use different fluorescent tags to label and track the subcellular structures and biomolecules within cells. However, not all of them are compatible with time lapse imaging, and the labeling itself can perturb the cells in undesirable ways. We hypothesized that phase image has the requisite information to identify and track nuclei within cells. By utilizing both traditional blob detection to generate binary mask labels from the stained channel images and the deep learning Mask RCNN model to train a detection and segmentation model, we managed to segment nuclei based only on phase images. The detection average precision is 0.82 when the IoU threshold is to be set 0.5. And the mean IoU for masks generated from phase images and ground truth masks from experts is 0.735. Without any ground truth mask labels during the training time, this is good enough to prove our hypothesis. This result enables the ability to detect nuclei without the need for exogenous labeling.  more » « less
Award ID(s):
1705464
NSF-PAR ID:
10110676
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Journal of Clinical Medicine
Volume:
8
Issue:
8
ISSN:
2077-0383
Page Range / eLocation ID:
1159
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Quantitative phase microscopy (QPM) enables studies of living biological systems without exogenous labels. To increase the utility of QPM, machine-learning methods have been adapted to extract additional information from the quantitative phase data. Previous QPM approaches focused on fluid flow systems or time-lapse images that provide high throughput data for cells at single time points, or of time-lapse images that require delayed post-experiment analyses, respectively. To date, QPM studies have not imaged specific cells over time with rapid, concurrent analyses during image acquisition. In order to study biological phenomena or cellular interactions over time, efficient time-dependent methods that automatically and rapidly identify events of interest are desirable. Here, we present an approach that combines QPM and machine learning to identify tumor-reactive T cell killing of adherent cancer cells rapidly, which could be used for identifying and isolating novel T cells and/or their T cell receptors for studies in cancer immunotherapy. We demonstrate the utility of this method by machine learning model training and validation studies using one melanoma-cognate T cell receptor model system, followed by high classification accuracy in identifying T cell killing in an additional, independent melanoma-cognate T cell receptor model system. This general approach could be useful for studying additional biological systems under label-free conditions over extended periods of examination.

     
    more » « less
  2. Observable reading behavior, the act of moving the eyes over lines of text, is highly stereotyped among the users of a language, and this has led to the development of reading detectors–methods that input windows of sequential fixations and output predictions of the fixation behavior during those windows being reading or skimming. The present study introduces a newmethod for reading detection using Region Ranking SVM (RRSVM). An SVM-based classifier learns the local oculomotor features that are important for real-time reading detection while it is optimizing for the global reading/skimming classification, making it unnecessary to hand-label local fixation windows for model training. This RRSVM reading detector was trained and evaluated using eye movement data collected in a laboratory context, where participants viewed modified web news articles and had to either read them carefully for comprehension or skim them quickly for the selection of keywords (separate groups). Ground truth labels were known at the global level (the instructed reading or skimming task), and obtained at the local level in a separate rating task. The RRSVM reading detector accurately predicted 82.5% of the global (article-level) reading/skimming behavior, with accuracy in predicting local window labels ranging from 72-95%, depending on how tuned the RRSVM was for local and global weights. With this RRSVM reading detector, a method now exists for near real-time reading detection without the need for hand-labeling of local fixation windows. With real-time reading detection capability comes the potential for applications ranging from education and training to intelligent interfaces that learn what a user is likely to know based on previous detection of their reading behavior. 
    more » « less
  3. Across basic research studies, cell counting requires significant human time and expertise. Trained experts use thin focal plane scanning to count (click) cells in stained biological tissue. This computer-assisted process (optical disector) requires a well-trained human to select a unique best z-plane of focus for counting cells of interest. Though accurate, this approach typically requires an hour per case and is prone to inter-and intra-rater errors. Our group has previously proposed deep learning (DL)-based methods to automate these counts using cell segmentation at high magnification. Here we propose a novel You Only Look Once (YOLO) model that performs cell detection on multi-channel z-plane images (disector stack). This automated Multiple Input Multiple Output (MIMO) version of the optical disector method uses an entire z-stack of microscopy images as its input, and outputs cell detections (counts) with a bounding box of each cell and class corresponding to the z-plane where the cell appears in best focus. Compared to the previous segmentation methods, the proposed method does not require time-and labor-intensive ground truth segmentation masks for training, while producing comparable accuracy to current segmentation-based automatic counts. The MIMO-YOLO method was evaluated on systematic-random samples of NeuN-stained tissue sections through the neocortex of mouse brains (n=7). Using a cross validation scheme, this method showed the ability to correctly count total neuron numbers with accuracy close to human experts and with 100% repeatability (Test-Retest). 
    more » « less
  4. Nowadays, to assess and document construction and building performance, large amount of visual data are captured and stored through camera equipped platforms such as wearable cameras, unmanned aerial/ground vehicles, and smart phones. However, due to the nonstop fashion in recording such visual data, not all of the frames in captured consecutive footages are intentionally taken, and thus not every frame is worthy of being processed for construction and building performance analysis. Since many frames will simply have non-construction related contents, before processing the visual data, the content of each recorded frame should be manually investigated depending on the association with the goal of the visual assessment. To address such challenges, this paper aims to automatically filter construction big visual data that requires no human annotations. To overcome challenges in pure discriminative approach using manually labeled images, we construct a generative model with unlabeled visual dataset, and use it to find construction-related frames in big visual dataset from jobsites. First, through composition-based snap point detection together with domain adaptation, we filter and remove most of accidently recorded frames in the footage. Then, we create discriminative classifier trained with visual data from jobsites to eliminate non-construction related images. To evaluate the reliability of the proposed method, we have obtained the ground truth based on human judgment for each photo in our testing dataset. Despite learning without any explicit labels, the proposed method shows a reasonable practical range of accuracy, which generally outperforms prior snap point detection. Through the case studies, the fidelity of the algorithm is discussed in detail. By being able to focus on selective visual data, practitioners will spend less time on browsing large amounts of visual data; rather spend more time on looking at how to leverage the visual data to facilitate decision-makings in built environments. 
    more » « less
  5. Efficient detection and observation of dynamic RNA changes remain a tremendous challenge. However, the continuous development of fluorescence applications in recent years enhances the efficacy of RNA imaging. Here we summarize some of these developments from different aspects. For example, single-molecule fluorescence in situ hybridization (smFISH) can detect low abundance RNA at the subcellular level. A relatively new aptamer, Mango, is widely applied to label and track RNA activities in living cells. Molecular beacons (MBs) are valid for quantifying both endogenous and exogenous mRNA and microRNA (miRNA). Covalent binding enzyme labeling fluorescent group with RNA of interest (ROI) partially overcomes the RNA length limitation associated with oligonucleotide synthesis. Forced intercalation (FIT) probes are resistant to nuclease degradation upon binding to target RNA and are used to visualize mRNA and messenger ribonucleoprotein (mRNP) activities. We also summarize the importance of some fluorescence spectroscopic techniques in exploring the function and movement of RNA. Single-molecule fluorescence resonance energy transfer (smFRET) has been employed to investigate the dynamic changes of biomolecules by covalently linking biotin to RNA, and a focus on dye selection increases FRET efficiency. Furthermore, the applications of fluorescence assays in drug discovery and drug delivery have been discussed. Fluorescence imaging can also combine with RNA nanotechnology to target tumors. The invention of novel antibacterial drugs targeting non-coding RNAs (ncRNAs) is also possible with steady-state fluorescence-monitored ligand-binding assay and the T-box riboswitch fluorescence anisotropy assay. More recently, COVID-19 tests using fluorescent clustered regularly interspaced short palindromic repeat (CRISPR) technology have been demonstrated to be efficient and clinically useful. In summary, fluorescence assays have significant applications in both fundamental and clinical research and will facilitate the process of RNA-targeted new drug discovery, therefore deserving further development and updating. 
    more » « less