An official website of the United States government
Abstract Motivation Cancer heterogeneity is observed at multiple biological levels. To improve our understanding of these differences and their relevance in medicine, approaches to link organ- and tissue-level information from diagnostic images and cellular-level information from genomics are needed. However, these ‘radiogenomic’ studies often use linear or shallow models, depend on feature selection, or consider one gene at a time to map images to genes. Moreover, no study has systematically attempted to understand the molecular basis of imaging traits based on the interpretation of what the neural network has learned. These studies are thus limited in their ability to understand the transcriptomic drivers of imaging traits, which could provide additional context for determining clinical outcomes. Results We present a neural network-based approach that takes high-dimensional gene expression data as input and performs non-linear mapping to an imaging trait. To interpret the models, we propose gene masking and gene saliency to extract learned relationships from radiogenomic neural networks. In glioblastoma patients, our models outperformed comparable classifiers (>0.10 AUC) and our interpretation methods were validated using a similar model to identify known relationships between genes and molecular subtypes. We found that tumor imaging traits had specific transcription patterns, e.g. edema and genes related to cellular invasion, and 10 radiogenomic traits were significantly predictive of survival. We demonstrate that neural networks can model transcriptomic heterogeneity to reflect differences in imaging and can be used to derive radiogenomic traits with clinical value. Availability and implementation https://github.com/novasmedley/deepRadiogenomics. Contact whsu@mednet.ucla.edu Supplementary information Supplementary data are available at Bioinformatics online.
more »
« less
Machine learning techniques to characterize functional traits of plankton from image data
Abstract Plankton imaging systems supported by automated classification and analysis have improved ecologists' ability to observe aquatic ecosystems. Today, we are on the cusp of reliably tracking plankton populations with a suite of lab‐based and in situ tools, collecting imaging data at unprecedentedly fine spatial and temporal scales. But these data have potential well beyond examining the abundances of different taxa; the individual images themselves contain a wealth of information on functional traits. Here, we outline traits that could be measured from image data, suggest machine learning and computer vision approaches to extract functional trait information from the images, and discuss promising avenues for novel studies. The approaches we discuss are data agnostic and are broadly applicable to imagery of other aquatic or terrestrial organisms.
more »
« less
- PAR ID:
- 10369897
- Author(s) / Creator(s):
- ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; more »
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Limnology and Oceanography
- Volume:
- 67
- Issue:
- 8
- ISSN:
- 0024-3590
- Format(s):
- Medium: X Size: p. 1647-1669
- Size(s):
- p. 1647-1669
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
As the basis of oceanic food webs and a key component of the biological carbon pump, planktonic organisms play major roles in the oceans. Their study benefited from the development of in situ imaging instruments, which provide higher spatio-temporal resolution than previous tools. But these instruments collect huge quantities of images, the vast majority of which are of marine snow particles or imaging artifacts. Among them, the In Situ Ichthyoplankton Imaging System (ISIIS) samples the largest water volumes (> 100 L s -1 ) and thus produces particularly large datasets. To extract manageable amounts of ecological information from in situ images, we propose to focus on planktonic organisms early in the data processing pipeline: at the segmentation stage. We compared three segmentation methods, particularly for smaller targets, in which plankton represents less than 1% of the objects: (i) a traditional thresholding over the background, (ii) an object detector based on maximally stable extremal regions (MSER), and (iii) a content-aware object detector, based on a Convolutional Neural Network (CNN). These methods were assessed on a subset of ISIIS data collected in the Mediterranean Sea, from which a ground truth dataset of > 3,000 manually delineated organisms is extracted. The naive thresholding method captured 97.3% of those but produced ~340,000 segments, 99.1% of which were therefore not plankton (i.e. recall = 97.3%, precision = 0.9%). Combining thresholding with a CNN missed a few more planktonic organisms (recall = 91.8%) but the number of segments decreased 18-fold (precision increased to 16.3%). The MSER detector produced four times fewer segments than thresholding (precision = 3.5%), missed more organisms (recall = 85.4%), but was considerably faster. Because naive thresholding produces ~525,000 objects from 1 minute of ISIIS deployment, the more advanced segmentation methods significantly improve ISIIS data handling and ease the subsequent taxonomic classification of segmented objects. The cost in terms of recall is limited, particularly for the CNN object detector. These approaches are now standard in computer vision and could be applicable to other plankton imaging devices, the majority of which pose a data management problem.more » « less
-
Abstract Underwater imaging enables nondestructive plankton sampling at frequencies, durations, and resolutions unattainable by traditional methods. These systems necessitate automated processes to identify organisms efficiently. Early underwater image processing used a standard approach: binarizing images to segment targets, then integrating deep learning models for classification. While intuitive, this infrastructure has limitations in handling high concentrations of biotic and abiotic particles, rapid changes in dominant taxa, and highly variable target sizes. To address these challenges, we introduce a new framework that starts with a scene classifier to capture large within‐image variation, such as disparities in the layout of particles and dominant taxa. After scene classification, scene‐specific Mask regional convolutional neural network (Mask R‐CNN) models are trained to separate target objects into different groups. The procedure allows information to be extracted from different image types, while minimizing potential bias for commonly occurring features. Using in situ coastal plankton images, we compared the scene‐specific models to the Mask R‐CNN model encompassing all scene categories as a single full model. Results showed that the scene‐specific approach outperformed the full model by achieving a 20% accuracy improvement in complex noisy images. The full model yielded counts that were up to 78% lower than those enumerated by the scene‐specific model for some small‐sized plankton groups. We further tested the framework on images from a benthic video camera and an imaging sonar system with good results. The integration of scene classification, which groups similar images together, can improve the accuracy of detection and classification for complex marine biological images.more » « less
-
Abstract Functional traits are increasingly used to assess changes in phytoplankton community structure and to link individual characteristics to ecosystem functioning. However, they are usually inferred from taxonomic identification or manually measured for each organism, both time consuming approaches. Instead, we focus on high throughput imaging to describe the main temporal variations of morphological changes of phytoplankton in Narragansett Bay, a coastal time‐series station. We analyzed a 2‐yr dataset of morphological features automatically extracted from continuous imaging of individual phytoplankton images (~ 105 million images collected by an Imaging FlowCytobot). We identified synthetic morphological traits using multivariate analysis and revealed that morphological variations were mainly due to changes in length, width, shape regularity, and chain structure. Morphological changes were especially important in winter with successive peaks of larger cells with increasing complexity and chains more clearly connected. Small nanophytoplankton were present year‐round and constituted the base of the community, especially apparent during the transitions between diatom blooms. High inter‐annual variability was also observed. On a weekly timescale, increases in light were associated with more clearly connected chains while more complex shapes occurred at lower nitrogen concentrations. On an hourly timescale, temperature was the determinant variable constraining cell morphology, with a general negative influence on length and a positive one on width, shape regularity, and chain structure. These first insights into the phytoplankton morphology of Narragansett Bay highlight the possible morphological traits driving the phytoplankton succession in response to light, temperature, and nutrient changes.more » « less
-
Manca, M.M. Granata (Ed.)In recent years, trait-based research on plankton has gained interest because of its potential to uncover general roles in ecology. While trait categories for phytoplankton and crustaceans have been posited, rotifer trait assessment has lagged behind. Here, we reviewed the literature to assess traits key to their life histories and provided a data matrix for the 138 valid genera of phylum Rotifera. We considered seven traits: habitat type, trophi type, presence of lorica and foot, predation defense attributes, corona type, and feeding traits. While most traits were morphological attributes and supposedly easy to assess, we were faced with several challenges regarding trait assignment. Feeding traits were especially difficult to assess for many genera because relevant information was missing. Our assembled trait matrix provides a foundation that will initiate additional research on rotifer functional diversity, diminish the misclassification of rotifer genera into trait categories, and facilitate studies across trophic levels.more » « less
