An official website of the United States government
Abstract Obtaining in situ measurements of biological microparticles is crucial for both scientific research and numerous industrial applications (e.g., early detection of harmful algal blooms, monitoring yeast during fermentation). However, existing methods are limited to offer timely diagnostics of these particles with sufficient accuracy and information. Here, we introduce a novel method for real‐time, in situ analysis using machine learning (ML)‐assisted digital inline holography (DIH). Our ML model uses a customized YOLOv5 architecture specialized for the detection and classification of small biological particles. We demonstrate the effectiveness of our method in the analysis of 10 plankton species with equivalent high accuracy and significantly reduced processing time compared to previous methods. We also applied our method to differentiate yeast cells under four metabolic states and from two strains. Our results show that the proposed method can accurately detect and differentiate cellular and subcellular features related to metabolic states and strains. This study demonstrates the potential of ML‐driven DIH approach as a sensitive and versatile diagnostic tool for real‐time, in situ analysis of both biotic and abiotic particles. This method can be readily deployed in a distributive manner for scientific research and manufacturing on an industrial scale.
more »
« less
Automated plankton classification from holographic imagery with deep convolutional neural networks
Abstract In situ digital inline holography is a technique which can be used to acquire high‐resolution imagery of plankton and examine their spatial and temporal distributions within the water column in a nonintrusive manner. However, for effective expert identification of an organism from digital holographic imagery, it is necessary to apply a computationally expensive numerical reconstruction algorithm. This lengthy process inhibits real‐time monitoring of plankton distributions. Deep learning methods, such as convolutional neural networks, applied to interference patterns of different organisms from minimally processed holograms can eliminate the need for reconstruction and accomplish real‐time computation. In this article, we integrate deep learning methods with digital inline holography to create a rapid and accurate plankton classification network for 10 classes of organisms that are commonly seen in our data sets. We describe the procedure from preprocessing to classification. Our network achieves 93.8% accuracy when applied to a manually classified testing data set. Upon further application of a probability filter to eliminate false classification, the average precision and recall are 96.8% and 95.0%, respectively. Furthermore, the network was applied to 7500 in situ holograms collected at East Sound in Washington during a vertical profile to characterize depth distribution of the local diatoms. The results are in agreement with simultaneously recorded independent chlorophyll concentration depth profiles. This lightweight network exemplifies its capability for real‐time, high‐accuracy plankton classification and it has the potential to be deployed on imaging instruments for long‐term in situ plankton monitoring.
more »
« less
- PAR ID:
- 10454551
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Limnology and Oceanography: Methods
- Volume:
- 19
- Issue:
- 1
- ISSN:
- 1541-5856
- Page Range / eLocation ID:
- p. 21-36
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)The characterization of particle and plankton populations, as well as microscale biophysical interactions, is critical to several important research areas in oceanography and limnology. A growing number of aquatic researchers are turning to holography as a tool of choice to quantify particle fields in diverse environments, including but not limited to, studies on particle orientation, thin layers, phytoplankton blooms, and zooplankton distributions and behavior. Holography provides a non-intrusive, free-stream approach to imaging and characterizing aquatic particles, organisms, and behavior in situ at high resolution through a 3-D sampling volume. Compared to other imaging techniques, e.g., flow cytometry, much larger volumes of water can be processed over the same duration, resolving particle sizes ranging from a few microns to a few centimeters. Modern holographic imaging systems are compact enough to be deployed through various modes, including profiling/towed platforms, buoys, gliders, long-term observatories, or benthic landers. Limitations of the technique include the data-intensive hologram acquisition process, computationally expensive image reconstruction, and coherent noise associated with the holograms that can make post-processing challenging. However, continued processing refinements, rapid advancements in computing power, and development of powerful machine learning algorithms for particle/organism classification are paving the way for holography to be used ubiquitously across different disciplines in the aquatic sciences. This review aims to provide a comprehensive overview of holography in the context of aquatic studies, including historical developments, prior research applications, as well as advantages and limitations of the technique. Ongoing technological developments that can facilitate larger employment of this technique toward in situ measurements in the future, as well as potential applications in emerging research areas in the aquatic sciences are also discussed.more » « less
-
Abstract We introduce a digital inline holography (DIH) method combined with deep learning (DL) for real-time detection and analysis of bacteria in liquid suspension. Specifically, we designed a prototype that integrates DIH with fluorescence imaging to efficiently capture holograms of bacteria flowing in a microfluidic channel, utilizing the fluorescent signal to manually identify ground truths for validation. We process holograms using a tailored DL framework that includes preprocessing, detection, and classification stages involving three specific DL models trained on an extensive dataset that included holograms of generic particles present in sterile liquid and five bacterial species featuring distinct morphologies, Gram stain attributes, and viability. Our approach, validated through experiments with synthetic data and sterile liquid spiked with different bacteria, accurately distinguishes between bacteria and particles, live and dead bacteria, and Gram-positive and negative bacteria of similar morphology, all while minimizing false positives. The study highlights the potential of combining DIH with DL as a transformative tool for rapid bacterial analysis in clinical and industrial settings, with potential extension to other applications including pharmaceutical screening, environmental monitoring, and disease diagnostics.more » « less
-
The small sizes of most marine plankton necessitate that plankton sampling occur on fine spatial scales, yet our questions often span large spatial areas. Underwater imaging can provide a solution to this sampling conundrum but collects large quantities of data that require an automated approach to image analysis. Machine learning for plankton classification, and high-performance computing (HPC) infrastructure, are critical to rapid image processing; however, these assets, especially HPC infrastructure, are only available post-cruise leading to an ‘after-the-fact’ view of plankton community structure. To be responsive to the often-ephemeral nature of oceanographic features and species assemblages in highly dynamic current systems, real-time data are key for adaptive oceanographic sampling. Here we used the new In-situ Ichthyoplankton Imaging System-3 (ISIIS-3) in the Northern California Current (NCC) in conjunction with an edge server to classify imaged plankton in real-time into 170 classes. This capability together with data visualization in a heavy.ai dashboard makes adaptive real-time decision-making and sampling at sea possible. Dual ISIIS-Deep-focus Particle Imager (DPI) cameras sample 180 L s -1 , leading to >10 GB of video per min. Imaged organisms are in the size range of 250 µm to 15 cm and include abundant crustaceans, fragile taxa (e.g., hydromedusae, salps), faster swimmers (e.g., krill), and rarer taxa (e.g., larval fishes). A deep learning pipeline deployed on the edge server used multithreaded CPU-based segmentation and GPU-based classification to process the imagery. AVI videos contain 50 sec of data and can contain between 23,000 - 225,000 particle and plankton segments. Processing one AVI through segmentation and classification takes on average 3.75 mins, depending on biological productivity. A heavyDB database monitors for newly processed data and is linked to a heavy.ai dashboard for interactive data visualization. We describe several examples where imaging, AI, and data visualization enable adaptive sampling that can have a transformative effect on oceanography. We envision AI-enabled adaptive sampling to have a high impact on our ability to resolve biological responses to important oceanographic features in the NCC, such as oxygen minimum zones, or harmful algal bloom thin layers, which affect the health of the ecosystem, fisheries, and local communities.more » « less
-
Abstract Underwater imaging enables nondestructive plankton sampling at frequencies, durations, and resolutions unattainable by traditional methods. These systems necessitate automated processes to identify organisms efficiently. Early underwater image processing used a standard approach: binarizing images to segment targets, then integrating deep learning models for classification. While intuitive, this infrastructure has limitations in handling high concentrations of biotic and abiotic particles, rapid changes in dominant taxa, and highly variable target sizes. To address these challenges, we introduce a new framework that starts with a scene classifier to capture large within‐image variation, such as disparities in the layout of particles and dominant taxa. After scene classification, scene‐specific Mask regional convolutional neural network (Mask R‐CNN) models are trained to separate target objects into different groups. The procedure allows information to be extracted from different image types, while minimizing potential bias for commonly occurring features. Using in situ coastal plankton images, we compared the scene‐specific models to the Mask R‐CNN model encompassing all scene categories as a single full model. Results showed that the scene‐specific approach outperformed the full model by achieving a 20% accuracy improvement in complex noisy images. The full model yielded counts that were up to 78% lower than those enumerated by the scene‐specific model for some small‐sized plankton groups. We further tested the framework on images from a benthic video camera and an imaging sonar system with good results. The integration of scene classification, which groups similar images together, can improve the accuracy of detection and classification for complex marine biological images.more » « less
