Underwater digital in‐line holography can provide high‐resolution, in situ imagery of marine particles and offers many advantages over alternative measurement approaches. However, processing of holograms requires computationally expensive reconstruction and processing, and computational cost increases with the size of the imaging volume. In this work, a processing pipeline is developed to extract targets from holograms where target distribution is relatively sparse without reconstruction of the full hologram. This is motivated by the desire to efficiently extract quantitative estimates of plankton abundance from a data set (>300,000 holograms) collected in the Northwest Atlantic using a large‐volume holographic camera. First, holograms with detectable targets are selected using a transfer learning approach. This was critical as a subset of the holograms were impacted by optical turbulence, which obscured target detection. Then, target diffraction patterns are detected in the hologram. Finally, targets are reconstructed and focused using only a small region of the hologram around the detected diffraction pattern. A search algorithm is employed to select distances for reconstruction, reducing the number of reconstructions required for 1 mm focus precision from 1000 to 31. When compared with full reconstruction techniques, this method detects 99% of particles larger than 0.1 mm2, a size class which includes most copepods and larger particles of marine snow, and 85% of those targets are sufficiently focused for classification. This approach requires 1% of the processing time required to compute full reconstructions, making processing of long time‐series, large imaging volume holographic data sets feasible.
In situ digital inline holography is a technique which can be used to acquire high‐resolution imagery of plankton and examine their spatial and temporal distributions within the water column in a nonintrusive manner. However, for effective expert identification of an organism from digital holographic imagery, it is necessary to apply a computationally expensive numerical reconstruction algorithm. This lengthy process inhibits real‐time monitoring of plankton distributions. Deep learning methods, such as convolutional neural networks, applied to interference patterns of different organisms from minimally processed holograms can eliminate the need for reconstruction and accomplish real‐time computation. In this article, we integrate deep learning methods with digital inline holography to create a rapid and accurate plankton classification network for 10 classes of organisms that are commonly seen in our data sets. We describe the procedure from preprocessing to classification. Our network achieves 93.8% accuracy when applied to a manually classified testing data set. Upon further application of a probability filter to eliminate false classification, the average precision and recall are 96.8% and 95.0%, respectively. Furthermore, the network was applied to 7500 in situ holograms collected at East Sound in Washington during a vertical profile to characterize depth distribution of the local diatoms. The results are in agreement with simultaneously recorded independent chlorophyll concentration depth profiles. This lightweight network exemplifies its capability for real‐time, high‐accuracy plankton classification and it has the potential to be deployed on imaging instruments for long‐term in situ plankton monitoring.
more » « less- NSF-PAR ID:
- 10454551
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Limnology and Oceanography: Methods
- Volume:
- 19
- Issue:
- 1
- ISSN:
- 1541-5856
- Page Range / eLocation ID:
- p. 21-36
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
null (Ed.)The characterization of particle and plankton populations, as well as microscale biophysical interactions, is critical to several important research areas in oceanography and limnology. A growing number of aquatic researchers are turning to holography as a tool of choice to quantify particle fields in diverse environments, including but not limited to, studies on particle orientation, thin layers, phytoplankton blooms, and zooplankton distributions and behavior. Holography provides a non-intrusive, free-stream approach to imaging and characterizing aquatic particles, organisms, and behavior in situ at high resolution through a 3-D sampling volume. Compared to other imaging techniques, e.g., flow cytometry, much larger volumes of water can be processed over the same duration, resolving particle sizes ranging from a few microns to a few centimeters. Modern holographic imaging systems are compact enough to be deployed through various modes, including profiling/towed platforms, buoys, gliders, long-term observatories, or benthic landers. Limitations of the technique include the data-intensive hologram acquisition process, computationally expensive image reconstruction, and coherent noise associated with the holograms that can make post-processing challenging. However, continued processing refinements, rapid advancements in computing power, and development of powerful machine learning algorithms for particle/organism classification are paving the way for holography to be used ubiquitously across different disciplines in the aquatic sciences. This review aims to provide a comprehensive overview of holography in the context of aquatic studies, including historical developments, prior research applications, as well as advantages and limitations of the technique. Ongoing technological developments that can facilitate larger employment of this technique toward in situ measurements in the future, as well as potential applications in emerging research areas in the aquatic sciences are also discussed.more » « less
-
more » « less
Abstract The rise of in situ plankton imaging systems, particularly high‐volume imagers such as the In Situ Ichthyoplankton Imaging System, has increased the need for fast processing and accurate classification tools that can identify a high diversity of organisms and nonliving particles of biological origin. Previous methods for automated classification have yielded moderate results that either can resolve few groups at high accuracy or many groups at relatively low accuracy. However, with the advent of new deep learning tools such as convolutional neural networks (CNNs), the automated identification of plankton images can be vastly improved. Here, we describe an image processing procedure that includes preprocessing, segmentation, classification, and postprocessing for the accurate identification of 108 classes of plankton using spatially sparse CNNs. Following a filtering process to remove images with low classification scores, a fully random evaluation of the classification showed that average precision was 84% and recall was 40% for all groups. Reliably classifying rare biological classes was difficult, so after excluding the 12 rarest taxa, classification accuracy for the remaining biological groups became > 90%. This method provides proof of concept for the effectiveness of an automated classification scheme using deep‐learning methods, which can be applied to a range of plankton or biological imaging systems, with the eventual application in a variety of ecological monitoring and fisheries management contexts.
-
The small sizes of most marine plankton necessitate that plankton sampling occur on fine spatial scales, yet our questions often span large spatial areas. Underwater imaging can provide a solution to this sampling conundrum but collects large quantities of data that require an automated approach to image analysis. Machine learning for plankton classification, and high-performance computing (HPC) infrastructure, are critical to rapid image processing; however, these assets, especially HPC infrastructure, are only available post-cruise leading to an ‘after-the-fact’ view of plankton community structure. To be responsive to the often-ephemeral nature of oceanographic features and species assemblages in highly dynamic current systems, real-time data are key for adaptive oceanographic sampling. Here we used the new In-situ Ichthyoplankton Imaging System-3 (ISIIS-3) in the Northern California Current (NCC) in conjunction with an edge server to classify imaged plankton in real-time into 170 classes. This capability together with data visualization in a heavy.ai dashboard makes adaptive real-time decision-making and sampling at sea possible. Dual ISIIS-Deep-focus Particle Imager (DPI) cameras sample 180 L s -1 , leading to >10 GB of video per min. Imaged organisms are in the size range of 250 µm to 15 cm and include abundant crustaceans, fragile taxa (e.g., hydromedusae, salps), faster swimmers (e.g., krill), and rarer taxa (e.g., larval fishes). A deep learning pipeline deployed on the edge server used multithreaded CPU-based segmentation and GPU-based classification to process the imagery. AVI videos contain 50 sec of data and can contain between 23,000 - 225,000 particle and plankton segments. Processing one AVI through segmentation and classification takes on average 3.75 mins, depending on biological productivity. A heavyDB database monitors for newly processed data and is linked to a heavy.ai dashboard for interactive data visualization. We describe several examples where imaging, AI, and data visualization enable adaptive sampling that can have a transformative effect on oceanography. We envision AI-enabled adaptive sampling to have a high impact on our ability to resolve biological responses to important oceanographic features in the NCC, such as oxygen minimum zones, or harmful algal bloom thin layers, which affect the health of the ecosystem, fisheries, and local communities.more » « less
-
null (Ed.)Urban flooding is a major natural disaster that poses a serious threat to the urban environment. It is highly demanded that the flood extent can be mapped in near real-time for disaster rescue and relief missions, reconstruction efforts, and financial loss evaluation. Many efforts have been taken to identify the flooding zones with remote sensing data and image processing techniques. Unfortunately, the near real-time production of accurate flood maps over impacted urban areas has not been well investigated due to three major issues. (1) Satellite imagery with high spatial resolution over urban areas usually has nonhomogeneous background due to different types of objects such as buildings, moving vehicles, and road networks. As such, classical machine learning approaches hardly can model the spatial relationship between sample pixels in the flooding area. (2) Handcrafted features associated with the data are usually required as input for conventional flood mapping models, which may not be able to fully utilize the underlying patterns of a large number of available data. (3) High-resolution optical imagery often has varied pixel digital numbers (DNs) for the same ground objects as a result of highly inconsistent illumination conditions during a flood. Accordingly, traditional methods of flood mapping have major limitations in generalization based on testing data. To address the aforementioned issues in urban flood mapping, we developed a patch similarity convolutional neural network (PSNet) using satellite multispectral surface reflectance imagery before and after flooding with a spatial resolution of 3 meters. We used spectral reflectance instead of raw pixel DNs so that the influence of inconsistent illumination caused by varied weather conditions at the time of data collection can be greatly reduced. Such consistent spectral reflectance data also enhance the generalization capability of the proposed model. Experiments on the high resolution imagery before and after the urban flooding events (i.e., the 2017 Hurricane Harvey and the 2018 Hurricane Florence) showed that the developed PSNet can produce urban flood maps with consistently high precision, recall, F1 score, and overall accuracy compared with baseline classification models including support vector machine, decision tree, random forest, and AdaBoost, which were often poor in either precision or recall. The study paves the way to fuse bi-temporal remote sensing images for near real-time precision damage mapping associated with other types of natural hazards (e.g., wildfires and earthquakes).more » « less
