skip to main content

Title: Visual Tracking of Deepwater Animals Using Machine Learning-Controlled Robotic Underwater Vehicles
The ocean is a vast three-dimensional space that is poorly explored and understood, and harbors unobserved life and processes that are vital to ecosystem function. To fully interrogate the space, novel algorithms and robotic platforms are required to scale up observations. Locating animals of interest and extended visual observations in the water column are particularly challenging objectives. Towards that end, we present a novel Machine Learning-integrated Tracking (or ML-Tracking) algorithm for underwater vehicle control that builds on the class of algorithms known as tracking-by-detection. By coupling a multi-object detector (trained on in situ underwater image data), a 3D stereo tracker, and a supervisor module to oversee the mission, we show how ML-Tracking can create robust tracks needed for long duration observations, as well as enable fully automated acquisition of objects for targeted sampling. Using a remotely operated vehicle as a proxy for an autonomous underwater vehicle, we demonstrate continuous input from the ML-Tracking algorithm to the vehicle controller during a record, 5+ hr continuous observation of a midwater gelatinous animal known as a siphonophore. These efforts clearly demonstrate the potential that tracking-by-detection algorithms can have on exploration in unexplored environments and discovery of undiscovered life in our ocean.
; ; ; ; ; ; ; ;
Award ID(s):
Publication Date:
Journal Name:
IEEE Winter Conference on Applications of Computer Vision
Page Range or eLocation-ID:
Sponsoring Org:
National Science Foundation
More Like this
  1. The deep chlorophyll maximum (DCM) layer is an ecologically important feature of the open ocean. The DCM cannot be observed using aerial or satellite remote sensing; thus, in situ observations are essential. Further, understanding the responses of microbes to the environmental processes driving their metabolism and interactions requires observing in a reference frame that moves with a plankton population drifting in ocean currents, i.e., Lagrangian. Here, we report the development and application of a system of coordinated robots for studying planktonic biological communities drifting within the ocean. The presented Lagrangian system uses three coordinated autonomous robotic platforms. The focal platformmore »consists of an autonomous underwater vehicle (AUV) fitted with a robotic water sampler. This platform localizes and drifts within a DCM community, periodically acquiring samples while continuously monitoring the local environment. The second platform is an AUV equipped with environmental sensing and acoustic tracking capabilities. This platform characterizes environmental conditions by tracking the focal platform and vertically profiling in its vicinity. The third platform is an autonomous surface vehicle equipped with satellite communications and subsea acoustic tracking capabilities. While also acoustically tracking the focal platform, this vehicle serves as a communication relay that connects the subsea robot to human operators, thereby providing situational awareness and enabling intervention if needed. Deployed in the North Pacific Ocean within the core of a cyclonic eddy, this coordinated system autonomously captured fundamental characteristics of the in situ DCM microbial community in a manner not possible previously.

    « less
  2. We report the development of novel fault detection and isolation (FDI) methods for model-based fault detection (MB-FD) and quotient-space fault isolation (QS-FI). This FDI approach performs MB-FD and QS-FI of single or multiple concurrent faults in plants and actuators simultaneously, without a priori knowledge of fault form, type, or dynamics. To detect faults, MB-FD characterizes deviation from nominal behavior using the plant velocity and plant and actuator parameters estimated by nullspace-based adaptive identification. To isolate (i.e. identify) faults, the QS-FI algorithm compares the estimated parameters to a nominal parameter class in progressively decreasing-dimensional quotient spaces of the parameter space. Amore »preliminary simulation study of these proposed FDI methods applied to a three degree-of-freedom uninhabited underwater vehicle plant model shows their ability to detect as well as isolate faults for the cases of both single and multiple simultaneous faults and suggests the generalizability of the MB-FD and QS-FI approaches to any well-defined second-order plant and actuator model whose parameters enter linearly: a broad class of systems which includes aerial vehicles, marine vehicles, spacecraft, and robot arms.« less
  3. The emerging sector of offshore kelp aquaculture represents an opportunity to produce biofuel feedstock to help meet growing energy demand. Giant kelp represents an attractive aquaculture crop due to its rapid growth and production, however precision farming over large scales is required to make this crop economically viable. These demands necessitate high frequency monitoring to ensure outplant success, maximum production, and optimum quality of harvested biomass, while the long distance from shore and large necessary scales of production makes in person monitoring impractical. Remote sensing offers a practical monitoring solution and nascent imaging technologies could be leveraged to provide dailymore »products of the kelp canopy and subsurface structures over unprecedented spatial scales. Here, we evaluate the efficacy of remote sensing from satellites and aerial and underwater autonomous vehicles as potential monitoring platforms for offshore kelp aquaculture farms. Decadal-scale analyses of the Southern California Bight showed that high offshore summertime cloud cover restricts the ability of satellite sensors to provide high frequency direct monitoring of these farms. By contrast, daily monitoring of offshore farms using sensors mounted to aerial and underwater drones seems promising. Small Unoccupied Aircraft Systems (sUAS) carrying lightweight optical sensors can provide estimates of canopy area, density, and tissue nitrogen content on the time and space scales necessary for observing changes in this highly dynamic species. Underwater color imagery can be rapidly classified using deep learning models to identify kelp outplants on a longline farm and high acoustic returns of kelp pneumatocysts from side scan sonar imagery signal an ability to monitor the subsurface development of kelp fronds. Current sensing technologies can be used to develop additional machine learning and spectral algorithms to monitor outplant health and canopy macromolecular content, however future developments in vehicle and infrastructure technologies are necessary to reduce costs and transcend operational limitations for continuous deployment in an offshore setting.« less
  4. We consider message-efficient continuous random sampling from a distributed stream, where the probability of inclusion of an item in the sample is proportional to a weight associated with the item. The unweighted version, where all weights are equal, is well studied, and admits tight upper and lower bounds on message complexity. For weighted sampling with replacement, there is a simple reduction to unweighted sampling with replacement. However, in many applications the stream may have only a few heavy items which may dominate a random sample when chosen with replacement. Weighted sampling without replacement (weighted SWOR) eludes this issue, since suchmore »heavy items can be sampled at most once. In this work, we present the first message-optimal algorithm for weighted SWOR from a distributed stream. Our algorithm also has optimal space and time complexity. As an application of our algorithm for weighted SWOR, we derive the first distributed streaming algorithms for tracking heavy hitters with residual error. Here the goal is to identify stream items that contribute significantly to the residual stream, once the heaviest items are removed. Residual heavy hitters generalize the notion of $\ell_1$ heavy hitters and are important in streams that have a skewed distribution of weights. In addition to the upper bound, we also provide a lower bound on the message complexity that is nearly tight up to a $łog(1/\eps)$ factor. Finally, we use our weighted sampling algorithm to improve the message complexity of distributed $L_1$ tracking, also known as count tracking, which is a widely studied problem in distributed streaming. We also derive a tight message lower bound, which closes the message complexity of this fundamental problem.« less
  5. Identifying dust aerosols from passive satellite images is of great interest for many applications. In this study, we developed five different machine-learning (ML) based algorithms, including Logistic Regression, K Nearest Neighbor, Random Forest (RF), Feed Forward Neural Network (FFNN), and Convolutional Neural Network (CNN), to identify dust aerosols in the daytime satellite images from the Visible Infrared Imaging Radiometer Suite (VIIRS) under cloud-free conditions on a global scale. In order to train the ML algorithms, we collocated the state-of-the-art dust detection product from the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) with the VIIRS observations along the CALIOP track. The 16more »VIIRS M-band observations with the center wavelength ranging from deep blue to thermal infrared, together with solar-viewing geometries and pixel time and locations, are used as the predictor variables. Four different sets of training input data are constructed based on different combinations of VIIRS pixel and predictor variables. The validation and comparison results based on the collocated CALIOP data indicate that the FFNN method based on all available predictor variables is the best performing one among all methods. It has an averaged dust detection accuracy of about 81%, 89%, and 85% over land, ocean and whole globe, respectively, compared with collocated CALIOP. When applied to off-track VIIRS pixels, the FFNN method retrieves geographical distributions of dust that are in good agreement with on-track results as well as CALIOP statistics. For further evaluation, we compared our results based on the ML algorithms to NOAA’s Aerosol Detection Product (ADP), which is a product that classifies dust, smoke, and ash using physical-based methods. The comparison reveals both similarity and differences. Overall, this study demonstrates the great potential of ML methods for dust detection and proves that these methods can be trained on the CALIOP track and then applied to the whole granule of VIIRS granule.« less