• RQ4: How to efficiently and robustly use limited resource communication channels to share information between a team of robots and between robots and operator? Utilizing an Autonomous Underwater Vehicle (AUV) to map an underwater structure presents the following dilemma. On one hand, if the AUV is close enough to map the observed details accurately, it is missing the big picture, i.e., in what direction is the most unknown part, and also, incremental SLAM generates drift. On the other hand, if the AUV is far enough to sense a large portion of the structure, details are missing due to the underwater sensing conditions. The proposed approach utilizes AUVs in two distinct roles: proximal observers are AUVs which operate near the underwater structure (see Fig. 1); distal observers are AUVs which operate at a distance, keeping a large portion of the structure together with the proximal observer(s) in their field of view; see Fig. 3a. Central to the proposed approach is the ability of the distal observers to detect and localize the proximal observers via a cooperative localization framework [1].
Wreck mapping has been studied using a variety of techniques all around the world. Photogrammetry of manually obtained images resulted in mosaics in Demesticha et al. [3], or from an ROV, see Nornes et al. [4]. While the Arrows EU project provides an overview of robotic technology use [5]. Menna et al. [6] provide a comprehensive review of techniques used. Mapping projects extend from Italy [7], Spain [8], Canada [9], Qatar [10], up to the arctic [11]. With the most famous wreck explorations of the Titanic [12] and the Antikythera [13] shipwrecks. Kurazume et al. [14], [15] first introduced the idea of localizing two robots based on mutual observations, termed it as cooperative positioning system, although in the literature it is most often termed as Cooperative Localization (CL) [1]. Dieudonné et al. [16] proved that CL with an arbitrary set of sensors is NP-hard, while Roumeliotis and Rekleitis [17] provided a theoretical analysis of factors that affect error growth in a CL system. Further studies have examined the performance [18], the sensing modalities [19], and consistency [20], proposing decentralized solutions [21] and integration with inertial sensors [22].
Active sensing methods for dense 3-D modeling have been reported in the literature, but none of them benefit from two or more tightly collaborating robots, such as the ones proposed here. Several approaches require enumerating and simulating sensing from discrete pose hypotheses in 6-D [23]- [27] incurring high computational cost; others are limited to 2-D slices of constant height or depth [28], [29] or require coarse initial models [30], [31]; while most operate on occupancy grids drastically reducing the resolution of the reconstructed surfaces. Exploration strategies [32], [33] that guide a robot towards frontier voxels without requiring sampling in pose space are closely related to our work, but they are limited to single robot, rely on a prior map, and do not model uncertainty. Multirobot 3-D reconstruction methods have been presented [34], but robots are assigned areas to map independently without tight cooperation.
The main idea of our approach is to have a team of co-robots collaborating with a human operator. There are two types of robots: proximal observers, which will operate close to the structure in order to produce an accurate map, and distal observers, which will be at distance maintaining the global picture of the structure and the pose of the proximal observer. Currently Cooperative Localization (see III-B); Motion strategies for navigating near obstacles (see III-C); Underwater State Estimation [35]; and photorealistic reconstruction (see III-E) have been explored, while, the remaining components of the proposed approach are still under investigation.
First, we need to address the problem of cooperative localization: its solution provides the relative pose between Distal and Proximal Observers. We recently proposed a deep learning based framework for detecting the relative pose between the two robots [2] from a single image.
To overcome the challenge of obtaining training data with accurate 6D poses, we utilize the Unreal Engine 4 to generate a rendered dataset by projecting the 3D model of the robot swimming over underwater images. However, these rendered images differ from real underwater images due to color loss and poor visibility quality in the underwater domain. Thus, we employ CycleGAN [36], an image-to-image translation network, to bridge the gap between rendered and real images, producing a synthetic dataset containing images that are closer in appearance to real underwater images. In effect, a CNN is trained on the synthetic dataset and then tested on real underwater images, as shown in Fig. 2.
We modified YOLOv3 [37] by adding a pose regression decoder, whereas the object detection decoder and backbone encoder remain unchanged. The pose regression decoder predicts projected keypoint locations of 8 corners corresponding to the 3D the model of the robot in an image, along with their confidence scores. An image is divided into grids, and each image patch corresponding to grid votes for the object detection box. Instead of using all the grid cells for 2D keypoint prediction, we select grid cells that fall inside the object bounding box, thus focusing on regions that belong to the robot. From these candidate keypoint predictions, the most dependable 2D keypoint candidates for each 3D keypoint are selected to yield a set of 2D-to-3D correspondences. More specifically, we select the 12 most confident keypoint predictions for each corner of the robot's 3D model. These selected 2D keypoints are used in the RANSAC-based PnP [38] algorithm to obtain a robust 6D pose estimate.
The proposed framework has been tested in different environments -pool, ocean -and different cameras including an Aqua2 robot and several GoPro cameras, demonstrating its robustness with respect to variations in underwater environment, camera intrinsics, and color calibration, Fig. 3. Moreover, we demonstrate better accuracy in terms of translation and orientation error on pool dataset compared to the state-of-the-art methods [39].
Our pose estimation framework might fail to produce a correct pose estimate sometimes when the bounding box detection is not accurate. Hence, relative pose of the robot from the observer can be determined accurately for each frame apart from failure cases. We are currently fusing other sensor data, e.g., IMU and pressure-based depth sensor to detect discrepancies in estimated pose across multiple frames and further improve the pose estimate.
The main goal of the proximal observer is to operate safely in proximity to the target structure and collect close observations. In the absence of prior information, the proximal observers will greedily explore the underwater structure. We plan to investigate two different directions:
1) Learning-based Exploration: A Deep Learning framework navigating the underwater robot to collect data in proximity to the structure is explored. Recently, we proposed a CNN framework [41] which is trained based on the way human divers collect data. In this framework, human operators annotated the training dataset, by marking the best direction of motion and orientation to guide the autonomous underwater robot for shipwreck coverage. Although due to the nature of the technique some limited unpredictable behavior is expected, we speculate that such techniques will avoid any issues that might arise from state estimation uncertainty.
2) Optimization-based Exploration: A path-optimizationbased technique is considered to navigate the robot near the structure maximizing visibility and information gain. Past work introduced AquaNav [42], a robust underwater navigation framework utilizing a path optimization planner that allows extensions in the form of cost functions. We will investigate the addition of novel cost functions maximizing visibility of target structures, determined on-line by a CNN similarly to [43]. Such a technique would require sufficient localization capabilities that could utilize a robust SLAM method such as SVIN2 [35], or can be provided locally by dead reckoning. We expect that although robust localization is needed, unlike the first approach, such a framework could ensure a predictable and safe performance with completeness, if paired with a high-level global planner. Preliminary results in simulation already show strong potential for this approach, as can be observed in Fig. 4.
The main objective of the distal observer is to track the proximal observers while simultaneously keeping in frame from a larger distance a bigger portion of the target structure. To achieve such behavior, the distal observers have to be capable to visually recognize and localize the proximal observers, predict their motions, visually recognize relevant segments of the target structure, and plan motions that satisfy the desired tracking objectives.
1) Proximal Observers Tracking: Prior work has already successfully addressed this issue for the Aqua2 underwater robot [2]. The method introduced a DNN technique trained on realistic simulated data that enabled an underwater robot using cameras to extract the relative position and orientation of another visible robot.
2) Structure Segment Recognition: Past research has already produced a CNN-based recognition method to corals [43]. We plan to investigate similar ways to recognise human-made underwater structures, such as shipwrecks, by employing DNNs trained on annotated datasets.
3) Motion Prediction: The distal observer has to produce motions that successfully track the proximal observer, thus the future positions of the tracked robot should be provided or extracted. This problem could be resolved either by utilizing a motion predictor combined with fast replanning, or, when allowed by the underwater communication conditions, the plans of the proximal observers could be shared directly to inform motion planning.
4) Navigation and Motion Planning: Given the map, areas of interest of the target underwater structure represented as segments, and a predicted trajectory of the proximal observer, the distal observer has to produce a sequence of motions that satisfy our objectives. Having already utilized AquaNav [42] -a powerful autonomous underwater navigation package for safe operation in proximity -similarly to the planning problem formulation of the proximal observers, we will focus on enhancing the optimization process with novel cost functions encouraging the visibility of the future positions of the tracked robot and the target structure at a desired distance.
Our goal in this project is to obtain photorealistic 3D models useful to the robots and their operators. Even though 3D point clouds may be sufficient for obstacle avoidance, they are not well suited for tasks such as visibility estimation or detection of geometric features such as openings and locations of high curvature. They are also a poor form of visualization for users to appreciate and understand underwater structures. We therefore aim to represent surface models using triangular meshes.
We propose a representation comprising a set of 3-D keypoints detected on the left (reference) image of the first stereo pair and reconstructed in 3-D via triangulation after they are matched on the right image. The keypoints are connected on the images to form triangles which are then lifted to 3-D after their vertices have been reconstructed. The proposed processing steps are as follows -see also Fig. 5, which shows a sketch of the approach.
1) Extract keypoints from the first image of the object and reconstruct them in 3-D. Harris corners or any other feature detection mechanism can be used here. 2) Apply 2D Delaunay triangulation on the keypoints on the image to obtain a mesh. 3) Select as active keypoint the one with the smallest expected reconstruction error still above the tolerance. 4) Compute the Next-Best View (NBV) for the active keypoint using the approach of Freundlich et al. [44], [45]. 5) Update position estimates and error covariance matrices for all visible keypoints based on new images and label as "explained" those with acceptable expected error. 6) Detect new keypoints in newly observed parts of the scene. 7) Insert as new keypoints the vertices that were generated due to triangle subdivision. 8) Attach new keypoints to the mesh and return to Step 3 unless all keypoints are explained. As the robot approaches a surface, triangles project onto larger areas in the image plane and new keypoints may be detected in their interior. When this occurs, the planarity of these triangles will be assessed and non-planar triangles will be subdivided to allow a tighter approximation of the surface. A classifier will be trained to make these decisions considering image appearance and reprojection errors computed by warping one image to another via the mesh. The output of this module will be a 3-D mesh comprising compact triangles, since their projections satisfy the Delaunay condition on some views, which will be texture-mapped by blending the input images with weights emphasizing fronto-parallel images observing the triangles at high resolution.
The underwater domain is a very harsh environment for communication, which is primarily based on acoustic devices characterized by very low data rate (tens of kilobits), relatively high packet loss, and distance-dependent performance [46]- [48]. This requires decisions over time on what information to share between robots and human operators.
We propose a hybrid representation comprising, on one hand, a set of 3-D keypoints and triangles, and, on the other hand, an octree-based occupancy grid [49]. The keypoints and triangles capture the details of the surface with the highest precision that has been achieved so far, while the octree naturally allows the hierarchical visualization and transmission of the current state of the estimated model.
Initially, a coarse level of the octree with a few bits per voxel, indicating whether it is free, occupied or unknown, will be transmitted. Information about voxels at finer levels will be transmitted subsequently, if their parent voxels have been subdivided. Additional information to give the operator a better view of the current surface estimate and the progress of the modeling process, such as 3-D reconstruction uncertainty within each voxel, will also be transmitted progressively. Finally, as the octree reaches a steady state and voxels at the finest resolution have been assigned stable labels, the list of keypoints and triangles will be transmitted. Note that each voxel is marked with the timestamp of the last transmission and last update, so that they can be possibly added back to the transmission list: a metric that identifies the change will determine the importance of retransmission, that can be used by the optimization method proposed in the following section.
We propose to optimize data transmission by performing a joint cross-layer optimization of the data representation at the application layer, and the operations of the communication network.
Our research on underwater structure mapping and inspection requires advanced state estimation, active sensing, and communications and will enable more robust situational awareness, autonomy, and robot coordination. The main contributions include the use of co-robots in tight cooperation that increases robustness and efficiency of the system inspecting an underwater structure by assigning different roles to the robots; an explicit cross-layer optimization for communication that improves the communication channel utilization under very limited bandwidth; a sensor data fusion approach that can be run in real time on-board the robot. These contributions will be deployed and demonstrated on inexpensive vehicles, such as the BlueROV2. The software developed to enable these operations will be open-sourced to facilitate research in the challenging field of marine robotics.
Taking a wider view, we aspire that our work will reduce the cost and risk of having divers perform exploration and mapping tasks, possibly under adverse conditions. We explicitly plan to rely on low-cost platforms to reduce the cost of underwater robotic operations. Reducing the cost and risk would enable broader mapping efforts leading to the discovery of historically important, valuable or otherwise interesting artifacts. Human divers will be engaged only after the robots have found promising evidence. Besides government entities and non-profit organizations, such as museums, the proposed research will benefit archaeologists who can monitor whether a recovery is needed for preservation as mandated by the UNESCO Convention on the Protection of the Underwater Cultural Heritage [50].