• Achieving lower computational cost over existing approaches. We evaluate the proposed method on the KITTI dataset [4] and the Oxford RobotCar dataset [5]. We demonstrate the robustness of our method against drastic visual appearance changes across seasons as recorded in the RobotCar dataset, and show that it achieves higher accuracy and computational efficiency over existing methods. Further performance improvement is achieved by augmenting the LiDAR descriptor with image intensity information.

In the field of vSLAM, ORB-SLAM2 [6] is a recent development that demonstrates high accuracy and computational efficiency. In ORB-SLAM2, loop closure is detected by Bag-of-Words (BoW) using ORB features [7]. A vocabulary tree is used in BoW to speed up feature matching and subsequent place queries. However, if the features are highly repetitive (e.g., plants), BoW may fail; an example is given in Fig. 6. Similarly, LSD-SLAM [8] adopts FAB-MAP [9] for place recognition. Other than BoW, Fisher vectors [10] and VLAD [11] also focus on 2D features. On the other hand, global image descriptors are also used to decide the similarity between images for place recognition. GIST [12] is one example which encodes spatial layout properties (spatial frequencies) of the scene. It exhibits high accuracy if the viewing angle does not significantly change.

Recently, researchers adopted deep learning to place recognition and achieved impressive performance (e.g., NetVLAD [13] and [14]). NetVLAD trained a convolutional neural network to extract learned features and proposed a generalized VLAD layer to describe the image automatically. Their accuracy is promising but their computational cost is usually high so that they are not widely used in real-time vSLAM systems.

Neither BoW nor GIST is robust against visual appearance change, which is not ideal for long term (e.g., from summer to winter) vSLAM applications, in addition to being computationally expensive. In ORB-SLAM2, place recognition runs in a separate execution thread to achieve real-time performance. Direct vSLAM systems (e.g., [15], [16]) have become popular in the past decade, which achieve higher performance in certain scenarios. Adapting BoW into direct vSLAM systems is challenging because features are not selected with the goal of being matched across frames. In LSD-SLAM mentioned above, an additional set of features are detected and matched separately, which are used specifically for place recognition, at a higher computational cost.

In LDSO [17], the point selection strategy of its direct vSLAM system [15] is tuned to flavor features that can be matched across frames to enable BoW. Our proposed approach for place recognition, however, is more elegant for direct vSLAM systems if stereo cameras are available.

A number of 3D place recognition methods have been designed for RGB-D cameras or LiDAR sensors. RGB-D Mapping [18] uses ICP [19] to detect loop closure and RANSAC [20] to get an initial pose for ICP. For LiDAR, place recognition methods can be categorized into local descriptors and global descriptors. Local descriptors use a subset of the points and describe them in a local neighborhood. Examples are Spin Image [21] and SHOT [22]. Spin Image describes a keypoint by a histogram of points lying in each bin of a vertical cylinder centered at that keypoint. SHOT creates a sphere around a keypoint and describes that keypoint by the histogram of normal angles in each bin in the sphere. Global methods describe the entire set of points. These methods can be more computationally efficient. Recent development includes NDT [23], M2DP [24], Scan Context [25], and DELIGHT [26]. NDT classifies keypoints into line, plane, and sphere classes according to their neighborhoods. A histogram of these three classes is created to represent the point cloud. M2DP projects points onto multiple planes, and the histogram of point count in each bin on each projection plane is concatenated to get a signature of the point cloud. Scan Context aligns the point cloud to the vertical direction and represents it by the histogram of the maximal height of each bin on the horizontal plane. DELIGHT focuses on LiDAR intensity; the scan sphere is divided into 16 parts and the histogram of LiDAR intensity in each part is concatenated to represent the point cloud.

Cieslewski et. al. [27] looked into the possibility of using the 3D points triangulated from Structure-from-Motion or vSLAM for place recognition. They proposed the NBLD descriptor [27] for the 3D points from a vision-based system. A keypoint is described by its neighborhood points in a vertical cylinder. The point density of each bin in the cylinder is calculated and compared with neighborhoods to create a binary descriptor of that keypoint. Ye et. al. [28] extended NBLD with a neural network. The vertical cylinder of NBLD is created in the same way; however, a neural network is trained to describe the cylinders, instead of calculating the point density. These are novel approaches in adopting point cloud descriptors into vision-based systems for place recognition.

In this work, we adapt global LiDAR descriptors into stereo-visual odometry for robust and efficient place recognition under visual appearance change. Direct vSLAM systems can easily adopt the proposed approach for place recognition without modifying their point selection strategy.

Similar to the idea of [27], our method recognizes places based on the 3D points generated by visual odometry. The main difference is that the visual odometry in this work is running on stereo cameras. Specifically, we use SO-DSO [29] as our stereo-visual odometry for its high accuracy and