Computer Science
Computer Science-2021.gz:143195
Computer Science
708799e5a18fc2f75e86ab1c72fee44a78052bb3
i3dLoc: Image-to-range Cross-domain Localization Robust to Inconsistent Environmental Conditions

We present a method for localizing a single camera with respect to a point cloud map in indoor and outdoor scenes. The problem is challenging because correspondences of local invariant features are inconsistent across the domains between image and 3D. The problem is even more challenging as the method must handle various environmental conditions such as illumination, weather, and seasonal changes. Our method can match equirectangular images to the 3D range projections by extracting cross-domain symmetric place descriptors. Our key insight is to retain condition-invariant 3D geometry features from limited data samples while eliminating the condition-related features by a designed Generative Adversarial Network. Based on such features, we further design a spherical convolution network to learn viewpoint-invariant symmetric place descriptors. We evaluate our method on extensive self-collected datasets, which involve \textit{Long-term} (variant appearance conditions), \textit{Large-scale} (up to $2km$ structure/unstructured environment), and \textit{Multistory} (four-floor confined space). Our method surpasses other current state-of-the-arts by achieving around $3$ times higher place retrievals to inconsistent environments, and above $3$ times accuracy on online localization. To highlight our method's generalization capabilities, we also evaluate the recognition across different datasets. With a single trained model, i3dLoc can demonstrate reliable visual localization in random conditions.


I. INTRODUCTION
Mobile robots and self-driving cars have entered our daily life in the recent years with the development of High-Definition maps-based accurate localization. Cameras have the huge potential to provide low-cost, compact and self-contained visual localization against point cloud maps. However, visual methods are inherently limited by inconsistent environmental conditions in the real world, e.g., illumination, weather, season and viewpoint differences. Whereas, accurate matching can be challenging to perform on point cloud data due to sensor sparsity with no sufficient texture feature guarantees. Transitional geometry-based methods [1] implicitly assume a static environment, such as stable lighting conditions, sunny weather, and fixed seasonal attributes. Recent learning-based visual localization methods are either constrained under limit environments [2] (structure road) or only fit for limited viewpoints [3] (forwards or backwards on the street). Current image-to-range localization methods are difficult to leverage in real-world applications, or can hardly address the above issues simultaneously.
To fill this gap, we propose i3dLoc -a 3D visual localization method with the assistance of offline 3D maps -invariant to environmental conditions and casual viewpoints. Compared Under an inconsistent environment, we aim to localize a 360 camera within a prior textureless point cloud map. Given the 360 image and initial position, we extract potential range projections, which we then evaluate using symmetric feature distance for 3D position estimation.
with Euclidean geometry features [1], we exploit the symmetric place descriptors between equirectangular visual inputs and range projections under condition invariant feature domain. As illustrated in Fig. 1, i3dLoc tackles the 3D visual localization task through two steps: (1) retaining geometry features from raw visual inputs which eliminates condition-related (day/night lighting conditions) factors; (2) extracting symmetric place descriptors even for inputs under vital viewpoint differences.
The major contributions of i3dLoc are: network performing on spherical convolution networks, where intrinsic characteristics of a spherical harmonica naturally help place descriptor matching under variant viewpoints. • We design an evaluation framework (includes Long-term, Large-scale and Multistory datasets), which can analyze the visual localization performance under significant environmental appearance changes, casual viewpoints and also the generalization ability for unseen datasets. In summary, i3dLoc provides a low-cost, condition-and viewpoint-invariant visual localization method for both indoor and outdoor large-scale environments. As demonstrated in the experimental results, i3dLoc outperforms the state-ofthe-art image-to-range visual localization methods [2] and significantly improves the localization accuracy of traditional visual SLAM methods [4]. When trained with all self-collected datasets under variant conditions, our method also shows great generalization ability for unseen indoor and outdoor environments, which makes it suitable for low-cost but robust localization for mobile robots.

II. RELATED WORKS
Visual localization has been well studied in the traditional SLAM framework; we refer to Lowry et.al [5] for an overview of approaches using cameras. The 3D point cloud based localization methods have also been well addressed [6,7,8] recently. Image-to-range localization based on consistent 3D maps and low-cost camera sensors have attracted more and more attention in recent years. Here we mainly concentrate on the related image-to-range localization approaches.
There are two main trends in image-to-range localization: the geometry based feature matching methods [9,1] and datadriven based visual localization methods [10,2]. Geometry features are usually carefully designed to bridge the description gap between image and point cloud domains and maintain the geometry's consistent nature. Xie et.al [9] propose to extract corner points of known calibration targets to maintain image-to-range feature connections. Compared with point features, line features in structured environments are more consistent for both image and range projection domains. Yu et.al [1] introduce an image-to-range registration method based on 2D and 3D line correspondences for place recognition.
Recent approaches have leveraged deep learning to develop data-driven visual localization frameworks that outperform classical methods both in accuracy and speed. In [2], Feng et.al propose an end-to-end deep network architecture to jointly learn the descriptors for 2D and 3D keypoints from images and point clouds. This method learns the cross-domain features through a weighted soft-margin triplet loss, while ignoring the underlying geometry connections between the two different domains. Sun et.al [11] introduce an image-to-range coarse localization method by building the feature connections within depth images, where the depth is estimated from a depth prediction network.
Contrary to the methods mentioned above, our method, i3dLoc exploits the more general image-to-range visual lo-calization solution. Similar to [2], we utilize a triplet-like loss to aggregate the learned 2D and 3D features into global descriptors. The major difference is that our method does not restrict the viewpoints and environmental conditions of visual inputs, which improves the generalization ability in real applications. Our method also show high robustness to unstructured environments, where corresponding features between image and range projections are hard to find.
III. METHODOLOGY As illustrated in Fig. 1, the main idea of i3dLoc is to find the corresponding place descriptors from 2D images and 3D range projections that are invariant to environmental conditions and viewpoints. To deal with the effects from appearance changes and viewpoint differences, i3dLoc mainly includes two modules: (1) a cross-domain transfer network to transform 2D equirectangular images into condition-invariant range projections; (2) a symmetric feature learning network to extract viewpoint-invariant descriptors even under casual viewpoints. To enable the end-to-end training, we design the domain transfer metric to enhance the domain adaptation, and the triplet-like learning metrics to bridge the feature similarity between 2D images and 3D range projections.

A. Cross-domain Transfer Learning
To generate constant geometry features from visual inputs under different environmental conditions, we construct a cross domain transfer network between 2D imagery and 3D range projections. Before we introduce the details, we will introduce the visual features from the point view of information entropy. Naturally, the condition-related feature Z C (illumination, weather, seasons) and invariant features Z G (geometry) in the image domain are tightly coupled. H(Z G , Z C |x) and I(Z G ; Z C |x) are the joint entropy and the mutual entropies conditioned on the given data samples x in the image domain. H(Z G |Z C , x) and H(Z C |Z G , x) are the conditional entropies based on Z C and Z G respectively. In the visual localization task, gathering visual data samples under all kinds of environmental conditions for each single area is difficult and time-consuming. To learn conditioninvariant place features with limited data samples, we mainly focus on the following three aspects: decreasing the joint entropy H(Z G , Z C |x), this approach can reduce the uncertainty between visual inputs and corresponding place features; improving the conditional entropy H(Z G |Z C , x), this can enhance place features capturing more non-geometry structures Z G from limited samples; reducing the mutual entropy I(Z G ; Z C |x), this can help separate the geometry features Z G from the non-geometry features Z C within the same visual distribution.
The joint entropy H p img (z|x) is intractable, since we can not access the data-label pair between the visual inputs x and their corresponding features {Z G , Z C }. Especially in the localization task, where each area has variants of visual appearances, i.e., there exists different kinds of combinations between Z G and Z C . An alternative approach is to optimize the upper bound of where KL is Kullback-Leibler divergence, which measures the distance of two data distributions. H p img θ 1 (log(F (z|x))) measures the uncertainty of the predicted place feature with a given sample data x. F (·) is the combination modules of q img φ1 and p lidar θ2 as depicted in Fig. 2, which can reconstruct the geometry features based on the original Z G . Since KL(·) ≥ 0, Eq. 1 can be rewritten as, where L Recon is the reconstruction loss between the original visual input x and the reconstructed imagex. The original To improve the conditional entropy H(Z G |Z C , x), we design a generative adversarial network (GAN) [12] for the where y in the first term is the real LiDAR projection, andŷ in the second term is the generated range predictions based on the estimated geometry features z G from the visual input x. As demonstrated by Goodfellow et.al [12], with iterative updating of the decoder and the discriminator module, GAN can pull the data distribution of generated data closer to the target data, and meanwhile improve the decoder's generalization ability. Finally, the mutual entropy I(z G ; z C |x) can be extended by, where, reducing the mutual entropy is equivalent to reducing the right-hand term in the above equation. We provide the paired data samples of visual images x and corresponding range projections y; both data can be extracted under the same viewpoint and y only contains the geometry features. The geometry feature distribution H(Z G |x) can be estimated from the encoder module {z G , z C } ∼ p img θ1 (z|x), and the ideal conditional geometry feature distribution is equal tô z G ∼ p lidar θ2 (z|y). We apply a soft triplet loss to reduce the feature difference between z G andẑ G , d(·) denotes the Euclidean distance and λ 1 is the hyperparameter to control the feature distance. Based on Eq. 2, 3 and 5, we can construct the conditional domain transfer module from visual inputs to range projections,

B. Symmetric Feature Learning
To learn viewpoint-invariant descriptors for both range predictions from visual inputs and range projections from 3D maps, we utilize the spherical harmonics to learn the place descriptors. As illustrated in Fig. 1, representing data in the spherical view is natural in 3D geometry analysis; however, it is difficult to adopt traditional convolution network in the spherical domain, because spaces between adjacent points in the spherical view are not uniform. Instead of traditional convolution, we apply the spherical convolution based on the property of spherical harmonics. Spherical convolution avoids space-varying distortions in the Euclidean space by convolving spherical signals in the harmonic domain. The mathematical model of spherical convolution into the harmonic domain shows its orientation-equivalent. Spherical convolution of SO(3) signals f and h (f, h are functions: where R, Q ∈ SO(3). As the proof in [13], spherical convolution is shown to be orientation-equivariant, where L Q (Q ∈ SO (3)) is a rotation operator for spherical signals. As depicted in Fig. 3, the convolution of two spherical signals f and h in the harmonics domain are computed by three steps. We first expand f and h to their spherical harmonic basis H f and H h , then compute the point-wise product of harmonic coefficients, and finally invert the spherical harmonic expansion. For more details, we suggest the reader refer to the original work in [14]. Intuitively, there exists spatial similarity in local outputs of spherical convolution. To leverage the viewpoint-invariant feature extraction, we utilize the VLAD layer [10], which can cluster the local features into the global place descriptors. With the assistance of our cross-domain transfer module, we can learn the conditional-and viewpoint-invariant place descriptors from visual inputs to match the static range projections.

C. Learning Metrics
To enable the end-to-end training for visual localization, we introduce triplet-like learning metrics to learn conditional-and viewpoint-invariant place descriptors. For the convenience of illustrating loss functions, we first describe the necessary definitions. The training tuple in both visual/LiDAR domains consists of four components: where S a is the spherical projections at the given position. {S rot } is a set of spherical representations manually rotated from {S a }, where the rotation angles are random sampled from ([0°, 30°, ...330°]). {S pos } denotes a set of spherical representations of 3D scans ("positive") whose distance to {S a } is within the threshold D pos , and {S neg } denotes a set of 3D scans ("negative") whose distance to {S a } is beyond D neg . In our applications, we set the threshold D pos = 5m and D neg = 20m. We construct paired tuples within visual domain S V and 3D map domain S L . Ideally, we want to minimize feature distances in both domains: is the function that encodes spherical representations into global descriptors by symmetric feature learning module, and d(·) denotes the Euclidean distance.
[.] + denotes the hinge loss, λ 2 and λ 3 are the constant thresholds to control the margins between the feature differences of different Euclidean distances. Meanwhile, we also define a domain learning metric to reduce the cross-domain feature differences: λ 4 and λ 5 are the constant thresholds to control the margins between the feature differences under visual/LiDAR domains. By combining the domain transfer metric L T ransf er and the above place learning metrics, the final joint learning metric can be written as, In our application, λ 1 , λ 2 and λ 4 is set to 0.5m and λ 3 and λ 5 is set to 1.0m. During the training procedure, we first train the domain transfer module with paired images and LiDAR projections; then we use the pertained transfer model for the conditional-and viewpoint-invariant place descriptors.

IV. EXPERIMENTS
In this section, we demonstrate the visual localization performance of i3dLoc on both indoor and outdoor datasets generated by our data collection platform as depicted in Fig. 4. In all experiments, we utilize a LiDAR device (Velodyne-VLP 16) and an omnidirectional camera (GoPro Max) mounting on the top of the payload, an inertial measurement unit (Xsense MTI 30, 0.5 • error in roll/pitch, 1 • error in yaw, 550mW), a mini PC (Intel NUC i7, 3.5 GHz, 28W) and an embedded GPU device (Nvidia Xavier, 8G memory). The network is trained on a GPU server with a single Nvidia 1080Ti GPU and 64G RAM. In the rest of this section, we detail the datasets, comparison methods, and evaluation metrics respectively. Then, we analyze qualitatively and quantitatively the performance The data-collection platform. This platform includes a LiDAR device, an 360 camera and an inertial measurement unit (IMU). We gathered the indoor and outdoor datasets via recording the raw data from above sensors. Since lacking the ground truth position in indoor or other GPS-denied environments, we use the LiDAR odometry outputs [15] as the ground truth estimation. of i3dLoc on place retrieval and online localization. Finally, we further discuss the current limitations and failure cases of i3dLoc.

Datasets. The training and evaluation dataset includes:
•  All the above datasets are collected by simultaneously holding the data-collection platform and recording LiDAR, IMU, and 360 images. To provide training/evaluation data, we first generate the global map with all the LiDAR sequences through a traditional LiDAR odometry method [15]. Since we are working in indoor and outdoor GPS-denied environments, we can not obtain ground truth position from a third-party system. In this paper, we use the LiDAR odometry estimation as the ground truth. Also, because the 3D offline map is generated using the same LiDAR odometry method, the standard division of relative ground truth noise is tiny. Based on this ground truth, we generate the paired images and 3D range projections. The range projections are generated by projecting points within 30m back to the keyframe. We resize the 2D images and 3D range projections to 64 × 64. Table. I shows the data splitting in the training and evaluation procedure into three different datasets. We evaluate the condition-invariant property on the Long-term dataset, which includes a different combination of environmental conditions. To investigate the viewpoint-invariant property, we generate the visual images on the same trajectory of different datasets but with casual viewpoint differences. And we also analyze the visual localization results when trained with non-rotated datasets but infer with the rotated datasets. Finally, we investigate the generalization ability by training one single model on the above three datasets and infer on unseen indoor and outdoor environments.

Evaluation Metrics and Methods
We first consider the place retrieval performance under changing conditions and viewpoints by comparing our method with several baseline methods, NetVLAD 1 [10], and 2D3DMatchNet [2]. We also combine our domain transfer module and NetVLAD, named i3d-Net, to compare with i3dLoc under viewpoint differences. For the 2D3DMatchNet, we manually imply their method for only position estimation without orientation estimation. Since all the above methods are image retrieval approaches that approximate the query's pose, we use the Average Recall at top 1% retrievals. The threshold for success retrieval distance is set to (10m). Please notes our localization evaluation does include position, but not orientation. All the above learningbased methods are trained under the same data configuration.
Secondly, we combine the visual localization and odometry for global localization. We investigate the online localization performance on the Large-scale dataset by evaluating the absolute pose error (APE) with Evaluation of Odometry tool 2 . We also compare with the pure visual odometry approach, ORB-SLAM 3 [4].

1) Condition-invariant Analysis:
We investigate conditioninvariant property on the Long-term and Multistory dataset. For fair comparison, all training/testing datasets are fed with fixed viewpoints. Fig. 5 shows the matching results of Longterm and Multistory datasets. The first column shows the raw equirectangular images from the 360 camera, and each image is taken from different lighting, season and weather conditions. Given the visual inputs, we analyze the matching results on fixed day-time conditions of different learning-based methods. The second column shows estimated geometry predictions by i3dLoc with the same domain-transfer module, where we can   notice that the predictions reduce the condition factors but keep the geometry outlines. The third column shows the range projections, which is generated by projecting the surrounding point cloud onto spherical view with the relative keyframe on the 3D map. We can see that the estimated range projections from different conditions share similiar geometric structures for both indoor and outdoor environments. The last three columns show the matching results of i3dLoc, NetVLAD and 2D3DMatchNet respectively. Compared to other methods, i3dLoc can provide reliable visual retrieval under variant environmental conditions. Table. II gives the quantitative analysis of average recall at top 1% retrievals of the different learning-based methods. To evaluate the generalization ability, except Long-term and Multistory datasets, we also analyze the visual localization results on the General datasets, which is the combination of Long-term and Multistory datasets, but with only 50% original training data. To further investigate the effects of our proposed domain transfer, we also compare the place retrieval performance under different combinations of loss metrics. The performance of i3dLoc (L reco ) has outperformed other learning-based baselines, and we can note that environmental conditions has significant effect on the place retrieval accuracy. However, only with reconstruct module has very limited improvements on the General dataset, which indicates that the learned domain transfer module has limited generalization ability on unseen datasets. When we combine L reco with the GAN module L G or with the mutual information module L M , the performance on General dataset can be further improved. Finally, with the complete cross-domain transfer module i3dLoc can outperform other learning-based methods on environments with complex environmental conditions, and also has higher generalization ability for unseen datasets.
2) Viewpoint-invariant Analysis: To evaluate the visual localization accuracy under variant viewpoints, we analyze the top 1% retrievals on different datasets with rotated datasets. Given the same trajectory, we provide the same visual inputs, but with range projections with different viewpoints. Since range projections are generated by projecting surrounding points onto the trajectory way points, thus we can generated different range projections by manually adding translation and orientation to the waypoints. As shown in the right three figures of Fig. 6, the y-axis represents the translation difference (from 0 ∼ 10m), x-axis represents the orientation difference (from 0 ∼ 180 • ). Here, we mainly show the comparison results with i3d-Net and 2D3DMatchNet. i3d-Net shows higher viewpoint-invariant property than 2D3DMatchNet, since the VLAD layer can extract an order-invariant place descriptor from local features. i3dLoc shows even higher robustness to all viewpoint differences, especially for orientations. Before  being sent to the VLAD layer, the extracted features from the symmetric learning module are orientation equivalent.
To investigate the generalization of the viewpoint invariant property on different datasets, in the left figure of Fig. 6, we also train/evaluate the performance among different datasets with rotated datasets. Here the General dataset is the combination of the other three datasets. We can note that when using the trained model from General dataset to infer others, i3dLoc can still provide reliable place retrievals. This indicates that our method has the potential to learn place descriptors for large-scale indoor and outdoor datasets at the same time.
We further analyze the average recall by training the networks with/without the rotated datasets. Based on the same cross-domain transfer module, we evaluate the performance between i3d-Net and i3dLoc. As we can see in Table. III, even when trained with non-rotated datasets, i3dLoc also has reliable average recall in both rotated and non-rotated evaluation. It indicates that our method, i3dLoc, has higher generalization ability when training only with limited viewpoints.

B. Online Localization Results
In this subsection, we further analyze the online localization performance by comparing both traditional visual odometry methods and learning-based approaches. We conduct this testing on the Large-scale dataset, and Fig. 7 shows the longest trajectory (2km) containing both structured and unstructured environments. The estimated trajectory from ORB-SLAM contains the accumulated odometry drift. Compared Fig. 7. The absolute position error analysis on Large-scale dataset. The first row shows the localization results on the 2km trajectory. The second row shows the image-to-range difference matrix, where x-and y-axis represent visual and located range features respectively. Red points mean the points with successful place retrieval. to other learning-based methods, i3dLoc can successfully provide reliable re-localization against the offline 3D point cloud map. Other learning-based methods can provide robust place retrieval at the early stage, but fail to find the correspondence in unstructured environments or aggressive viewpoints. As we can see in the enlarged map on Fig. 7, i3dLoc can successfully follow the aggressive trajectory while other methods failed. Compared to i3dLoc, i3d-Net is more sensitive to local viewpoints differences. We also plot the cosine feature differences between image queries (x-axis) and the matched range projections (y-axis), where the red points indicates successful retrievals (distance to ground truth within 10m). Table. IV analyzes the localization accuracy, GPU usage and inferencing time of different methods in online visual localization. Since the visual localization are combined with the same ORB-SLAM odometry, the localization accuracy of all learning-based methods are better than pure ORB-SLAM. i3dLoc surpasses all other learning-based methods, while consuming less GPU memory. The above property of i3dLoc makes it feasible to run on the embedded system (Nvidia Xavier) for low-cost robots in long-term SLAM and navigation tasks.

C. Discussion
In Fig. 8, we use the same pre-trained model of i3dLoc for online visual localization in both outdoor and Multistory indoor environments. To demonstrate the similarity among place descriptors, we cluster the extracted descriptors into 10 classes with each trajectory based on the K-means method. Places sharing with the similar geometric structures are clus-tered with the same labels. We can notice that the same areas fall into the same place classes. i3dLoc has the viewpointinvariant property, which helps it deal with aggressive viewpoint changes, such as place B in the outdoor environment, and place E, F and a G, H in indoor multistory environments. However, i3dLoc can only provide coarse localization results, and can not deal with places with continuous 3D geometry structures, i.e., long-corridor and indoor confined spaces.

V. CONCLUSIONS
This paper presents a novel image-to-range localization method, i3dLoc, under inconsistent environments. The advantage of i3dLoc is that it extracts condition-and viewpointinvariant features based on our cross-domain transfer learning module and our symmetric feature learning module. The experiments on long-term, large-scale, and indoor and outdoor environments demonstrate that our method can surpass both traditional visual SLAM methods and learning-based visual localization methods. We also evaluate our method's generalization ability for different environmental conditions and limited viewpoints, which indicates our method can provide reliable place retrieval when trained with both indoor and outdoor environments under variant conditions and viewpoints. It leaves us an interesting question: can we enable incremental place feature learning for robotics? In future work, we aim to provide an incremental place feature learning method to enable lifelong visual localization for real-world robots.