In this paper we present CMRNet, a realtime approach based on a Convolutional Neural Network to localize an RGB image of a scene in a map built from LiDAR data. Our network is not trained on the working area, i.e. CMRNet does not learn the map. Instead it learns to match an image to the map. We validate our approach on the KITTI dataset, processing each frame independently without any tracking procedure. CMRNet achieves 0.26m and 1.05deg median localization accuracy on the sequence 00 of the odometry dataset, starting from a rough pose estimate displaced up to 3.5m and 17deg. To the best of our knowledge this is the first CNN-based approach that learns to match images from a monocular camera to a given, preexisting 3D LiDAR-map.
The work presented in this paper has been inspired by Schneider al @cite_10 , which used 3D scans from a LIDAR and RGB images as the input of a novel CNN, RegNet. Their goal was to provide a CNN-based method for calibrating the extrinsic parameters of a camera a LIDAR sensor. Taking inspiration from that work, in this paper we propose a novel approach that has the advantages of both the categories described above. Differently from the aforementioned literature contribution, which exploits the data gathered from a synchronized single activation of a 3D LIDAR and a camera image, the inputs of our approach are a complete 3D LIDAR map of the environment, together with a single image and a rough initial guess of the camera pose. Eventually, the output consists of an accurate 6-DoF camera pose localization. It is worth to notice that having a single LIDAR scan taken at the same time as the image imply that the observed scene is exactly the same. In our case, instead, the 3D map usually depicts a different configuration, road users are not present, making the matching more challenging.
@cite_10
2963270286
This paper presents a method for calibrating the extrinsic and intrinsic parameters of a camera and a lidar scanner. The approach uses normalised mutual information to compare an image with a lidar scan. A camera model that takes into account orientation, location and focal length is used to create a 2D lidar image, with the intensity of the pixels dened by the angle of the normals in the lidar scan. Particle swarm optimisation is used to nd the optimal model parameters. The method is successfully validated in a natural environment with images collected by a hyperspectral and a 3D lidar scanner.