To the best of our knowledge, 2D3D-MatchNet [1]} is the only prior work for general image-point cloud registration. It extracts images keypoints with SIFT [2]}, and point cloud keypoints with ISS [3]}. The image and point cloud patches around the keypoints are fed into each branch of a Siamese-like network and trained with triplet loss to extract cross-modal descriptors. At inference, it is a standard pipeline that consists of RANSAC-based descriptor matching and EPnP [4]} solver. Despite its greatly simplified experimental settings where the point clouds and images are captured at nearby timestamps with almost zero relative rotation, the low inlier rate of correspondences reveals the struggle for a deep network to learn common features across the drastically different modalities.
Another work [5]} establishes 2D-3D line correspondences between images and prior Lidar maps, but they requires accurate initialization, e.g., from a SLAM/Odometry system. In contrast, the general image-to-point cloud registration, including 2D3D-MatchNet [1]} and our DeepI2P do not rely on another accurate localization system. Some other works [7]}, [8]} focus on image-to-point cloud place recognition / retrieval without estimating the relative rotation and translation.
