Images-to-image registrations [1]}, [2]} are done in the \(\mathbb {P}^2\)  space because of the lack of depth information. This is usually the first step to the computation of the projective transformation or SfM. Typical methods are usually based on feature matching. A set of features such as SIFT [3]} or ORB [4]} are extracted from both source and target images. Correspondences are then established based on the extracted features, which can be used to solve for the rotation, translation using Bundle Adjustment [5]}, [6]}, Perspective-n-Point solvers [7]}, etc. Such techniques have been applied in modern SLAM systems [8]}, [9]}, [10]}. However, such methods are based on feature descriptors in the image modality to establish correspondences, and do not work for our general image-to-point cloud registration task.
