Many works [1]}, [2]}, [3]} have proven that multi-modal learning is capable of strengthening feature representation by actively relating the compositional information across different modalities such as image, text, and audio. Afterwards, various works utilize both point clouds and image data to improve the understanding of 3D vision, which are mostly targeted at 3D detection. Namely, AVOD [4]} and MV3D [5]} make region proposals by mapping the LiDAR point clouds to bird's eye view. Then Qi \(et\)  \(al.\)  [6]} and Wang \(et\)  \(al.\)  [7]} attempt to localize the points by proposing 2D regions with 2D CNN object detector and then transforming the corresponding 2D pixels to 3D space. Pointpainting [8]} projects the points into the image segmentation mask and detects 3D objects via LiDAR-based detector. Similarly, PI-RCNN [9]} employs semantic feature maps and makes predictions through point-wise fusion. 3D-CVF [10]} transfers the camera-view features using auto-calibrated projection and fuses the multi-modal features with gated fusion networks. More recently, some works [11]}, [12]} try to make use of multi-modal information in a self-supervised manner and transfer the learned representation to the downstream tasks.
