Depth perception plays a crucial role in computer vision, enabling it to tackle tasks such as autonomous driving, object manipulation, and more. The 3D structure of a sensed environment can be inferred through passive and active sensing technologies. The former has been deployed for decades using stereo [1]}, [2]}, structure-from-motion [3]} or multi-view-stereo [4]}. Each of these methods has its flaws and constraints. For instance, stereo depth perception requires two calibrated cameras and struggles where the scene lacks texture.
On the other hand, active sensing relies on specialized sensors, and in the case of LiDARs (Light Detection And Ranging), flooding the scene with a laser beam and computing the distance of each point by measuring the traveling time of the ray. Despite being accurate, LiDARs struggle, for instance, when sensing not Lambertian surfaces due to multi-path interference or subsurface scattering. Moreover, the resulting point cloud is sparse and not coupled with any visual information. Thus, it is common to jointly use it with a standard camera and project the LiDAR point cloud over the camera image plane, resulting in a sparse depth map. However, this procedure raises a fundamental issue due to the different points of view of the two devices and the intrinsic sparsity of the LiDAR's output. Specifically, it leads to wrong depth values in the final RGB-D image, as shown in Figure [fig:teaser]1. Therefore, LiDAR depth map filtering is a serious problem to be tackled.
