We describe a method to infer dense depth from camera motion and sparse depth as estimated using a visual-inertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to self-supervision,' measuring photometric consistency across time, forward-backward pose consistency, and geometric compatibility with the sparse point cloud. We also launch the first visual-inertial + depth dataset, which we hope will foster additional exploration into combining the complementary strengths of visual and inertial sensors. To compare our method to prior work, we adopt the unsupervised KITTI depth completion benchmark, and show state-of-the-art performance on it.
Rotation Parameterization To construct the photometric consistency loss during training, an auxiliary pose network is needed if no camera poses are available. While the translational part of the relative pose can be modeled as @math , the rotational part belongs to the special orthogonal group @math @cite_2 , which is represented by a @math matrix. @cite_3 uses quaternions, which require an additional norm constraint; this is a soft constraint imposed in the loss function, and thus is not guaranteed. @cite_14 @cite_5 @cite_29 use Euler angles which requires the composition of several matrices that may result in the rotation matrix to no longer be orthogonal. We use the exponential map on @math to map the output of the pose network to a rotation matrix. Though theoretically similar, we empirically found that the exponential map is more beneficial than the Euler angles in sec:experiments_results .
@cite_14 @cite_29 @cite_3 @cite_2 @cite_5
1977259876 2737630486 2754329383 2576915720
Parameterizing three degree-of-freedom (DOF) rotations is difficult to do well. Many graphics applications demand that we be able to compute and differentiate positions and orientations of articulated figures with respect to their rotational (and other) parameters, as well as integrate differential equations, optimize rotation parameters, and interpolate orientations. Widely used parameterizations such as Euler angles and quaternions are well suited to only a few of these operations. The exponential map maps a vector in R 3 describing the axis and magnitude of a three-DOF rotation to the corresponding rotation. Several graphics researchers have applied it with limited success to interpolation of orientations, but it has been virtually ignored with respect to the other operations mentioned above. In this paper we present formulae for computing, differentiating, and integrating three-DOF rotations with the exponential map. We show that our formulation is numerically stable in the face of machine precision issues, and that for most applications all singularities in the map can be avoided through a simple technique of dynamic reparameterization. We demonstrate how to use the exponential map to solve both the "freely rotating body" problem, and the important ball-and-socket joint required to accurately model shoulder and hip joints in articulated figures. Examining several common graphics applications, we explain the benefits of our formulation of the exponential map over Euler angles and quaternions, including robustness, small state vectors, lack of explicit constraints, good modeling capabilities, simplicity of solving ordinary differential equations, and good interpolation behavior. Convolutional Neural Networks (CNNs) have been applied to camera relocalization, which is to infer the pose of the camera given a single monocular image. However, there are still many open problems for camera relocalization with CNNs. We delve into the CNNs for camera relocalization. First, a variant of Euler angles named Euler6 is proposed to represent orientation. Then a data augmentation method named pose synthesis is designed to reduce spsarsity of poses in the whole pose space to cope with overfitting in training. Third, a multi-task CNN named BranchNet is proposed to deal with the complex coupling of orientation and translation. The network consists of several shared convolutional layers and splits into two branches which predict orientation and translation, respectively. Experiments on the 7Scenes dataset show that incorporating these techniques one by one into an existing model PoseNet always leads to better results. Together these techniques reduce the orientation error by 15.9 and the translation error by 38.3 compared to the state-of-the-art model Bayesian PoseNet. We implement BranchNet on an Intel NUC mobile platform and reach a speed of 43 fps, which meets the real-time requirement of many robotic applications. We present a novel method to fuse the power of deep networks with the computational efficiency of geometric and probabilistic localization algorithms. In contrast to other methods that completely replace a classical visual estimator with a deep network, we propose an approach that uses a convolutional neural network to learn difficult-to-model corrections to the estimator from ground-truth training data. To this end, we derive a novel loss function for learning SE(3) corrections based on a matrix Lie groups approach, with a natural formulation for balancing translation and rotation errors. We use this loss to train a deep pose correction network (DPC-Net) that predicts corrections for a particular estimator, sensor and environment. Using the KITTI odometry dataset, we demonstrate significant improvements to the accuracy of a computationally-efficient sparse stereo visual odometry pipeline, that render it as accurate as a modern computationally-intensive dense estimator. Further, we show how DPC-Net can be used to mitigate the effect of poorly calibrated lens distortion parameters. Translating or rotating an input image should not affect the results of many computer vision tasks. Convolutional neural networks (CNNs) are already translation equivariant: input image translations produce proportionate feature map translations. This is not the case for rotations. Global rotation equivariance is typically sought through data augmentation, but patch-wise equivariance is more difficult. We present Harmonic Networks or H-Nets, a CNN exhibiting equivariance to patch-wise translation and 360-rotation. We achieve this by replacing regular CNN filters with circular harmonics, returning a maximal response and orientation for every receptive field patch. H-Nets use a rich, parameter-efficient and fixed computational complexity representation, and we show that deep feature maps within the network encode complicated rotational invariants. We demonstrate that our layers are general enough to be used in conjunction with the latest architectures and techniques, such as deep supervision and batch normalization. We also achieve state-of-the-art classification on rotated-MNIST, and competitive results on other benchmark challenges.