In this paper we present an on-manifold sequence-to-sequence learning approach to motion estimation using visual and inertial sensors. It is to the best of our knowledge the first end-to-end trainable method for visual-inertial odometry which performs fusion of the data at an intermediate feature-representation level. Our method has numerous advantages over traditional approaches. Specifically, it eliminates the need for tedious manual synchronization of the camera and IMU as well as eliminating the need for manual calibration between the IMU and camera. A further advantage is that our model naturally and elegantly incorporates domain specific information which significantly mitigates drift. We show that our approach is competitive with state-of-the-art traditional methods when accurate calibration data is available and can be trained to outperform them in the presence of calibration and synchronization errors.
VO algorithms estimate the incremental ego-motion of a camera. A traditional VO algorithm, illustrated in Fig. a., operates by extracting features in an image, matching the features between the current and successive images and then computing the optical flow. The motion can then be computed using the optical flow. The fast semi-direct monocular visual odometry (SVO) algorithm @cite_11 is an example of a state-of-the-art VO algorithm. It is designed to be fast and robust by operating directly on the image patches, not relying on slow feature extraction. Instead, it uses probabilistic depth filters on patches of the image itself. The depth filters are then updated through whole image alignment. This visual odometry algorithm is efficient and runs in real-time on an embedded platform. Its probabilistic formulation, however, makes it difficult to tune and it also requires a bootstrapping procedure to start the process. As expected, its performance depends heavily on the hardware to prevent tracking failures - typically global shutter cameras operating at higher than 50 fps needs to be used to ensure the odometry estimates remain accurate.
@cite_11
2910508725
We present a visual odometry (VO) algorithm for a multi-camera system and robust operation in challenging environments. Our algorithm consists of a pose tracker and a local mapper. The tracker estimates the current pose by minimizing photometric errors between the most recent keyframe and the current frame. The mapper initializes the depths of all sampled feature points using plane-sweeping stereo. To reduce pose drift, a sliding window optimizer is used to refine poses and structure jointly. Our formulation is flexible enough to support an arbitrary number of stereo cameras. We evaluate our algorithm thoroughly on five datasets. The datasets were captured in different conditions: daytime, night-time with near-infrared (NIR) illumination and nighttime without NIR illumination. Experimental results show that a multi-camera setup makes the VO more robust to challenging environments, especially night-time conditions, in which a single stereo configuration fails easily due to the lack of features.