Estimating scene flow in RGB-D videos is attracting much interest of the computer vision researchers, due to its potential applications in robotics. The state-of-the-art techniques for scene flow estimation typically rely on the knowledge of scene structure of the frame and the correspondence between frames. However, with the availability of large RGB-D data captured from depth sensors, learning representations for estimation of scene flow has become possible. This paper introduces a first effort to apply a deep learning method for direct estimation of scene flow by presenting a fully convolutional neural network with an encoder-decoder (ED) architecture. The proposed network SceneEDNet involves estimation of three dimensional motion vectors of all the scene points from sequence of stereo images. The training for direct estimation of scene flow is done using consecutive pairs of stereo images and corresponding scene flow ground truth.
Motion field flow prediction in videos using CNNs, still remains an unsolved problem due to the complexity inherent into the videos for structure prediction. None of the deep learning based approaches for optical flow could produce better results compared to the state-of.the-art handcrafted techniques. Hence, mere extension of the optical flow prediction techniques to estimate scene flow, does not work well. However, the recent works in per-pixel prediction tasks using CNNs show the possibility that scene flow estimation can be performed by training a deep network on large dataset. Motivated by the recent efforts on optical flow estimation, we propose a technique to directly estimate the scene flow in x, y and z directions from the RGB-D videos, by introducing an encoder-decoder architecture. We apply the proposed method on FlyingThings3D @cite_22 dataset. Although the results of the proposed approach cannot outperform the state-of-the-art handcrafted techniques, however, this first attempt of CNN architecture for direct estimation of scene flow can be enhanced further to be used for dealing with large datasets.
@cite_22
2259424905
Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluation of scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.