Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> b: DEEP STRUCTURE METHODS <s> This paper introduces an approach to accomplish motion segmentation from a moving stereo camera based on deep learning. Previous work on moving object detection mostly use point features based on 3D geometric constraints. However, point features require good features, and are hard to detect or to be matched correctly in situations where objects have smooth textures. To alleviate this problem, learning high-level spatio-temporal features unsupervisedly from raw image data based on Reconstruction Independent Component Analysis (RICA) autoencoders is proposed. Despite the power of the new spatio-temporal features, these features cannot not learn and be used to interpret 3D geometry of dynamic scenes, which is critical for moving object detection from moving cameras. As detected moving points based on 3D geometric constraints still contain valuable information of 3D scene as well as the camera egomotion, we propose a framework that incorporates both the detected moving point results and the learned spatio-temporal features as inputs to Recursive Neural Networks (RNN) that performs motion segmentation. Both features effectively complement each other. The proposed approach is demonstrated with real-world stereo video data that contains multiple moving objects, and has achieved 26% better detection rate over the existing 3D geometric-based moving points detector. <s> BIB001 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> b: DEEP STRUCTURE METHODS <s> We segment moving objects in videos by ranking spatio-temporal segment proposals according to “moving objectness”; how likely they are to contain a moving object. In each video frame, we compute segment proposals using multiple figure-ground segmentations on per frame motion boundaries. We rank them with a Moving Objectness Detector trained on image and motion fields to detect moving objects and discard over/under segmentations or background parts of the scene. We extend the top ranked segments into spatio-temporal tubes using random walkers on motion affinities of dense point trajectories. Our final tube ranking consistently outperforms previous segmentation methods in the two largest video segmentation benchmarks currently available, for any number of proposals. Further, our per frame moving object proposals increase the detection rate up to 7% over previous state-of-the-art static proposal methods. <s> BIB002 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> b: DEEP STRUCTURE METHODS <s> As an important and challenging topic in computer vision, moving object detection is typically posed as a problem of scene analysis, which usually requires both robust feature description and effective statistical modeling. In general, the conventional detection methods make use of pre-defined hand-crafted features and sophisticated background models for scene analysis. Therefore, they usually have a low generalization capability of adapting to different scenes with diverse spatio-temporal motion information. In the face of high-definition video data, sophisticated statistical modeling often suffers from an expensive computation or memory cost because of its low efficiency in evaluation and parallelization. In order to address this issue, we propose a deep learning based block-wise scene analysis method equipped with a binary spatio-temporal scene model. Based on the stacked denoising autoencoder, the deep learning module of the proposed method aims to learn an effective deep image representation encoding the intrinsic scene information, which leads to the robustness of feature description. Furthermore, the proposed binary scene model captures the spatio-temporal scene distribution information in the Hamming space, which ensures the high efficiency of moving object detection. Experimental results on several datasets demonstrate the effectiveness and efficiency of the proposed method. <s> BIB003 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> b: DEEP STRUCTURE METHODS <s> Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos. <s> BIB004 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> b: DEEP STRUCTURE METHODS <s> This paper tackles the task of semi-supervised video object segmentation, i.e., the separation of an object from the background in a video, given the mask of the first frame. We present One-Shot Video Object Segmentation (OSVOS), based on a fully-convolutional neural network architecture that is able to successively transfer generic semantic information, learned on ImageNet, to the task of foreground segmentation, and finally to learning the appearance of a single annotated object of the test sequence (hence one-shot). Although all frames are processed independently, the results are temporally coherent and stable. We perform experiments on two annotated video segmentation databases, which show that OSVOS is fast and improves the state of the art by a significant margin (79.8% vs 68.0%). <s> BIB005
The deep structure based methods usually builds a hierarchical learning model of multiple layers for recognition, classification, segmentation, prediction, etc. Although they may require more training data and need higher computational cost than shallow structure methods, their performances are usually better than shallow structure based methods. Recently, with the high development of deep learning models, many different kinds of deep structure based motion segmentation methods have been proposed. For example, Fragkiadaki et al. BIB002 developed a Convolutional Neural Network (CNN) based 'Moving Objectness Detector' for moving object segmentation. In each frame, they compute segment proposals according to this 'Moving Objectness Detector'. They segment moving objects in dynamic scenarios by ranking spatio-temporal segment proposals and discard under/over segmentations or background parts of the scene. Lin and Wang BIB001 proposed a Recursive Neural Networks (RNN) based deep learning framework to accomplish motion segmentation in moving stereo camera videos. This framework first detects moving points and learns highlevel spatio-temporal features from raw images based on Reconstruction Independent Component Analysis (RICA). Then it incorporates both the detected moving points and the learned spatio-temporal features as inputs of RNN to perform motion segmentation. The pipeline of the framework is shown in Fig.6 . Zhang et al. BIB003 developed a deep learning based block-wise scene analysis method for moving object detection, equipped with a binary spatio-temporal scene model. The deep learning module, which is based on denoising autoencoder, aims at learning effective deep representation to encode the intrinsic scene information. The binary scene model, which captures the spatio-temporal scene distribution in the Hamming space, ensures the robust moving object detection. Sevilla-Lara et al. BIB004 introduced a CNN based semantic segmentation algorithm with localized layers in the optical flow field. This method first learns the types of objects in the scene, defines different image motion models depending on the type of object, and then segments the scene into objects of different types. Murali et al.  proposed a Transition State Clustering with Deep Learning (TSCDL) algorithm, which is a novel unsupervised algorithm leveraging video and kinematic data for task-level segmentation in robot-assisted minimally invasive surgery applications. Caelles et al. BIB005 deal with the task of semi-supervised object segmentation in one-shot videos, e.g. the segmentation of an object from the background part of a video, given the mask of the first frame. That is, they develop a CNN based OneShot Video Object Segmentation (OSVOS) approach, which is able to transfer generic semantic information learned from ImageNet successively, to the part of foreground segmentation, and then learn the appearance of an annotated object in the test sequence (and thus one-shot) BIB005 .