Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> I. INTRODUCTION <s> Abstract This paper describes our approach to real-time detection of camera motion and moving object segmentation in videos acquired from moving cameras. As far as we know, none of the proposals reported in the literature are able to meet real-time requirements. In this work, we present an approach based on a color segmentation followed by a region-merging on motion through Markov Random Fields (MRFs). The technique we propose is inspired to a work of Gelgon and Bouthemy (Pattern Recognition 33 (2000) 725–40), that has been modified to reduce computational cost in order to achieve a fast segmentation (about 10 frame per second). To this aim a modified region matching algorithm (namely Partitioned Region Matching) and an innovative arc-based MRF optimization algorithm with a suitable definition of the motion reliability are proposed. Results on both synthetic and real sequences are reported to confirm validity of our solution. <s> BIB001 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> I. INTRODUCTION <s> Segmentation of objects in image sequences is very important in many aspects of multimedia applications. In second-generation image/video coding, images are segmented into objects to achieve efficient compression by coding the contour and texture separately. As the purpose is to achieve high compression performance, the objects segmented may not be semantically meaningful to human observers. The more recent applications, such as content-based image/video retrieval and image/video composition, require that the segmented objects be semantically meaningful. Indeed, the recent multimedia standard MPEG-4 specifies that a video is composed of meaningful video objects. Although many segmentation techniques have been proposed in the literature, fully automatic segmentation tools for general applications are currently not achievable. This paper provides a review of this important and challenging area of segmentation of moving objects. We describe common approaches including temporal segmentation, spatial segmentation, and the combination of temporal-spatial segmentation. As an example, a complete segmentation scheme, which is an informative part of MPEG-4, is summarized. <s> BIB002 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> I. INTRODUCTION <s> The subspace segmentation problem is fundamental in many applications. The goal is to cluster data drawn from an unknown union of subspaces. In this paper we state the problem and describe its connection to other areas of mathematics and engineering. We then review the mathematical and algorithmic methods created to solve this problem and some of its particular cases. We also describe the problem of motion tracking in videos and its connection to the subspace segmentation problem and compare the various techniques for solving it. <s> BIB003 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> I. INTRODUCTION <s> A major difference between amateur and professional video lies in the quality of camera paths. Previous work on video stabilization has considered how to improve amateur video by smoothing the camera path. In this paper, we show that additional changes to the camera path can further improve video aesthetics. Our new optimization method achieves multiple simultaneous goals: 1) stabilizing video content over short time scales; 2) ensuring simple and consistent camera paths over longer time scales; and 3) improving scene composition by automatically removing distractions, a common occurrence in amateur video. Our approach uses an $L^{1}$ camera path optimization framework, extended to handle multiple constraints. Two passes of optimization are used to address both low-level and high-level constraints on the camera path. The experimental and user study results show that our approach outputs video that is perceptually better than the input, or the results of using stabilization only. <s> BIB004 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> I. INTRODUCTION <s> Interactive image segmentation aims at classifying the image pixels into foreground and background classes given some foreground and background markers. In this paper, we propose a novel framework for interactive image segmentation that builds upon graph-based manifold ranking model, a graph-based semi-supervised learning technique which can learn very smooth functions with respect to the intrinsic structure revealed by the input data. The final segmentation results are improved by overcoming two core problems of graph construction in traditional models: graph structure and graph edge weights. The user provided scribbles are treated as the must-link and must-not-link constraints. Then we model the graph as an approximatively k-regular sparse graph by integrating these constraints and our extended neighboring spatial relationships into graph structure modeling. The content and labels driven locally adaptive kernel parameter is proposed to tackle the insufficiency of previous models which usually employ a unified kernel parameter. After the graph construction, a novel three-stage strategy is proposed to get the final segmentation results. Due to the sparsity and extended neighboring relationships of our constructed graph and usage of superpixels, our model can provide nearly real-time, user scribble insensitive segmentations which are two core demands in interactive image segmentation. Last but not least, our framework is very easy to be extended to multi-label segmentation, and for some less complicated scenarios, it can even get the segmented object through single line interaction. Experimental results and comparisons with other state-of-the-art methods demonstrate that our framework can efficiently and accurately extract foreground objects from background. <s> BIB005 </s> Dependent Motion Segmentation in Moving Camera Videos: A Survey <s> I. INTRODUCTION <s> Extracting background features for estimating the camera path is a key step in many video editing and enhancement applications. Existing approaches often fail on highly dynamic videos that are shot by moving cameras and contain severe foreground occlusion. Based on existing theories, we present a new, practical method that can reliably identify background features in complex video, leading to accurate camera path estimation and background layering. Our approach contains a local motion analysis step and a global optimization step. We first divide the input video into overlapping temporal windows, and extract local motion clusters in each window. We form a directed graph from these local clusters, and identify background ones by finding a minimal path through the graph using optimization. We show that our method significantly outperforms other alternatives, and can be directly used to improve common video editing applications such as stabilization, compositing and background reconstruction. <s> BIB006
Motion segmentation refers to the process of decomposing a video (or an image sequence) into background and moving objects. It is a fundamental step for many computer vision and pattern recognition tasks, especially for video based applications such as video coding, video surveillance, activity recognition, content-based retrieval, visualization, robotics, traffic monitoring and so on BIB001 . Extensive studies have been conducted on motion segmentation for stationary camera scenarios. However, little effort has been put on the review of dependent motion segmentation methods. Recently, with the development of first-person view devices and tecnnologies, more and more researchers have focused on the researches of motion segmentation in dynamic scenarios, which are performed on moving camera videos. Dependent motion segmentation is an essential part to many applications. It is because for many situations, we only have interest on the camera motion, such as video stabilization, structurefrom-motion, video editing BIB004 , BIB006 etc.; but for other applications, we just focus on object motions, such as action recognition, content-based retrieval, activity recognition etc. Thus, we need to separate between the camera motion and object motions for further applications. Moving camera videos usually refer to the videos taken by a moving camera in the 3D real world. A common theme in moving camera motion segmentation is to use image plane motion (optical flow) or trajectories as a surrogate for realworld object motion BIB002 - BIB003 . Image plane motion can be used directly as a cue for clustering or to compensate for the camera motion so that the pixelwise model from the previous frame can be adjusted in order to remain accurate. Motion segmentation in stationary camera videos is relatively straightforward, where a pixelwise background model can accurately classify pixels as background or foreground. However, separating moving objects from moving camera videos is more challenging. In moving camera videos, the camera motion causes most image pixels to move, where pixelwise models become inadequate. Moreover, since the camera motion and object motions are mixed and dependent with each other, this produces motion dependence between the camera motion and object motions. The observed 2D image motion is generated by the 3D motions of both moving camera and moving objects in the scene. The mixture and motion dependence make the motion segmentation in moving camera video much more challenging. In BIB005 , motion segmentation for dynamic scenarios is also named as 'dependent motion segmentation'. It is a very challenging task in computer vision especially for the scenes generated by unconstrained and unpredictable camera motions. A large number of works have been proposed for the motion segmentation in moving camera videos. However, the performance of these techniques are still fall behind human perceptions. In this paper, we present a review on the most important and latest developed motion segmentation algorithms for moving camera videos. We categorize these methods into different groups according to their principles and features. We also give the detailed explanation of their strengths and weaknesses and provide possible suggestions on new directions for dependent motion segmentation. The organization of this paper is as follows: Section 2 provides an overview of the problem of dependent motion segmentation with common issues and main features. According to these issues and features, a classification of existing dependent motion segmentation techniques is proposed in this paper. Section 3 introduces the principles of motion flow modeling in both 2D and 3D scenes. In Section 4, the main ideas of recent developed dependent motion segmentation methods are described and analyzed. Their strengths and weaknesses are also analyzed in this section. Finally, this paper summarizes each category of methods and suggests new further research directions in Section 5.