What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> A classical problem of imaging—the matting problem—is separation of a non-rectangular foreground image from a (usually) rectangular background image—for example, in a film frame, extraction of an actor from a background scene to allow substitution of a different background. Of the several attacks on this difficult and persistent problem, we discuss here only the special case of separating a desired foreground image from a background of a constant, or almost constant, backing color. This backing color has often been blue, so the problem, and its solution, have been called blue screen matting. However, other backing colors, such as yellow or (increasingly) green, have also been used, so we often generalize to constant color matting. The mathematics of constant color matting is presented and proven to be unsolvable as generally practiced. This, of course, flies in the face of the fact that the technique is commonly used in film and video, so we demonstrate constraints on the general problem that lead to solutions, or at least significantly prune the search space of solutions. We shall also demonstrate that an algorithmic solution is possible by allowing the foreground object to be shot against two constant backing colors—in fact, against two completely arbitrary backings so long as they differ everywhere. <s> BIB001 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> A common method for real-time segmentation of moving regions in image sequences involves "background subtraction", or thresholding the error between an estimate of the image without moving objects and the current image. The numerous approaches to this problem differ in the type of background model used and the procedure used to update the model. This paper discusses modeling each pixel as a mixture of Gaussians and using an on-line approximation to update the model. The Gaussian, distributions of the adaptive mixture model are then evaluated to determine which are most likely to result from a background process. Each pixel is classified based on whether the Gaussian distribution which represents it most effectively is considered part of the background model. This results in a stable, real-time outdoor tracker which reliably deals with lighting changes, repetitive motions from clutter, and long-term scene changes. This system has been run almost continuously for 16 months, 24 hours a day, through rain and snow. <s> BIB002 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> In this paper, we propose a novel approach for foreground layer extraction using flash/no-flash image pairs, which we call flash cut. Flash cut is based on the simple observation that only the foreground is significantly brightened by the flash and the background appearance change is very small, if the background is distant. Changes due to flash, motion, and color information are fused in an MRF framework to produce high quality segmentation results. Flash cut handles some amount of camera shake, and foreground motion, which makes it practical for anyone with a flash-equipped camera to use. We validate our approach on a variety of indoor and outdoor examples. <s> BIB003 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> Locating moving objects in a video sequence is the first step of many computer vision applications. Among the various motion-detection techniques, background subtraction methods are commonly implemented, especially for applications relying on a fixed camera. Since the basic inter-frame difference with global threshold is often a too simplistic method, more elaborate (and often probabilistic) methods have been proposed. These methods often aim at making the detection process more robust to noise, background motion and camera jitter. In this paper, we present commonly-implemented background subtraction algorithms and we evaluate them quantitatively. In order to gauge performances of each method, tests are performed on a wide range of real, synthetic and semi-synthetic video sequences representing different challenges. <s> BIB004 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> In this paper, we present Paint Selection, a progressive painting-based tool for local selection in images. Paint Selection facilitates users to progressively make a selection by roughly painting the object of interest using a brush. More importantly, Paint Selection is efficient enough that instant feedback can be provided to users as they drag the mouse. We demonstrate that high quality selections can be quickly and effectively "painted" on a variety of multi-megapixel images. <s> BIB005 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> Background subtraction algorithms define the background as parts of a scene that are at rest. Traditionally, these algorithms assume a stationary camera, and identify moving objects by detecting areas in a video that change over time. In this paper, we extend the concept of ‘subtracting’ areas at rest to apply to video captured from a freely moving camera. We do not assume that the background is well-approximated by a plane or that the camera center remains stationary during motion. The method operates entirely using 2D image measurements without requiring an explicit 3D reconstruction of the scene. A sparse model of background is built by robustly estimating a compact trajectory basis from trajectories of salient features across the video, and the background is ‘subtracted’ by removing trajectories that lie within the space spanned by the basis. Foreground and background appearance models are then built, and an optimal pixel-wise foreground/background labeling is obtained by efficiently maximizing a posterior function. <s> BIB006 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> In this paper, we propose a novel explicit image filter called guided filter. Derived from a local linear model, the guided filter computes the filtering output by considering the content of a guidance image, which can be the input image itself or another different image. The guided filter can be used as an edge-preserving smoothing operator like the popular bilateral filter [1], but it has better behaviors near edges. The guided filter is also a more generic concept beyond smoothing: It can transfer the structures of the guidance image to the filtering output, enabling new filtering applications like dehazing and guided feathering. Moreover, the guided filter naturally has a fast and nonapproximate linear time algorithm, regardless of the kernel size and the intensity range. Currently, it is one of the fastest edge-preserving filters. Experiments show that the guided filter is both effective and efficient in a great variety of computer vision and computer graphics applications, including edge-aware smoothing, detail enhancement, HDR compression, image matting/feathering, dehazing, joint upsampling, etc. <s> BIB007 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> With the invention of the low-cost Microsoft Kinect sensor, high-resolution depth and visual (RGB) sensing has become available for widespread use. The complementary nature of the depth and visual information provided by the Kinect sensor opens up new opportunities to solve fundamental problems in computer vision. This paper presents a comprehensive review of recent Kinect-based computer vision algorithms and applications. The reviewed approaches are classified according to the type of vision problems that can be addressed or enhanced by means of the Kinect sensor. The covered topics include preprocessing, object tracking and recognition, human activity analysis, hand gesture analysis, and indoor 3-D mapping. For each category of methods, we outline their main algorithmic contributions and summarize their advantages/differences compared to their RGB counterparts. Finally, we give an overview of the challenges in this field and future research trends. This paper is expected to serve as a tutorial and source of references for Kinect-based computer vision researchers. <s> BIB008 </s> What is the Truth: A Survey of Video Compositing Techniques <s> II. GENERATING THE MASK <s> In the film industry, many tricks have been employed using the integration of a 3D object with a real actor. Usually, attaching a 3D object with a real actor is a costly process because of the usage of an expensive motion capture system. This paper presents a system using a medium-cost motion capture system and a chroma-keying technique for generating a video footage of an actor with an integrated 3D object (e.g. amputated arm). The result of the proposed system shows the attaching process of different 3D objects with a real actor who is combined with a new background scene in the same viewpoint. <s> BIB009
For specifying the location of an object that is wanted to be composited with other footage, a mask is generated. This mask specifies the spatial location of the interesting pixels. This mask has the same dimensions of the original image. Each pixel in the mask has a value between zero (black) and one (white). Black pixels refer to that the corresponding pixel in the original footage is not intervention in the blending process. White pixels refer to that the corresponding pixel in the original footage is one of the interesting pixels. This mask is used on the blending step that is presented in Sec. IV. For generating the mask, many techniques were developed. Transparent images were introduced by Alvy Ray and Ed Catmull  . Alpha channel is used in image formats to express the transparency in images. In which, the alpha channel contains the mask that specifies which pixels on the RGB channels are opaque and which pixels are transparent. Alpha channel is used for straightforward blending using layer-based technique that is presented in Sec. IV. For computer generated footage, the mask that specifies the alpha channel of the footage is automatically generated with the original footage. Fig. 1 shows an example of the alpha channel of a computer generated image. The problem appears in the footages that do not contain the alpha channel such as real camera images or videos. Many techniques were presented for extracting the interesting objects from a given footage. Image matting techniques require interactions of the user by drawing scribbles that determine the areas of background objects, foreground objects, and the matte. Image matting techniques extract the alpha matte of a given image, such as Poisson matting [8] , closed form matting  , and guided filter BIB007 . Paint selection technique that was presented by J. Liu BIB005 facilitates extracting interesting objects from the given images using a brush. Flash Cut that was presented by J. Sun et al. BIB003 uses a pair of flash and no-flash images. In the flash image, the foreground objects are brightened more than the similar foreground objects that are in the no-flash images. While, the background is not affected by the flash. By using this observation, Flash Cut separates the foreground objects from a given pair of flash and no-flash images. Adobe Photoshop presents a set of tools for extracting an object from a given image such as Lasso tool, Polygonal Lasso tool, and Magnetic Lasso tool  . All of those techniques and tools give good results for extracting specific objects from a still image. For static objects and a static camera, extracting specific objects from a given video is the same as from a static image. However, for animated objects or a non-static camera, there are other techniques used for the extraction process. Many animated foreground extraction techniques were presented. In which, the binary mask is represented by the following equation. where B t (p) is the binary value of the binary mask of pixel p, D is the distance between the intensity value of the pixel p in frame F at time t and the corresponding pixel in the clear frame (Background frame) C, and T is a predefined threshold BIB004 . Where, the clear frame C is a frame that has no foreground objects. In Equ. 1, the problem is how to get the clear frame C. The traditional techniques use a reference frame without any object as a clear frame  . Those techniques are not sufficient because of illumination conditions cannot be the same in each frame as in the reference one BIB009 . Some methods model the clear frame using Gaussian distribution, such as in  . Other methods use multi-modal probability density functions (PDF) for modeling the clear frame BIB002 . Where, each pixel in the clear frame is modeled dynamically as a Mixture of Gaussians (MOG). P. Kelly et al. presented a technique for estimating background pixels to be used in the foreground extraction process. A new term d is used to represent the color/brightness difference value for detecting changes in shadow and lighting, as shown in the following equation where d E is the Euclidean distance between the intensity of the current pixel and the most common background pixel, V C is the brightness in the clear frame, and V i is the brightness of the pixel at the current frame. By using the new term d, any background pixel in the variable illumination regions is considered one of the background pixels. O. Barnich et al.  presented a new algorithm called (ViBe). ViBe adapts the background model using intensity values of a set of pixels that are located at the same location in the past frames. All of those techniques assume that the camera is a static one within the duration of modeling the background. Y. Sheikh et al. BIB006 reconstructed the scene using a sparse model of the background for handling the case of a non-static camera and background. However, most of the previous techniques for generating the mask of an animated object need special conditions and complex computations to achieve good results. On the other hand, chroma-keying techniques are considered the best solution for integrating any object into another background. A uniform color is used as a background during recording the interesting objects for extracting those objects from the footages. This color is red, green, or blue. However, the red color is very close to the skin's color of a human who is usually required to be separated from the scene for the integration purpose with another background. The blue channel is the noisiest channel and most of captured frame's details are in the red and green channels. channel is usually used by chroma-keying techniques  . In chroma-keying, there are two types of the green screen's illumination  , as shown in Fig. 2 . The first type is called "back-drop". In back-drop, the green screen is located away behind the interesting object. The interesting object's illumination is separated from the illumination of the green screen for avoiding the shadow of the object to be appearing on the green screen. The back-drop is used for simple tilt/pan shots. However, in the case of a free camera movement, backdrop is hard to be used. The second illumination type is called "cyc" that is used in the case of a free camera movement. In cyc, a set of soft lighting is used to illuminate the green stage uniformly. In chroma-keying, the important step is how to generate the binary mask that is called "matte". Luminance version for each frame is used for generating the matte  . For each frame, the luminance version L is driven from the following equation where R, G, and B are the red, green, and blue channels, respectively, see Fig. 3 . The generated luminance version is used for generating the matte using a predefined threshold. Where, each pixel in L smaller than the threshold is considered a green pixel. Another technique for generating the matte uses Euclidean distance for calculating the distance between each pixel with a predefined color, which is usually green. If the calculated distance smaller than a predefined threshold, this pixel is considered a green pixel. Color difference matte is another technique for generating the matte  by generating a raw matte r that is driven from the following equation Each pixel that is smaller than a predefined threshold in the generated raw matte r is considered a green pixel. Many techniques and systems were presented to improve the chroma-keying process   BIB001  . Many commercial software products introduce chroma-keying filters, such as Adobe After Effects  and Final Cut Pro  . Fig. 4 shows the generated masks using different chromakeying techniques. Nowadays, cheap depth sensors are available. Where, the RGB images become RGBD with a new channel for storing the depth information for each pixel. Microsoft Kinect sensor (http://www.microsoft.com/enus/kinectforwindows/) is one of the depth sensors. Microsoft Kinect captures the depth information using an InfraRed (IR) camera and IR projector. An IR laser is emitted from the IR projector. This IR laser is captured by the IR camera for getting the depth information using triangulation  . The captured depth values are represented as a depth frame. This depth frame can be used as a mask for separating objects that approximately have the same range of depth values from the camera. By comparing the foreground extraction process using RGB BIB008 . However, the cheap depth sensors have restrictions on the resolution of the captured frames. The generated mask is used for extracting the interesting object from the original footages to be used on the blending stage. This process is performed by multiplying the generated mask by the original frame.