Detecting Faces in Images : A Survey <s> Skin Color <s> A new method was introduced to detect skin color and facial pattern regions from NTSC television pictures. About 4000 skin colors in the TV picture were measured and the chromaticities r, g of those colors were calculated. A skin color chart was made from measured regions and color transformation matrix was calculated for printing from a TV picture <s> BIB001 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> This paper describes a new approach to the integration and control of continuously operating visual processes. Visual processes are expressed as transformations which map signals from virtual sensors into commands for devices. These transformations define reactive processes which tightly couple perception and action. Such transformations may be used to control robotic devices, including fixation an active binocular head, as well as the to select and control the processes which interpret visual data. <s> BIB002 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> The authors present a real-time face tracker. The system has achieved a rate of 30+ frames/second using an HP-9000 workstation with a frame grabber and a Canon VC-Cl camera. It can track a person's face while the person moves freely (e.g., walks, jumps, sits down and stands up) in a room. Three types of models have been employed in developing the system. First, they present a stochastic model to characterize skin color distributions of human faces. The information provided by the model is sufficient for tracking a human face in various poses and views. This model is adaptable to different people and different lighting conditions in real-time. Second, a motion model is used to estimate image motion and to predict the search window. Third, a camera model is used to predict and compensate for camera motion. The system can be applied to teleconferencing and many HCI applications including lip reading and gaze tracking. The principle in developing this system can be extended to other tracking problems such as tracking the human hand. <s> BIB003 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> There are many applications where it is desirable to segment a video image into regions defined by color. Among these are the recognition of gesture from the image (as opposed to instrumented gloves), facial expression and orientation, and video teleconferencing. In these examples, the important elements of the images are human hands and face, which share common skin coloration of the subject. This paper describes an approach to the identification of skin-colored regions of the image that is robust in terms of variations in skin pigmentation in a single subject, differences in skin pigmentation across a population of potential users, and subject clothing and image background. The paper also discusses the potential for being robust over a wide range of illuminating conditions. <s> BIB004 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> This paper describes the techniques used to separate the hand from a cluttered background in a gesture recognition system. Target colors are identified using a histogram-like structure called a Color Predicate, which is trained in real-time using a novel algorithm. Running on standard PC hardware, the segmentation is of sufficient speed and quality to support an interactive user interface. The method has shown its flexibility in a range of different office environments, segmenting users with many different skin-tones. Variations have been applied to other problems including finding face candidates in video sequences. <s> BIB005 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> Recognition of human faces out of still images or image sequences is a research field of fast increasing interest. At first, facial regions and facial features like eyes and mouth have to be extracted. In the present paper we propose an approach that copes with problems of these first two steps. We perform face localization based on the observation that human faces are characterized by their oval shape and skin-color, also in the case of varying light conditions. For that we segment faces by evaluating shape and color (HSV) information. Then face hypotheses are verified by searching for facial features inside of the face-like regions. This is done by applying morphological operations and minima localization to intensity images. <s> BIB006 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> The authors present a new approach for automatically segmentation and tracking of faces in color images. Segmentation of faces is performed by evaluating color and shape information. First, skin-like regions are determined based on the color attributes hue and saturation. Then regions with elliptical shape are selected as face hypotheses. They are verified by searching for facial features in their interior. After a face is reliably detected it is tracked over time. Tracking is realized by using an active contour model. The exterior forces of the snake are defined based on color features. They push or pull snaxels perpendicular to the snake. Results for tracking are shown for an image sequence consisting of 150 frames. <s> BIB007 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> This paper proposed a new method for the full face detection from complex backgrounds. Based on feature parameters of space gray-level dependence (SGLD) matrix, the face-texture model composed by a set of inequalities was derived. A color information utilization incorporated with the face-texture feature was also investigated. Using the face-texture model, we designed a kind of scanning scheme for face detection in color scenes, in which the orange-like parts including the face areas were enhanced by utilizing the I component of the YIQ color system. The experiments showed that this method can locate the face position in the complex backgrounds effectively. <s> BIB008 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> A real-time system is described for automatically detecting, modeling and tracking faces in 3D. A closed loop approach is proposed which utilizes structure from motion to generate a 3D model of a face and then feed back the estimated structure to constrain feature tracking in the next frame. The system initializes by using skin classification, symmetry operations, 3D warping and eigenfaces to find a face. Feature trajectories are then computed by SSD or correlation-based tracking. The trajectories are simultaneously processed by an extended Kalman filter to stably recover 3D structure, camera geometry and facial pose. Adaptively weighted estimation is used in this filter by modeling the noise characteristics of the 2D image patch tracking technique. In addition, the structural estimate is constrained by using parametrized models of facial structure (eigen-heads). The Kalman filter's estimate of the 3D state and motion of the face predicts the trajectory of the features which constrains the search space for the next frame in the video sequence. The feature tracking and Kalman filtering closed loop system operates at 25 Hz. <s> BIB009 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> Visual processes to detect and track faces for video compression and transmission. The system is based on an architecture in which a supervisor selects and activates visual processes in cyclic manner. Control of visual processes is made possible by a confidence factor which accompanies each observation. Fusion of results into a unified estimation for tracking is made possible by estimating a covariance matrix with each observation. Visual processes for face tracking are described using blink detection, normalised color histogram matching, and cross correlation (SSD and NCC). Ensembles of visual processes are organised into processing states so as to provide robust tracking. Transition between states is determined by events detected by processes. The result of face detection is fed into recursive estimator (Kalman filter). The output from the estimator drives a PD controller for a pan/tilt/zoom camera. The resulting system provides robust and precise tracking which operates continuously at approximately 20 images per second on a 150 megahertz computer workstation. <s> BIB010 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> We describe a face modeling system which estimates complete facial structure and texture from a real-time video stream. The system begins with a face trading algorithm which detects and stabilizes live facial images into a canonical 3D pose. The resulting canonical texture is then processed by a statistical model to filter imperfections and estimate unknown components such as missing pixels and underlying 3D structure. This statistical model is a soft mixture of eigenfeature selectors which span the 3D deformations and texture changes across a training set of laser scanned faces. An iterative algorithm is introduced for determining the dimensional partitioning of the eigenfeatures to maximize their generalization capability over a cross-validation set of data. The model's abilities to filter and estimate absent facial components are then demonstrated over incomplete 3D data. This ultimately allows the model to span known and regress unknown facial information front stabilized natural video sequences generated by a face tracking algorithm. The resulting continuous and dynamic estimation of the model's parameters over a video sequence generates a compact temporal description of the 3D deformations and texture changes of the face. <s> BIB011 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> In this paper, we present visual tracking techniques for multimodal human computer interaction. First, we discuss techniques for tracking human faces in which human skin-color is used as a major feature. An adaptive stochastic model has been developed to characterize the skin-color distributions. Based on the maximum likelihood method, the model parameters can be adapted for different people and different lighting conditions. The feasibility of the model has been demonstrated by the development of a real-time face tracker. The system has achieved a rate of 30-t- frames/second using a low-end workstation with a framegrabber and a camera. We also present a top-down approach for tracking facial features such as eyes, nostrils, and lip comers. These real-time visual tracking techniques have been successfully applied to many applications such as gaze tracking, and lipreading. The face tracker has been combined with a microphone array for extracting speech signal from a specific person. The gaze tracker has been combined with a speech recognizer in a multimodal interface for controlling a panoramic image viewer. <s> BIB012 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> This paper proposes an object oriented face detection method using range and color information. Objects are segmented from the background using the stereo disparity histogram that represents the range information of the objects. A matching pixel count (MPC) disparity measure is introduced to enhance the matching accuracy, and removes the effect of unexpected noise in the boundary region. For the high-performance implementation of the MPC disparity histogram, redundancy operations inherent to the area-based search operation are removed. To detect facial regions among segmented objects, a skin-color transform technique is used with the generalized face color distribution (GFCD) modeled by a 2D Gaussian function in a normalized color space. Using GFCD, the input color image can be transformed into a gray-level image enhancing only facial color components. To detect facial information only in the defined range, both results from range segmentation and color transforms are combined effectively. The experimental results show that the proposed algorithm works well in various environments with multiple human objects. Moreover, the processing time for a test image is not exceeding 2 seconds in general purpose workstations. The range information of the objects can be useful in MPEG-4 where natural and synthetic images can be mixed and synthesized. <s> BIB013 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> As we know, a robust approach to face and facial features detection must be able to handle the variation issues such as changes in imaging conditions, face appearances and image contents. Here we present a method which utilizes color, local symmetry and geometry information of human face based on various models. The algorithm first detects most likely face regions or ROIs (Region-Of-Interest) from the image using face color model and face outline model, produces a face color similarity map. Then it performs local symmetry detection within these ROIs to obtain a local symmetry similarity map. These two maps are fused to obtain potential facial feature points. Finally similarity matching is performed to identify faces between the fusion map and face geometry model under affine transformation. The output results are the detected faces with confidence values. Experimental results have demonstrated its validity and robustness to identify faces under certain variations. <s> BIB014 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> This paper presents a statistics-based method for estimating the position and size of a face in a complex background. Face position and size are estimated based on robust statistical measurements which are derived from two one-dimensional histograms obtained by projecting the result of skin color filtering. The proposed algorithm also utilizes a linear Kalman filter and a simple nonlinear filter to perform smooth tracking and remove jitter. The algorithm has been implemented and tested under a wide range of real-world conditions. It has consistently provided performance which satisfies the following requirements: (1) the ability to automatically determine the initial position and size of a face and track it in a complex background; (2) is insensitive to partial occlusions and shadows; (3) is insensitive to face orientation and scale changes; and (4) is also insensitive to lighting condition changes. In addition, the algorithm is computationally simple so that it can be executed in real-time. <s> BIB015 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> We describe an algorithm for detecting human faces and facial features, such as the location of the eyes, nose and mouth. First, a supervised pixel-based color classifier is employed to mark all pixels that are within a prespecified distance of ''skin color'', which is computed from a training set of skin patches. This color-classification map is then smoothed by Gibbs random field model-based filters to define skin regions. An ellipse model is fit to each disjoint skin region. Finally, we introduce symmetry-based cost functions to search the center of the eyes, tip of nose, and center of mouth within ellipses whose aspect ratio is similar to that of a face. <s> BIB016 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> We propose a new method to detect human faces in color images. A human skin color model is built to capture the chromatic properties based on multivariate statistical analysis. Given a color image, multiscale segmentation is used to generate homogeneous regions at multiple different scales. From the coarsest to the finest scale, regions of skin color are merged until the shape is approximately elliptic. Postprocessing is performed to determine whether a merged region contains a human face and include the facial features of non-skin color such as eyes and mouth if necessary. Experimental results show that human faces in color images can be detected regardless of size, orientation and viewpoint. <s> BIB017 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> This paper is concerned with estimating a probability density function of human skin color, using a finite Gaussian mixture model, whose parameters are estimated through the EM algorithm. Hawkins' statistical test on the normality and homoscedasticity (common covariance matrix) of the estimated Gaussian mixture models is performed and McLachlan's bootstrap method is used to test the number of components in a mixture. Experimental results show that the estimated Gaussian mixture model fits skin images from a large database. Applications of the estimated density function in image and video databases are presented. <s> BIB018 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> We developed Name-It, a system that associates faces and names in news videos. It processes information from the videos and can infer possible name candidates for a given face or locate a face in news videos by name. To accomplish this task, the system takes a multimodal video analysis approach: face sequence extraction and similarity evaluation from videos, name extraction from transcripts, and video-caption recognition. <s> BIB019 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> The use of adaptive Gaussian mixtures to model the colour distributions of objects is described. These models are used to perform robust, real-time tracking under varying illumination, viewing geometry and camera parameters. Observed log-likelihood measurements were used to perform selective adaptation. <s> BIB020 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> Computer vision is moving into a new era in which the aim is to develop visual skills for robots that allow them to interact with a dynamic, unconstrained environment. To achieve this aim, new kinds of vision algorithms need to be developed which run in real time and subserve the robot's goals. Two fundamental goals are determining the identity of an object with a known location, and determining the location of a known object. Color can be successfully used for both tasks. ::: This dissertation demonstrates that color histograms of multicolored objects provide a robust, efficient cue for indexing into a large database of models. It shows that color histograms are stable object representations in the presence of occlusion and over change in view, and that they can differentiate among a large number of objects. For solving the identification problem, it introduces a technique called Histogram Intersection, which matches model and image histograms and a fast incremental version of Histogram Intersection which allows real-time indexing into a large database of stored models. It demonstrates techniques for dealing with crowded scenes and with models with similar color signatures. For solving the location problem it introduces an algorithm called Histogram Backprojection which performs this task efficiently in crowded scenes. <s> BIB021 </s> Detecting Faces in Images : A Survey <s> Skin Color <s> The existence of large image datasets such as photos on the World Wide Web make it possible to build powerful generic models for low-level image attributes like color using simple histogram learning techniques. We describe the construction of color models for skin and non-skin classes from a dataset of nearly 1 billion labeled pixels. These classes exhibit a surprising degree of separability which we exploit by building a skin pixel detector that achieves an equal error rate of 88%. We compare the performance of histogram and mixture models in skin detection and find histogram models to be superior in accuracy and computational cost. Using aggregate features computed from the skin detector we build a remarkably effective detector for naked people. We believe this work is the most comprehensive and detailed exploration of skin color models to date. <s> BIB022
Human skin color has been used and proven to be an effective feature in many applications from face detection to hand tracking. Although different people have different skin color, several studies have shown that the major difference lies largely between their intensity rather than their chrominance  ,  , BIB003 . Several color spaces have been utilized to label pixels as skin including RGB BIB009 , BIB011 , BIB019 , normalized RGB BIB001 , BIB002 ,  , BIB003 , BIB010 ,  , BIB012 , BIB013 , BIB014 , BIB015 , HSV (or HSI) BIB004 , BIB005 , BIB006 , BIB007 , YCrCb  ,  , YIQ  , BIB008 , YES BIB016 , CIE XYZ  , and CIE LUV BIB017 . Many methods have been proposed to build a skin color model. The simplest model is to define a region of skin tone pixels using Cr; Cb values  , i.e., RðCr; CbÞ, from samples of skin color pixels. With carefully chosen thresholds, ½Cr 1 ; Cr 2 and ½Cb 1 ; Cb 2 , a pixel is classified to have skin tone if its values ðCr; CbÞ fall within the ranges, i.e., Cr 1 Cr Cr 2 and Cb 1 Cb Cb 2 . Crowley and Coutaz used a histogram hðr; gÞ of ðr; gÞ values in normalized RGB color space to obtain the probability of obtaining a particular RGBvector given that the pixel observes skin BIB002 , BIB010 . In other words, a pixel is classified to belong to skin color if hðr; gÞ ! , where is a threshold selected empirically from the histogram of samples. Saxe and Foulds proposed an iterative skin identification method that uses histogram intersection in HSV color space BIB004 . An initial patch of skin color pixels, called the control seed, is chosen by the user and is used to initiate the iterative algorithm. To detect skin color regions, their method moves through the image, one patch at a time, and presents the control histogram and the current histogram from the image for comparison. Histogram intersection BIB021 is used to compare the control histogram and current histogram. If the match score or number of instances in common (i.e., intersection) is greater than a threshold, the current patch is classified as being skin color. Kjeldsen and Kender defined a color predicate in HSV color space to separate skin regions from background BIB005 . In contrast to the nonparametric methods mentioned above, Gaussian density functions BIB017 , BIB013 , BIB017 and a mixture of Gaussians BIB009 , BIB011 , BIB018 are often used to model skin color. The parameters in a unimodal Gaussian distribution are often estimated using maximum-likelihood BIB017 , BIB013 , BIB017 . The motivation for using a mixture of Gaussians is based on the observation that the color histogram for the skin of people with different ethnic background does not form a unimodal distribution, but rather a multimodal distribution. The parameters in a mixture of Gaussians are usually estimated using an EM algorithm BIB009 , BIB018 . Recently, Jones and Rehg conducted a large-scale experiment in which nearly 1 billion labeled skin tone pixels are collected (in normalized RGB color space) BIB022 . Comparing the performance of histogram and mixture models for skin detection, they find histogram models to be superior in accuracy and computational cost. Color information is an efficient tool for identifying facial areas and specific facial features if the skin color model can be properly adapted for different lighting environments. However, such skin color models are not effective where the spectrum of the light source varies significantly. In other words, color appearance is often unstable due to changes in both background and foreground lighting. Though the color constancy problem has been addressed through the formulation of physics-based models  , several approaches have been proposed to use skin color in varying lighting conditions. McKenna et al. presented an adaptive color mixture model to track faces under varying illumination conditions BIB020 . Instead of relying on a skin color model based on color constancy, they used a stochastic model to estimate an object's color distribution online and adapt to accommodate changes in the viewing and lighting conditions. Preliminary results show that their system can track faces within a range of illumination conditions. However, this method cannot be applied to detect faces in a single image. Skin color alone is usually not sufficient to detect or track faces. Recently, several modular systems using a combination of shape analysis, color segmentation, and motion information for locating or tracking heads and faces in an image sequence have been developed  , BIB017 , BIB003 , BIB020 , BIB006 . We review these methods in the next section.