Detecting Faces in Images : A Survey <s> Facial Features <s> This paper describes a computational approach to edge detection. The success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points. These goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution. We define detection and localization criteria for a class of edges, and present mathematical forms for these criteria as functionals on the operator impulse response. A third criterion is then added to ensure that the detector has only one response to a single edge. We use the criteria in numerical optimization to derive detectors for several common image features, including step edges. On specializing the analysis to step edges, we find that there is a natural uncertainty principle between detection and localization performance, which are the two main goals. With this principle we derive a single operator shape which is optimal at any scale. The optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a Gaussian-smoothed image. We extend this simple detector using operators of several widths to cope with different signal-to-noise ratios in the image. We present a general method, called feature synthesis, for the fine-to-coarse integration of information from operators at different scales. Finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge. <s> BIB001 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> The paper obtains the exact distribution of Bookstein's shape variables under his plausible model for landmark data. We consider its properties including invariances, marginal distributions and the relationship with Kendall's uniform measure. Particular cases for triangles and quadrilaterals are highlighted. A normal approximation to the distribution is obtained, extending Bookstein's result for three landmarks. The adequacy of these approximations is also studied. <s> BIB002 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> The authors develop previous theories of the analysis of deformation of apparent contours under viewer motion. Earlier results showing how surface curvature can be inferred from acceleration of image features are generalized for arbitrary viewer motion and perspective projection. It is shown that relative image acceleration, based on parallax measurements, is robust to uncertainties in robot motion. The theory has been implemented and extensively tested in a real-time (15 frames per second) tracking system based on deformable contours (snakes). It is shown that focusing attention by means of snakes allows rapid, robust computation of surface curvature, including discrimination of extremal and occluding contours. > <s> BIB003 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> An algorithm for locating quasi-frontal views of human faces in cluttered scenes is presented. The algorithm works by coupling a set of local feature detectors with a statistical model of the mutual distances between facial features it is invariant with respect to translation, rotation (in the plane), and scale and can handle partial occlusions of the face. On a challenging database with complicated and varied backgrounds, the algorithm achieved a correct localization rate of 95% in images where the face appeared quasi-frontally. > <s> BIB004 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> Present approaches to human face detection have made several assumptions that restrict their ability to be extended to general imaging conditions. We identify that the key factor in a generic and robust system is that of exploiting a large amount of evidence, related and reinforced by model knowledge through a probabilistic framework. In this paper, we propose a face detection framework that groups image features into meaningful entities-using perceptual organization, assigns probabilities to each of them, and reinforce there probabilities using Bayesian reasoning techniques. True hypotheses of faces will be reinforced to a high probability. The detection of faces under scale, orientation and viewpoint variations will be examined in a subsequent paper. <s> BIB005 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> Human face detection has always been an important problem for face, expression and gesture recognition. Though numerous attempts have been made to detect and localize faces, these approaches have made assumptions that restrict their extension to more general cases. We identify that the key factor in a generic and robust system is that of using a large amount of image evidence, related and reinforced by model knowledge through a probabilistic framework. In this paper, we propose a feature-based algorithm for detecting faces that is sufficiently generic and is also easily extensible to cope with more demanding variations of the imaging conditions. The algorithm detects feature points from the image using spatial filters and groups them into face candidates using geometric and gray level constraints. A probabilistic framework is then used to reinforce probabilities and to evaluate the likelihood of the candidate as a face. We provide results to support the validity of the approach and demonstrate its capability to detect faces under different scale, orientation and viewpoint. <s> BIB006 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> An e$cient face detection algorithm which can detect multiple faces oriented in any directions in a cluttered environment is proposed. In this paper, a morphology-based technique is "rst devised to perform eye-analogue segmentation. Next, the previously located eye-analogue segments are used as guides to search for potential face regions. Then, each of these potential face images is normalized to a standard size and fed into a trained backpropagation neural network for identi"cation. In this detection system, the morphology-based eye-analogue segmentation process is able to reduce the background part of a cluttered image by up to 95%. This process signi"cantly speeds up the subsequent face detection procedure because only 5}10% of the regions of the original image remain for further processing. Experiments demonstrate that an approximately 94% success rate is reached, and that the relative false detection rate is very low. ( 2000 Published by Elsevier Science Ltd on behalf of Pattern Recognition Society. <s> BIB007 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> We introduce a very large family of binary features for two-dimensional shapes. The salient ones for separating particular shapes are determined by inductive learning during the construction of classification trees. There is a feature for every possible geometric arrangement of local topographic codes. The arrangements express coarse constraints on relative angles and distances among the code locations and are nearly invariant to substantial affine and nonlinear deformations. They are also partially ordered, which makes it possible to narrow the search for informative ones at each node of the tree. Different trees correspond to different aspects of shape. They are statistically and weakly dependent due to randomization and are aggregated in a simple way. Adapting the algorithm to a shape family is then fully automatic once training samples are provided. As an illustration, we classified handwritten digits from the NIST database; the error rate was 0.7 percent. <s> BIB008 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> Under a weak perspective camera model, the image plane coordinates in different views of a planar object are related by an affine transformation. Because of this property, researchers have attempted to use affine invariants for recognition. However, there are two problems with this approach: (1) objects or object classes with inherent variability cannot be adequately treated using invariants; and (2) in practice the calculated affine invariants can be quite sensitive to errors in the image plane measurements. In this paper we use probability distributions to address both of these difficulties. Under the assumption that the feature positions of a planar object can be modeled using a jointly Gaussian density, we have derived the joint density over the corresponding set of affine coordinates. Even when the assumptions of a planar object and a weak perspective camera model do not strictly hold, the results are useful because deviations from the ideal can be treated as deformability in the underlying object model. <s> BIB009 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> Recent advances in human face detection algorithms have seen varying degrees of success using numerous approaches. We identify that a feature-based approach is able to detect faces efficiently over large viewpoint and illumination variations. In this paper, we will enhance the approach by proposing the use of active contour models to detect the face boundary, and subsequently use it to verify face candidates. We present a method to initialize the active contour model, and show how the resulting information can be used to verify true face candidates. Further verification of the face hypothesis is achieved by checking for consistency with motion. We present results of testing the system under a wide range of imaging conditions, demonstrating its capability and robustness. <s> BIB010 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> We present a neural network-based upright frontal face detection system. A retinally connected neural network examines small windows of an image and decides whether each window contains a face. The system arbitrates between multiple networks to improve performance over a single network. We present a straightforward procedure for aligning positive face examples for training. To collect negative examples, we use a bootstrap algorithm, which adds false detections into the training set as training progresses. This eliminates the difficult task of manually selecting nonface training examples, which must be chosen to span the entire space of nonface images. Simple heuristics, such as using the fact that faces rarely overlap in images, can further improve the accuracy. Comparisons with several other state-of-the-art face detection systems are presented, showing that our system has comparable performance in terms of detection and false-positive rates. <s> BIB011 </s> Detecting Faces in Images : A Survey <s> Facial Features <s> Major topics: Human processing of face recognition and its relevance to forensic systems * Face coding * Connectionist methods and support vector machines (SVM) * Hybrid methods for face recognition * Predictive learning and performance evaluation. <s> BIB012
Sirohey proposed a localization method to segment a face from a cluttered background for face identification  . It uses an edge map (Canny detector BIB001 ) and heuristics to remove and group edges so that only the ones on the face Fig. 2 . A typical face used in knowledge-based top-down methods: Rules are coded based on human knowledge about the characteristics (e.g., intensity distribution and difference) of the facial regions  . contour are preserved. An ellipse is then fit to the boundary between the head region and the background. This algorithm achieves 80 percent accuracy on a database of 48 images with cluttered backgrounds. Instead of using edges, Chetverikov and Lerch presented a simple face detection method using blobs and streaks (linear sequences of similarly oriented edges)  . Their face model consists of two dark blobs and three light blobs to represent eyes, cheekbones, and nose. The model uses streaks to represent the outlines of the faces, eyebrows, and lips. Two triangular configurations are utilized to encode the spatial relationship among the blobs. A low resolution Laplacian image is generated to facilitate blob detection. Next, the image is scanned to find specific triangular occurrences as candidates. A face is detected if streaks are identified around a candidate. Graf et al. developed a method to locate facial features and faces in gray scale images  . After band pass filtering, morphological operations are applied to enhance regions with high intensity that have certain shapes (e.g., eyes). The histogram of the processed image typically exhibits a prominent peak. Based on the peak value and its width, adaptive threshold values are selected in order to generate two binarized images. Connected components are identified in both binarized images to identify the areas of candidate facial features. Combinations of such areas are then evaluated with classifiers, to determine whether and where a face is present. Their method has been tested with head-shoulder images of 40 individuals and with five video sequences where each sequence consists of 100 to 200 frames. However, it is not clear how morphological operations are performed and how the candidate facial features are combined to locate a face. Leung et al. developed a probabilistic method to locate a face in a cluttered scene based on local feature detectors and random graph matching BIB004 . Their motivation is to formulate the face localization problem as a search problem in which the goal is to find the arrangement of certain facial features that is most likely to be a face pattern. Five features (two eyes, two nostrils, and nose/lip junction) are used to describe a typical face. For any pair of facial features of the same type (e.g., lefteye, right-eye pair), their relative distance is computed, and over an ensemble of images the distances are modeled by a Gaussian distribution. A facial template is defined by averaging the responses to a set of multiorientation, multiscale Gaussian derivative filters (at the pixels inside the facial feature) over a number of faces in a data set. Given a test image, candidate facial features are identified by matching the filter response at each pixel against a template vector of responses (similar to correlation in spirit). The top two feature candidates with the strongest response are selected to search for the other facial features. Since the facial features cannot appear in arbitrary arrangements, the expected locations of the other features are estimated using a statistical model of mutual distances. Furthermore, the covariance of the estimates can be computed. Thus, the expected feature locations can be estimated with high probability. Constellations are then formed only from candidates that lie inside the appropriate locations, and the most face-like constellation is determined. Finding the best constellation is formulated as a random graph matching problem in which the nodes of the graph correspond to features on a face, and the arcs represent the distances between different features. Ranking of constellations is based on a probability density function that a constellation corresponds to a face versus the probability it was generated by an alternative mechanism (i.e., nonface). They used a set of 150 images for experiments in which a face is considered correctly detected if any constellation correctly locates three or more features on the faces. This system is able to achieve a correct localization rate of 86 percent. Instead of using mutual distances to describe the relationships between facial features in constellations, an alternative method for modeling faces was also proposed by the Leung et al.  , BIB009 . The representation and ranking of the constellations is accomplished using the statistical theory of shape, developed by Kendall  and Mardia and Dryden BIB002 . The shape statistics is a joint probability density function over N feature points, represented by ðx i ; y i Þ, for the ith feature under the assumption that the original feature points are positioned in the plane according to a general 2N-dimensional Gaussian distribution. They applied the same maximum-likelihood (ML) method to determine the location of a face. One advantage of these methods is that partially occluded faces can be located. However, it is unclear whether these methods can be adapted to detect multiple faces effectively in a scene. In BIB005 , BIB006 , Yow and Cipolla presented a featurebased method that uses a large amount of evidence from the visual image and their contextual evidence. The first stage applies a second derivative Gaussian filter, elongated at an aspect ratio of three to one, to a raw image. Interest points, detected at the local maxima in the filter response, indicate the possible locations of facial features. The second stage examines the edges around these interest points and groups them into regions. The perceptual grouping of edges is based on their proximity and similarity in orientation and strength. Measurements of a region's characteristics, such as edge length, edge strength, and intensity variance, are computed and stored in a feature vector. From the training data of facial features, the mean and covariance matrix of each facial feature vector are computed. An image region becomes a valid facial feature candidate if the Mahalanobis distance between the corresponding feature vectors is below a threshold. The labeled features are further grouped based on model knowledge of where they should occur with respect to each other. Each facial feature and grouping is then evaluated using a Bayesian network. One attractive aspect is that this method can detect faces at different orientations and poses. The overall detection rate on a test set of 110 images of faces with different scales, orientations, and viewpoints is 85 percent BIB010 . However, the reported false detection rate is 28 percent and the implementation is only effective for faces larger than 60 Â 60 pixels. Subsequently, this approach has been enhanced with active contour models BIB003 , BIB010 . Fig. 4 summarizes their featurebased face detection method. Takacs and Wechsler described a biologically motivated face localization method based on a model of retinal feature extraction and small oscillatory eye movements  . Their algorithm operates on the conspicuity map or region of interest, with a retina lattice modeled after the magnocellular ganglion cells in the human vision system. The first phase computes a coarse scan of the image to estimate the location of the face, based on the filter responses of receptive fields. Each receptive field consists of a number of neurons which are implemented with Gaussian filters tuned to specific orientations. The second phase refines the conspicuity map by scanning the image area at a finer resolution to localize the face. The error rate on a test set of 426 images (200 subjects from the FERET database) is 4.69 percent. Han et al. developed a morphology-based technique to extract what they call eye-analogue segments for face detection BIB007 . They argue that eyes and eyebrows are the most salient and stable features of human face and, thus, useful for detection. They define eye-analogue segments as edges on the contours of eyes. First, morphological operations such as closing, clipped difference, and thresholding are applied to extract pixels at which the intensity values change significantly. These pixels become the eyeanalogue pixels in their approach. Then, a labeling process is performed to generate the eye-analogue segments. These segments are used to guide the search for potential face regions with a geometrical combination of eyes, nose, eyebrows and mouth. The candidate face regions are further verified by a neural network similar to BIB011 . Their experiments demonstrate a 94 percent accuracy rate using a test set of 122 images with 130 faces. Recently, Amit et al. presented a method for shape detection and applied it to detect frontal-view faces in still intensity images BIB012 . Detection follows two stages: focusing and intensive classification. Focusing is based on spatial arrangements of edge fragments extracted from a simple edge detector using intensity difference. A rich family of such spatial arrangements, invariant over a range of photometric and geometric transformations, is defined. From a set of 300 training face images, particular spatial arrangements of edges which are more common in faces than backgrounds are selected using an inductive method developed in BIB008 . Meanwhile, the CART algorithm  is applied to grow a classification tree from the training images and a collection of false positives identified from generic background images. Given a test image, regions of interest are identified from the spatial arrangements of edge fragments. Each region of interest is then classified as face or background using the learned CART tree. Their experimental results on a set of 100 images from the Olivetti (now AT&T) data set  report a false positive rate of 0.2 percent per 1,000 pixels and a false negative rate of 10 percent.