Detecting Faces in Images : A Survey <s> Performance Evaluation <s> Provides a unified, comprehensive and up-to-date treatment of both statistical and descriptive methods for pattern recognition. The topics treated include Bayesian decision theory, supervised and unsupervised learning, nonparametric techniques, discriminant analysis, clustering, preprosessing of pictorial data, spatial filtering, shape description techniques, perspective transformations, projective invariants, linguistic procedures, and artificial intelligence techniques for scene analysis. <s> BIB001 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> In this paper we present a visual learning technique that maximizes the discrimination between positive and negative examples in a training set. We demonstrate our technique in the context of face detection with complea background without color or motion information, which has proven to be a challenging problem. We use a family of discrete Markov processes to model the face and background patterns and estimate the probability models using the data statistics. Then, we convert the learning process into an optimization, selecting the Markov process that optimizes the information-based discrim.ination between the two classes. The detection process is carried out by computing the likelihood ratio using the probability model obtained from the learning procedure. We show that because of the discrete nature of these models, the detection process is at least two orders of magnitude less computationally expensive than neural network approaches. However, no improvement in terms of correct-answer/false-alarm tradeoff is achieved. <s> BIB002 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> A new performance evaluation paradigm for computer vision systems is proposed. In real situation, the complexity of the input data and/or of the computational procedure can make traditional error propagation methods infeasible. The new approach exploits a resampling technique recently introduced in statistics, the bootstrap. Distributions for the output variables are obtained by perturbing the nuisance properties of the input, i.e., properties with no relevance for the output under ideal conditions. From these bootstrap distributions, the confidence in the adequacy of the assumptions embedded into the computational procedure for the given input is derived. As an example, the new paradigm is applied to the task of edge detection. The performance of several edge detection methods is compared both for synthetic data and real images. The confidence in the output can be used to obtain an edgemap independent of the gradient magnitude. <s> BIB003 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> A new method for evaluating edge detection algorithms is presented and applied to measure the relative performance of algorithms by Canny, Nalwa-Binford, Iverson-Zucker, Bergholm, and Rothwell. The basic measure of performance is a visual rating score which indicates the perceived quality of the edges for identifying an object. The process of evaluating edge detection algorithms with this performance measure requires the collection of a set of gray-scale images, optimizing the input parameters for each algorithm, conducting visual evaluation experiments and applying statistical analysis methods. The novel aspect of this work is the use of a visual task and real images of complex scenes in evaluating edge detectors. The method is appealing because, by definition, the results agree with visual evaluations of the edge images. <s> BIB004 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> We present a neural network-based upright frontal face detection system. A retinally connected neural network examines small windows of an image and decides whether each window contains a face. The system arbitrates between multiple networks to improve performance over a single network. We present a straightforward procedure for aligning positive face examples for training. To collect negative examples, we use a bootstrap algorithm, which adds false detections into the training set as training progresses. This eliminates the difficult task of manually selecting nonface training examples, which must be chosen to span the entire space of nonface images. Simple heuristics, such as using the fact that faces rarely overlap in images, can further improve the accuracy. Comparisons with several other state-of-the-art face detection systems are presented, showing that our system has comparable performance in terms of detection and false-positive rates. <s> BIB005 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> In this paper, we describe a statistical method for 3D object detection. We represent the statistics of both object appearance and "non-object" appearance using a product of histograms. Each histogram represents the joint statistics of a subset of wavelet coefficients and their position on the object. Our approach is to use many such histograms representing a wide variety of visual attributes. Using this method, we have developed the first algorithm that can reliably detect human faces with out-of-plane rotation and the first algorithm that can reliably detect passenger cars over a wide range of viewpoints. <s> BIB006 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> Two of the most critical requirements in support of producing reliable face-recognition systems are a large database of facial images and a testing procedure to evaluate systems. The Face Recognition Technology (FERET) program has addressed both issues through the FERET database of facial images and the establishment of the FERET tests. To date, 14,126 images from 1,199 individuals are included in the FERET database, which is divided into development and sequestered portions of the database. In September 1996, the FERET program administered the third in a series of FERET face-recognition tests. The primary objectives of the third test were to 1) assess the state of the art, 2) identify future areas of research, and 3) measure algorithm performance. <s> BIB007 </s> Detecting Faces in Images : A Survey <s> Performance Evaluation <s> In real-world environments it usually is difficult to specify target operating conditions precisely, for example, target misclassification costs. This uncertainty makes building robust classification systems problematic. We show that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions. In some cases, the performance of the hybrid actually can surpass that of the best known classifier. This robust performance extends across a wide variety of comparison frameworks, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization. The hybrid also is efficient to build, to store, and to update. The hybrid is based on a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull (ROCCH) method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers. The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses. Finally, we point to empirical evidence that a robust hybrid classifier indeed is needed for many real-world problems. <s> BIB008
In order to obtain a fair empirical evaluation of face detection methods, it is important to use a standard and representative test set for experiments. Although many face detection methods have been developed over the past Fig. 15 . Sample images of profile faces from Schneiderman and Kanade's data set BIB006 . This data set contains images with faces in profile views and some with facial expressions. decade, only a few of them have been tested on the same data set. Table 4 summarizes the reported performance among several appearance-based face detection methods on two standard data sets described in the previous section. Although Table 4 shows the performance of these methods on the same test set, such an evaluation may not characterize how well these methods will compare in the field. There are a few factors that complicate the assessment of these appearance-based methods. First, the reported results are based on different training sets and different tuning parameters. The number and variety of training examples have a direct effect on the classification performance. However, this factor is often ignored in performance evaluation, which is an appropriate criteria if the goal is to evaluate the systems rather than the learning methods. The second factor is the training time and execution time. Although the training time is usually ignored by most systems, it may be important for real-time applications that require online training on different data sets. Third, the number of scanning windows in these methods vary because they are designed to operate in different environments (i.e., to detect faces within a size range). For example, Colmenarez and Huang argued that their method scans more windows than others and, thus, the number of false detections is higher than others BIB002 . Furthermore, the criteria adopted in reporting the detection rates is usually not clearly described in most systems. Fig. 16a shows a test image and Fig. 16b shows some subimages to be classified as a face or nonface. Suppose that all the subimages in Fig. 16b are classified as face patterns, some criteria may consider all of them as "successful" detections. However, a more strict criterion (e.g., each successful detection must contain all the visible eyes and mouths in an image) may classify most of them as false alarms. It is clear that a uniform criteria should be adopted to assess different classifiers. In BIB005 , Rowley et al. adjust the criteria until the experimental results match their intuition of what a correct detection is, i.e., the square window should contain the eyes and also the mouth. The criteria they eventually use is that the center of the detected bounding box must be within four pixels and the scale must be within a factor of 1.2 (their scale step size) of ground truth (recorded manually). Finally, the evaluation criteria may and should depend on the purpose of the detector. If the detector is going to be used to count people, then the sum of false positives and false negatives is appropriate. On the other hand, if the detector is to be used to verify that an individual is who he/she claims to be (validation), then it may be acceptable for the face detector to have additional false detections since it is unlikely that these false detections will be acceptable images of the individual, i.e., the validation process will reject the false detections. In other words, the penalty or cost of one type of error should be properly weighted such that one can build an optimal classifier using Bayes decision rule (See Sections 2.2-2.4 in BIB001 ). This argument is supported by a recent study which points out the accuracy of the classifier (i.e., detection rate in face detection) is not an appropriate goal for many of the realworld task BIB008 . One reason is that classification accuracy assumes equal misclassification costs. This assumption is problematic because for most real-world problems one type of classification error is much more expensive than another. In some face detection applications, it is important that all the existing faces are detected. Another reason is accuracy maximization assumes that the class distribution is known for the target environment. In other words, we assume the test data sets represent the "true" working environment for the face detectors. However, this assumption is rarely justified. When detection methods are used within real systems, it is important to consider what computational resources are required, particularly, time and memory. Accuracy may need to be sacrificed for for speed. The scope of the considered techniques in evaluation is also important. In this survey, we discuss at least four different forms of the face detection problem: 1. Localization in which there is a single face and the goal is to provide a suitable estimate of position; scale to be used as input for face recognition. 2. In a cluttered monochrome scene, detect all faces. 3. In color images, detect (localize) all faces. 4. In a video sequence, detect and localize all faces. An evaluation protocol should be carefully designed when assessing these different detection situations. It should be noted that there is a potential risk of using a universal though modest sized standard test set. As researchers develop new methods or "tweak" existing ones to get better performance on the test set, they engage in a subtle form of the unacceptable practice of "testing on the training set." As a consequence, the latest methods may perform better against this hypothetical test set but not actually perform better in practice. This can be obviated by having a sufficiently large and representative universal test set. Alternatively, methods could be evaluated on a smaller test set if that test set is randomly chosen (generated) each time the method is evaluated. In summary, fair and effective performance evaluation requires careful design of protocols, scope, and data sets. Such issues have attracted much attention in numerous vision problems BIB003 , BIB004 ,  , BIB007 . However, performing this evaluation or trying to declare a "winner" is beyond the scope of this survey. Instead, we hope that either a consortium of researchers engaged in face detection or a third party will take on this task. Until then, we hope that when applicable, researchers will report the result of their methods on the publicly available data sets described here. As a first step toward this goal, we have collected sample face detection codes and evaluation tools at http://vision. ai.uiuc.edu/mhyang/face-detection-survey.html.