Detecting Faces in Images : A Survey <s> INTRODUCTION <s> We have developed a near-real-time computer system that can locate and track a subject's head, and then recognize the person by comparing characteristics of the face to those of known individuals. The computational approach taken in this system is motivated by both physiology and information theory, as well as by the practical requirements of near-real-time performance and accuracy. Our approach treats the face recognition problem as an intrinsically two-dimensional (2-D) recognition problem rather than requiring recovery of three-dimensional geometry, taking advantage of the fact that faces are normally upright and thus may be described by a small set of 2-D characteristic views. The system functions by projecting face images onto a feature space that spans the significant variations among known face images. The significant features are known as "eigenfaces," because they are the eigenvectors (principal components) of the set of faces; they do not necessarily correspond to features such as eyes, ears, and noses. The projection operation characterizes an individual face by a weighted sum of the eigenface features, and so to recognize a particular face it is necessary only to compare these weights to those of known individuals. Some particular advantages of our approach are that it provides for the ability to learn and later recognize new faces in an unsupervised manner, and that it is easy to implement using a neural network architecture. <s> BIB001 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> We describe a computer program which understands a greyscale image of a face well enough to locate individual face features such as eyes and mouth. The program has two distinct components: modules designed to locate particular face features, usually in a restricted area; and the overall control strategy which activates modules on the basis of the current solution state, and assesses and integrates the results of each module. <s> BIB002 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> Abstract Humans detect and identify faces in a scene with little or no effort. However, building an automated system that accomplishes this task is very difficult. There are several related subproblems: detection of a pattern as a face, identification of the face, analysis of facial expressions, and classification based on physical features of the face. A system that performs these operations will find many applications, e.g. criminal identification, authentication in secure systems, etc. Most of the work to date has been in identification. This paper surveys the past work in solving these problems. The capability of the human visual system with respect to these problems is also discussed. It is meant to serve as a guide for an automated system. Some new approaches to these problems are also briefly discussed. <s> BIB003 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> The first step for human face recognition is to locate the head boundary in a head-and-shoulders image. An approach that uses adaptive contour models or "snakes" is described to solve this problem. Since we have a priori knowledge of the shape of a head, this active contour model is tailor-made for representing the head boundary. In this paper, a reliable method to locate the approximate position of the head and to estimate the head boundary is proposed. The effect of the parameters for snakes is investigated by locating the head boundary, and a best set of the parameters is suggested. A fast algorithm based on the greedy algorithm for active contour modeling is also presented. The computational complexity of this new algorithm is analyzed and compared with the greedy algorithm. This fast algorithm has a performance capability comparable to the greedy algorithm and reduces the execution time by more than 30% on the average. <s> BIB004 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> The goal of this paper is to present a critical survey of existing literature on human and machine recognition of faces. Machine recognition of faces has several applications, ranging from static matching of controlled photographs as in mug shots matching and credit card verification to surveillance video images. Such applications have different constraints in terms of complexity of processing requirements and thus present a wide range of different technical challenges. Over the last 20 years researchers in psychophysics, neural sciences and engineering, image processing analysis and computer vision have investigated a number of issues related to face recognition by humans and machines. Ongoing research activities have been given a renewed emphasis over the last five years. Existing techniques and systems have been tested on different sets of images of varying complexities. But very little synergism exists between studies in psychophysics and the engineering literature. Most importantly, there exists no evaluation or benchmarking studies using large databases with the image quality that arises in commercial and law enforcement applications In this paper, we first present different applications of face recognition in commercial and law enforcement sectors. This is followed by a brief overview of the literature on face recognition in the psychophysics community. We then present a detailed overview of move than 20 years of research done in the engineering community. Techniques for segmentation/location of the face, feature extraction and recognition are reviewed. Global transform and feature based methods using statistical, structural and neural classifiers are summarized. > <s> BIB005 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> Previous efforts at facial expression recognition have been based on the Facial Action Coding System (FACS), a representation developed in order to allow human psychologists to code expression from static facial "mugshots." We develop new more accurate representations for facial expression by building a video database of facial expressions and then probabilistically characterizing the facial muscle activation associated with each expression using a detailed physical model of the skin and muscles. This produces a muscle based representation of facial motion, which is then used to recognize facial expressions in two different ways. The first method uses the physics based model directly, by recognizing expressions through comparison of estimated muscle activations. The second method uses the physics based model to generate spatio temporal motion energy templates of the whole face for each different expression. These simple, biologically plausible motion energy "templates" are then used for recognition. Both methods show substantially greater accuracy at expression recognition than has been previously achieved. > <s> BIB006 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> Visual processes to detect and track faces for video compression and transmission. The system is based on an architecture in which a supervisor selects and activates visual processes in cyclic manner. Control of visual processes is made possible by a confidence factor which accompanies each observation. Fusion of results into a unified estimation for tracking is made possible by estimating a covariance matrix with each observation. Visual processes for face tracking are described using blink detection, normalised color histogram matching, and cross correlation (SSD and NCC). Ensembles of visual processes are organised into processing states so as to provide robust tracking. Transition between states is determined by events detected by processes. The result of face detection is fed into recursive estimator (Kalman filter). The output from the estimator drives a PD controller for a pan/tilt/zoom camera. The resulting system provides robust and precise tracking which operates continuously at approximately 20 images per second on a 150 megahertz computer workstation. <s> BIB007 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> Two novel variants of dynamic link architecture that are based on mathematical morphology and incorporate coefficients which weigh the contribution of each node in elastic graph matching according to its discriminatory power are developed. They are the so called Morphological Dynamic Link Architecture and the Morphological Signal Decomposition-Dynamic Lint Architecture. The proposed variants are tested for face authentication in a cooperative scenario where the candidates claim an identity to be checked. Their performance is evaluated in terms of their receiver operating characteristic and the equal error rate achieved in M2VTS database. An equal error rate in the range 3.7-6.8% is reported. <s> BIB008 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> Two variants of dynamic link matching based on mathematical morphology are developed and tested for frontal face authentication, namely, the morphological dynamic link architecture and the morphological signal decomposition-dynamic link architecture. Local coefficients which weigh the contribution of each node in elastic graph matching according to its discriminatory power are derived. The performance of the proposed algorithms is evaluated in terms of their receiver operating characteristic and the equal error rate (EER) achieved in the M2VTS database. The comparison with other frontal face authentication algorithms developed within M2VTS project indicates that morphological dynamic link architecture with discriminatory power coefficients is ranked as the best algorithm in terms of the EER. <s> BIB009 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> We address the problem of robust face identification in the presence of pose, lighting, and expression variation. Previous approaches to the problem have assumed similar models of variation for each individual, estimated from pooled training data. We describe a method of updating a first order global estimate of identity by learning the class-specific correlation between the estimate and the residual variation during a sequence. This is integrated with an optimal tracking scheme, in which identity variation is decoupled from pose, lighting and expression variation. The method results in robust tracking and a more stable estimate of facial identity under changing conditions. <s> BIB010 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> The objects that surround us --desks, cars, shoes and coats --are deaf and blind; this limits their ability to adapt to our needs and thus to be useful. We have therefore developed computer systems that can follow people's actions, recognizing their faces, gestures, and expressions. Using this technology we have begun to make "smart rooms" and "smart clothes" that can help people in day-to-day life without chaining them to keyboards, pointing devices or special goggles. <s> BIB011 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> Smart environments, wearable computers, and ubiquitous computing in general are the coming "fourth generation" of computing and information technology. But that technology will be a stillbirth without new interfaces for interaction, minus a keyboard or mouse. To win wide consumer acceptance, these interactions must be friendly and personalized; the next generation interfaces must recognize people in their immediate environment and, at a minimum, know who they are. In this article, the authors discuss face recognition technology, how it works, problems to be overcome, current technologies, and future developments and possible applications. Twenty years ago, the problem of face recognition was considered among the most difficult in artificial intelligence and computer vision. Today, however, there are several companies that sell commercial face recognition software that is capable of high-accuracy recognition with databases of more than 1,000 people. The authors describe the face recognition technology used, explaining the algorithms for face recognition as well as novel applications, such as behavior monitoring that assesses emotions based on facial expressions. <s> BIB012 </s> Detecting Faces in Images : A Survey <s> INTRODUCTION <s> We present an approach to real-time person tracking in crowded and/or unknown environments using integration of multiple visual modalities. We combine stereo, color, and face detection modules into a single robust system, and show an initial application in an interactive, face-responsive display. Dense, real-time stereo processing is used to isolate users from other objects and people in the background. Skin-hue classification identifies and tracks likely body parts within the silhouette of a user. Face pattern detection discriminates and localizes the face within the identified body parts. Faces and bodies of users are tracked over several temporal scales: short-term (user stays within the field of view), medium-term (user exits/reenters within minutes), and long term (user returns after hours or days). Short-term tracking is performed using simple region position and size correspondences, while medium and long-term tracking are based on statistics of user appearance. We discuss the failure modes of each individual module, describe our integration method, and report results with the complete system in trials with thousands of users. <s> BIB013
W ITH the ubiquity of new information technology and media, more effective and friendly methods for human computer interaction (HCI) are being developed which do not rely on traditional devices such as keyboards, mice, and displays. Furthermore, the ever decreasing price/ performance ratio of computing coupled with recent decreases in video image acquisition cost imply that computer vision systems can be deployed in desktop and embedded systems  , BIB011 , BIB012 . The rapidly expanding research in face processing is based on the premise that information about a user's identity, state, and intent can be extracted from images, and that computers can then react accordingly, e.g., by observing a person's facial expression. In the last five years, face and facial expression recognition have attracted much attention though they have been studied for more than 20 years by psychophysicists, neuroscientists, and engineers. Many research demonstrations and commercial applications have been developed from these efforts. A first step of any face processing system is detecting the locations in images where faces are present. However, face detection from a single image is a challenging task because of variability in scale, location, orientation (up-right, rotated), and pose (frontal, profile). Facial expression, occlusion, and lighting conditions also change the overall appearance of faces. We now give a definition of face detection: Given an arbitrary image, the goal of face detection is to determine whether or not there are any faces in the image and, if present, return the image location and extent of each face. The challenges associated with face detection can be attributed to the following factors: . Pose. The images of a face vary due to the relative camera-face pose (frontal, 45 degree, profile, upside down), and some facial features such as an eye or the nose may become partially or wholly occluded. . Presence or absence of structural components. Facial features such as beards, mustaches, and glasses may or may not be present and there is a great deal of variability among these components including shape, color, and size. . Facial expression. The appearance of faces are directly affected by a person's facial expression. . Occlusion. Faces may be partially occluded by other objects. In an image with a group of people, some faces may partially occlude other faces. . Image orientation. Face images directly vary for different rotations about the camera's optical axis. . Imaging conditions. When the image is formed, factors such as lighting (spectra, source distribution and intensity) and camera characteristics (sensor response, lenses) affect the appearance of a face. There are many closely related problems of face detection. Face localization aims to determine the image position of a single face; this is a simplified detection problem with the assumption that an input image contains only one face BIB004 ,  . The goal of facial feature detection is to detect the presence and location of features, such as eyes, nose, nostrils, eyebrow, mouth, lips, ears, etc., with the assumption that there is only one face in an image BIB002 ,  . Face recognition or face identification compares an input image (probe) against a database (gallery) and reports a match, if any BIB001 , BIB003 , BIB005 . The purpose of face authentication is to verify the claim of the identity of an individual in an input image BIB008 , BIB009 , while face tracking methods continuously estimate the location and possibly the orientation of a face in an image sequence in real time BIB007 , BIB010 , BIB013 . Facial expression recognition concerns identifying the affective states (happy, sad, disgusted, etc.) of humans BIB006 ,  . Evidently, face detection is the first step in any automated system which solves the above problems. It is worth mentioning that many papers use the term "face detection," but the methods and the experimental results only show that a single face is localized in an input image. In this paper, we differentiate face detection from face localization since the latter is a simplified problem of the former. Meanwhile, we focus on face detection methods rather than tracking methods. While numerous methods have been proposed to detect faces in a single image of intensity or color images, we are unaware of any surveys on this particular topic. A survey of early face recognition methods before 1991 was written by Samal and Iyengar BIB003 . Chellapa et al. wrote a more recent survey on face recognition and some detection methods BIB005 . Among the face detection methods, the ones based on learning algorithms have attracted much attention recently and have demonstrated excellent results. Since these datadriven methods rely heavily on the training sets, we also discuss several databases suitable for this task. A related and important problem is how to evaluate the performance of the proposed detection methods. Many recent face detection papers compare the performance of several methods, usually in terms of detection and false alarm rates. It is also worth noticing that many metrics have been adopted to evaluate algorithms, such as learning time, execution time, the number of samples required in training, and the ratio between detection rates and false alarms. Evaluation becomes more difficult when researchers use different definitions for detection and false alarm rates. In this paper, detection rate is defined as the ratio between the number of faces correctly detected and the number faces determined by a human. An image region identified as a face by a classifier is considered to be correctly detected if the image region covers more than a certain percentage of a face in the image (See Section 3.3 for details). In general, detectors can make two types of errors: false negatives in which faces are missed resulting in low detection rates and false positives in which an image region is declared to be face, but it is not. A fair evaluation should take these factors into consideration since one can tune the parameters of one's method to increase the detection rates while also increasing the number of false detections. In this paper, we discuss the benchmarking data sets and the related issues in a fair evaluation. With over 150 reported approaches to face detection, the research in face detection has broader implications for computer vision research on object recognition. Nearly all model-based or appearance-based approaches to 3D object recognition have been limited to rigid objects while attempting to robustly perform identification over a broad range of camera locations and illumination conditions. Face detection can be viewed as a two-class recognition problem in which an image region is classified as being a "face" or "nonface." Consequently, face detection is one of the few attempts to recognize from images (not abstract representations) a class of objects for which there is a great deal of within-class variability (described previously). It is also one of the few classes of objects for which this variability has been captured using large training sets of images and, so, some of the detection techniques may be applicable to a much broader class of recognition problems. Face detection also provides interesting challenges to the underlying pattern classification and learning techniques. When a raw or filtered image is considered as input to a pattern classifier, the dimension of the feature space is extremely large (i.e., the number of pixels in normalized training images). The classes of face and nonface images are decidedly characterized by multimodal distribution functions and effective decision boundaries are likely to be nonlinear in the image space. To be effective, either classifiers must be able to extrapolate from a modest number of training samples or be efficient when dealing with a very large number of these high-dimensional training samples. With an aim to give a comprehensive and critical survey of current face detection methods, this paper is organized as follows: In Section 2, we give a detailed review of techniques to detect faces in a single image. Benchmarking databases and evaluation criteria are discussed in Section 3. We conclude this paper with a discussion of several promising directions for face detection in Section 4.  Though we report error rates for each method when available, tests are often done on unique data sets and, so, comparisons are often difficult. We indicate those methods that have been evaluated with a publicly available test set. It can be assumed that a unique data set was used if we do not indicate the name of the test set.