A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> Two of the most critical requirements in support of producing reliable face-recognition systems are a large database of facial images and a testing procedure to evaluate systems. The Face Recognition Technology (FERET) program has addressed both issues through the FERET database of facial images and the establishment of the FERET tests. To date, 14,126 images from 1199 individuals are included in the FERET database, which is divided into development and sequestered portions. In September 1996, the FERET program administered the third in a series of FERET face-recognition tests. The primary objectives of the third test were to (1) assess the state of the art, (2) identify future areas of research, and (3) measure algorithm performance on large databases. <s> BIB001 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> The eyes are the most important facial landmarks on the human face. The accuracy of face normalization, which is critical to the performance of the following face analysis steps, depends on the locations of the two eyes, as well as their relatively constant interocular distance. In this paper, we propose a novel GaborEye model for eye localization. Based on the special gray distribution in the eye-and-brow region, a proper Gabor kernel is adaptively chosen to convolute with the face image to highlight the eye-and-brow region, which can be exploited to segment the two pupil regions efficiently. After getting the region of the pupil, a fast radial symmetry operator is used to locate the center of the pupil. Extensive experiments show that the method can accurately locate the pupils, and it is robust to the variations of face poses, expressions, accessories and illuminations. <s> BIB002 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> In this paper, we compare the performance of descriptors computed for local interest regions, as, for example, extracted by the Harris-Affine detector [Mikolajczyk, K and Schmid, C, 2004]. Many different descriptors have been proposed in the literature. It is unclear which descriptors are more appropriate and how their performance depends on the interest region detector. The descriptors should be distinctive and at the same time robust to changes in viewing conditions as well as to errors of the detector. Our evaluation uses as criterion recall with respect to precision and is carried out for different image transformations. We compare shape context [Belongie, S, et al., April 2002], steerable filters [Freeman, W and Adelson, E, Setp. 1991], PCA-SIFT [Ke, Y and Sukthankar, R, 2004], differential invariants [Koenderink, J and van Doorn, A, 1987], spin images [Lazebnik, S, et al., 2003], SIFT [Lowe, D. G., 1999], complex filters [Schaffalitzky, F and Zisserman, A, 2002], moment invariants [Van Gool, L, et al., 1996], and cross-correlation for different types of interest regions. We also propose an extension of the SIFT descriptor and show that it outperforms the original method. Furthermore, we observe that the ranking of the descriptors is mostly independent of the interest region detector and that the SIFT-based descriptors perform best. Moments and steerable filters show the best performance among the low dimensional descriptors. <s> BIB003 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> The accuracy of face alignment affects the performance of a face recognition system. Since face alignment is usually conducted using eye positions, an accurate eye localization algorithm is therefore essential for accurate face recognition. In this paper, we first study the impact of eye locations on face recognition accuracy, and then introduce an automatic technique for eye detection. The performance of our automatic eye detection technique is subsequently validated using FRGC 1.0 database. The validation shows that our eye detector has an overall 94.5% eye detection rate, with the detected eyes very close to the manually provided eye positions. In addition, the face recognition performance based on the automatic eye detection is shown to be comparable to that of using manually given eye positions. <s> BIB004 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> We address the task of accurately localizing the eyes in face images extracted by a face detector, an important problem to be solved because of the negative effect of poor localization on face recognition accuracy. We investigate three approaches to the task: a regression approach aiming to directly minimize errors in the predicted eye positions, a simple Bayesian model of eye and non-eye appearance, and a discriminative eye detector trained using AdaBoost. By using identical training and test data for each method we are able to perform an unbiased comparison. We show that, perhaps surprisingly, the simple Bayesian approach performs best on databases including challenging images, and performance is comparable to more complex state-of-the-art methods. <s> BIB005 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> This paper presents a novel and efficient facial image representation based on local binary pattern (LBP) texture features. The face image is divided into several regions from which the LBP feature distributions are extracted and concatenated into an enhanced feature vector to be used as a face descriptor. The performance of the proposed method is assessed in the face recognition problem under different challenges. Other applications and several extensions are also discussed <s> BIB006 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> We present a method for precise eye localization that uses two Support Vector Machines trained on properly selected Haar wavelet coefficients. The evaluation of our technique on many standard databases exhibits very good performance. Furthermore, we study the strong correlation between the eye localization error and the face recognition rate. <s> BIB007 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> Automatic facial feature localization has been a long-standing challenge in the field of computer vision for several decades. This can be explained by the large variation a face in an image can have due to factors such as position, facial expression, pose, illumination, and background clutter. Support Vector Machines (SVMs) have been a popular statistical tool for facial feature detection. Traditional SVM approaches to facial feature detection typically extract features from images (e.g. multiband filter, SIFT features) and learn the SVM parameters. Independently learning features and SVM parameters might result in a loss of information related to the classification process. This paper proposes an energy-based framework to jointly perform relevant feature weighting and SVM parameter learning. Preliminary experiments on standard face databases have shown significant improvement in speed with our approach. <s> BIB008 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> In this paper, we describe a novel type of feature for fast and accurate face detection. The feature is called Locally Assembled Binary (LAB) Haar feature. LAB feature is basically inspired by the success of Haar feature and Local Binary Pattern (LBP) for face detection, but it is far beyond a simple combination. In our method, Haar features are modified to keep only the ordinal relationship (named by binary Haar feature) rather than the difference between the accumulated intensities. Several neighboring binary Haar features are then assembled to capture their co-occurrence with similar idea to LBP. We show that the feature is more efficient than Haar feature and LBP both in discriminating power and computational cost. Furthermore, a novel efficient detection method called feature-centric cascade is proposed to build an efficient detector, which is developed from the feature-centric method. Experimental results on the CMU+MIT frontal face test set and CMU profile test set show that the proposed method can achieve very good results and amazing detection speed. <s> BIB009 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> In this paper we address the problem of eye localization for the purpose of face matching in low and standard definition image and video content. In addition to an explorative study that aimed at discovering the effect of eye localization accuracy on face matching performance, we also present a probabilistic eye localization method based on well-known multi-scale local binary patterns (LBPs). These patterns provide a simple but powerful spatial description of texture, and are robust to the noise typical to low and standard definition content. The extensive evaluation involving multiple eye localizers and face matchers showed that the shape of the eye localizer error distribution has a big impact on face matching performance. Conditioned by the error distribution shape and the minimum required eye localization accuracy, eye localization can boost the performance of naive face matchers and allow for more efficient face matching without degrading its performance. The evaluation also showed that our proposed method has superior accuracy with respect to the state-of-the-art on eye localization, and that it fulfills the criteria for improving the face matching performance and efficiency mentioned above. <s> BIB010 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> Making recognition more reliable under uncontrolled lighting conditions is one of the most important challenges for practical face recognition systems. We tackle this by combining the strengths of robust illumination normalization, local texture-based face representations, distance transform based matching, kernel-based feature extraction and multiple feature fusion. Specifically, we make three main contributions: 1) we present a simple and efficient preprocessing chain that eliminates most of the effects of changing illumination while still preserving the essential appearance details that are needed for recognition; 2) we introduce local ternary patterns (LTP), a generalization of the local binary pattern (LBP) local texture descriptor that is more discriminant and less sensitive to noise in uniform regions, and we show that replacing comparisons based on local spatial histograms with a distance transform based similarity metric further improves the performance of LBP/LTP based face recognition; and 3) we further improve robustness by adding Kernel principal component analysis (PCA) feature extraction and incorporating rich local appearance cues from two complementary sources-Gabor wavelets and LBP-showing that the combination is considerably more accurate than either feature set alone. The resulting method provides state-of-the-art performance on three data sets that are widely used for testing recognition under difficult illumination conditions: Extended Yale-B, CAS-PEAL-R1, and Face Recognition Grand Challenge version 2 experiment 4 (FRGC-204). For example, on the challenging FRGC-204 data set it halves the error rate relative to previously published methods, achieving a face verification rate of 88.1% at 0.1% false accept rate. Further experiments show that our preprocessing method outperforms several existing preprocessors for a range of feature sets, data sets and lighting conditions. <s> BIB011 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> We present a novel approach to address the representation issue and the matching issue in face recognition (verification). Firstly, our approach encodes the micro-structures of the face by a new learning-based encoding method. Unlike many previous manually designed encoding methods (e.g., LBP or SIFT), we use unsupervised learning techniques to learn an encoder from the training examples, which can automatically achieve very good tradeoff between discriminative power and invariance. Then we apply PCA to get a compact face descriptor. We find that a simple normalization mechanism after PCA can further improve the discriminative ability of the descriptor. The resulting face representation, learning-based (LE) descriptor, is compact, highly discriminative, and easy-to-extract. To handle the large pose variation in real-life scenarios, we propose a pose-adaptive matching method that uses pose-specific classifiers to deal with different pose combinations (e.g., frontal v.s. frontal, frontal v.s. left) of the matching face pair. Our approach is comparable with the state-of-the-art methods on the Labeled Face in Wild (LFW) benchmark (we achieved 84.45% recognition rate), while maintaining excellent compactness, simplicity, and generalization ability across different datasets. <s> BIB012 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> We present a generic detection/localization algorithm capable of searching for a visual object of interest without training. The proposed method operates using a single example of an object of interest to find similar matches, does not require prior knowledge (learning) about objects being sought, and does not require any preprocessing step or segmentation of a target image. Our method is based on the computation of local regression kernels as descriptors from a query, which measure the likeness of a pixel to its surroundings. Salient features are extracted from said descriptors and compared against analogous features from the target image. This comparison is done using a matrix generalization of the cosine similarity measure. We illustrate optimality properties of the algorithm using a naive-Bayes framework. The algorithm yields a scalar resemblance map, indicating the likelihood of similarity between the query and all patches in the target image. By employing nonparametric significance tests and nonmaxima suppression, we detect the presence and location of objects similar to the given query. The approach is extended to account for large variations in scale and rotation. High performance is demonstrated on several challenging data sets, indicating successful detection of objects in diverse contexts and under different imaging conditions. <s> BIB013 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> The Scale-Invariant Feature Transform (or SIFT) algorithm is a highly robust method to extract and consequently match distinctive invariant features from images. These features can then be used to reliably match objects in diering images. The algorithm was rst proposed by Lowe [12] and further developed to increase performance resulting in the classic paper [13] that served as foundation for SIFT which has played an important role in robotic and machine vision in the past decade. <s> BIB014 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Appearance features extraction and representation <s> This paper presents a new eye localization method via Multiscale Sparse Dictionaries (MSD). We built a pyramid of dictionaries that models context information at multiple scales. Eye locations are estimated at each scale by fitting the image through sparse coefficients of the dictionary. By using context information, our method is robust to various eye appearances. The method also works efficiently since it avoids sliding a search window in the image during localization. The experiments in BioID database prove the effectiveness of our method. <s> BIB015
To build a reliable statistic appearance model, one has to decide at first where and how to obtain a proper appearance representation. Most methods extract appearance features from a small patch image of eyes. Some subtle but important considerations are needed here. For example, how large should the patch be? Should it cover the pupil, the iris, and the eyelids only, or should it cover much context like the eyebrow? Unfortunately, most current discussions on these topics are empirical in nature. Intuitively, a relatively larger eye patch contains more discriminative information and thus tends to reduce the risk of false positive, but this will be at the cost of losing generalization capability as they will be less likely to be good representatives of eyes. In BIB002 , it is shown that incorporating the eyebrow into the eye patch leads to better localization performance on FERET BIB001 and YALE  face databases. But in general, a cross validation procedure is recommended to search the best suitable setting BIB005 . Given an eye patch, several visual feature sets, from middle to high level, can be extracted from it. Each feature set is simply a transformation of a set of neighboring raw pixel values, designed to be invariant to certain changes. Since no single feature descriptor will satisfy all the needs, selecting the ones to use in practice is mostly application-driven and the factors that need to be taken into account mainly include: (1) the invariance properties it provides, e.g., to lighting changes or to variations in scale, orientation, and other affine transformations; (2) the information encoded and the discriminability preserved; (3) the computational efficiency. The first criterion helps to clarify which kind of change one wishes to compensate for after preprocessing the patches. The second one is essential to the subsequent statistical modeling/processing, while the third one concerns the computational aspects of the feature set. Popular feature set descriptors include those in frequency domain, e.g., Harr wavelets features, Gabor features, and those in the spatial domain, especially various gradient-based features, such as Local Binary Patterns (LBP, BIB006 ), Scale Invariant Feature Transform (SIFT BIB014 ) and Gradient Location-Orientation Histogram (GLOH BIB003 ). Some of them are illustrated in Fig. 6 . We are not aware of any comparative work concerning the effect of different feature sets on the task of eye localization, but in the more general context of computer vision, Mikolajczyk and Schmid BIB003 reviewed some recently developed view-invariant local image descriptors and experimentally compared their performance. They found that GLOH BIB003 and SIFT BIB014 are the two best performed descriptors among others in their settings. In BIB010 , the authors presented a method based on the multiscale LBP feature sets for eye localization in low and standard definition content (see  with multi-scale Gabor feature sets). In general LBP feature BIB006 is good at coding the details of appearance and texture, whereas Gabor features encode global shape and appearance over a range of coarse scales. Both representations are rich in information and computationally efficient and hence are widely used in the field of facial analysis. It is worth mentioning that eye localization is a complex task for which it is useful to include a lighting normalization stage before feature extraction and combine multiple types of feature sets BIB011 . The multi-scale method used in BIB010 is helpful to alleviate the problem of choosing the right patch for eye representation by properly fusing multi-scale context information. A similar idea is adopted in BIB015 , where a pyramid of dictionaries is offline built at multiple scales (see Fig. 7 ). For online localization, the dictionaries are sequentially applied from the largest scale to the smallest one. This fitting procedure, however, involves solving an l 1 problem and is generally time consuming. Besides usual feature descriptors which directly apply certain linear or nonlinear transformations on the given eye patches, recently there has been a trend to construct feature sets statistically BIB004 BIB008 BIB007 BIB009 BIB012 BIB013 . Usually these feature sets are obtained by embedding the aforementioned middle-level feature sets into another feature space with desired properties (e.g., compact, discriminative, etc.). These welcome properties are introduced by learning from training examples either discriminatively or generatively. For example, in BIB004 the authors presented a method which learned an optimal discriminant feature space from training data by minimizing the empirical Bayes error using nonparametric discriminant analysis. Since the learned feature space is tuned to highlight the contrast between the positive eye samples and the false ones, it is expected to be able to characterize eye patterns better. In BIB008 , the authors proposed an energy-based framework to jointly perform discriminative feature sets learning and eye localization, in which the parameters of statistical feature sets are optimized to maximize the generalization performance of an SVM classifier. Similarly, BIB007 proposed to group different sets of features hierarchically and built a cascade of SVM classifiers on them (see Fig. 8 ). Compared to common feature descriptors, the major advantage of statistic-based feature sets lies in their stableness in handling uncertainty of image data, but at the cost of more computational efforts and of the needs for plenty of representative training data to ensure good performance.