A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> A machine for screwing down the spoke nuts of a bicycle wheel while controlling the torque applied to each nut in response to the position of the nut on the wheel in relation to the order of tightening of the nuts. The torque applied is progressively greater from nut to nut starting from the first until a maximum torque is applied after the spokes over half the wheel circumference have been tightened. <s> BIB001 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> In this paper we present a computationally efficient framework for part-based modeling and recognition of objects. Our work is motivated by the pictorial structure models introduced by Fischler and Elschlager. The basic idea is to represent an object by a collection of parts arranged in a deformable configuration. The appearance of each part is modeled separately, and the deformable configuration is represented by spring-like connections between pairs of parts. These models allow for qualitative descriptions of visual appearance, and are suitable for generic recognition problems. We address the problem of using pictorial structure models to find instances of an object in an image as well as the problem of learning an object model from training examples, presenting efficient algorithms in both cases. We demonstrate the techniques by learning models that represent faces and human bodies and using the resulting models to locate the corresponding objects in novel images. <s> BIB002 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> The accuracy of face alignment affects the performance of a face recognition system. Since face alignment is usually conducted using eye positions, an accurate eye localization algorithm is therefore essential for accurate face recognition. In this paper, we first study the impact of eye locations on face recognition accuracy, and then introduce an automatic technique for eye detection. The performance of our automatic eye detection technique is subsequently validated using FRGC 1.0 database. The validation shows that our eye detector has an overall 94.5% eye detection rate, with the detected eyes very close to the manually provided eye positions. In addition, the face recognition performance based on the automatic eye detection is shown to be comparable to that of using manually given eye positions. <s> BIB003 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> Sliding window classifiers are among the most successful and widely applied techniques for object localization. However, training is typically done in a way that is not specific to the localization task. First a binary classifier is trained using a sample of positive and negative examples, and this classifier is subsequently applied to multiple regions within test images. We propose instead to treat object localization in a principled way by posing it as a problem of predicting structured data: we model the problem not as binary classification, but as the prediction of the bounding box of objects located in images. The use of a joint-kernelframework allows us to formulate the training procedure as a generalization of an SVM, which can be solved efficiently. We further improve computational efficiency by using a branch-and-bound strategy for localization during both training and testing. Experimental evaluation on the PASCAL VOC and TU Darmstadt datasets show that the structured training procedure improves performance over binary training as well as the best previously published scores. <s> BIB004 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> Straightforward classification using kernelized SVMs requires evaluating the kernel for a test vector and each of the support vectors. For a class of kernels we show that one can do this much more efficiently. In particular we show that one can build histogram intersection kernel SVMs (IKSVMs) with runtime complexity of the classifier logarithmic in the number of support vectors as opposed to linear for the standard approach. We further show that by precomputing auxiliary tables we can construct an approximate classifier with constant runtime and space requirements, independent of the number of support vectors, with negligible loss in classification accuracy on various tasks. This approximation also applies to 1 - chi2 and other kernels of similar form. We also introduce novel features based on a multi-level histograms of oriented edge energy and present experiments on various detection datasets. On the INRIA pedestrian dataset an approximate IKSVM classifier based on these features has the current best performance, with a miss rate 13% lower at 10-6 False Positive Per Window than the linear SVM detector of Dalal & Triggs. On the Daimler Chrysler pedestrian dataset IKSVM gives comparable accuracy to the best results (based on quadratic SVM), while being 15times faster. In these experiments our approximate IKSVM is up to 2000times faster than a standard implementation and requires 200times less memory. Finally we show that a 50times speedup is possible using approximate IKSVM based on spatial pyramid features on the Caltech 101 dataset with negligible loss of accuracy. <s> BIB005 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> We have created the first image search engine based entirely on faces. Using simple text queries such as "smiling men with blond hair and mustaches," users can search through over 3.1 million faces which have been automatically labeled on the basis of several facial attributes. Faces in our database have been extracted and aligned from images downloaded from the internet using a commercial face detector, and the number of images and attributes continues to grow daily. Our classification approach uses a novel combination of Support Vector Machines and Adaboost which exploits the strong structure of faces to select and train on the optimal set of features for each attribute. We show state-of-the-art classification results compared to previous works, and demonstrate the power of our architecture through a functional, large-scale face search engine. Our framework is fully automatic, easy to scale, and computes all labels off-line, leading to fast on-line search performance. In addition, we describe how our system can be used for a number of applications, including law enforcement, social networks, and personal photo management. Our search engine will soon be made publicly available. <s> BIB006 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> We present methods for training high quality object detectors very quickly. The core contribution is a pair of fast training algorithms for piece-wise linear classifiers, which can approximate arbitrary additive models. The classifiers are trained in a max-margin framework and significantly outperform linear classifiers on a variety of vision datasets. We report experimental results quantifying training time and accuracy on image classification tasks and pedestrian detection, including detection results better than the best previous on the INRIA dataset with faster training. <s> BIB007 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> The literature on the topic has shown a strong correlation between the degree of precision of face localization and the face recognition performance. Hence, there is a need for precise facial feature detectors, as well as objective measures for their evaluation and comparison. In this paper, we will present significant improvements to a previous method for precise eye center localization, by integrating a module for mouth localization. The technique is based on Support Vector Machines trained on optimally chosen Haar wavelet coefficients. The method has been tested on several public databases; the results are reported and compared according to a standard error measure. The tests show that the algorithm achieves high precision of localization. <s> BIB008 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> This paper introduces a class of correlation filters called average of synthetic exact filters (ASEF). For ASEF, the correlation output is completely specified for each training image. This is in marked contrast to prior methods such as synthetic discriminant functions (SDFs) which only specify a single output value per training image. Advantages of ASEF training include: insensitivity to over-fitting, greater flexibility with regard to training images, and more robust behavior in the presence of structured backgrounds. The theory and design of ASEF filters is presented using eye localization on the FERET database as an example task. ASEF is compared to other popular correlation filters including SDF, MACE, OTF, and UMACE, and with other eye localization methods including Gabor Jets and the OpenCV cascade classifier. ASEF is shown to outperform all these methods, locating the eye to within the radius of the iris approximately 98.5% of the time. <s> BIB009 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> We present two novel methods for face verification. Our first method - “attribute” classifiers - uses binary classifiers trained to recognize the presence or absence of describable aspects of visual appearance (e.g., gender, race, and age). Our second method - “simile” classifiers - removes the manual labeling required for attribute classification and instead learns the similarity of faces, or regions of faces, to specific reference people. Neither method requires costly, often brittle, alignment between image pairs; yet, both methods produce compact visual descriptions, and work on real-world images. Furthermore, both the attribute and simile classifiers improve on the current state-of-the-art for the LFW data set, reducing the error rates compared to the current best by 23.92% and 26.34%, respectively, and 31.68% when combined. For further testing across pose, illumination, and expression, we introduce a new data set - termed PubFig - of real-world images of public figures (celebrities and politicians) acquired from the internet. This data set is both larger (60,000 images) and deeper (300 images per individual) than existing data sets of its kind. Finally, we present an evaluation of human performance. <s> BIB010 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> To detect multiple objects of interest, the methods based on Hough transform use non-maxima supression or mode seeking in order to locate and to distinguish peaks in Hough images. Such postprocessing requires tuning of extra parameters and is often fragile, especially when objects of interest tend to be closely located. In the paper, we develop a new probabilistic framework that is in many ways related to Hough transform, sharing its simplicity and wide applicability. At the same time, the framework bypasses the problem of multiple peaks identification in Hough images, and permits detection of multiple objects without invoking nonmaximum suppression heuristics. As a result, the experiments demonstrate a significant improvement in detection accuracy both for the classical task of straight line detection and for a more modern category-level (pedestrian) detection problem. <s> BIB011 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> We describe an object detection system based on mixtures of multiscale deformable part models. Our system is able to represent highly variable object classes and achieves state-of-the-art results in the PASCAL object detection challenges. While deformable part models have become quite popular, their value had not been demonstrated on difficult benchmarks such as the PASCAL data sets. Our system relies on new methods for discriminative training with partially labeled data. We combine a margin-sensitive approach for data-mining hard negative examples with a formalism we call latent SVM. A latent SVM is a reformulation of MI--SVM in terms of latent variables. A latent SVM is semiconvex, and the training problem becomes convex once latent information is specified for the positive examples. This leads to an iterative training algorithm that alternates between fixing latent values for positive examples and optimizing the latent SVM objective function. <s> BIB012 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> Making recognition more reliable under uncontrolled lighting conditions is one of the most important challenges for practical face recognition systems. We tackle this by combining the strengths of robust illumination normalization, local texture-based face representations, distance transform based matching, kernel-based feature extraction and multiple feature fusion. Specifically, we make three main contributions: 1) we present a simple and efficient preprocessing chain that eliminates most of the effects of changing illumination while still preserving the essential appearance details that are needed for recognition; 2) we introduce local ternary patterns (LTP), a generalization of the local binary pattern (LBP) local texture descriptor that is more discriminant and less sensitive to noise in uniform regions, and we show that replacing comparisons based on local spatial histograms with a distance transform based similarity metric further improves the performance of LBP/LTP based face recognition; and 3) we further improve robustness by adding Kernel principal component analysis (PCA) feature extraction and incorporating rich local appearance cues from two complementary sources-Gabor wavelets and LBP-showing that the combination is considerably more accurate than either feature set alone. The resulting method provides state-of-the-art performance on three data sets that are widely used for testing recognition under difficult illumination conditions: Extended Yale-B, CAS-PEAL-R1, and Face Recognition Grand Challenge version 2 experiment 4 (FRGC-204). For example, on the challenging FRGC-204 data set it halves the error rate relative to previously published methods, achieving a face verification rate of 88.1% at 0.1% false accept rate. Further experiments show that our preprocessing method outperforms several existing preprocessors for a range of feature sets, data sets and lighting conditions. <s> BIB013 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> Despite active research and significant progress in the last 30 years, eye detection and tracking remains challenging due to the individuality of eyes, occlusion, variability in scale, location, and light conditions. Data on eye location and details of eye movements have numerous applications and are essential in face detection, biometric identification, and particular human-computer interaction tasks. This paper reviews current progress and state of the art in video-based eye detection and tracking in order to identify promising techniques as well as issues to be further addressed. We present a detailed review of recent eye models and techniques for eye detection and tracking. We also survey methods for gaze estimation and compare them based on their geometric properties and reported accuracies. This review shows that, despite their apparent simplicity, the development of a general eye detection technique involves addressing many challenges, requires further theoretical developments, and is consequently of interest to many other domains problems in computer vision and beyond. <s> BIB014 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> Face alignment is a crucial step in face recognition tasks. Especially, using landmark localization for geometric face normalization has shown to be very effective, clearly improving the recognition results. However, no adequate databases exist that provide a sufficient number of annotated facial landmarks. The databases are either limited to frontal views, provide only a small number of annotated images or have been acquired under controlled conditions. Hence, we introduce a novel database overcoming these limitations: Annotated Facial Landmarks in the Wild (AFLW). AFLW provides a large-scale collection of images gathered from Flickr, exhibiting a large variety in face appearance (e.g., pose, expression, ethnicity, age, gender) as well as general imaging and environmental conditions. In total 25,993 faces in 21,997 real-world images are annotated with up to 21 landmarks per image. Due to the comprehensive set of annotations AFLW is well suited to train and test algorithms for multi-view face detection, facial landmark localization and face pose estimation. Further, we offer a rich set of tools that ease the integration of other face databases and associated annotations into our joint framework. <s> BIB015 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> We present a unified model for face detection, pose estimation, and landmark estimation in real-world, cluttered images. Our model is based on a mixtures of trees with a shared pool of parts; we model every facial landmark as a part and use global mixtures to capture topological changes due to viewpoint. We show that tree-structured models are surprisingly effective at capturing global elastic deformation, while being easy to optimize unlike dense graph structures. We present extensive results on standard face benchmarks, as well as a new “in the wild” annotated dataset, that suggests our system advances the state-of-the-art, sometimes considerably, for all three tasks. Though our model is modestly trained with hundreds of faces, it compares favorably to commercial systems trained with billions of examples (such as Google Picasa and face.com). <s> BIB016 </s> A literature survey on robust and efficient eye localization in real-life scenarios <s> Conclusion and prospect <s> In this paper we describe a detector of facial landmarks based on the Deformable Part Models. We treat the task of landmark detection as an instance of the structured output classification problem. We propose to learn the parameters of the detector from data by the Structured Output Support Vector Machines algorithm. In contrast to the previous works, the objective function of the learning algorithm is directly related to the performance of the resulting detector which is controlled by a user-defined loss function. The resulting detector is real-time on a standard PC, simple to implement and it can be easily modified for detection of a different set of landmarks. We evaluate performance of the proposed landmark detector on a challenging “Labeled Faces in the Wild” (LFW) database. The empirical results demonstrate that the proposed detector is consistently more accurate than two public domain implementations based on the Active Appearance Models and the Deformable Part Models. We provide an open-source implementation of the proposed detector and the manual annotation of the facial landmarks for all images in the LFW database. <s> BIB017
Eyes are arguably one of the most salient features of the human face, and locating eyes precisely and efficiently is of great importance for a wide range of real-world applications. It is not uncommon to mistakenly think that eye localization is a simple task since eyes are just a simple structure in the face. However, eyes have its unique geometric, photometric and motion characteristics, and changes due to these three characteristics would lead to a very complicated nonlinear manifold of eyes. Developing effective eye models to describe this manifold and performing efficient searching within this manifold is therefore of both academic and practical interest. In the recent thirty years, intensive studies on this problem have resulted in a great amount of achievements. BIB015 http://lrs.icg.tugraz.at/research/aflw/ IMM Face BIB001 http://www2.imm.dtu.dk/aam/datasets/ datasets.html Code and toolBox Hough transform BIB011 http://graphics.cs.msu.ru/en/science/ research/machinelearning/hough Structured output regression BIB004 https://sites.google.com/site/ christophlampert/software Pictorial structures BIB002 BIB012 http://people.cs.uchicago.edu/ rbg/latent/ Enhanced Pictorial Model  http://parnec.nuaa.edu.cn/xtan/data/ eyedetector.html Max-Margin Additive BIB007 http://www.cs.berkeley.edu/ smaji/ projects/add-models/ Fast Intersection Kernel BIB005 http://www.cs.berkeley.edu/ smaji/ projects/fiksvm/ Lighting processing BIB013 http://lear.inrialpes.fr/people/triggs/src/ amfg07-demo-v1.tar.gz Landmark Localization BIB016 http://www.ics.uci.edu/ xzhu/face/ Eye+Mouth BIB008 http://homes.di.unimi.it/ lipori/download. html Annotation tool BIB017 http://cmp.felk.cvut.cz/ uricamic/ flandmark/ OCOFTools BIB009 http://www.cs.colostate.edu/ ross/ ocof_toolset_2012/ Fig. 13 . The 3D model used to define the centers of the eyes and other facial landmarks BIB015 . In this paper, we attempt to provide a comprehensive survey of current researches in this area. In particular, we have focused on the overall difficulties and challenges in real-life scenarios and described the current state of the art in dealing with these challenges. The ad hoc methods are mostly reviewed from the perspective of pattern recognition and computer vision with the bias on how to learn generalizable, flexible and efficient statistical eye models from a small number of training images, which, we believe, could help us to develop a clear understanding of this particular problem. We also pay much attention to the practical issues concerning the development of a robust eye localization system, ranging from various preprocessing methods to postprocessing ones. Furthermore, we reviewed the relative localization performance of current algorithms on popular face databases and discussed various factors (e.g., the design of ground truth) that should be taken into account when evaluating a system. Finally, it is worth mentioning that some closely related problems are deliberately ignored in this paper, such as gaze estimation and eye tracking, which are also very important in practice. For these topics, we refer to BIB014 for a detailed discussion. Despite of many efforts devoted to eye localization during the last several decades, we have to admit that this problem is far from being resolved and several promising research directions could be suggested. Firstly, from the application's point of view, locating eyes under uneven lighting and occlusion, variable face pose, and low image resolution remains largely problematic. Each of these difficulties is non-trivial and is of great academic interest as well. While the final solution of these problems depends heavily on the advance of related areas, such as computer vision, pattern recognition and machine learning, promising progress can be made in developing new problem-specific but flexible feature extraction techniques and constructing effective mathematical model for lighting, pose, occlusion and image enhancement. In addition, considering that human can perform the task of object detection and recognition task effectively and efficiently, insights into the mechanism of human brain and neural network will definitely promote development in eye localization, for example, through deriving novel visual attention model. Secondly, accuracy and efficiency are conflicting in the modeling of eyes. It is still worth discussing on how to find feature sets and to develop optimization techniques compromising the requirements for both the accuracy and efficiency. 'Coarse-to-fine' and 'branch-and-bound' are currently two major fast searching techniques for real-time eye localization. However, constructing a BIB003 4715 (v1) 99.0 a The error measured in pixels is converted into normalized localization error (cf. Eq. (2)) by assuming that the eyes distance is 40 pixels for faces in LFW  . cascade of classifiers with increasing complexity may miss the global solution, while the branch and bound searching avoids this problem but its usage depends on the heuristic exploring of specific model structure. We are therefore expecting to see more effective optimization models in the next few years which accurately model the eyes while being fitted efficiently. Thirdly, there are needs to develop benchmark eye databases with carefully designed ground truth. Although current face databases contain useful eye patterns, they are not originally designed for eye localization and the ground truth are mainly provided for face recognition, which cannot be used for other eyerelated tasks such as gaze estimation. Furthermore, many face images collected in the laboratory have eyes looking in the frontal direction, which is rarely the case in reality. While the LFW database  is closed to the real-life scenarios, it lacks ground truth for eyes partly due to the difficulties involved in labeling those realistic face images. Fortunately, we see that at least two large-scale annotated face databases, i.e., FaceTracer BIB006 and AFLW BIB015 with images harvested from the web, have emerged to meet the need for ground truth. Finally, the question of whether the application areas for eye localization actually need precise eye locations bears mentioning. Specifically, it is usually assumed that better face alignment leads to better performance. However, it has been shown that accurate recognition is possible (e.g., using high-level attribute features BIB010 ) without requiring face alignment and, hence, do not rely on eye localization. Will the existence of such robust methods remove the need for eye localization algorithms? Probably not, but this is a topic that needs to be addressed. After all, it is well-known that the warping operation has the side effect of image distortion. Nevertheless, the aforementioned difficulties do not mean that addressing this problem cannot be achieved under the current technique framework. As shown in Table 3 , considerable efforts in this field are very encouraging. Due to the inherent complexity of this problem and its wide practical applications, we believe that this area will draw increasing attention from a variety of fields beyond computer vision, pattern recognition and machine learning.