A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Evaluation Metrics and Comparison <s> Facial landmark detection has long been impeded by the problems of occlusion and pose variation. Instead of treating the detection task as a single and independent problem, we investigate the possibility of improving detection robustness through multi-task learning. Specifically, we wish to optimize facial landmark detection together with heterogeneous but subtly correlated tasks, e.g. head pose estimation and facial attribute inference. This is non-trivial since different tasks have different learning difficulties and convergence rates. To address this problem, we formulate a novel tasks-constrained deep model, with task-wise early stopping to facilitate learning convergence. Extensive evaluations show that the proposed task-constrained learning (i) outperforms existing methods, especially in dealing with faces with severe occlusion and pose variation, and (ii) reduces model complexity drastically compared to the state-of-the-art method based on cascaded deep model [21]. <s> BIB001 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Evaluation Metrics and Comparison <s> Abstract This paper presents a comprehensive survey of facial feature point detection with the assistance of abundant manually labeled images. Facial feature point detection favors many applications such as face recognition, animation, tracking, hallucination, expression analysis and 3D face modeling. Existing methods are categorized into two primary categories according to whether there is the need of a parametric shape model: parametric shape model-based methods and nonparametric shape model-based methods. Parametric shape model-based methods are further divided into two secondary classes according to their appearance models: local part model-based methods (e.g. constrained local model) and holistic model-based methods (e.g. active appearance model). Nonparametric shape model-based methods are divided into several groups according to their model construction process: exemplar-based methods, graphical model-based methods, cascaded regression-based methods, and deep learning based methods. Though significant progress has been made, facial feature point detection is still limited in its success by wild and real-world conditions: large variations across poses, expressions, illuminations, and occlusions. A comparative illustration and analysis of representative methods provides us a holistic understanding and deep insight into facial feature point detection, which also motivates us to further explore more promising future schemes. <s> BIB002 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Evaluation Metrics and Comparison <s> In this paper, we propose Deep Alignment Network (DAN), a robust face alignment method based on a deep neural network architecture. DAN consists of multiple stages, where each stage improves the locations of the facial landmarks estimated by the previous stage. Our method uses entire face images at all stages, contrary to the recently proposed face alignment methods that rely on local patches. This is possible thanks to the use of landmark heatmaps which provide visual information about landmark locations estimated at the previous stages of the algorithm. The use of entire face images rather than patches allows DAN to handle face images with large variation in head pose and difficult initializations. An extensive evaluation on two publicly available datasets shows that DAN reduces the state-of-the-art failure rate by up to 70%. Our method has also been submitted for evaluation as part of the Menpo challenge. <s> BIB003
For providing comparable results regardless of the image size and camera focus, in the literature it is common to measure the distance between the ground truth and the detection result by Normalized Mean Error (NME) e, calculated as: where S, and S * represent the detected shape and the shape of the ground truth, respectively. d * is a normalizing distance, which could be the inter-ocular distance (IOD) or inter-pupil distance. Many times, it is used as the bounding box diagonal or geometric mean of image length, with the height as d * instead if the distance between the two eyes is too small on 3D/large pose datasets. Another metric that is based on the N ME is the Cumulative Error Distribution (CED) curve. The CED generally represents the proportion of images in the test set having an error below a given threshold. This curve provides a visual result of the algorithm performance in different situations, and the Area Under the Curve (AUC) provides a qualitative result of how the algorithm performs at progressive mean errors: where e is the normalized error, f (e) is the cumulative error distribution (CED) function, and α is the upper bound that is used to calculate the definite integration. A bigger AUC value generally means that the algorithm has better performance. The failure rate is used to measure the robustness of an algorithm. A threshold of NME was chosen to be a threshold of failure, and the proportion of failed detection was calculated to represent the capacity of handling the difficult images. Now, we are ready to provide a comparison of different facial landmark extraction methods, as well as different deep-compression models. This comparison includes traditional cascaded regression methods and deep-learning-based facial landmark extraction methods. Zhang et al. BIB001 and Kowalski et al. BIB003 both provide a good benchmark on several popular methods by measuring the normalized inter-ocular distance error. Table 1 shows the performance of seven non-deep-learning 2D facial landmark extraction methods and eight deep learning facial landmark extraction methods evaluated on 300W by the normalized inter-ocular distance error. The failure rate is not included in the table since the choice of threshold is not objective. Table 2 reports a comparison of the Mean Error (%) (normalized by face size) of different video facial landmark extraction methods on 300VW. In general, the deep-learning-based algorithms outperform the others. However, considering the performances, all of the cascaded-regression based methods can run in real-time even with a Matlab implementation BIB002 , while some deep-learning-based methods can achieve real-time detection on a powerful GPU or CPU, but most runtimes on CPU are not satisfying. 