A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We demonstrate a fast, robust method of interpreting face images using an Active Appearance Model (AAM). An AAM contains a statistical model of shape and grey level appearance which can generalise to almost any face. Matching to an image involves finding model parameters which minimise the difference between the image and a synthesised face. We observe that displacing each model parameter from the correct value induces a particular pattern in the residuals. In a training phase, the AAM learns a linear model of the correlation between parameter displacements and the induced residuals. During search it measures the residuals and uses this model to correct the current parameters, leading to a better fit. A good overall match is obtained in a few iterations, even from poor starting estimates. We describe the technique in detail and show it matching to new face images. <s> BIB001 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Humans detect and interpret faces and facial expressions in a scene with little or no effort. Still, development of an automated system that accomplishes this task is rather difficult. There are several related problems: detection of an image segment as a face, extraction of the facial expression information, and classification of the expression (e.g., in emotion categories). A system that performs these operations accurately and in real time would form a big step in achieving a human-like interaction between man and machine. The paper surveys the past work in solving these problems. The capability of the human visual system with respect to these problems is discussed, too. It is meant to serve as an ultimate goal and a guide for determining recommendations for development of an automatic facial expression analyzer. <s> BIB002 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We describe a new method of matching statistical models of appearance to images. A set of model parameters control modes of shape and gray-level variation learned from a training set. We construct an efficient iterative matching algorithm by learning the relationship between perturbations in the model parameters and the induced image errors. <s> BIB003 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> The capacity to estimate the head pose of another person is a common human ability that presents a unique challenge for computer vision systems. Compared to face detection and recognition, which have been the primary foci of face-related vision research, identity-invariant head pose estimation has fewer rigorously evaluated systems or generic solutions. In this paper, we discuss the inherent difficulties in head pose estimation and present an organized survey describing the evolution of the field. Our discussion focuses on the advantages and disadvantages of each approach and spans 90 of the most innovative and characteristic papers that have been published on this topic. We compare these systems by focusing on their ability to estimate coarse and fine head pose, highlighting approaches that are well suited for unconstrained environments. <s> BIB004 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Despite active research and significant progress in the last 30 years, eye detection and tracking remains challenging due to the individuality of eyes, occlusion, variability in scale, location, and light conditions. Data on eye location and details of eye movements have numerous applications and are essential in face detection, biometric identification, and particular human-computer interaction tasks. This paper reviews current progress and state of the art in video-based eye detection and tracking in order to identify promising techniques as well as issues to be further addressed. We present a detailed review of recent eye models and techniques for eye detection and tracking. We also survey methods for gaze estimation and compare them based on their geometric properties and reported accuracies. This review shows that, despite their apparent simplicity, the development of a general eye detection technique involves addressing many challenges, requires further theoretical developments, and is consequently of interest to many other domains problems in computer vision and beyond. <s> BIB005 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB006 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We describe a very simple framework for deriving the most-well known optimization problems in Active Appearance Models (AAMs), and most importantly for providing efficient solutions. Our formulation results in two optimization problems for fast and exact AAM fitting, and one new algorithm which has the important advantage of being applicable to 3D. We show that the dominant cost for both forward and inverse algorithms is a few times mN which is the cost of projecting an image onto the appearance subspace. This makes both algorithms not only computationally realizable but also very attractive speed-wise for most current systems. Because exact AAM fitting is no longer computationally prohibitive, we trained AAMs in-the-wild with the goal of investigating whether AAMs benefit from such a training process. Our results show that although we did not use sophisticated shape priors, robust features or robust norms for improving performance, AAMs perform notably well and in some cases comparably with current state-of-the-art methods. We provide Matlab source code for training, fitting and reproducing the results presented in this paper at http://ibug.doc.ic.ac.uk/resources. <s> BIB007 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Many computer vision problems (e.g., camera calibration, image alignment, structure from motion) are solved through a nonlinear optimization method. It is generally accepted that 2nd order descent methods are the most robust, fast and reliable approaches for nonlinear optimization of a general smooth function. However, in the context of computer vision, 2nd order descent methods have two main drawbacks: (1) The function might not be analytically differentiable and numerical approximations are impractical. (2) The Hessian might be large and not positive definite. To address these issues, this paper proposes a Supervised Descent Method (SDM) for minimizing a Non-linear Least Squares (NLS) function. During training, the SDM learns a sequence of descent directions that minimizes the mean of NLS functions sampled at different points. In testing, SDM minimizes the NLS objective using the learned descent directions without computing the Jacobian nor the Hessian. We illustrate the benefits of our approach in synthetic and real examples, and show how SDM achieves state-of-the-art performance in the problem of facial feature detection. The code is available at www.humansensing.cs. cmu.edu/intraface. <s> BIB008 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We present a very efficient, highly accurate, "Explicit Shape Regression" approach for face alignment. Unlike previous regression-based approaches, we directly learn a vectorial regression function to infer the whole facial shape (a set of facial landmarks) from the image and explicitly minimize the alignment errors over the training data. The inherent shape constraint is naturally encoded into the regressor in a cascaded learning framework and applied from coarse to fine during the test, without using a fixed parametric shape model as in most previous methods. To make the regression more effective and efficient, we design a two-level boosted regression, shape indexed features and a correlation-based feature selection method. This combination enables us to learn accurate models from large training data in a short time (20 min for 2,000 training images), and run regression extremely fast in test (15 ms for a 87 landmarks shape). Experiments on challenging data show that our approach significantly outperforms the state-of-the-art in terms of both accuracy and efficiency. <s> BIB009 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Human faces captured in real-world conditions present large variations in shape and occlusions due to differences in pose, expression, use of accessories such as sunglasses and hats and interactions with objects (e.g. food). Current face landmark estimation approaches struggle under such conditions since they fail to provide a principled way of handling outliers. We propose a novel method, called Robust Cascaded Pose Regression (RCPR) which reduces exposure to outliers by detecting occlusions explicitly and using robust shape-indexed features. We show that RCPR improves on previous landmark estimation methods on three popular face datasets (LFPW, LFW and HELEN). We further explore RCPR's performance by introducing a novel face dataset focused on occlusion, composed of 1,007 faces presenting a wide range of occlusion patterns. RCPR reduces failure cases by half on all four datasets, at the same time as it detects face occlusions with a 80/40% precision/recall. <s> BIB010 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Face landmarking, defined as the detection and localization of certain characteristic points on the face, is an important intermediary step for many subsequent face processing operations that range from biometric recognition to the understanding of mental states. Despite its conceptual simplicity, this computer vision problem has proven extremely challenging due to inherent face variability as well as the multitude of confounding factors such as pose, expression, illumination and occlusions. The purpose of this survey is to give an overview of landmarking algorithms and their progress over the last decade, categorize them and show comparative performance statistics of the state of the art. We discuss the main trends and indicate current shortcomings with the expectation that this survey will provide further impetus for the much needed high-performance, real-life face landmarking operating at video rates. <s> BIB011 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> In modern face recognition, the conventional pipeline consists of four stages: detect => align => represent => classify. We revisit both the alignment step and the representation step by employing explicit 3D face modeling in order to apply a piecewise affine transformation, and derive a face representation from a nine-layer deep neural network. This deep network involves more than 120 million parameters using several locally connected layers without weight sharing, rather than the standard convolutional layers. Thus we trained it on the largest facial dataset to-date, an identity labeled dataset of four million facial images belonging to more than 4, 000 identities. The learned representations coupling the accurate model-based alignment with the large facial database generalize remarkably well to faces in unconstrained environments, even with a simple classifier. Our method reaches an accuracy of 97.35% on the Labeled Faces in the Wild (LFW) dataset, reducing the error of the current state of the art by more than 27%, closely approaching human-level performance. <s> BIB012 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> This paper addresses the problem of Face Alignment for a single image. We show how an ensemble of regression trees can be used to estimate the face's landmark positions directly from a sparse subset of pixel intensities, achieving super-realtime performance with high quality predictions. We present a general framework based on gradient boosting for learning an ensemble of regression trees that optimizes the sum of square error loss and naturally handles missing or partially labelled data. We show how using appropriate priors exploiting the structure of image data helps with efficient feature selection. Different regularization strategies and its importance to combat overfitting are also investigated. In addition, we analyse the effect of the quantity of training data on the accuracy of the predictions and explore the effect of data augmentation using synthesized data. <s> BIB013 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> In this paper we provide the first, to the best of our knowledge, Bayesian formulation of one of the most successful and well-studied statistical models of shape and texture, i.e. Active Appearance Models (AAMs). To this end, we use a simple probabilistic model for texture generation assuming both Gaussian noise and a Gaussian prior over a latent texture space. We retrieve the shape parameters by formulating a novel cost function obtained by marginalizing out the latent texture space. This results in a fast implementation when compared to other simultaneous algorithms for fitting AAMs, mainly due to the removal of the calculation of texture parameters. We demonstrate that, contrary to what is believed regarding the performance of AAMs in generic fitting scenarios, optimization of the proposed cost function produces results that outperform discriminatively trained state-of-the-art methods in the problem of facial alignment "in the wild". <s> BIB014 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> The development of facial databases with an abundance of annotated facial data captured under unconstrained 'in-the-wild' conditions have made discriminative facial deformable models the de facto choice for generic facial landmark localization. Even though very good performance for the facial landmark localization has been shown by many recently proposed discriminative techniques, when it comes to the applications that require excellent accuracy, such as facial behaviour analysis and facial motion capture, the semi-automatic person-specific or even tedious manual tracking is still the preferred choice. One way to construct a person-specific model automatically is through incremental updating of the generic model. This paper deals with the problem of updating a discriminative facial deformable model, a problem that has not been thoroughly studied in the literature. In particular, we study for the first time, to the best of our knowledge, the strategies to update a discriminative model that is trained by a cascade of regressors. We propose very efficient strategies to update the model and we show that is possible to automatically construct robust discriminative person and imaging condition specific models 'in-the-wild' that outperform state-of-the-art generic face alignment strategies. <s> BIB015 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> This paper presents a highly efficient, very accurate regression approach for face alignment. Our approach has two novel components: a set of local binary features, and a locality principle for learning those features. The locality principle guides us to learn a set of highly discriminative local binary features for each facial landmark independently. The obtained local binary features are used to jointly learn a linear regression for the final output. Our approach achieves the state-of-the-art results when tested on the current most challenging benchmarks. Furthermore, because extracting and regressing local binary features is computationally very cheap, our system is much faster than previous methods. It achieves over 3, 000 fps on a desktop or 300 fps on a mobile phone for locating a few dozens of landmarks. <s> BIB016 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Abstract This paper presents a comprehensive survey of facial feature point detection with the assistance of abundant manually labeled images. Facial feature point detection favors many applications such as face recognition, animation, tracking, hallucination, expression analysis and 3D face modeling. Existing methods are categorized into two primary categories according to whether there is the need of a parametric shape model: parametric shape model-based methods and nonparametric shape model-based methods. Parametric shape model-based methods are further divided into two secondary classes according to their appearance models: local part model-based methods (e.g. constrained local model) and holistic model-based methods (e.g. active appearance model). Nonparametric shape model-based methods are divided into several groups according to their model construction process: exemplar-based methods, graphical model-based methods, cascaded regression-based methods, and deep learning based methods. Though significant progress has been made, facial feature point detection is still limited in its success by wild and real-world conditions: large variations across poses, expressions, illuminations, and occlusions. A comparative illustration and analysis of representative methods provides us a holistic understanding and deep insight into facial feature point detection, which also motivates us to further explore more promising future schemes. <s> BIB017 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We present a novel face alignment framework based on coarse-to-fine shape searching. Unlike the conventional cascaded regression approaches that start with an initial shape and refine the shape in a cascaded manner, our approach begins with a coarse search over a shape space that contains diverse shapes, and employs the coarse solution to constrain subsequent finer search of shapes. The unique stage-by-stage progressive and adaptive search i) prevents the final solution from being trapped in local optima due to poor initialisation, a common problem encountered by cascaded regression approaches; and ii) improves the robustness in coping with large pose variations. The framework demonstrates real-time performance and state-of-the-art results on various benchmarks including the challenging 300-W dataset. <s> BIB018 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> The problem of face alignment has been intensively studied in the past years. A large number of novel methods have been proposed and reported very good performance on benchmark dataset such as 300W. However, the differences in the experimental setting and evaluation metric, missing details in the description of the methods make it hard to reproduce the results reported and evaluate the relative merits. For instance, most recent face alignment methods are built on top of face detection but from different face detectors. In this paper, we carry out a rigorous evaluation of these methods by making the following contributions: 1) we proposes a new evaluation metric for face alignment on a set of images, i.e., area under error distribution curve within a threshold, AUC$_\alpha$, given the fact that the traditional evaluation measure (mean error) is very sensitive to big alignment error. 2) we extend the 300W database with more practical face detections to make fair comparison possible. 3) we carry out face alignment sensitivity analysis w.r.t. face detection, on both synthetic and real data, using both off-the-shelf and re-retrained models. 4) we study factors that are particularly important to achieve good performance and provide suggestions for practical applications. Most of the conclusions drawn from our comparative analysis cannot be inferred from the original publications. <s> BIB019 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors. <s> BIB020 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Facial landmark detection is an important yet challenging task for real-world computer vision applications. This paper proposes an effective and robust approach for facial landmark detection by combining data- and model-driven methods. First, a fully convolutional network (FCN) is trained to compute response maps of all facial landmark points. Such a data-driven method could make full use of holistic information in a facial image for global estimation of facial landmarks. After that, the maximum points in the response maps are fitted with a pre-trained point distribution model (PDM) to generate the initial facial shape. This model-driven method is able to correct the inaccurate locations of outliers by considering the shape prior information. Finally, a weighted version of regularized landmark mean-shift (RLMS) is employed to fine-tune the facial shape iteratively. This estimation-correction-tuning process perfectly combines the advantages of the global robustness of the data-driven method (FCN), outlier correction capability of the model-driven method (PDM), and non-parametric optimization of RLMS. Results of extensive experiments demonstrate that our approach achieves state-of-the-art performances on challenging data sets, including 300W, AFLW, AFW, and COFW. The proposed method is able to produce satisfying detection results on face images with exaggerated expressions, large head poses, and partial occlusions. <s> BIB021 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Biometric systems consist of devices, procedures, and algorithms used to recognize people based on their physiological or behavioral features, known as biometric traits. Computational intelligence (CI) approaches are widely adopted in establishing identity based on biometrics and also to overcome non-idealities typically present in the samples. Typical areas include sample enhancement, feature extraction, classification, indexing, fusion, normalization, and anti-spoofing. In this context, computational intelligence plays an important role in performing of complex non-linear computations by creating models from the training data. These approaches are based on supervised as well as unsupervised training techniques. This work presents computational intelligence techniques applied to biometrics, from both a theoretical and an application point of view. <s> BIB022 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> Over the last two decades, face alignment or localizing fiducial facial points has received increasing attention owing to its comprehensive applications in automatic face analysis. However, such a task has proven extremely challenging in unconstrained environments due to many confounding factors, such as pose, occlusions, expression and illumination. While numerous techniques have been developed to address these challenges, this problem is still far away from being solved. In this survey, we present an up-to-date critical review of the existing literatures on face alignment, focusing on those methods addressing overall difficulties and challenges of this topic under uncontrolled conditions. Specifically, we categorize existing face alignment techniques, present detailed descriptions of the prominent algorithms within each category, and discuss their advantages and disadvantages. Furthermore, we organize special discussions on the practical aspects of face alignment in-the-wild, towards the development of a robust face alignment system. In addition, we show performance statistics of the state of the art, and conclude this paper with several promising directions for future research. <s> BIB023 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We present a novel convolutional neural network (CNN) design for facial landmark coordinate regression. We examine the intermediate features of a standard CNN trained for landmark detection and show that features extracted from later, more specialized layers capture rough landmark locations. This provides a natural means of applying differential treatment midway through the network, tweaking processing based on facial alignment. The resulting Tweaked CNN model (TCNN) harnesses the robustness of CNNs for landmark detection, in an appearance-sensitive manner without training multi-part or multi-scale models. Our results on standard face landmark detection and face verification benchmarks show TCNN to surpasses previously published performances by wide margins. <s> BIB024 </s> A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning <s> Introduction <s> We present a probabilistic generative model for tracking by prediction the dynamics of affective spacial expressions in videos. The model relies on Bayesian filter sampling of facial landmarks conditioned on motor action parameter dynamics; namely, trajectories shaped by an autoregressive Gaussian Process Latent Variable state-space. The analysis-by-synthesis approach at the heart of the model allows for both inference and generation of affective expressions. Robustness of the method to occlusions and degradation of video quality has been assessed on a publicly available dataset. <s> BIB025
The face represents a key component in conveying verbal and non-verbal information during interactions between humans. By looking at the face, humans are able to extract a lot of non-verbal information, like identity, intent, and emotion. In the field of computer vision, to automatically extract such information, the detection of facial landmarks ( Figure 1 ) is usually an important step, and several facial analysis techniques are based on the precise detection of landmarks. Head pose estimation BIB004 , and facial expression recognition BIB002 algorithms are strongly based on the facial shape information, given by landmark positions. For instance, facial landmarks around the eyes can provide an initial guess of the pupil center positions for eye detection, as well as eye-gaze tracking BIB005 . In facial recognition, landmarks on a 2D image are usually used along with a 3D head model to frontalize the face and reduce within-subject variability, which improves recognition accuracy BIB012 . Further, facial information obtained through facial landmarks can improve applications in the fields of human and computer interaction, entertainment, security surveillance, and medical applications  BIB024 BIB021 BIB022 . Facial landmark detection algorithms seek to identify the locations of the facial key landmark points on facial images or videos. Such key points are the main points that describe the unique location of a facial component (e.g., eye corner), or an interpolated point that connects those dominant points around the facial components, as well as facial contour. Formally, given a facial image denoted as I, a landmark detection algorithm predicts the locations of D landmarks x = {x 1 , y 1 , x 2 , y 2 , . . . , x D , y D }, where x and y represent the image coordinates of the facial landmarks. Facial landmark extraction is challenging for several reasons: Firstly, facial appearance changes significantly across subjects under different facial expressions and head poses; secondly, facial occlusions by other objects, or self-occlusion due to extreme head poses, leads to incomplete facial appearance information; and thirdly, environmental conditions such as illumination can affect the appearance of the face on facial images. Sample image with predicted facial landmarks. An ensemble of randomized regression trees is used to detect 194 landmarks on a face from a single image BIB013 . Over the past few years, there have been important developments in facial landmark detection algorithms. We noticed that the first works focused on less challenging facial images without the said facial variations. Also, facial landmark detection algorithms usually aimed at handling several variations within fixed categories, and the facial images were usually collected with controlled conditions. For instance, in controlled conditions, facial poses and facial expressions can only be in certain categories. More recently, researchers began focusing on challenging in-the-wild conditions, in which facial images may undergo arbitrary facial expressions, head poses, illumination, facial occlusions, etc. In general, there is still a lack of a robust method that can handle all of those variations  BIB024 . Algorithms for facial landmark extraction can be split into two categories: generative, and discriminative algorithms. The first type builds an object representation, and then searches for the region most similar to the object without taking background information into account. Discriminative methods train an online binary classifier to adaptively separate the object from the background, which are more robust against appearance variations of an object. Generative models include part-based generative models, such as Active Shape Models (ASM)   , and holistic generative models, such as Active Appearance Models (AAM) BIB003 BIB001 BIB007 BIB014 BIB025 that model the shape of the face and its appearance as probabilistic distributions. Among discriminative models, cascaded regression models (CRMs) BIB013 BIB008 BIB009 BIB015  BIB016 BIB010 BIB018 have gained widespread popularity due to its excellent performance and low complexity. Several exhaustive surveys exist for facial landmark detection and tracking  BIB011 BIB019 BIB017 BIB023 . This survey is focused on landmark extraction and tracking using deep-learning techniques. We focus on deep-learning methods for facial landmark detection, as since the arrival of deep-learning neural networks BIB006 BIB020 , deep-learning methods have achieved state-of-art performance in identity and face recognition, object detection, and other research fields mentioned above. In the last few years, there appeared even some closely related tasks to facial landmark extraction in the literature, such as video facial landmark extraction and 3D facial landmark extraction. The first consists of tracking facial landmarks in successive video frames of the same person taking advantage of temporal redundancy and identity. The hardest challenge is represented by in-the-wild video frames. The aim of 3D facial landmark extraction is to detect facial landmarks in arbitrary poses, for recovery of the projected 3D locations of invisible facial landmarks, given a 2D image. This article will consider the newest deep-learning methods for facial landmark extraction in 2D image and video. An analysis of many methods is given, and a comparison between performance is reported based on the performance measures used in the literature. The most common datasets used for both training and assessment of performance are reported and briefly analyzed. Eventually, the main challenges and future research directions will be provided.