---
abstract: 'Active illumination is a prominent complement to enhance 2D face recognition and make it more robust, e.g., to spoofing attacks and low-light conditions. In the present work we show that it is possible to adopt active illumination to enhance state-of-the-art 2D face recognition approaches with 3D features, while bypassing the complicated task of 3D reconstruction. The key idea is to project over the test face a high spatial frequency pattern, which allows us to simultaneously recover real 3D information plus a standard 2D facial image. Therefore, state-of-the-art 2D face recognition solution can be transparently applied, while from the high frequency component of the input image, complementary 3D facial features are extracted. Experimental results on ND-2006 dataset show that the proposed ideas can significantly boost face recognition performance and dramatically improve the robustness to spoofing attacks.'
author:
- |
    J. Matias Di Martino\
    Department of Physics\
    Universidad de la Republica\
    Montevideo, Uruguay\
    `matiasdm@fing.edu.uy`\
    Fernando Suzacq\
    Department of Electrical Engineering\
    Universidad de la Republica\
    Montevideo, Uruguay\
    fsuzacq@gmail.com\
    Mauricio Delbracio[^1]\
    Department of Electrical Engineering\
    Universidad de la Republica\
    Montevideo, Uruguay\
    mdelbra@fing.edu.uy\
    Qiang Qiu\
    Department of Electrical Engineering\
    Duke University\
    Durham, NC, USA\
    qiuqiang@gmail.com\
    Guillermo Sapiro\
    Department of Electrical Engineering\
    Duke University\
    Durham, NC, USA\
    guillermo.sapiro@gmail.com\
bibliography:
- 'bibfile.bib'
title: |
    *Differential 3D Facial Recognition:*\
    Adding 3D to your State-of-the-Art 2D Method
---

Introduction {#sec:introduction}
============

Two-dimensional face recognition has become extremely popular as it can be ubiquitously deployed and large datasets are available. In the past several years, tremendous progress has been achieved in making 2D approaches more robust and useful in real-world applications. Though 2D face recognition has surpassed human performance in certain conditions, challenges remain to make it robust to facial poses, uncontrolled ambient illumination, aging, low-light conditions, and spoofing attacks [@Ding2016; @kemelmacher2016megaface; @Nech2017; @taigman2014deepface]. In the present work we address some of these issues by enhancing the captured RGB facial image with 3D information as illustrated in Figure \[fig:teaser\]. High resolution cameras became ubiquitous, although for 2D face recognition, we only need a facial image of moderate or low resolution. For example latest phones frontal camera have a very high resolution (e.g., $3088\times2320$ pixels) while the resolution of the input to most face recognition systems is limited to $224\times224$ pixels [@arcface; @parkhi2015deep; @schroff2015facenet; @taigman2014deepface; @Zulqarnain2018]. This means that, in the context of face recognition, we are drastically underutilizing most of the resolution of captured images. We propose an alternative to use the discarded portion of the spectra and extract real 3D information by projecting a high frequency light pattern. Hence, a low resolution version of the RGB image remains approximately invariant allowing the use of standard 2D approaches, while 3D information is extracted efficiently from the local deformation of the projected patterns.

![Real 3D face recognition is possible by capturing one single RGB image if a high frequency pattern is projected. The low frequency components of the captured image can be fed into a state-of-the-art 2D face recognition method, while the high frequency components encode local depth information that can be used to extract 3D facial features. It is important to highlight that, in contrast with most existing 3D alternatives, the proposed approach provides real 3D information, not 3D hallucination from the RGB input. As a result, state-of-the-art 2D face recognition methods can be enhanced with real 3D information.[]{data-label="fig:teaser"}](figures/teaser-3dface.pdf){width=".7\columnwidth"}

The proposed solution to extract 3D facial features has key differences with the two common approaches presented in existing literature: 3D hallucination [@eigen2014depth; @Huber2015; @Liu2015; @Pini2018] and 3D reconstruction [@Zafeiriou2013; @Zou2005]. We will discuss these differences in detail in the following section. We illustrate the main limitation of 3D hallucination in the context of face recognition in Figure \[fig:3dillustration\], which emphasizes the lack of real 3D information on a standard RGB input image. We demonstrate that it is possible to extract actual 3D facial features bypassing the ill-posed problem of explicit depth estimation. Our contributions are summarized as follows:

-   Analyzing the spectral content of thousands of facial images, we design a high frequency light pattern that simultaneously allow us to retrieve a standard 2D low resolution facial image plus a 3D gradient facial representation.

-   We propose an effective and modular solution that achieves 2D and 3D information decomposition and facial feature extraction in a data-driven fashion (bypassing a 3D facial reconstruction).

-   We show that by defining an adequate distance function in the space of the feature embedding, we can leverage the advantages of both 2D and 3D features. We can transparently exploit existing state-of-the-art 2D methods and improve their robustness, e.g., to spoofing attacks.

Related Work {#sec:relatedWork}
============

To recognize or validate the identity of a subject from a 2D color photograph is a longstanding problem of computer vision and has been largely studied for over forty years [@kaya1972basic; @zhao2003face]. Recent advances in machine learning, and in particular, the success of deep neural networks, reshaped the field and yielded more efficient, accurate, and reliable 2D methods such as: ArcFace[@arcface], VGG-Face [@parkhi2015deep], DeepFace [@taigman2014deepface], and FaceNet [@schroff2015facenet]. In spite of this, spoofing attacks and variations in pose, expression and illumination are still active challenges and significant efforts are being made to address them [@Cao2018; @Hayat2017; @He2018; @Kumar2018; @Lezama2017; @Liu2018; @Tran2017; @Yu2017; @Zhao2018; @Zou2005]. For example, Deng et al. [@Deng2018] attempt to handle large pose discrepancy between samples. To that end, they propose an adversarial facial UV map completion GAN. Complementing previous approaches that seek for robust feature representations, several works propose more robust loss and metric functions [@Liu2017; @Wang2018].

#### 3D hallucination from single RGB.

To enhance 2D approaches a common trend is to hallucinate a 3D representation from an input RGB image which is used to extract 3D features [@blanz2003face; @Dou2017; @eigen2014depth; @Huber2015; @Liu2015; @Pini2018]. For example, Cui et al. [@Cui2018] introduce a cascade of networks that simultaneously recover depth from an RGB input while seeking for separability of individual subjects. The estimated depth information is then used as a complementary modality to RGB.

#### 3D face recognition.

The approaches described previously share an important practical advantage that at the same time is their weakness, they extract all the information from a standard (RGB) 2D photograph of the face. As depicted in Figure \[fig:3dillustration\] a single image does not contain actual 3D information. To overcome this intrinsic limitation different ideas have been proposed and datasets with 3D facial information are becoming more popular [@Zulqarnain2018]. For example, Zafeiriou et al. [@Zafeiriou2013] propose a four-light source photometric stereo (PS). A similar idea is elaborated by Zou et al. [@Zou2005] who propose to use active near-infrared illumination and combine a pair of input images to extract an illumination invariant face representation. Despite the previous mentioned techniques, performing a 3D facial reconstruction is still a challenging and complicated task. Many strategies have been proposed to tackle this problem, including time delay based [@marks1992system], image cue based [@eigen2015predicting; @eigen2014depth; @laina2016deeper; @prados2006shape; @saxena2009make3d], and triangulation based methods [@ayubi2010pulse; @DiMartino2015one; @li2014some; @Rosman2016; @zhang2006high]. Although there has been great recent development, available technology for 3D scanning is still too complicated to be ubiquitously deployed [@di2018one; @hartley2003multiple; @zhang2010recent; @zhang2013handbook]. The proposed solution has two key features that make it, to the best of our knowledge, different from existing alternatives. (a) Because the projected pattern is of a high spatial frequency, we can recover a standard (low resolution) RGB facial image that can be fed into state-of-the-art 2D face recognition methods. (b) We avoid the complicated task of 3D facial reconstruction and instead, extract local 3D features from the local deformation of the projected pattern. In that sense our ideas can be implemented exploiting existing and future 2D solutions. In addition, our approach is different from those that hallucinate 3D information. As discussed before and illustrated in Figure \[fig:3dillustration\] this task requires a strong prior of the scene which is ineffective, for example, if a spoofing attack is presented (see the example provided in Figure \[fig:3dmm\] in the supplementary material).

Proposed Approach {#sec:ProposedApproach}
=================

#### Notation.

Let ${\mathcal{I}}\subset{\mathbb{R}}^{H\times W\times C}$ denote the space of images with $H\times W$ pixels and $C$ color channels, and ${\mathcal{X}}_{n}\subset{\mathbb{R}}^{n}$ a space of n-dimensional column vectors (in the context of this work associated to a facial feature embedding). ${\mathcal{I}}_{rgb}$ denotes the set of RGB images ($C=3$), while ${\mathcal{I}}_{\nabla z}$ is used to denote the space of two channel images ($C=2$) associated to the gradient of a single-channel image $z\in{\mathbb{R}}^{H\times W \times 1}$. (The first/second channel represents the partial derivative with respect to the first/second coordinate.)

![image](figures/framework.png){width=".9\textwidth"}

#### Combining depth and RGB information.

The proposed approach consists of three main modules as illustrated in Figure \[fig:proposedFramework\]: $g:{\mathcal{I}}_{rgb}\rightarrow {\mathcal{I}}_{rgb}\times{\mathcal{I}}_{\nabla z}$ performs a decomposition of the input image into texture and depth information, $f_{rgb}:{\mathcal{I}}_{rgb}\rightarrow {\mathcal{X}}_{n/2}$, and $f_{\nabla z}:{\mathcal{I}}_{\nabla z}\rightarrow {\mathcal{X}}_{n/2}$ extract facial features associated to the facial texture and depth respectively. These three components are illustrated in Figure \[fig:proposedFramework\] in blue, yellow, and green, respectively. (We decided to have three modules instead of a single *end-to-end* design for several reasons that will be discussed below.) We denote the facial feature extraction from the input image as $f_{\theta} : {\mathcal{I}}_{rgb} \rightarrow {\mathcal{X}}_n$, where $f_\theta(I) = (f_{rgb}(I_{rgb}), f_{\nabla z}(I_{\nabla z}))^T$ with $\{I_{rgb},I_{\nabla z}\} = g(I)$. The subscript $\theta$ represent the parameters of the mapping $f$, which can be decomposed in three groups $\theta = (\theta_g, \theta_{rgb}, \theta_{\nabla z})$, associated to the image decomposition, RGB feature extraction, and depth feature extraction respectively. In the following we discuss how these parameters are optimized for each specific task, which is one of the advantages of formulating the problem in a modular fashion. Once texture and depth facial information is extracted into a suitable vector representation $x = f_{\theta}(I)$ (as illustrated in Algorithm \[alg:facial\_embedding\]), we can select a distance measure $d:{\mathcal{X}}_n\times{\mathcal{X}}_n\rightarrow {\mathbb{R}}^+$ to compare facial samples and estimate whether they have a high likelihood of belonging to the same subject or not. It is worth noticing that faces are embedded into a space in which the first half of the dimensions are associated to information extracted from the RGB representation while the other half codes depth information. These two sources of information may have associated different confidence levels (depending on the conditions at deployment). We address this in detail in Section \[sec:distanceDesign\] and propose an anisotropic distance adapted to our solution, and capable of leveraging the good performance of 2D solutions in certain conditions, while improving robustness and handling spoofing attacks in a continuous and unified fashion.

Decompose the input image into texture and depth gradient information. $\left\{ I_{rgb}, I_{\nabla z}\right\} = g(I)$ Extract facial information from each component. $x_{rgb} = f_{rgb}(I_{rgb})$ $x_{\nabla z} = f_{\nabla z}\left(I_{\nabla z}\right)$ Combine texture and depth information. $x = $ Concatenate$(x_{rgb}, x_{\nabla z})$ **return** $x$

Pattern design. {#sec:pattern design}
---------------

When a pattern of light $p(x,y)$ is projected over a surface with a height map $z(x,y)$, it is perceived by a camera located along the $x$-axis with a deformation given by $p(x+\phi(x,y),y)$ ($\phi(x,y) \propto z(x,y)$). A detailed description of active stereo geometry is provided in the supplementary material Section \[sec:reviewActiveStereo\]. Let us denote $I_0(x,y)$ the image we would acquire under homogeneous illumination, and $p(x,y)$ the intensity profile of the projected light. Without loss of generality we assume the system baseline is parallel to the $x$ axis. The image acquired by the camera when the projected light is modulated with a profile $p(x,y)$ is $$\label{eq:pat_desing_1}
    I(x,y) = I_0(x,y)p(x+\phi(x,y), y).$$ We will restrict to periodic modulation patterns and let $T$ denote the pattern spatial period, we also define $f_0 \stackrel{def}{=} \frac 1 T$. To simplify the system design and analysis, lets also restrict to periodic patterns that are invariant to the $y$ coordinate. In these conditions we can express $p(x,y) = \sum_{n=-\infty}^{+\infty} a_n\, e^{i2\pi n f_0 x}$ where $a_n$ represent the coefficients of the Fourier series of $p$. (Note that because of the invariance with respect to the $y$ coordinate, the coefficients $a_n$ are constant instead of a function of $y$.) Equation  can be expressed as $$\label{eq:pat_design_2}
    I(x,y) = \sum_{n=-\infty}^{+\infty} I_0(x,y)\, a_n\, e^{i2\pi n f_0 (x+\phi(x,y))}.$$ Defining $q_n(x,y)\stackrel{def}{=}I_0(x,y)\,a_n\,e^{i2\pi n f_0 \phi(x,y)}$, Equation  can be expressed as [@takeda1983fourier] $$\label{eq:pat_design_3}
    I(x,y) = \sum_{n=-\infty}^{+\infty} q_n(x,y) e^{i2\pi n f_0 x}.$$ Applying the 2D Fourier Transform (FT) in both sides of Equation  and using standard properties of the FT [@distributions] we obtain $$\label{eq:pat_design_4}
    \tilde{I}(f_x,f_y) = \sum_{n=-\infty}^{+\infty} \tilde{q_n}(f_x - nf_0, f_y).$$ We denote as $\tilde{I}$ the FT of $I$ and use $(f_x,f_y)$ to represent the 2D frequency domain associated to $x$ and $y$ axis respectively.

Equation  shows that the FT of the acquired image can be decomposed into the components $\tilde{q_n}$ centered at $(nf_0,0)$. In the context of this section, we refer to a function $h(x,y)$ being smooth if $$\frac{\|\tilde{h}(f_x,f_y)\|}{\|\tilde{h}(0,0)\|}<10^{-3}\ \ \  \forall \ \ \ |f_x|>\frac{f_0}{2} .$$ Assuming $I_0(x,y)$ and $\phi(x,y)$ are smooth (we empirically validate this hypothesis below), the components $\tilde{q_n}$ can be isolated as illustrated in Figure \[fig:fringesDescomposition\]. The central component is of particular interest, $q_0(x,y) = a_0\, I_0(x,y)$ captures the facial texture information and can be recovered from $I(x,y)$ if $f_0$ is large enough (we provide a more precise quantitative analysis in what follows). On the other hand, relative (gradient) 3D information can be retrieved from the components $\{q_0, q_1\}$ as we show in Proposition \[prop:gradient\_depth\].

\[prop:gradient\_depth\] Gradient depth information is encoded in the components $\{q_0(x,y), q_1(x,y)\}$.

We define the wrapping function $\mathcal{W}(u)={\mbox{atan}}({\mbox{tan}}(u))$. This function wraps the real set into the interval $\left(-\pi/2,\pi/2\right]$ [@ghiglia1998unwrapping]. This definition can be extended to vector inputs wrapping the modulus of the vector field while keeping its direction unchanged, i.e., $\mathcal{W}(\vec{u})=\frac{\mathcal{W}(\|\vec{u}\|)}{\|\vec{u}\|}\vec{u}$ if $\|\vec{u}\|\neq 0$ and $\mathcal{W}(\vec{u})=\vec{0}$ if $\|\vec{u}\| = 0$. From $q_1(x,y)$ and $q_0(x,y)$ we can compute[^2] $$\label{eq:prop_prof_1}
    \phi_{\mathcal{W}}(x,y) = \frac{1}{2\pi f_0}\, {\mbox{atan}}\left(\frac{\mbox{Im}\{\frac{q_1(x,y)}{q_0(x,y)}\}}{\mbox{Re}\{\frac{q_1(x,y)}{q_0(x,y)}\}}\right)$$ where $\phi_{\mathcal{W}}$ denotes the wrapped version of $\phi$. Moreover, $\phi_{\mathcal{W}}(x,y)=\phi(x,y) + \pi k(x,y)$ with $k(x,y)\in\mathbb{N}$ (wrapping introduces shifts of magnitude multiple of $\pi$). Computing the gradient both sides leads to $\nabla\phi_{\mathcal{W}}(x,y)=\nabla\phi(x,y) + \pi \nabla k(x,y)$ where $\|\nabla k(x,y)\|\in\mathbb{N}$. Assuming the magnitude of the gradient of $\phi(x,y)$ is bounded by $\pi/2$ and considering that $\|\nabla k(x,y)\|\in\mathbb{N}$, we can apply the wrapping function both sides of the previous equality to obtain $\mathcal{W}(\nabla\phi_{\mathcal{W}})(x,y)=\nabla\phi(x,y)$ which proves (recall Equation ) that the gradient of $\phi$ can be extracted from the components $q_0$ and $q_1$. To conclude the proof, we use the property of linearity of the gradient operation and the fact that $\phi(x,y)$ is proportional to the depth map of the scene (see Equation  and Section \[sec:reviewActiveStereo\] in the supplementary material).

![Faces average spectral content. The first column illustrates the mean luminance and depth map for the faces in the dataset ND-2006. The second column shows the mean Fourier Transform of the faces luminance and depth respectively. The third column shows the profile across different sections of the 2D Fourier domain. Columns two and three represent the absolute value of the Fourier transform in logarithmic scale. Faces are registered using the eyes landmarks and the size normalized to $480\times480$ pixels.[]{data-label="fig:face_spectrum"}](figures/face_spectrum.png){width=".6\columnwidth"}

#### Analytic versus data-driven texture and gradient depth extraction

The previous analysis shows that closed forms can be obtained to extract texture and depth gradient information. However, to compute these expressions is necessary to isolate different spectral components $\tilde{q_n}$. To that end, filters need to be carefully designed. The design of these filters is challenging, e.g., one need to control over-smoothing versus introducing ringing artifact which are drastically amplified by a posterior gradient computation [@DiMartino2015one; @zhang2006high]. To overcome these challenges, we chose to perform a depth (gradient) and texture decomposition in a data-driven fashion, which as we show in Section \[sec:Experiments\], provides an efficient and effective solution.

#### Bounds on $f_0$ and optimal spectral orientation.

As discussed above, the projected pattern $p(x,y)$ should have a large fundamental frequency $f_0$. In addition, the orientation of the fringes and the system baseline can be optimized if faces present a narrower spectral content in a particular direction. We study the texture and depth spectrum of the facial images of ND-2006 dataset (this dataset provides ground truth facial texture and depth information). We observed (see Figure \[fig:face\_spectrum\]) that for facial images sampled at a $480\times480$ spatial resolution, most of the energy is concentrated in a third of the discrete spectral domain (observe the extracted one dimensional profiles of the spectrum shown at the left side of Figure \[fig:face\_spectrum\]). In addition, we observe that the spectral content of facial images is approximately isotropic. See, for example, Figure \[fig:face\_spectrum\] and observe how for 1-dimensional sections across different orientations the 2D spectra envelope is almost constant. We conclude that the orientation of the fringes does not play a significant role in the context of facial analysis. In addition, we conclude that the fringes width should be smaller than $7$mm (distance measure over the face).[^3]

Network training and the advantages of modularity. {#sec:networkDesign}
--------------------------------------------------

As described previously, the parameters of the proposed solution can be split in three groups $\theta = (\theta_g, \theta_{rgb}, \theta_{\nabla z})$. This is an important practical property and we designed the proposed solution to meet this condition (in contrast to an end-to-end approach). Let us define $\mathcal{B}_{1}$, $\mathcal{B}_{2}$, and $\mathcal{B}_{3}$ three datasets containing ground truth depth information, ground truth identity for rgb facial images, and ground truth identity for depth facial images, respectively. More precisely, $\mathcal{B}_{1}=\{(I_i(x,y), {I_0}_i(x,y), z_i(x,y)),\ i=1,\,...,\,n_1\}$, $\mathcal{B}_{2}=\{({I_0}_i(x,y), y_i),\ i=1,\,...,\,n_2\}$, and $\mathcal{B}_{3}=\{(z_i(x,y), y_i),\ i=1,\,...,\,n_3\}$, where $I_i(x,y)$ denotes a (facial or generic) RGB image acquired under the projection of the designed pattern, ${I_0}_i(x,y)$ represents (facial or generic) standard RGB images, $z_i(x,y)$ denotes a gray image representing the depth of the scene, and $y_i$ a scalar integer representing the subject id. We denote as $\{g_1(I), g_2(I)\} = g(I)$ the RGB and gradient depth components estimated by the decomposition operation $g$. We partitioned the parameters of $g$ into two sets of dedicated kernels $\theta_g = \{{\theta_g}_1, {\theta_g}_2\}$, the first group focuses on retrieving the texture component while the second group retrieves the depth gradient. These parameters can be optimized as $$\begin{aligned}
\label{eq:loss_g}
    {\theta_g}_1 = \operatorname*{argmin}\sum_{({I_0}_i,I_i)\in\mathcal{B}_1} \|g_1(I_i) - {I_0}_i \|_2^2 \\
    {\theta_g}_2 = \operatorname*{argmin}\sum_{({z}_i,I_i)\in\mathcal{B}_1} \|g_2(I_i) - \nabla z_i \|_2^2.\end{aligned}$$ (We also evaluated training a shared set of kernels trained with an unified loss, this alternative is harder to train in practice, due to the natural difference between the dynamic range and sparsity of gradient images compared with texture images.) For texture and depth facial feature extraction, we tested models inspired in the Xception architecture [@chollet2017xception]). Additional details are provided in the supplementary material Section \[app:implementation\_details\]. To train these models we add an auxiliary fully connected layer on top of the facial embedding (with as many neurons as identities in the train set) and minimize the cross-entropy between the ground truth and the predicted labels. More precisely, let us denote $\hat{f}_{rgb}(I_{rgb})=[p_1,...,p_c]$ the output of the fully connected layer associated to the embedding $f_{rgb}(I_{rgb})$ where $p_i$ denotes the probability associated to the id $i$, $$\begin{aligned}
    \theta_{rgb} = \operatorname*{argmin}\sum_{({I_0}_i, y_i)\in\mathcal{B}_2} \sum_{c} -{\bf{1}}_{y_i=c} \log(\hat{f}_{rgb}({I_0}_i)[c]) \\
    \theta_{\nabla z} = \operatorname*{argmin}\sum_{({z}_i, y_i)\in\mathcal{B}_3} \sum_{c} -{\bf{1}}_{y_i=c} \log(\hat{f}_{\nabla z}(\nabla z_i)[c])\end{aligned}$$ where ${\bf{1}}_{y_i=c}$ denotes the indicator function. (Of course one can choose other alternative losses to train these modules, see e.g., [@arcface; @Liu2017; @Wang2018; @Zheng2018].) As described above, the proposed design allows to leverage information from three types of datasets ($\mathcal{B}_1$, $\mathcal{B}_2$, $\mathcal{B}_3$). This has an important practical advantage as 2D facial and 3D generic datasets are more abundant, and the pattern dependant set $\mathcal{B}_1$ can be of modest size as $\#(\theta_g)\ll\#(\theta_{rgb})$.

Distance design. {#sec:distanceDesign}
----------------

Once different modules are set we can compute the facial embedding of test images following the procedure described in Algorithm \[alg:facial\_embedding\]. Let us define $x^a\in{\mathcal{X}}_n$ and $x^b\in{\mathcal{X}}_n$ the feature embedding of two facial images $I_a$ and $I_b$ respectively. Recall that the first $n/2$ elements of $x$ are associated to features extracted from (a recovered) RGB facial image while the remaining elements are associated to depth information, i.e., $x = (x_{rgb}[1],..., x_{rgb}[n/2], x_{\nabla z}[1], ..., x_{\nabla z}[n/2])^T$. We define the distance between two feature representations $x^a = (x_{rgb}^a, x_{\nabla z}^a)$, and $x^b = (x_{rgb}^b, x_{\nabla z}^b)$ as $$\label{eq:distance}
\begin{array}{l}
    \displaystyle d_{\alpha,\beta,\gamma}(x^a,x^b) \stackrel{def}{=}  (1-\gamma)\, d_c(x^a_{rgb},\,x^b_{rgb}) \vspace{2mm} \\
     \ \ +\ \gamma\, d_c(x^a_{\nabla z},\,x^b_{\nabla z}) \left(1 + \left(\frac{d_c(x^a_{\nabla z},\,x^b_{\nabla z})}{\beta}\right)^\alpha \right).
\end{array}$$ $d_c:{\mathcal{X}}_{n/2}\times{\mathcal{X}}_{n/2}\rightarrow[0, 1]$ denotes the cosine distance, $\gamma\in[0,1]$ sets the relative weight of RGB and depth features, and $\alpha,\beta \in {\mathbb{R}}$ define a non-linear response for the distance between depth features. As we will describe in the following, this provides robustness against common cases of spoofing attacks. Intuitively, $\gamma$ allows us to set the relative confidence associated to RGB and depth features, for example, $\gamma=1/2$ gives the same weight to RGB and depth features, while $\gamma=0$ ($\gamma=1$) ignores the distance between samples in the depth (RGB) embedding space. This is important in practice, as is common to obtain substantially more data to train RGB models than depth ones ($|\mathcal{B}_2| \gg |\mathcal{B}_3|$). This suggests that in good test conditions (e.g., good lighting) one may trust more RGB features over depth features ($\gamma<1/2$). As we will empirically show in the following section, when two facial candidates are compared, $d_{\alpha,\infty,\gamma}(x^a,x^b) =  (1-\gamma)\, \delta(x^a_{rgb},\,x^b_{rgb}) + \gamma\, \delta(x^a_{\nabla z},\,x^b_{\nabla z})$ is an effective distance choice. However, it does not handle robustly common cases of spoofing attacks. The most common deployments of spoofing attacks imitate the facial texture more accurately than the facial depth [@OULU_NPU_2017; @Liu2018spoofing; @Zhang2016], therefore, the global distance between two samples should be large when the distance of the depth features is large (i.e. above a certain threshold). To that end, we introduce an additional non-linear term controlled by parameters $\beta$ and $\alpha$, for $\delta(x^a_{\nabla z}, x^b_{\nabla z}) < \beta$ the standard cosine distance dominates while for large values the distance will be amplified in a non-linear fashion.

Experiments and Discussion {#sec:Experiments}
==========================

#### Data.

Three public dataset are used for experimental validation: FaceScrub [@FaceScrub], CASIA Anti-spoofing [@zhang2012face], and ND-2006 [@nd2006]. FaceScrub contains $100k$ RGB (2D) facial images of $530$ different subjects, and is used to train the texture-based facial embedding. CASIA dataset contains $150$ genuine videos (recording a person) and $450$ videos of different types of spoofing attacks, the data was collected for $50$ subjects. We use this dataset to simulate and imitate the texture properties of images of spoofing attacks. ND-2006 is one of the larges publicly available datasets with 2D and 3D facial information, it contains $13k$ images of $888$ subjects. We used this set to demonstrate that differential 3D features can be extracted from a single RGB input, to compare RGB features with 3D features extracted from the differential 3D input, and to show that when 2D and 3D information is properly combined, the best properties of each can be obtained.

#### Texture and differential 3D decomposition.

In Section \[sec:pattern design\] we discussed how real 3D information and texture information can be coded and later extracted using a single RGB image. In addition, we argue that this decomposition can be learned efficiently and effectively in a data-driven fashion. To that end, we tested simple network architectures composed of standard convolutional layers (a full description of these architectures and the training protocols are provided as supplementary material). Using ground truth texture and depth facial information, we simulated the projection of the designed pattern over the $888$ subjects provided in ND-2006 dataset. Illustrative results are presented in Figure \[fig:nd2006patterns\] and in the supplementary material. The 3D geometrical model and a detailed description of the simulation process is provided in Section \[app:sec:sim\_proj\_pat\]. Though the simulation of the deformation of a projected pattern can be computed in a relatively simple manner (if the depth information is known), the inverse problem is analytically hard [@DiMartino2015one; @Rosman2016; @zhang2013handbook].

![Active light projection. From left to right: ground truth RGB facial image, 3D facial scanner, and finally the image we would acquire if the designed high frequency pattern is projected over the face. Two random samples from ND-2006 are illustrated.[]{data-label="fig:nd2006patterns"}](figures/nd2006patterns.png){width=".6\columnwidth"}

Despite the previous, we observed that a stack of convolutional layers can efficiently learn how to infer from the image with the projected pattern, both depth gradient information, and the standard (2D) facial image. Figure \[fig:texture\_descomposition\] illustrates some results for subjects in the test set. The first column corresponds to the input to the network, the second column the ground truth texture information, and the third column the retrieved texture information. The architecture of the network and the training protocol is described in detail in the supplementary material Section \[app:implementation\_details\]. As we can see in the examples illustrated in Figure \[fig:texture\_descomposition\], an accurate low resolution texture representation of the face can be achieved in general, and visible artifact are observed only in the regions where the depth is discontinuous (see for example, the regions illustrated at the bottom of Figure \[fig:texture\_descomposition\]).

![Examples of the facial texture recovered from the image with the projected pattern. The first column, shows the input image (denoted as $I$ in Algorithm \[alg:facial\_embedding\]). The second column shows the ground truth, and the third column the texture recovered by the network $I_{rgb}$. This examples are from the test set and the images associated to these subjects were never seen during the training phase. []{data-label="fig:texture_descomposition"}](figures/texture_recovery.jpg){width=".6\columnwidth"}

Figure \[fig:depth\_gradient\_descomposition\] illustrates the ground truth and the retrieved depth gradient (again, for random samples from the test set). To estimate the 3D information, we feed to a different branch of convolutional layers the gray version of the input image. These layers are fully described in the supplementary material Table \[tab:DescompositionNetwork\]. A gray input image is considered instead of a color one because the projected pattern is achromatic, and therefore, no 3D information is encoded in the colors of the image. In addition, we crop the input image to exclude the edges of the face. (Facial registration and cropping is performed automatically using dlib [@king2009dlib] facial landmarks.) As discussed in Section \[sec:ProposedApproach\], and in particular, in the proof of Proposition \[prop:gradient\_depth\], the deformation of the projected fringes only provide local gradient information if the norm of the gradient of the depth is bounded. In other words, where the scene present depth discontinuities, no local depth information can be extracted by our proposed approach. This is one of the main reasons why differential 3D information can be exploited for face recognition, while bypassing the more complicated task of a 3D facial reconstruction.

![Differential depth information extracted from the image with the projected pattern. The first row illustrates the input image (depth information can be extracted from a gray version of the input as the designed patter is achromatic). The second and third row show the ground truth and the retrieved $x$ and $y$ partial derivatives of the depth respectively.[]{data-label="fig:depth_gradient_descomposition"}](figures/results_i3d.jpg){width="\columnwidth"}

One of the advantages of the proposed approach is that it extracts local depth information, and therefore, the existence of depth discontinuities does not affect the estimation on the smooth portion of the face. This is illustrated in Figure \[fig:depth\_extraction\_verification\] (a)-(b), where a larger facial patch is fed into the network. The decomposition module is composed exclusively of convolutional layers, and therefore, images of arbitrary size can be evaluated. Figure \[fig:depth\_extraction\_verification\]-(a) shows the input to the network, and Figure \[fig:depth\_extraction\_verification\]-(b) the first channel of the output (for compactness we display only the x-partial derivative). As we can see, the existence of depth discontinuities does not affect the prediction in the interior of the face (we consider the prediction outside this region as noise and we replace it by $0$ for visualization).

![Is the network really extracting depth information? In this figure we show the output of the network for two inputs generated using identical facial texture but different depth ground truth data. (a) Image obtained when the projected pattern is projected over the face with the real texture and the real 3D profile. (b) Output of the network when we input (a) (only the x-partial derivative is displayed for compactness). (c) Image obtained when the projected pattern is projected over a flat surface with the texture of the real face. (d) Output of the network when the input is (c). None of these images were seen during training.[]{data-label="fig:depth_extraction_verification"}](figures/spoofingExample1.png){width=".7\columnwidth"}

Several algorithms have been proposed to hallucinate 3D information from a 2D facial image [@eigen2014depth; @Huber2015; @Liu2015; @Pini2018]. In order to verify that our decomposition network is extracting real depth information (in lieu of hallucinating it from texture cues), we simulated an image where the pattern is projected over a surface with identical texture but with a planar 3D shape (as in the example illustrated in Figure \[fig:3dillustration\]). Figure \[fig:depth\_extraction\_verification\] (a) shows the image acquired when the fringes are projected over the ground truth facial depth, and (c) when instead the depth is set to $0$ (without modifying the texture information). The first component of the output (x-partial derivative) is shown in (b) and (d), as we can see, the network is actually extracting true depth information (from the deformation of the fringes) and not hallucinating 3D information from texture cues. (As we will see next, this property is particularly useful for joint face recognition and spoofing prevention.)

#### 2D and 3D face recognition.

Once the input image is decomposed into a (standard) texture image and depth gradient information, we can proceed to extract 2D and 3D facial features from each component. To this end, state-of-the-art network architectures are evaluated. Our method is agnostic to the RGB and depth feature extractors, moreover, as the retrieved texture image is close to a standard RGB facial images (in sense of the L2-norm), any pre-train 2D feature extractor can be used (e.g., [@arcface; @parkhi2015deep; @schroff2015facenet; @taigman2014deepface; @Zulqarnain2018]). In the experiments presented in this section we tested a network based on the Xception architecture [@chollet2017xception] (details are provided as supplementary material). For the extraction of texture features, the network is trained using FaceScrub [@FaceScrub] dataset (as we previously described, this is a public dataset of 2D facial images). The module that extracts 3D facial features is trained using $2/3$ of the subjects of ND-2006 dataset, leaving the remaining subjects exclusively for testing. The output of each module is a 512-dimensional feature vector (see, e.g., Figure \[fig:proposedFramework\]), hence the concatenation of 2D+3D features leads to a 1024-dimensional feature vector. Figure \[fig:embedding\] illustrates a 2D embedding of the texture features, the depth features, and the combination of both. The 2D mapping is learned by optimizing the t-SNE [@maaten2008visualizing] over the train partition, then a random subset of test subjects are mapped for visualization. As we can see, 3D features favor the compactness and increase the distance between clusters associated to different subjects.

![image](figures/embedding.jpg){width="\textwidth"}

To test the recognition performance, the images of the test subjects are partitioned into two sets: gallery and probe. For all the images in both sets, the 2D and 3D feature embedding is computed (using the pre-trained networks described before). Then, for each image in the probe set, the $n$ nearest neighbors in the gallery set are selected. The distance between each sample (in the embedding space) is measured using the distance defined in Section \[sec:ProposedApproach\], Equation . For each sample in the probe set, we consider the classification as accurate, if at least one of the $n$ nearest neighbors is a sample from the same subject. The Rank-n accuracy is the percentage of samples in the probe set accurately classified.

![Rank-n accuracy for 2D, 3D, and 2D+3D face recognition. As discussed in Section \[sec:ProposedApproach\] the value of $\gamma$ can be set to weight texture and depth information in the classification decision. The extreme cases are $\gamma=0$ (only texture is considered) and $\gamma=1$ (only depth is considered). These extreme cases are illustrated in yellow and blue respectively, while intermediate solutions ($0<\gamma<1$) are presented in tones of green.[]{data-label="fig:gammarec"}](figures/rank_acc_varying_gamma.png){width=".6\columnwidth"}

Figure \[fig:gammarec\] and Table \[tab:Rank-n\_rgb\_vs\_depth\] show the Rank-n accuracy when: only 2D features ($\gamma=0$), only 3D features ($\gamma=1$), or a combination of both ($0<\gamma<1$) is considered. As explained in Section \[sec:distanceDesign\], the value of $\gamma$ can be used to balance the weight of texture and depth features. As we can see, in all the cases a combination of texture and depth information outperforms each of them individually. This is an expected result as classification tends to improve when independent sources of information are combined [@Kuncheva2004]. $\gamma$ is an hyper-parameter that should be set depending on the conditions at deployment. In our particular experiments the best results are obtained for $\gamma=0.3$, which suggests that RGB features are slightly more reliable than depth features. This is an expected result as the module that extract RGB features is typically trained in a much larger datasets (2D facial images became ubiquitous). We believe this may change if, for example, testing is performed under low light conditions [@Lezama2017]. Testing this hypothesis is one of the potential path for future research. In the experiment discussed so far, we ignored the role of $\beta$ and $\alpha$ (i.e., we set $\beta=\infty$ and $\alpha=1$). As we will discuss in the following, these parameters become relevant to achieve jointly face recognition and spoofing prevention.

[L[3.2cm]{} C[1cm]{} C[1cm]{} C[1cm]{} C[1cm]{}]{} Rank-n Accuracy & 1 & 2 & 5 & 10\
RGB baseline ($\gamma=0$) & $78.5$ & $82.6$ & $87.7$ & $90.6$\
Depth baseline ($\gamma=1$) & $77.2$ & $81.4$ & $87.4$ & $90.1$\
(our) $\gamma=0.3$ & $\bf{90.6}$ & $\bf{93.2}$ & $\bf{95.6}$ & $\bf{96.4}$\
(our) $\gamma=0.5$ & $88.6$ & $91.0$ & $94.4$ & $94.9$\
(our) $\gamma=0.8$ & $85.0$ & $87.9$ & $91.5$ & $93.0$\

#### Robustness to spoofing attacks.

Spoofing attack are simulated to test face recognition models, in particular, how robust these frameworks are under (unseen) spoofing attacks. As in the present work we focus on the combination of texture and depth based features, the simulation of spoofing attacks must account for realistic texture and depth models. The models for the synthesis of spoofing attacks are described in detail in the supplementary material Section \[app:implementation\_spoofing\].

![Examples of samples from live subjects and spoofing attacks. From left to right: (1) the ground truth texture, (2) the ground truth depth, (3) the input to our system (image with the projected pattern), (4) the recovered texture component (one of the outputs of the decomposition network), (5)/(6) recovered $x$/$y$ depth partial derivative. The first four rows correspond to spoofing samples (as explained in Section \[app:implementation\_spoofing\]), and the bottom five rows to genuine samples from live subjects.[]{data-label="fig:more_spoofing_examples"}](figures/moreExamples.jpg){width="\columnwidth"}

Figure \[fig:more\_spoofing\_examples\] illustrates spoofing samples (first four rows) and genuine samples (bottom five rows). The first two columns correspond to the ground truth texture and depth information, the third column illustrates the input to our system, and the last three columns correspond to the outputs of the decomposition network. These three last images are fed into the feature extraction modules for the extraction of texture and depth based features respectively, as illustrated in Figure \[fig:proposedFramework\]. It is extremely important to highlight, that spoofing samples are included exclusively at testing time. In other worlds, during all the training process the entire framework is agnostic to the existence of spoofing examples. If the proposed framework is capable of extracting real 3D facial features, it should be inherently robust to most common types of spoofing attacks. As discussed before, the combination of texture and depth based features improves recognition accuracy. On the other hand, when spoofing attacks are included, we observe that texture based features are more vulnerable to spoofing attacks (see for example figure \[fig:more\_spoofing\_examples\] and \[fig:spoofing\_roc\]). To simultaneously exploit the best of each feature component, we design a non-linear distance as described in Equation . Figure \[fig:distance\_illustration\] illustrates the properties of the defined distance for different values of $\alpha$ and $\beta$. As it can be observed, for those genuine samples (relative distances lower than $\beta$) the non linear component can be ignored and the distance behave as the euclidean distance with a relative modulation set by $\gamma$. On the other hand, if the distance between the depth components is above the threshold $\beta$, it will dominate the overall distance achieving a more robust response to spoofing attacks.

![Illustration of the properties of the distance function defined in . On the left side we illustrate the role of the parameter $\alpha$, and on the right, we compare the proposed distance and the standard euclidean distance. As can be observed, both measures are numerically equivalent in the region $[-\beta/2,\beta/2]\times[-\beta/2,\beta/2]$, but the proposed measure gives a higher penalty to vectors whose $u$ coordinate exceeds the value $\beta$.[]{data-label="fig:distance_illustration"}](figures/distance_spoofing.png){width=".9\columnwidth"}

To quantitatively evaluate the robustness against spoofing attacks, spoofing samples are generated for all the subjects in the test set. As before, the test set is separated into a gallery and a probe set and the generated spoofing samples are aggregated into the probe set. For each image in the probe set, the distance to a sample of the same subject in the gallery set is evaluated. If this distance is below a certain threshold $\lambda$, the image is labeled as genuine, otherwise, the image is labeled as spoofing. Comparing the classification label with the ground truth label we obtain the number of true positive (genuine classified as genuine), false positive (spoofing classified as genuine), true negative (spoofing classified as spoofing), and false negative (genuine classified as spoofing). Changing the value of the threshold $\lambda$ we can control the number of false positive versus the number of false negatives as illustrated in Figure \[fig:spoofing\_roc\].

![False acceptance rate and false rejection rate under the presence of spoofing attacks. On color blue we illustrate the RGB baseline ($\gamma=0$), on the other extreme, the red curve illustrates the performance when only depth features are considered. The combination of RGB and depth features is illustrated in tones of green for different values of $\alpha$ and $\beta$ (in this experiment we set $\gamma=0.3$).[]{data-label="fig:spoofing_roc"}](figures/roc_gamma.png){width=".6\columnwidth"}

Figure \[fig:spoofing\_roc\] shows the ratio of false positive and false negative for $\lambda\in[0,2]$. As before the distance between the samples is computed using the definition provided in , in blue/red the RGB/depth baseline is illustrated, the other set of curves (displayed in green tones) correspond to a combination of texture and depth features with $\gamma=0.3$ and different values of $\alpha$ and $\beta$. In Table \[tab:spoofing\_tpr\] the ratio of true positive is reported for a fixed ratio of false positives. The ACER measure (last column) corresponds to the average between the ratio of spoofing and genuine samples misclassified.

[L[4.95cm]{}C[1.8cm]{} C[1.8cm]{} C[1.5cm]{}]{} & TPR% @FPR=$10^{-3}$ & TPR% @FPR=$10^{-2}$ & ACER %\
RGB baseline ($\gamma=0$) & $21.8$ & $24.0$ & $38.9$\
Depth baseline ($\gamma=1$) & $\bf{88.4}$ & $\bf{97.1}$ & $4.0$\
(our) $\gamma=0.3,\ \beta=0.35\ \alpha=2$ & $85.5$ & $96.9$ & $4.5$\
(our) $\gamma=0.3,\ \beta=0.35\ \alpha=5$ & $83.8$ & $\bf{97.1}$ & $4.0$\
(our) $\gamma=0.3,\ \beta=0.35\ \alpha=10$ & $85.0$ & $95.6$ & $\bf{3.9}$\
(our) $\gamma=0.3,\ \beta=0.4\ \alpha=2$ & $82.6$ & $96.9$ & $4.7$\
(our) $\gamma=0.3,\ \beta=0.4\ \alpha=5$ & $86.4$ & $\bf{97.1}$ & $4.4$\
(our) $\gamma=0.3,\ \beta=0.4\ \alpha=10$ & $81.8$ & $\bf{97.1}$ & $4.1$\
(our) $\gamma=0.3,\ \beta=0.5\ \alpha=2$ & $86.4$ & $96.4$ & $5.3$\
(our) $\gamma=0.3,\ \beta=0.5\ \alpha=5$ & $82.8$ & $95.6$ & $5.7$\
(our) $\gamma=0.3,\ \beta=0.5\ \alpha=10$ & $85.0$ & $94.4$ & $5.9$\

#### Testing variations on the ambient illumination.

To test the impact of variations on lighting conditions we simulated test samples under different ambient illumination, implementation details are described in the supplementary material Section \[app:simulating\_lighting\_conditions\]. Table \[tab:diff\_light\_acc\] compares the rank-5 accuracy of 2D features and 2D+3D features as the power of the ambient illumination increases. As described in the supplementary material, the ambient illumination is modeled with random orientation, and therefore, the more powerful the illumination is the more diversity between the test and the gallery samples is introduced.

[L[4.95cm]{}C[2.0cm]{} C[2.0cm]{} C[2.0cm]{}]{} Rank-5 Accuracy & power=100% & power=150% & power=200%\
RGB baseline ($\gamma=0$) & $89.2$ & $81.2$ & $53.9$\
(our) $\gamma=0.5$ & $93.6$ & $90.7$ & $80.7$\

In the present experiments, we assumed that both the projected pattern and the ambient illumination have similar spectral content. In practice, one can project the pattern, e.g., on the infrared band. This would make the system invisible to the user, and reduce the sensitivity of 3D features to variations on the ambient illuminations. We provide a hardware implementation feasibility study and illustrate how the proposed ideas can be deployed in practice in the supplementary material Section \[app:hardware implementation\].

#### Improving state of the art 2D face recognition.

To test how the proposed ideas can impact the performance of state-of-the-art 2D face recognition systems, we evaluated our features in combination with texture based features obtained with ArcFace [@arcface]. ArcFace is a powerful method pre-trained on very large datasets, on ND-2006 examples it achieves perfect recognition accuracy (100% rank-1 accuracy). When ArcFace is combined with the proposed 3D features, the accuracy remains excellent (100% rank-1 accuracy), i.e., adding the proposed 3D features does not negatively affects robust 2D solutions. On the other hand, 3D features improve ArcFace on challenging conditions as we discuss in the following. Interesting results are observed when ArcFace is tested under spoofing attacks, as we show in Table \[tab:spoofing\_tpr\_arcface\], ArcFace fails to detect spoofing attacks. ArcFace becomes more robust when it is combined with 3D features, improving from nearly $0$ TPR@FPR($10^{-3}$) to $84\%$. In summary, as 2D methods improve and become more accurate, our 3D features do not affect them negatively when they work well, while improve their robustness in challenging situations.

[L[4.95cm]{}C[2.0cm]{} C[2.0cm]{} C[1.5cm]{}]{} & TPR% @FPR=$10^{-3}$ & TPR% @FPR=$10^{-2}$ & ACER %\
ArcFace ($\gamma=0$) & $0$ & $0$ & $46.2$\
ArcFace + 3D $(\gamma=0.5)$ & $84.7$ & $94.7$ & $7.9$\

Conclusions {#sec:Conclusions}
===========

We proposed an effective and modular alternative to enhance 2D face recognition methods with actual 3D information. A high frequency pattern is designed to exploit the high resolution cameras ubiquitous in modern smartphones and personal devices. Depth gradient information is coded in the high frequency spectrum of the captured image while a standard texture facial image can be recovered to exploit state-of-the-art 2D face recognition methods. We show that the proposed method can be used to simultaneously leverage 3D information and texture information. This allows us to enhance state-of-the-art 2D methods improving their accuracy and making them robust, e.g., to spoofing attack.

Acknowledgments {#acknowledgments .unnumbered}
===============

Work partially supported by ARO, ONR, NSF, and NGA.

[^1]: Now at Google Research.

[^2]: We assume images are extended in an even fashion outside the image domain, to guaranteed that $a_1\in{\mathbb{R}}$ and avoid an additional offset term.

[^3]: This numerical results is obtained by approximating the bounding box of the face as a $20cm\times 20cm$ region, sampled with $480\times 480$ pixels which corresponds to a pixel length of $2.4mm$, a third of the spectral band correspond to signal of a period of $6$ pixels which leads to a binary fringe of at least $7.2mm$ wide.
