Face Beneath the Ink: Synthetic Data and Tattoo Removal with Application to Face Recognition

Systems that analyse faces have seen significant improvements in recent years and are today used in numerous application scenarios. However, these systems have been found to be negatively affected by facial alterations such as tattoos. To better understand and mitigate the effect of facial tattoos in facial analysis systems, large datasets of images of individuals with and without tattoos are needed. To this end, we propose a generator for automatically adding realistic tattoos to facial images. Moreover, we demonstrate the feasibility of the generation by using a deep learning-based model for removing tattoos from face images. The experimental results show that it is possible to remove facial tattoos from real images without degrading the quality of the image. Additionally, we show that it is possible to improve face recognition accuracy by using the proposed deep learning-based tattoo removal before extracting and comparing facial features.


I. INTRODUCTION
Facial analysis systems are deployed in various applications ranging from medical analysis to border control. Such facial analysis systems are known to be negatively affected by facial occlusions [1]. A specific kind of facial alteration that partially occludes a face is a face tattoo. Facial tattoos have become more appealing recently and have been described as a mainstream trend in several major newspapers [2], [3]. Ensuring inclusiveness and accessibility for all individuals, independent of physical appearance, is imperative in developing fair facial analysis systems. In this regard, facial tattoos are especially challenging, as they cause permanent alterations where ink is induced into the dermis layer of the skin. For instance, Ibsen et al. investigated in [4] the impact of facial tattoos and paintings on state-of-the-art face recognition systems. The authors showed that tattoos might impair the recognition accuracy and thus the security of such a facial analysis system.
In coherence with the findings in [4], it is of interest to make facial analysis systems more robust towards facial tattoos. One way to do this is face completion, where missing or occluded parts of a face are reconstructed; such approaches have, for instance, shown to improve face recognition performance for some occlusions [5]. An additional benefit of using face completion over approaches like occlusion-aware face recognition is the potential to use the reconstructed facial image for other The authors are with the Biometrics and Internet Security Research Group at Hochschule Darmstadt, Germany E-mail: mathias.ibsen@h-da.de purposes, e.g. visualising how a face might look without the occlusion or preventing that tattoos are used for recognition purposes; which is something that raises ethical issues as discussed in [6].
However, one major problem with face completion for tattoo removal is the lack of sufficient and high-quality training data, as no extensive database of facial tattoos is currently available.
The main focus of this work is, therefore, two-fold. First, we propose a method for synthetically adding tattoos to facial images, which we use to create a large database of facial images with tattoos. The proposed method uses face detection and landmark localisation to divide the face into regions, whereafter suitable placements of tattoos are found. Subsequently, we approximate depth and construct depth and cut-out maps which are used to realistically blend tattoos onto a face. It has recently been shown that synthetic data can be beneficial for face analysis tasks and be a good alternative to real data [7]. Secondly, we show the usefulness of our synthetic data by training a deep learning-based model for tattoo removal (as illustrated in Fig. 1) and evaluate the impact of removing facial tattoos on a state-of-the-art face recognition system using a database comprising real facial images with tattoos.
The approach for synthetically adding tattoos to a facial image in a fully automated way is, to the authors' best knowledge, the first of its kind. The proposed generator can be used to create large databases which can be used in related fields such as tattoo detection or studying the effects of tattoos on human perception. Additionally, we are the first to measure the effect of removing facial tattoos on face recognition systems. The code for synthetically adding tattoos to face images will be made publicly available 1 .
In summary, this work makes the following contributions: • A novel algorithm for synthetically adding facial tattoos to face images. • An algorithm for removing tattoos from facial images trained on only facial images with synthetically added tattoos. We refer to this algorithm as TRNet. • An experimental analysis of the quality of the tattoo removal. • Showcasing the application of tattoo removal in a face recognition system by conducting an experimental analysis on the effect of removing facial tattoos on a face recognition system. The outline of the remaining article is as follows: Sect. II describes prominent related works, Sect. III describes an automated approach for synthetically blending tattoos to facial images, which is used in Sect. IV to generate a database of facial images with tattoos. Sect. V and VI show the feasibility of the synthetic generation by training a deep learning-based model for tattoo removal and evaluating if it can improve biometric recognition performance, respectively. Finally, Sect. VII provides a summary of this work.

II. RELATED WORK
The following subsections summarise related works w.r.t. synthetic data generation for facial analysis (Sect. II-A), facial alterations (Sect. II-B), and facial completion (Sect. II-C).

A. Synthetic Data Generation for Face Analysis
Synthetically generated data has seen many application scenarios in face analysis, most notably for addressing the lack of training data. Synthetic data has especially become relevant with the recent advances in deep learning-based algorithms, which usually require a large amount of training data. Privacy regulations, e.g. the European General Data Protection Regulation [8], make sharing and distributing largescale face databases impracticable as face images are classified as a special category of personal data when used for biometric identification. As an alternative, researchers have explored the use of synthetic data. The generation of realistic-looking synthetic face data has especially become feasible with the recent advances in Generative Adversarial Networks (GANs), 1 Generation code will be released upon acceptance of this manuscript first proposed by Goodfellow et al. in [9]. Prominent work in this field includes StyleGAN, which was first introduced in [10] by Karras et al. and showed, at the time, state-ofthe-art performance for synthesising facial images. Since the original work, two improved versions of StyleGAN have been proposed [11], [12]. Much current research in this area focuses on GAN-inversion, where existing face images are encoded into the latent space of a generator. Thereafter, the resulting latent code can be shifted in the latent space, whereby the inverted image of the shifted vector results in an alteration of the original image. The technique can, for instance, be used for face age progression [13]. In addition to the face, some research has also been conducted for other biometric modalities, e.g. fingerprint [14], [15], [16] and iris [17], [18].
Little work has been conducted regarding synthetic data generation of facial images with tattoos. However, in [19] the authors proposed a method for transforming digital portrait images into realistic-looking tattoos. In [20], the author also shows examples of tattoo images added to facial and body images using an existing GAN for drawing art portraits; however, details about this approach are not scientifically documented.

B. Facial Alterations
Facial alterations can occur in either the physical or digital domain and cause permanent or temporary changes of a face. Several studies have explored the impact of physical and digital alterations on face recognition systems. In the physical domain, predominantly the effects of makeup and plastic surgery on face recognition have been studied [21]. In [22] the authors collected a database of 900 individuals to analyse the effect of plastic surgery and found that the tested algorithms were unable to effectively account for the appearance changes caused by plastic surgery. More recently, Rathgeb et al. showed in [23], using a database of mostly ICAO-quality face images [24] captured before and after various types of facial plastic surgeries, that different tested state-of-the-art face recognition systems maintained almost perfect verification performance at an operationally relevant threshold corresponding to a False Match Rate (FMR) of 0.1%. Numerous works have addressed the impact of temporary alterations on face recognition systems. In [25], Dantcheva et al. found that makeup can hinder reliable face recognition; similar conclusions were drawn by Wang et al. in [26] where they investigated the impact of human faces under disguise and makeup. The previous work shows that makeup might be successfully used for identity concealment; in [27], the authors additionally showed that makeup could also be used for presentation attacks with the goal of impersonating another identity. In [28], the authors found that especially high-quality makeup-based presentation attacks can hamper the security of face recognition systems. In [29], the authors found that disguised faces severely affect recognition performance, especially for occlusions near the periocular region. The database used by the authors includes different types of disguises, including facial paintings. Coherent with these findings, Ibsen Ferrara et al. were among the first to show that digital alterations can impair the security of face recognition systems. Especially notable is their work in [30] where they showed the possibility of attacking face recognition systems using morphed images. Specifically, they showed that if a highquality morphed image is infiltrated into a face recognition system (e.g. stored in a passport), it is likely that individuals contributing to the morph are positively authenticated by the biometric system. Since then, there have been numerous works on face recognition systems under morphing attacks. For a comprehensive survey, the reader is referred to [31]. Facial retouching is another area which has seen some attention in the research community. While some early works showed that face recognition can be significantly affected by retouching, Rathgeb et al. showed more recently that face recognition systems might be robust to slight alterations caused by retouching [32]. Similar improvements have been shown for geometrical distortions, e.g. stretching [33]. A more recent threat that has arrived with the prevalence of deep-learning techniques is so-called DeepFakes [34], which can be used to spread misinformation and as such lead to a loss of trust in digital content. Many researchers are working on the detection or generation of deep learning-based alterations. Several arduous challenges and benchmarks have already been established, for instance, the recent Deepfake Detection Challenge [35] where the top model only achieved an accuracy of approximately 65% on previously unseen data. Generation and detection of deep learning-based alterations are continuously evolving and remain a cat-and-mouse game; interested readers are referred to [36] for a comprehensive survey.

C. Facial Completion
Most methods for face completion 2 build upon deep learning-based algorithms, which are trained on paired images where each pair contains a non-occluded face and a corresponding occluded face. In [37], the authors proposed an approach for general image completion and showed its applicability for facial completion. In this work, the authors leveraged a fully convolutional neural network trained with global and local context discriminators. Similar work was done in [38] where the authors occluded faces by adding random squares of noise pixels. Subsequently, they trained an autoencoder to reconstruct the occluded part of the face using global and local adversarial losses as well as a semantic parsing loss. Motivated by the prevalence of VR/AR displays which can hinder face-to-face communication, Zhao et al. [39] proposed a new generative architecture with an identity preserving loss. In [40], Song et al. used landmark detection to estimate the geometry of a face and used it, together with the occluded face image, as input to an encoder-decoder architecture for reconstructing the occluded parts of the face. The proposed approach allows generating diverse results by altering the estimated facial geometry. More recently, Din et al. [41] employed a GAN-based architecture for the unmasking of masked facial images. The proposed architecture consists of two stages where the first stage detects the masked area of the face and creates a binary segmentation map. The segmentation map is then used in the second stage for facial completion using a GAN-based architecture with two discriminators where one focuses on the global structure and the other on the occluded parts of the face. In [5], it was found that facial completion can improve face recognition performance.

III. FACIAL TATTOO GENERATOR
To address the lack of existing databases of image pairs of individuals before and after they got facial tattoos, we propose an automated approach for synthetically adding facial tattoos to images. An overview of the proposed generation is depicted in Fig. 2. The process of synthetically adding tattoos to a facial image can be split into two main steps which are described in the following subsections: (1) finding the placement of tattoos in a face and (2) blending the tattoos to the face.

A. Placement of Tattoos
To find suitable placements of tattoos in a face, we start by localising the facial region and detecting landmarks of the face. To this end, we use dlib [42] which returns a list of 68 landmarks as shown in Fig. 3. Thereafter, the landmarks are used to divide the face into small regions of triangles by performing a fixed Delaunay triangulation. The regions are then extended to the forehead by using the length of the nose as an estimate. Each region now constitutes a possible placement of a tattoo; however, such a division is inadequate for the placement of larger tattoos. Therefore, the face is divided into six larger regions. The division of the face into large and small regions gives high controllability in the data generation. As indicated, some regions are excluded, i.e. regions around the nostrils, mouth and nose. These regions are estimated based on the detected landmarks. The division of a face into regions is illustrated in Fig. 4. The regions make it possible to avoid placing tattoos in heavily bearded areas or on top of glasses if such information is available about the facial images during the generation phase. In our work, we do not use beard or glass detectors; however, for some of the images information about beard or glasses is available which we use to avoid placing tattoos in the affected regions.
A tattoo can now be placed in one of the six predefined regions, or the regions can be further combined to allow placing the tattoos in larger areas of the face. A combined region is simply a new region consisting of several smaller regions. The exact placement of a tattoo within a region depends on a pre-selected generation strategy. The generation strategy determines (1) possible regions where a tattoo can be placed, (2) the selection of tattoos, and (3) the size and placement of a tattoo within a region. An example is illustrated in Fig. 5 where one of the cheeks is selected as a possible region, whereafter the largest unoccupied subset within that region is found. Thereafter, the tattoo is placed by estimating its largest possible placement within the selected subset without altering the original aspect ratio of the tattoo. In this work, we use a database comprising more than 600 distinct tattoo templates, which mainly consist of real tattoo designs collected from acquired tattoo books. The selection of which tattoos to place depends on the generation strategies, which are further described in Sect. III-C.

B. Blending
To blend the tattoos to faces, various image manipulations are performed.
Given a facial image and placement of tattoos (see Sect. III-A); each tattoo is placed and overlayed on the facial image by multiplying the tattoo layer with the facial image. Afterwards, the tattoo is displaced to match the contours of the face using displacement mapping. Areas of the tattoo which have been displaced outside the face or inside the mouth, nostrils and eyes are cut out. This is done by using cut-out maps (see Fig. 2), which are calculated from the landmarks detected by dlib in the placement phase. Lastly, the tattoo is made more realistic by colour adjustment, Gaussian blurring, and lowering the opacity of the tattoo.
As previously stated, displacement mapping is used for mapping tattoos to the contours of a face. It is a technique which utilises depth information of texture maps to alter the positions of pixels according to the depth information in the provided map. Contrary to other approaches, such as bump mapping, it alters the source image by displacing pixels. In displacement mapping, a map M containing values in the range 0-255 is used to displace pixels in a source image I. In general, a specific pixel, I(x, y), is displaced in one direction if M (x, y) is less than the theoretical average pixel value of the map (127.5); else it is displaced in the opposite direction. For the displacement technique used in this work, a pixel in the source image is displaced both vertically and horizontally.
More specifically, let c be a coefficient, let (x, y) ∈ I, and let (x, y) ∈ M . The distance for displacing a pixel, I(x, y), in the vertical and horizontal direction is then: M (x, y) − 127.5 127.5 PRNet [43] is used to generate the depth maps used in this work. PRNet is capable of performing 3D face reconstruction from 2D face images, and as such, it can also approximate depth maps from 2D facial images. An example of a depth map generated using PRNet is shown in Fig. 6a. (a) Depth image generated by PR-Net (b) Transformed depth image Fig. 6: Example of (a) a depth map generated from a facial image using PRNet and (b) after it has been transformed.
As seen in Fig. 6a, the pixel values in the face region are rather bright, and there is little contrast. The small contrast between the pixel values and the high offset from the theoretical average pixel value implies that the depth map will not work very well as tattoos will be displaced too much in certain regions and too little in others. Therefore, to make the displacement more realistic, the depth map generated by PRNet is transformed by increasing the contrast and lowering the brightness of the map. Fig. 6b shows an example of a transformed depth map, and as it can be seen, the pixel values are much closer to the theoretical average value than the unaltered map, while the contrast around the nose, eyes and mouth are still high. Fig. 7 shows an example where two facial tattoos are displaced to match the contours of a face. Black ink tends to change in colour slightly over time due to the pigment used in black ink. Therefore, for colour adjustment, all pixels of a tattoo which is similar to pure black are selected and changed to simulate different colours of grey, green, and blue, which causes black tattoos to appear different for different facial images. The colour adjustments of black pixels are determined per tattoo, and as such slight variations can occur between different tattoos in the same facial image. Examples are given in Fig. 9.

C. Generation Strategies
By varying how tattoos are selected and placed (Sect. III-A), many different types of facial images with tattoos can be generated. For the database used in this work, we employed two different strategies. In the first strategy, a desired coverage percent of tattoos on a face is randomly chosen from a specified range. Subsequently, tattoos are arbitrarily selected and placed on facial regions until the resulting coverage approximates the desired coverage. The coverage of a tattoo on a face is calculated based on the total amount of facial regions to which tattoos can be placed (see Fig. 4c) and the number of non-transparent pixels in the placed tattoos. In the second strategy, a specific region is always selected. Using the first strategy, it is possible to create databases where tattoos are placed arbitrarily until a selected coverage percent has been reached (see Fig. 10a-10c). Using the latter approach allows for more controlled placement of tattoos, e.g. placing tattoos in the entire face region (Fig. 10d) or in a specific region (Fig. 10e-10f).

IV. SYNTHETIC TATTOO DATABASE
This section describes the generation of a large database of facial images with tattoos. The database is used in section V to train deep learning-based models for removing tattoos from facial images. To generate the synthetic tattoo database, subsets of original images from the FERET [44], FRGCv2 [45], and CelebA [46] datasets were used. An overview of the generated database is given in Table I. For the FERET and FRGCv2 datasets, different generation strategies were used, including facial images where tattoos have been placed randomly, with specific coverage ranging from 5% to 25% as well as placement of single tattoos. For the single tattoos, we generated two versions: one version where the tattoo is placed in the entire facial region and another where portrait tattoos are blended to a random region in the face. For the CelebA database, which is more uncontrolled, facial tattoos were placed randomly. To simulate varying image qualities, a data augmentation was performed by randomly applying differing degrees of JPEG compression or Gaussian blur to all the images. Tattoo images and corresponding original (bona fide) images were paired such that similar augmentation was applied to corresponding images. Examples of images in the generated database are depicted in Fig. 11.

V. TATTOO REMOVAL
To evaluate the realism of the proposed data generation and its use in real-world applications, two models are trained for the task of tattoo removal. Sect. V-A briefly describes the different models used for removing tattoos. Sect. V-B describes different metrics for evaluating the quality of the tattoo removal which is then evaluated in Sect. V-C.

A. Models
Two different deep learning-based methods were trained for removing tattoos from facial images: pix2pix is a supervised conditional GAN for image-to-image translation [47]. For the generator, a U-Net architecture is used, whereas the discriminator is based on a PatchGAN classifier which divides the image into N × N patches and discriminates between bona fide (i.e., real images) and fake images. Tattoo Removal Net (TRNet) is a U-net architecture [20] [48] with spectral normalization and self-attention trained on our synthetic data (Sect.IV). The U-Net architecture is depicted in Fig. 12; the encoder is based on ResNet34 and the decoder consist of four main blocks and utilize PixelShuffling [49]. The loss function is a combination of feature loss (perceptual loss) from [50], gram matrix style loss [51], and pixel (L1) loss. For the gram matrix loss and the feature loss, blocks from a pre-trained VGG-16 model is used [52] [48].

B. Quality Metrics
To evaluate the quality of the different tattoo removal models, we use three different metrics commonly used in the literature: Peak signal-to-noise ratio (PSNR) is a measurement of error between an input and an output image and is calculated as follows: where MAX I is the theoretical maximum pixel value (i.e. 255 for 8 bit channels) and MSE (X, Y ) is the mean squared error between the ground truth image X and the inpainted image Y . The PSNR is measured in decibel and a higher value indicates better quality of the reconstructed image. Mean Structural Similarity Index (MSSIM) as given in [53], is defined as follows: where X and Y are the ground truth image and inpainted image, respectively, M is the number of local windows in an image and x i and y i are the image content of the i'th local window. The SSIM over local window patches (x, y) is defined as: SSIM (x, y) = (2µ x µ y + C 1 )(2σ xy + C 2 ) (µ 2 x + µ 2 y + C 1 ) + (σ 2 x + σ 2 y + C 2 ) where µ x and µ y are the mean values of the local window patches x and y, respectively; σ 2 x + σ 2 y are their local variances and σ xy is the local covariance of x and y. C 1 and C 2 are constants set based on the same parameter settings as Wang et al. [53], i.e. C 1 ≈ 6.55, C 2 ≈ 58.98. MSSIM returns a value in the range of 0 to 1, where 1 means that X and Y are identical. Visual Information Fidelity (VIF) is a full reference image quality assessment measurement proposed by Sheikh and Bovik in [54]. VIF is derived from a statistical model for natural scenes as well as models for image distortion and the human visual system. V IF (X, Y ) returns a value in range of 0 to 1, where 1 indicates that the ground truth and inpainted images are identical. We use the pixel domain version as implemented in [55].

C. Removal Quality Results
We use a total of 41 facial images with tattoos where the tattoos have been manually removed using PhotoShop; we refer to these as our ground truth images. Examples of using the different deep learning-based methods for removing tattoos are given in Fig. 14. As seen, the best model (TRNet) is able to remove most tattoos with only a few artefacts, whereas the other models perform less well and, for some images, alter the face or fail to remove all tattoos accurately. Different quality scores are reported in Tab. II which shows that the TRNet model performs best in most scenarios especially when only looking at the inner part of the face.
While the tattoo removal performs well in many scenarios, there are also some extreme cases where it does not work so well. Examples of removing large coverage of tattoos from facial images are shown in Fig. 15. Depicted example images clearly show the limitations of the presented approach.

VI. APPLICATION TO FACE RECOGNITION
In this section, we describe how tattoo removal can be integrated and used in a face recognition system. A face recognition system consists of several preprocessing modules such as face alignment and quality estimation. These modules help to minimise factors which are unimportant for face recognition and ensure that only images of sufficient quality are used during authentication. As part of the preprocessing, we propose to use the deep learning-based removal algorithms described in Sect. V. While facial tattoos can be seen as distinctive and helpful in identifying individuals, tattoo removal is useful for face recognition in cases where only one of the face images in a comparison contains tattoos [4]. In our experiments, we trained the classifiers to remove facial tattoos from aligned images and as such will assume that our input images have already been aligned since our focus is on improving feature extraction and comparison. Note that the proposed tattoo removal method could also be retrained on unaligned images and placed before the detection module to improve detection accuracy.

A. Experimental Setup
In the following, we describe the database, the employed face recognition system, and metrics used to evaluate the biometric performance: Database for the evaluation, we use the publicly available database HDA Facial Tattoo and Painting Database 3 , which consists of 250 image pairs of individuals with and without real facial tattoos. The database was originally collected by Ibsen et al. in [4]. The images have all been aligned using the RetinaFace facial detector [56]. Examples of original image-pairs (before tattoo removal) are given in Fig. 16. These pairs of images are used for evaluating the performance of a face recognition system. For evaluating the effect of tattoo removal, the models described in Sect. V-A are employed on the facial images containing tattoos, whereafter the resulting images are used during the evaluation. Fig. 16: Examples of image-pairs in the HDA facial tattoo and painting database.
Face recognition system to evaluate the applicability of tattoo removal for face recognition, we use the established ArcFace pre-trained model (LResNet100E-IR,ArcFace@ms1m-refine-v2) with the RetinaFace facial detector. Recognition performance metrics the effect of removing facial tattoos is evaluated empirically [57]. Specifically, we measure the FNMR at operationally relevant thresholds corresponding to a FMR of 0.1% and 1%:

B. Experimental Results
The effect of removing tattoos on the computed comparison scores is visualised in Fig. 17. As can be seen, the comparison scores are not significantly affected for the pix2pix model which only showed moderate capabilities of removing tattoos from facial images. However, for TRNet, which has been trained on the synthetic database, it is shown that the dissimilarity score on average gets lower, which indicates that the recognition performance might improve.  Table III shows the biometric performance scores calculated on the tattooed images and the inpainted facial images for the different used models. The scores indicate that realistic removal of tattoos (TRNet) might improve face recognition performance since we can observe that, compared to the baseline (tattooed), the EER is halved, and the FNMR at an FMR of 1% is reduced to 0%. The results indicate that a tattoo removal module can be integrated into the processing chain of a face recognition system and help make it more robust towards facial tattoos.

VII. SUMMARY
In this paper, we proposed an automatic approach for blending tattoos onto facial images and showed that it is possible to use synthetic data to train a deep learning-based facial tattoo removal algorithm, thereby enhancing the performance of a state-of-the-art face recognition system. To create a facial image with tattoos, the face is first divided into face regions using landmark detection whereafter tattoo placements can be found. Subsequently, deep reconstruction maps and cut-out maps can be estimated from the input image. Thereafter, the information is combined to realistically blend tattoos to the facial image. Using this approach, we created a large database of facial images with tattoos and used it to train a deep learning-based algorithm for removing tattoos. Experimental results show a high quality of the tattoo removal. To further show the feasibility of the reconstruction, we evaluated the effect of removing facial tattoos on a state-of-the-art face recognition system and found that it can improve automated face recognition performance.