Cross modal face matching between the thermal and visible spectrum is a much desired capability for night-time surveillance and security applications. Due to a very large modality gap, thermal-to-visible face recognition is one of the most challenging face matching problem. In this paper, we present an approach to bridge this modality gap by a significant margin. Our approach captures the highly non-linear relationship between the two modalities by using a deep neural network. Our model attempts to learn a non-linear mapping from the visible to the thermal spectrum while preserving the identity information. We show substantive performance improvement on three difficult thermal---visible face datasets. The presented approach improves the state-of-the-art by more than 10 on the UND-X1 dataset and by more than 15---30 on the NVESD dataset in terms of Rank-1 identification. Our method bridges the drop in performance due to the modality gap by more than 40 .
One of the very first comparative studies on visible and thermal face recognition was performed by @cite_19 . They concluded that "LWIR thermal imagery of human faces is not only a valid biometric, but almost surely a superior one to comparable visible imagery." A good survey on single model and cross modal face recognition methods can be found in @cite_16 .
@cite_19 @cite_16
2021242760 1991410148 2204652942 2963156682
Although visible face recognition has been an active area of research for several decades, cross-modal face recognition has only been explored by the biometrics community relatively recently. Thermal-to-visible face recognition is one of the most difficult cross-modal face recognition challenges, because of the difference in phenomenology between the thermal and visible imaging modalities. We address the cross-modal recognition problem using a partial least squares (PLS) regression-based approach consisting of preprocessing, feature extraction, and PLS model building. The preprocessing and feature extraction stages are designed to reduce the modality gap between the thermal and visible facial signatures, and facilitate the subsequent one-vs-all PLS-based model building. We incorporate multi-modal information into the PLS model building stage to enhance cross-modal recognition. The performance of the proposed recognition algorithm is evaluated on three challenging datasets containing visible and thermal imagery acquired under different experimental scenarios: time-lapse, physical tasks, mental tasks, and subject-to-camera range. These scenarios represent difficult challenges relevant to real-world applications. We demonstrate that the proposed method performs robustly for the examined scenarios. We present a series of long-wave-infrared (LWIR) polarimetric-based thermal images of facial profiles in which polarization-state information of the image-forming radiance is retained and displayed. The resultant polarimetric images show enhanced facial features, additional texture, and details that are not present in corresponding conventional thermal imagery. It has been generally thought that conventional thermal imagery (MidIR or LWIR) could not produce the detailed spatial information required for reliable human identification due to the so-called “ghosting” effect often seen in thermal imagery of human subjects. By using polarimetric information, we are able to extract subtle surface features of the human face, thus improving subject identification. Polarimetric image sets considered include the conventional thermal intensity image, S0, the two Stokes images, S1 and S2, and a Stokes image product called the degree-of-linear-polarization image. Face recognition research has primarily focused on the visible spectrum, due to the prevalence and low cost of visible cameras. However, face recognition in the visible spectrum is sensitive to illumination variations, and is infeasible in low-light or nighttime settings. In contrast, thermal imaging acquires naturally emitted radiation from facial skin tissue, and is therefore ideal for nighttime surveillance and intelligence gathering operations. However, conventional thermal face imagery lacks textural and geometrics details that are present in visible spectrum face signatures. In this work, we further explore the impact of polarimetric imaging in the LWIR spectrum for face recognition. Polarization-state information provides textural and geometric facial details unavailable with conventional thermal imaging. Since the frequency content of the conventional thermal, polarimetric thermal, and visible images is quite different, we propose a spatial correlation based procedure to optimize the filtering of polarimetric thermal and visible face images to further facilitate cross-spectrum face recognition. Additionally, we use a more extensive gallery database to more robustly demonstrate an improvement in the performance of cross-spectrum face recognition using polarimetric thermal imaging. Thermal to visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or extract robust features from these modalities for cross-modal matching. In this paper, we take a different approach in which we make use of the attributes extracted from the visible image to synthesize the attribute-preserved visible image from the input thermal image for cross-modal matching. A pre-trained VGG-Face network is used to extract the attributes from the visible image. Then, a novel Attribute Preserved Generative Adversarial Network (AP-GAN) is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. Finally, a deep network is used to extract features from the synthesized image and the input visible image for verification. Extensive experiments on the ARL Polarimetric face dataset show that the proposed method achieves significant improvements over the state-of-the-art methods.