Thermal to visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or extract robust features from these modalities for cross-modal matching. In this paper, we take a different approach in which we make use of the attributes extracted from the visible image to synthesize the attribute-preserved visible image from the input thermal image for cross-modal matching. A pre-trained VGG-Face network is used to extract the attributes from the visible image. Then, a novel Attribute Preserved Generative Adversarial Network (AP-GAN) is proposed to synthesize the visible image from the thermal image guided by the extracted attributes. Finally, a deep network is used to extract features from the synthesized image and the input visible image for verification. Extensive experiments on the ARL Polarimetric face dataset show that the proposed method achieves significant improvements over the state-of-the-art methods.
We investigate the performance of polarimetric imaging in the long-wave infrared (LWIR) spectrum for cross-modal face recognition. For this work, polarimetric imagery is generated as stacks of three components: the conventional thermal intensity image (referred to as S0), and the two Stokes images, S1 and S2, which contain combinations of different polarizations. The proposed face recognition algorithm extracts and combines local gradient magnitude and orientation information from S0, S1, and S2 to generate a robust feature set that is well-suited for cross-modal face recognition. Initial results show that polarimetric LWIR-to-visible face recognition achieves an 18 increase in Rank-1 identification rate compared to conventional LWIR-to-visible face recognition. We conclude that a substantial improvement in automatic face recognition performance can be achieved by exploiting the polarization-state of radiance, as compared to using conventional thermal imagery. The problem of face identication in the Mid-Wave InfraRed (MWIR) spectrum is studied in order to understand the performance of intra-spectral (MWIR to MWIR) and cross-spectral (visible to MWIR) matching. The contributions of this work are two-fold. First, a database of 50 subjects is assembled and used to illustrate the challenges associated with the problem. Second, a set of experiments is performed in order to demonstrate the possibility of MWIR intra-spectral and cross-spectral matching. Experiments show that images captured in the MWIR band can be eciently matched to MWIR images using existing techniques (originally not designed to address such a problem). These results are comparable to the baseline results, i.e., when comparing visible to visible face images. Experiments also show that cross-spectral matching (the heterogeneous problem, where gallery and probe sets have face images acquired in dierent spectral bands) is a very challenging problem. In order to perform cross-spectral matching, we use multiple texture descriptors and demonstrate that fusing these descriptors improves recognition performance. Experiments on a small database, suggests that the problem of cross-spectral matching requires further investigation.© (2012) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only. We present a series of long-wave-infrared (LWIR) polarimetric-based thermal images of facial profiles in which polarization-state information of the image-forming radiance is retained and displayed. The resultant polarimetric images show enhanced facial features, additional texture, and details that are not present in corresponding conventional thermal imagery. It has been generally thought that conventional thermal imagery (MidIR or LWIR) could not produce the detailed spatial information required for reliable human identification due to the so-called “ghosting” effect often seen in thermal imagery of human subjects. By using polarimetric information, we are able to extract subtle surface features of the human face, thus improving subject identification. Polarimetric image sets considered include the conventional thermal intensity image, S0, the two Stokes images, S1 and S2, and a Stokes image product called the degree-of-linear-polarization image. Although visible face recognition has been an active area of research for several decades, cross-modal face recognition has only been explored by the biometrics community relatively recently. Thermal-to-visible face recognition is one of the most difficult cross-modal face recognition challenges, because of the difference in phenomenology between the thermal and visible imaging modalities. We address the cross-modal recognition problem using a partial least squares (PLS) regression-based approach consisting of preprocessing, feature extraction, and PLS model building. The preprocessing and feature extraction stages are designed to reduce the modality gap between the thermal and visible facial signatures, and facilitate the subsequent one-vs-all PLS-based model building. We incorporate multi-modal information into the PLS model building stage to enhance cross-modal recognition. The performance of the proposed recognition algorithm is evaluated on three challenging datasets containing visible and thermal imagery acquired under different experimental scenarios: time-lapse, physical tasks, mental tasks, and subject-to-camera range. These scenarios represent difficult challenges relevant to real-world applications. We demonstrate that the proposed method performs robustly for the examined scenarios. Heterogeneous face recognition (HFR) involves matching two face images from alternate imaging modalities, such as an infrared image to a photograph or a sketch to a photograph. Accurate HFR systems are of great value in various applications (e.g., forensics and surveillance), where the gallery databases are populated with photographs (e.g., mug shot or passport photographs) but the probe images are often limited to some alternate modality. A generic HFR framework is proposed in which both probe and gallery images are represented in terms of nonlinear similarities to a collection of prototype face images. The prototype subjects (i.e., the training set) have an image in each modality (probe and gallery), and the similarity of an image is measured against the prototype images from the corresponding modality. The accuracy of this nonlinear prototype representation is improved by projecting the features into a linear discriminant subspace. Random sampling is introduced into the HFR framework to better handle challenges arising from the small sample size problem. The merits of the proposed approach, called prototype random subspace (P-RS), are demonstrated on four different heterogeneous scenarios: 1) near infrared (NIR) to photograph, 2) thermal to photograph, 3) viewed sketch to photograph, and 4) forensic sketch to photograph. Face recognition research has primarily focused on the visible spectrum, due to the prevalence and low cost of visible cameras. However, face recognition in the visible spectrum is sensitive to illumination variations, and is infeasible in low-light or nighttime settings. In contrast, thermal imaging acquires naturally emitted radiation from facial skin tissue, and is therefore ideal for nighttime surveillance and intelligence gathering operations. However, conventional thermal face imagery lacks textural and geometrics details that are present in visible spectrum face signatures. In this work, we further explore the impact of polarimetric imaging in the LWIR spectrum for face recognition. Polarization-state information provides textural and geometric facial details unavailable with conventional thermal imaging. Since the frequency content of the conventional thermal, polarimetric thermal, and visible images is quite different, we propose a spatial correlation based procedure to optimize the filtering of polarimetric thermal and visible face images to further facilitate cross-spectrum face recognition. Additionally, we use a more extensive gallery database to more robustly demonstrate an improvement in the performance of cross-spectrum face recognition using polarimetric thermal imaging. ABSTRACT In low light conditions, visible light face identication is infeasible due to the lack of illumination. For nighttimesurveillance, thermal imaging is commonly used because of the intrinsic emissivity of thermal radiation from thehuman body. However, matching thermal images of faces acquired at nighttime to the predominantly visiblelight face imagery in existing government databases and watch lists is a challenging task. The diculty arisesfrom the signicant dierence between the faces thermal signature and its visible signature (i.e. the modalitygap). To match the thermal face to the visible face acquired by the two dierent modalities, we applied facerecognition algorithms that reduce the modality gap in each step of face identication, from low-level analysis tomachine learning techniques. Specically, partial least squares-discriminant analysis (PLS-DA) based approacheswere used to correlate the thermal face signatures to the visible face signatures, yielding a thermal-to-visible faceidentication rate of 49.9 . While this work makes progress for thermal-to-visible face recognition, more eortsneed to be devoted to solving this dicult task. Successful development of a thermal-to-visible face recognitionsystem would signicantly enhance the Nations nighttime surveillance capabilities.Keywords: thermal, visible, thermal-to-visible face recognition, face, recognition, multi, modal