Thermal cameras and applications: a survey <s> Facial analysis <s> A method for face identification has been developed by exploiting thermal image processing techniques. The method is based on 2-dimensional detection of the temperature distribution of the face, using infrared rays. The front-view face in the input image is normalized in terms of location and size, followed by measurement of the temperature distribution, the locally averaged temperature and the shape factors of face. The measured temperature distribution and the locally averaged temperature are separately used as input data for a neural network, while the values of shape factors are used for supervised classification. By integrating information from the NN and supervised classification, the face was identified with excellent accuracy. The shortcoming of visible ray image analysis that the accuracy of face identification is strongly influenced by lighting condition including variation of shadow, reflection and darkness is considered to be perfectly overcome by the present method exploiting infrared rays. <s> BIB001 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> A method using thermal image processing and a neural network is presented for recognition of facial expression. The method is based on 2-dimensional detection of temperature distribution of the face, using infrared rays. The front-view face in the input image is normalized in terms of the size and the location, followed by measurement of the local temperature difference between the averaged neutral and the unknown expression faces. For measuring the local temperature difference, several blocks on the face are selected, based on the studies in the fields of psychology, especially FACS-AU, and computer vision with visible ray, while the blocks including cheekbone region are selected experimentally for improving recognition accuracy. The local temperature difference caused by the rearrangement of face muscle and the inner temperature change is used as input data for a neural network. By using the backpropagation method, neutral, happy, surprised and sad expressions were recognized with 90% accuracy. <s> BIB002 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> A key problem for face recognition has been accurate identification under variable illumination conditions. Conventional video cameras sense reflected light so that image grayvalues are a product of both intrinsic skin reflectivity and external incident illumination, thus obfuscating the intrinsic reflectivity of skin. Thermal emission from skin, on the other hand, is an intrinsic measurement that can be isolated from external illumination. We examine the invariance of Long-Wave InfraRed (LWIR) imagery with respect to different illumination conditions from the viewpoint of performance comparisons of two well-known face recognition algorithms applied to LWIR and visible imagery. We develop rigourous data collection protocols that formalize face recognition analysis for computer vision in the thermal IR. <s> BIB003 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> Recent research has demonstrated distinct advantages of using thermal infrared imaging for improving face recognition performance. While conventional video cameras sense reflected light, thermal infrared cameras primarily measure emitted radiation from objects such as faces. Visible and thermal infrared image data collections of frontal faces have been on-going at NIST for over two years, producing the most comprehensive face database known to involve thermal infrared imagery. Rigorous experimentation with this database has revealed consistently superior recognition performance of algorithms when applied to thermal infrared, particularly under variable illumination conditions. Physical phenomenology responsible for this observation is analyzed. An end-to-end face recognition system incorporating simultaneous coregistered thermal infrared and visible has been developed and tested indoors with good performance. <s> BIB004 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as Thermal Minutia Points (TMP) and constitute the feature database. To render the method robust to facial pose variations we collect for each subject to be stored in the database five (5) different pose images (center, mid-left profile, left profile, mid-right profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a sizeable database of thermal facial images collected in our lab. The good experimental results show that the proposed methodology has merit. More important, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area. <s> BIB005 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> This paper studies the problem of determining facial orientation without correspondences in distortion-related infrared images. To improve estimation accuracy and reduce sensitivity to noise and unavoidable error, a simple and robust method based on the single linkage clustering is proposed to simultaneously detect inlier set and estimate orientation angle under contaminated data. An iterative strategy is adopted to avoid random choice of link distance in the single linkage clustering. The experimental results indicate that the proposed method is substantially superior to the moment method. This method can also be extended to detect arbitrary objects with mirror symmetrical or nearly symmetrical property. <s> BIB006 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> The current dominant approaches to face recognition rely on facial characteristics that are on or over the skin. Some of these characteristics have low permanency can be altered, and their phenomenology varies significantly with environmental factors (e.g., lighting). Many methodologies have been developed to address these problems to various degrees. However, the current framework of face recognition research has a potential weakness due to its very nature. We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs) and constitute the feature database. To render the method robust to facial pose variations, we collect for each subject to be stored in the database five different pose images (center, midleft profile, left profile, midright profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a multipose database of thermal facial images collected in our laboratory, as well as on the time-gap database of the University of Notre Dame. The good experimental results show that the proposed methodology has merit, especially with respect to the problem of low permanence over time. More importantly, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area <s> BIB007 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> In this work we present an efficient approach for physiological features extraction from infrared face images. The physiological features represent the network of blood vessels under the face skin. These networks are unique to each individual. The proposed technique permits the construction of Faceprints (similar to fingerprints) from thermal infrared images of faces. The extracted faceprint is unique and can be used in infrared face recognition. <s> BIB008 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> This paper presents a novel approach to handle the challenges of face recognition. In this work thermal face images are considered, which minimizes the affect of illumination changes and occlusion due to moustache, beards, adornments etc. The proposed approach registers the training and testing thermal face images in polar coordinate, which is capable to handle complicacies introduced by scaling and rotation. Polar images are projected into eigenspace and finally classified using a multi-layer perceptron. In the experiments we have used Object Tracking and Classification Beyond Visible Spectrum (OTCBVS) database benchmark thermal face images. Experimental results show that the proposed approach significantly improves the verification and identification performance and the success rate is 97.05%. <s> BIB009 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> Head pose estimation is one of key steps for human and robot interaction because head pose is highly related with the attention of human to robot. Although head pose estimation has progressed in the past few decades, existing algorithms for visual imagery are not fast enough for smooth human and robot interaction. This paper proposes an algorithm to estimate head pose of a previously unseen head from thermal images, motivated by the fact that thermal imagery provides us the possibility to develop light-weight computing algorithm for estimating head pose. The proposed algorithm has multiple merits. Firstly, it is a fully-automatic algorithm. That does not require any manual initialization. Second, its operation is very simple. It recognizes the head pose from a single image. Third, it is a light-weight computing algorithm. This is mainly because it has a fast head localization procedure. The proposed algorithm can estimate head pose at 110 fps. Last, it is a very robust algorithm in head localization. The performance in head pose estimation is promising. <s> BIB010 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> In this paper, we address the problem of pedestrians tracking in cluttered scenes using location, color and thermal cues. The Dezert–Smarandache (DSm) theoretical framework is used to combine the measurements provided by the sensors into a single and unified frame. The use of DSm Theory allows modeling the conflicts that might arise between the sensors due to the presence of clutter and high levels of occlusion. The location cue is integrated as a prior knowledge, which increases the robustness of the tracking. A belief measure is derived and used as a step in a particle filtering algorithm. Finally, experimental results are given, where the developed approach is used to track walking persons in indoor scenes with high levels of occlusion and clutter. <s> BIB011 </s> Thermal cameras and applications: a survey <s> Facial analysis <s> Accurate tracking of facial tissue in thermal infrared imaging is challenging because it is affected not only by positional but also physiological (functional) changes. This paper presents a particle filter tracker driven by a probabilistic template function with both spatial and temporal smoothing components, which is capable of adapting to abrupt positional and physiological changes. The method was tested on tracking facial regions of subjects under varying physiological and environmental conditions in 25 thermal clips. It demonstrated robustness and accuracy, outperforming other strategies. This new method promises improved performance in a number of biomedical applications that involve physiological measurements on the face, such as unobtrusive sleep and stress studies. <s> BIB012
Face detection is the first step in many applications, including face recognition, head pose analysis, or even some full person detection systems. Since the face is normally not covered by clothes, a thermal camera can capture the direct skin temperature of the face. Krotosky et al. [98] and Mekyska et al. [121] proposed head detection systems based on a combination of temperature and shape. Face recognition using thermal cameras eliminates the effects of illumination changes and eases the segmentation step, but it can also introduce some challenges due to the different heat patterns of a subject, caused by different activity levels or emotions such as anxiety. One of the very early approaches is neural networks BIB001 . Socolinsky and Selinger BIB003 and Wolff et al. BIB004 compared the use of thermal images in face recognition to visual images using appearance-based methods. The thermal images yield better results than visual images here. However, it has not yet been tested how different activity levels of the subjects and extreme ambient temperature will affect the recognition rate. In BIB009 , a thermal face recognition algorithm has been developed using the techniques of polar transformation, eigenspace projection, and classification using a multilayer perceptron. Gault et al. [56] tested the use of different parts of the face for facial recognition, and concluded that using the upper part of the face gives a better recognition rate than using the whole face. Buddharaju et al. BIB005 BIB007 proposed a face recognition system using characteristic and time-invariant physiological information as features. The recognition of common facial expressions is another task of great interest. Neural networks have also been used as an early approach here BIB002 . Using a sparse dataset of 120 images, showing four different expressions from one person, the system showed good results. Jiang and Kang [80] proposed a system to recognise facial expressions by analysing the geometry and local characteristics. Also facial orientations are of interest in many vision systems. A few papers have proposed systems to estimate the head pose. Wu et al. BIB006 calculated the roll angle of frontal face images, while Yu et al. BIB010 proposed a system to estimate the yaw angle of the head. Kato et al. [92] proposed a system for detecting the driver's posture in a car. First, the face area is detected, and then the posture is classified as leftward, frontward or rightward. Measuring the heat distribution in the face can give information about the anxiety level [141], the emotion of car drivers [95] , or it can be used for automatic blush detection [66] . For such systems to work automatically, it is important that the system is able to follow the tissue of interest over time. Zhou et al. BIB012 proposed such a tracking system, using a particle filter tracker. For biometric identification, Akhloufi and Bendada BIB008 proposed the use of thermal 'faceprints'. These faceprints capture facial physiological features, representing the network of blood vessels under the skin. BIB011 Guzman et al. [61, 62] also proposed the use of thermal face images for biometric identification. They extracted the superficial blood vessels from MWIR images with skeletonization. Figure 11 