Thermal cameras and applications: a survey <s> Fig. 12 <s> We present a comprehensive performance study of multiple appearance-based face recognition methodologies, on visible and thermal infrared imagery. We compare algorithms within the same imaging modality as well as between them. Both identification and verification scenarios are considered, and appropriate performance statistics reported for each case. Our experimental design is aimed at gaining full understanding of algorithm performance under varying conditions, and is based on Monte Carlo analysis of performance measures. This analysis reveals that under many circumstances, using thermal infrared imagery yields higher performance, while in other cases performance in both modalities is equivalent. Performance increases further when algorithms on visible and thermal infrared imagery are fused. Our study also provides a partial explanation for the multiple contradictory claims in the literature regarding performance of various algorithms on visible data sets. <s> BIB001 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> In the current context of increased surveillance and security, more sophisticated surveillance systems are needed. One idea relies on the use of pairs of video (visible spectrum) and thermal infrared (IR) cameras located around premises of interest. To automate the system, a dedicated image processing approach is required, which is described in the paper. The first step in the proposed study is to collect a database of known scenarios both indoor and outdoor with a few pedestrians. These image sequences (video and TIR) are synchronized, geometrically corrected and temperature calibrated. The next step is to develop a segmentation strategy to extract the regions of interest (ROI) corresponding to pedestrians in the images. The retained strategy exploits the motion in the sequences. Next, the ROIs are grouped from image to image separately for both video and TIR sequences before a fusion algorithm proceeds to track and detect humans. This insures a more robust performance. Finally, specific criteria of size and temperature relevant to humans are introduced as well. Results are presented for a few typical situations. <s> BIB002 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> We present a comparative study of face recognition performance with visible and thermal infrared imagery, emphasizing the influence of time-lapse between enrollment and testing images. Most previous research in this area, with few exceptions, focused on results obtained when enrollment and testing images were acquired in the same session. We show that the performance difference between visible and thermal recognition in a time-lapse scenario is smaller than previously believed, and in fact is not statistically significant on existing data sets. <s> BIB003 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> This paper presents the results of several large-scale studies of face recognition employing visible-light and infrared (IR) imagery in the context of principal component analysis. We find that in a scenario involving time lapse between gallery and probe, and relatively controlled lighting, (1) PCA-based recognition using visible-light images outperforms PCA-based recognition using infrared images, (2) the combination of PCA-based recognition using visible-light and infrared imagery substantially outperforms either one individually. In a same-session scenario (i.e., near-simultaneous acquisition of gallery and probe images) neither modality is significantly better than the other. These experimental results reinforce prior research that employed a smaller data set, presenting a convincing argument that, even across a broad experimental spectrum, the behaviors enumerated above are valid and consistent. <s> BIB004 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> In uncontrolled environments, with dynamic background and lighting changes, performing efficient and real-time foreground - background segmentation is very challenging. This work is based on the hypothesis that the combination of long wave infrared (LWIR) (8-12 mum) and colour cameras can significantly improve the robustness of moving objects extraction. Pros and cons of colour and thermal imagers in outdoor video monitoring applications are discussed. In order to fuse information from both sensors, we favoured an approach based on "analytical fusion" rather than "representative fusion". Starting from a state-of-the-art algorithm for moving objects extraction in colour video (non-parametric codebook model), we first adapted the method for processing of "red-green-blue-thermal" video format. A preliminary objective performance evaluation of detection accuracy is presented. Original image sequences grabbed with co-aligned thermal and visible fields of view was used. Finally, some improvements to the original codebook model are proposed. <s> BIB005 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> The illumination variation problem is one of the well-known problems in face recognition in uncontrolled environment. In this paper an extensive and up-to-date survey of the existing techniques to address this problem is presented. This survey covers the passive techniques that attempt to solve the illumination problem by studying the visible light images in which face appearance has been altered by varying illumination, as well as the active techniques that aim to obtain images of face modalities invariant to environmental illumination. <s> BIB006 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> Mini Unmanned Aerial Vehicles (mUAVs) have the potential to assist Wilderness Search and Rescue groups by providing a bird's eye view of the search area. This paper proposes a method for augmenting visible-spectrum searching with infrared sensing in order to make use of thermal search clues. It details a method for combining the color and heat information from these two modalities into a single fused display to reduce needed screen space for remote field use. To align the video frames for fusion, a method for simultaneously pre-calibrating the intrinsic and extrinsic parameters of the cameras and their mount using a single multi-spectral calibration rig is also presented. A user study conducted to validate the proposed image fusion methods showed no reduction in performance when detecting various objects of interest in this single-screen fused display compared to side-by-side viewing. Most significantly, the users' increased performance on a simultaneous auditory task showed that their cognitive load was reduced when using the fused display. <s> BIB007 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> With the cost decline of Infrared (IR) cameras, it is envisage that more IR camera will be deployed in vision surveillance system for around-clock day and night surveillance. Infrared camera sense radiation emitted by an object at a non-zero absolute temperature in the infrared spectrum which is not available in visual image, but lose information, such as texture, color and geometric, which is available in visual cameras. Fusion of IR and visual images can enhance features in both kinds of images. And more impressively, it can reveal some new features that might not be present either in IR images or in visual images. In this paper, a statistical signal processing approach based on expectation maximization (EM) is proposed for IR and visual image fusion. The sensor images are described as the true scene corrupted by additive Gaussian distortion. At each iteration of the EM, the fusion result is obtained by using covariance intersection (CI) in the E-step, while the model parameters are updated in the M-step. The simulation results using the real IR and visual images demonstrate the effectiveness of the proposed method. <s> BIB008 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> We present a new Tri-band Color Low-light Observation (TRICLOBS) system The TRICLOBS is an all-day all-weather surveillance and navigation tool. Its sensor suite consists of two digital image intensifiers and an uncooled longwave infrared microbolometer. This sensor suite registers the visual, near-infrared and longwave infrared bands of the electromagnetic spectrum. The optical axes of the three cameras are aligned, using two dichroic beam splitters. A fast lookup-table based color transform (the Color-the-Night color mapping principle) is used to represent the TRICLOBS image in natural daylight colors (using information in the visual and NIR bands) and to maximize the detectability of thermal targets (using the LWIR signal). A bottom-up statistical visual saliency mode is deployed in the initial optimization of the color mapping for surveillance and navigation purposes. Extensive observer experiments will result in further optimization of the color representation for a range of different tasks. <s> BIB009 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> This paper presents two methods for fusion of infrared (IR) and visible surveillance images. The first method combines Curvelet Transform (CT) with Discrete Wavelet Transform (DWT). As wavelets do not represent long edges well while curvelets are challenged with small features, our objective is to combine both to achieve better performance. The second approach uses Discrete Wavelet Packet Transform (DWPT), which provides multiresolution in high frequency band as well and hence helps in handling edges better. The performance of the proposed methods have been extensively tested for a number of multimodal surveillance images and compared with various existing transform domain fusion methods. Experimental results show that evaluation based on entropy, gradient, contrast etc., the criteria normally used, are not enough, as in some cases, these criteria are not consistent with the visual quality. It also demonstrates that the Petrovic and Xydeas image fusion metric is a more appropriate criterion for fusion of IR and visible images, as in all the tested fused images, visual quality agrees with the Petrovic and Xydeas metric evaluation. The analysis shows that there is significant increase in the quality of fused image, both visually and quantitatively. The major achievement of the proposed fusion methods is its reduced artifacts, one of the most desired feature for fusion used in surveillance applications. <s> BIB010 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> The next generation of automotive Night Vision Enhancement systems offers automatic pedestrian recognition with a ::: performance beyond current Night Vision systems at a lower cost. This will allow high market penetration, covering the ::: luxury as well as compact car segments. ::: Improved performance can be achieved by fusing a Far Infrared (FIR) sensor with a Near Infrared (NIR) sensor. ::: However, fusing with today's FIR systems will be too costly to get a high market penetration. The main cost drivers of ::: the FIR system are its resolution and its sensitivity. Sensor cost is largely determined by sensor die size. Fewer and ::: smaller pixels will reduce die size but also resolution and sensitivity. Sensitivity limits are mainly determined by ::: inclement weather performance. Sensitivity requirements should be matched to the possibilities of low cost FIR optics, ::: especially implications of molding of highly complex optical surfaces. As a FIR sensor specified for fusion can have ::: lower resolution as well as lower sensitivity, fusing FIR and NIR can solve performance and cost problems. ::: To allow compensation of FIR-sensor degradation on the pedestrian detection capabilities, a fusion approach called ::: MultiSensorBoosting is presented that produces a classifier holding highly discriminative sub-pixel features from both ::: sensors at once. The algorithm is applied on data with different resolution and on data obtained from cameras with ::: varying optics to incorporate various sensor sensitivities. As it is not feasible to record representative data with all ::: different sensor configurations, transformation routines on existing high resolution data recorded with high sensitivity ::: cameras are investigated in order to determine the effects of lower resolution and lower sensitivity to the overall ::: detection performance. This paper also gives an overview of the first results showing that a reduction of FIR sensor ::: resolution can be compensated using fusion techniques and a reduction of sensitivity can be compensated. <s> BIB011 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> In the current context of increased surveillance and security, more sophisticated and robust surveillance systems are needed. One idea relies on the use of pairs of video (visible spectrum) and thermal infrared (IR) cameras located around premises of interest. To automate the system, a robust person detection algorithm and the development of an efficient technique enabling the fusion of the information provided by the two sensors becomes necessary and these are described in this chapter. Recently, multi-sensor based image fusion system is a challenging task and fundamental to several modern day image processing applications, such as security systems, defence applications, and intelligent machines. Image fusion techniques have been actively investigated and have wide application in various fields. It is often a vital pre-processing procedure to many computer vision and image processing tasks which are dependent on the acquisition of imaging data via sensors, such as IR and visible. One such task is that of human detection. To detect humans with an artificial system is difficult for a number of reasons as shown in Figure 1 (Gavrila, 2001). The main challenge for a vision-based pedestrian detector is the high degree of variability with the human appearance due to articulated motion, body size, partial occlusion, inconsistent cloth texture, highly cluttered backgrounds and changing lighting conditions. <s> BIB012 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> Infrared (IR) cameras capture thermal radiations emitted by objects in a scene while Electro-Optical (EO) cameras picture reflected colors from a scene. These sources of different modalities produce complementary data of the same panorama. In aeronautics, medical imaging and rescue operations, these sensors are often utilized simultaneously. The objective of this work is to integrate the content of both streams in order to deliver a unique image presenting more visual cues than any of the originals taken separately. To reach that goal, the synthesis of a robust algorithm is required. The Empirical Mode Decomposition (EMD) decomposes image signals into Intrinsic Mode Functions (IMFs). In this paper, we proposed a novel approach that integrates the IR and EO IMFs using principles of neural science for multi-spectral fusion. We show how to integrate IR and EO information utilizing spatial opponent processing. We report the performance on images from Octec ltd and compare our result to their Wavelet-based fusion output. Mutual Information is the metric employed for fusion quality assessment in this work. Positive results have been obtained for tests conducted on IR and EO images containing complementary information. <s> BIB013 </s> Thermal cameras and applications: a survey <s> Fig. 12 <s> In this paper we present a novel method of face identification using different levels of pixel fusion (e.g. ratios for pixel information taken from the visual and thermal images are, 2:3, 1:1, 3:2 and 7:3) and classification of fused images using different kernels of Support Vector Machine (SVM). Visual imagery has been broadly used in face identification systems, but these are very sensitive to illumination changes. This limitation has been overcome by the Infrared (IR) spectrum that provides simpler and more robust solution to boost the identification performance in uncontrolled environments and deliberate attempts to obscure identity. But IR imagery is sensitive to temperature changes in the surrounding environment and variations in the heat patterns of the face and it is opaque to glass. All these facts degrade the face identification efficiency. This drove us to fuse information from both visual and thermal spectra, which have the potential to improve face identification performance as fusion of thermal and visual images provide improved images with more compact information. Once we get fused images those are reduced in dimension using Eigenvalue Decomposition based Candid Co-variance free Incremental Principal Component Analysis (EVD-CCIPCA) and these reduced fused images are classified using the three different kernels of SVM. The three kernels used here are: linear, polynomial and gaussian RBF. SVM is primarily a classifier method that performs classification tasks by constructing hyperplanes in a multidimensional space that separates cases of different class labels. In this paper, we have used multiclass SVM to carry out identification on face images and Quadratic Programming (QP) optimization method to train the SVM. For experiments, IRIS Thermal/Visual Face Database is used. Experimental results show that 97.28% is the highest average success rate achieved on the fused images of 70% visual and 30% thermal images using the linear kernel. However, the highest success rate of 100% is achieved for classes 4 and 10 in several cases. <s> BIB014
Thermal and visible image of a calibration chessboard consisting of two cardboard sheets, where only the white layer has been heated before assembling Fusion can take place at different levels of the processing, often described as pixel level, feature level, or decision level BIB012 . In pixel-level fusion, the images need to be spatially registered, as discussed above, so that the same pixel positions of all images correspond to the same location in the real world. The images are then combined pixel-by-pixel using a fusion algorithm. With feature-level fusion, the features are found in all images individually and then fused into a joint feature set. Decision-level fusion follows the individual processing until the evaluation of the observed scene is done. At the end-stage, the decisions, or classifications, are combined into one result. The choice of fusion level will depend on the application. Methods for fusing visible and infrared videos are discussed in BIB007 and BIB013 , where the two images are aligned and combined using an overlay image with the heat information. In [133] , two fusion methods were compared, one named a general fusion model (pixel level) and the other method named a combination module (feature level). The combination module has the best performance tested over six sequences. Shah et al. BIB010 proposed a combination of curvelet and wavelet transforms, as well as a discrete wavelet packet transform approach for fusing visual and thermal images. In BIB008 , a statistical approach based on expectation maximisation was proposed. Lallier and Farooq [103] used an adaptive weighting method, which enhances unnatural objects in each modality before fusion. An alternative approach is to keep both colour and thermal information as separate channels in a new 'Red-Green-Blue-Thermal' video format BIB005 . The spatial resolution of thermal cameras is still low, and the price for the best resolution available is high. Using cheap thermal sensors with a low spatial resolution can still improve surveillance when combined with the use of a visual camera. Jones et al. [86] used a thermal sensor with 16 × 16 pixels to give an indication of where to search in the visual image. They concentrated on calibrating the two cameras and found the correspondence between the images. By fusing the visual and thermal videos, super-resolution of the thermal images can be constructed [85] . A prototype using three cameras to combine both the visual, NIR, and LWIR bands is proposed in BIB009 . Sensors other than visual cameras can also be combined with thermal cameras in fusion systems. Fardi et al. [43] fused the data from a thermal camera and a laser scanner to obtain robust pedestrian detection for tracking. Schweiger et al. BIB011 fused near-infrared sensors and low resolution, far-infrared sensors to obtain a low-cost night vision system. Fused image systems find application in a variety of areas, which we now summarise. In surveillance systems, it is popular to use fused image modalities, due to the necessity of a robust system that can work both night and day, indoors and outdoors [101, 132] . In both surveillance and night vision systems for cars, the detection of pedes-trians can be improved by fusing the image modalities BIB002 131, 108 Face detection and face recognition have been thoroughly investigated and standard methods exist using visual cameras. However, illumination changes still have a negative impact on the performance of these systems BIB006 . Research has been conducted on whether these problems could be overcome by extending the systems with a thermal sensor. Socolinsky and Selinger BIB003 showed a significant improvement by fusing the visible and thermal images in a timelapse experiment. Work has been done on face recognition using both pixel-level fusion [127, 97, BIB014 , feature-level fusion BIB001 , and decision-level fusion [145, 129, BIB004 . Heo et al. [67] proposed fusion at both pixel and decision levels, while Bebis et al.  tested two different fusion schemes. They all reported an improved performance when fusing the different image modalities.