Thermal cameras and applications: a survey <s> Image fusion <s> This paper presents a multi-perspective (i.e., four camera views) multi-modal (i.e., thermal infrared and color) video based system for robust and real-time 3D tracking of important body parts.The multi-perspective characteristics of the system provides 3Dtrajectory of the body parts, while the multi-modal characteristics of the system provides robustness and reliability of feature detection and tracking. The application context for this research is that of intelligent vehicles and driver assistance systems. Experimental results demonstrate effectiveness of the proposed system. <s> BIB001 </s> Thermal cameras and applications: a survey <s> Image fusion <s> Thermal analysis using finite element modeling (FEM) is widely used in automotive industry nowadays. FEM can give simulation results at design stage. However, as FEM is a kind of simulations, it normally suffers from deviations with experimental results. Therefore, experimental measurement is necessary. To measure the 3D temperature distribution on a car body, we normally use a number of temperature sensors and then record the temperature values according to the positions of the sensors. It can provide the temperature values at different positions. Using this information, we can obtain the 3D temperature distribution on a car body. However, this method is time consuming and inconvenient, as we have to place a number of sensors on a car body, measure its corresponding positions as well as process the measured temperature values. In contrast, we firstly take a sequence of 2D thermal infrared images of the car body to have 2D temperature distribution and then reconstruct the 3D temperature distribution from the 2D images using computer vision method. It provides a more convenient method to acquire 3D temperature distribution compared to using a number of temperature sensors. The experimental results show the feasibility of our method. <s> BIB002 </s> Thermal cameras and applications: a survey <s> Image fusion <s> Accurate and efficient thermal-infrared (IR) camera calibration is important for advancing computer vision research within the thermal modality. This paper presents an approach for geometrically calibrating individual and multiple cameras in both the thermal and visible modalities. The proposed technique can be used to correct for lens distortion and to simultaneously reference both visible and thermal-IR cameras to a single coordinate frame. The most popular existing approach for the geometric calibration of thermal cameras uses a printed chessboard heated by a flood lamp and is comparatively inaccurate and difficult to execute. Additionally, software toolkits provided for calibration either are unsuitable for this task or require substantial manual intervention. A new geometric mask with high thermal contrast and not requiring a flood lamp is presented as an alternative calibration pattern. Calibration points on the pattern are then accurately located using a clustering-based algorithm which utilizes the maximally stable extremal region detector. This algorithm is integrated into an automatic end-to-end system for calibrating single or multiple cameras. The evaluation shows that using the proposed mask achieves a mean reprojection error up to 78% lower than that using a heated chessboard. The effectiveness of the approach is further demonstrated by using it to calibrate two multiple-camera multiple-modality setups. Source code and binaries for the developed software are provided on the project Web site. <s> BIB003 </s> Thermal cameras and applications: a survey <s> Image fusion <s> In this work, we propose a novel fast and accurate method based on keypoints and temporal information to solve the registration problem on planar scenes with moving objects for infrared-visible stereo pairs. A keypoint descriptor and a temporal buffer (reservoir) filled with matched keypoints are used in order to find the homography transformation for the registration. Inside a given frame, the problem of registration is formulated as correspondences between noisy polygon vertices. Sections of polygons are matched locally to find the corresponding vertices inside a frame. These correspondences are then accumulated temporally using a reservoir of matches for homography calculation. Results show that our method outperforms two recent state-of-the-art global registration methods by a large margin in almost all tested videos. <s> BIB004
As discussed in Sect. 1, visual cameras and thermal cameras have both advantages and disadvantages in computer vision applications. Since the limitations of the different technologies are independent and often do not occur simultaneously, it can be beneficial to combine these different types of images. Occlusion is a well-known problem across all modalities. Separating partly occluded persons or objects of the same temperature can be very difficult in thermal images, as they have the same pixel intensity. Including depth information or colour edges can help disambiguate in this situation. The most common combination of cameras is thermal and visual. This is due to the low price and well-known characteristics of visual cameras and the ensuing advantages of augmenting colour with temperature. The main challenges are how to align and fuse the different image modalities. There is not necessarily any relation between brightness level in the different spectra, thus many mutual information alignment methods are not appropriate [133] . Often, corresponding points are manually selected to calculate a planar homography, and then warp one of the images. Automatic alignment techniques that rely on the correlation between edge orientations are presented in [74, 75] , and a method that calculates the homography from automatically detected keypoints is presented in BIB004 . The standard chessboard method for geometric calibration, correction of lens distortion, and alignment of the cameras relies on colour difference, and cannot be used for thermal cameras without some changes. Cheng et al. BIB001 and Prakash et al. [146] reported that when heating the board with a flood lamp, the difference in emissivity of the colours will result in an intensity difference in the thermal image. However, a more crisp chessboard pattern can be obtained by constructing a chessboard of two different materials, with large difference in thermal emissivity and/or temperature BIB003 . This approach is also applied in [68] using a copper plate with milled checker patterns in front of a different base material, and in BIB002 with a metal wire in front of a plastic board. When these special chessboards are heated by a heat gun, hairdryer or similar, a clear chessboard pattern will be visible in the thermal image, due to the different emissivity of the materials. At the same time, it is also visible in the visual image, due to colour difference. Figure 12 shows thermal and RGB pictures from a calibration test. The chessboard consists of two cardboard sheets, where the white base sheet has been heated right before assembling the board.