Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Abstract This paper presents a vision-based approach for tracking people on a mobile robot using thermal images. The approach combines a particle filter with two alternative measurement models that are suitable for real-time tracking. With this approach a person can be detected independently from current light conditions and in situations where no skin colour is visible. In addition, the paper presents a comprehensive, quantitative evaluation of the different methods on a mobile robot in an office environment, for both single and multiple persons. The results show that the measurement model that was learned from local grey-scale features could improve on the performance of the elliptic contour model, and that both models could be combined to further improve performance with minimal extra computational cost. <s> BIB001 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This paper presents a method for pedestrian detection and tracking using a night vision video camera installed on the vehicle. To deal with the nonrigid nature of human appearance on the road, a two-step detection/tracking method is proposed. The detection phase is performed by a support vector machine (SVM) with size-normalized pedestrian candidates and the tracking phase is a combination of Kalman filter prediction and mean shift tracking. The detection phase is further strengthened by information obtained by a road detection module that provides key information for pedestrian validation. Experimental comparisons have been carried out on gray-scale SVM recognition vs. binary SVM recognition and entire body detection vs. upper body detection. <s> BIB002 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This paper describes an approach for pedestrian detection in infrared images. The developed system has been implemented on an experimental vehicle equipped with an infrared camera and preliminarily tested in different situations. It is based on the localization of warm symmetrical objects with specific size and aspect ratio; since also road infrastructures and other road participants may have such characteristics, a set of matched filters was added in order to reduce false detections. A final validation process, based on the human shape's morphological characteristics, is used to build the list of pedestrian appearing in the scene. No temporal correlation, nor motion cues are used in this first part of the project. <s> BIB003 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Falls are a major health hazard for the elderly and a major obstacle to independent living. The estimated incidence of falls for both institutionalized and independent persons aged over 75 is at least 30 percent per year. In the SIMBAD (Smart Inactivity Monitor using Array-Based Detectors) project, we've developed an intelligent fall detector based on a low-cost array of infrared detectors. A field trial and user research indicate that SIMBAD could significantly enhance the functionality and effectiveness of existing monitoring systems and community alarm systems. <s> BIB004 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This paper describes a system for pedestrian detection in infrared images, which has been implemented on an experimental vehicle equipped with an infrared camera. The proposed system has been tested in many situations and has proven to be efficient and with a very low false-positive rate. It is based on a multiresolution localization of warm symmetrical objects with specific size and aspect ratio; anyway, because road infrastructures and other road participants may also have such characteristics, a set of matched filters is included in order to reduce false detections. A final validation process, based on human shape's morphological characteristics, is used to build the list of pedestrian appearing in the scene. Neither temporal correlation nor motion cues are used in this first part of the project: the processing is based on the analysis of single frames only. <s> BIB005 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This paper describes a modular tracking system designed to improve the performance of a pedestrian detector. The tracking system consists of two modules, a labeler and a predictor. The former associates a tracking identifier to each pedestrian, keeping memory of the past history; this is achieved by merging the detector and predictor outputs combined with data about vehicle motion. The predictor, basically a Kalman filter, estimates the new pedestrian position by observing his previous movements. Its output helps the labeler to improve the match between the pedestrians detected in the new frame and those observed in the previous shots (feedback). If a pedestrian is occluded by some obstacle for a short while, the system continues tracking its movement using motion parameters. Moreover, it is able to reassign the same tracking ID in case the occlusion disappears in a short time. This behavior helps to correct temporary mis-recognitions that occur when the detector fails. The system has been tested using a quantitative performance evaluation tool, giving promising results. <s> BIB006 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This paper describes an approach for pedestrian detection in stereo infrared images. The developed system has been implemented on an experimental vehicle equipped with two infrared cameras and preliminarily tested in different situations. It is based on the localization and distance estimation of warm areas in the scene; the algorithm groups areas with similar position and considers only results with specific size and aspect ratio. A final validation process, based on the head shape's morphological and thermal characteristics, is used to build the list of potential pedestrian appearing in the scene. Neither temporal correlation, nor motion cues are used in this processing. <s> BIB007 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Thispaperpresents acomplete methodforpedes- trian detection applied toinfrared images. First, westudy an imagedescriptor basedonhistograms oforiented gradients (HOG),associated witha Support Vector Machine(SVM) classifier andevaluate itsefficiency. Afterhaving tunedthe HOG descriptor andtheclassifier, weinclude this methodin acomplete system, whichdeals withstereo infrared images. Thisapproach gives goodresults forwindowclassification, andapreliminary testapplied onavideosequence proves thatthis approach isverypromising. I.INTRODUCTION Since thelast fewyears now,thedevelopment ofdriving assistance systems hasbeenveryactive inorder toin- crease thevehicle anditsenvironment safety. Atthe present time, themainobjective inthisdomainisto provide thedrivers withsomeinformation concerning its environment andanypotential hazard. Oneamongall useful information isthedetection andlocalization ofa pedestrian infront ofavehicle. Thisproblem ofdetecting pedestrians isaverydifficult problemthathasessentially beenaddressed using vision sensors, imageprocessing andpattern recognition techniques. Inparticular, detecting pedestrians thanks to images isacomplex challenge duetotheir appearance andposevariability. Inthecontext ofdaylight vision, several approaches havebeenproposed andarebased on different imageprocessing techniques ormachine learning (9), (5), (12). Recently, owingtothedevelopment oflow-cost infrared cameras, night vision systems havegained moreand moreinterest, thusincreasing theneedofautomatic detection ofpedestrians atnight. Thisproblem of detecting pedestrians frominfrared images hasbeen investigated byvarious research teamsinthelast years. Themainmethodology isbasedon extracting cues (symmetry, shape-independent <s> BIB008 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> In this paper we present a robust method for person detection using far infrared images. To extract initial nominated head regions, thresholding and morphological operations are applied using intensity information. Among these regions, some of wrongly extracted regions are removed using the pattern of person head based on the local maximums of Sobel edge image. After the head regions are segmented, the person body and legs region are roughly estimated by the ratios. The histograms of Sobel edge of such estimated regions are used to confirm the segmented head. This method can be applicable to person detection at both near and far distances in indoor and outdoor scenes. Moreover, we propose another novel algorithm using the movement pattern of gravity centers. It is a very simple way, especially valid for images at near distances. Our experiments demonstrate the effectiveness of the proposed method and the advantages in dealing with person detection for night vision applications. Finally, image fusion of visible and far infrared cameras is discussed. <s> BIB009 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Use of IR images is advantageous for many surveillance applications where the systems must operate around the clock and external illumination is not always available. We investigate the methods derived from visible spectrum analysis for the task of human detection. Two feature classes (edgelets and HOG features) and two classification models(AdaBoost and SVM cascade) are extended to IR images. We find out that it is possible to get detection performance in IR images that is comparable to state-of-the-art results for visible spectrum images. It is also shown that the two domains share many features, likely originating from the silhouettes, in spite of the starkly different appearances of the two modalities. <s> BIB010 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This article presents a validator stage for a pedestrian detection system based on the use of probabilistic models for the infrared domain. Four different models are employed in order to recognize the pose of the pedestrians; open, almost open, almost closed and fully closed legs are detected. In an attempt to overcome the drawbacks of template-matching in far infrared images, two different approaches are proposed. The algorithm has been tested on an experimental vehicle in different situations and a receiver operating characteristic has been computed. <s> BIB011 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> We present in this article a human detection and tracking algorithm using infrared vision in order to have reliable information on a room occupation. We intend to use this information to limit energetic consumption (light, heating). We perform first, a foreground segmentation with a Gaussian background model. A tracking step based on connected components intersections allows to collect information on 2D displacements of moving objects in the image plane. A classification based on a cascade of boosted classifiers is used for the recognition. Experimental results show the efficiency of the proposed algorithm. <s> BIB012 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Recent advances in the field of Unmanned Aerial Vehicles (UAVs) make flying robots suitable platforms for carrying sensors and computer systems capable of performing advanced tasks. This paper presents a technique which allows detecting humans at a high frame rate on standard hardware onboard an autonomous UAV in a real-world outdoor environment using thermal and color imagery. Detected human positions are geolocated and a map of points of interest is built. Such a saliency map can, for example, be used to plan medical supply delivery during a disaster relief effort. The technique has been implemented and tested on-board the UAVTech1 autonomous unmanned helicopter platform as a part of a complete autonomous mission. The results of flight- tests are presented and performance and limitations of the technique are discussed. <s> BIB013 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Thermography, or thermal visualization is a type of infrared visualization. Thermographic cameras are used in many heavy factories like metal recycling factories, wafer production factories and etc for monitoring the temperature conditions of the machines. Besides, thermographic camera can be used to detect trespassers in environment with poor lighting condition, whereby, the conventional digital cameras are less applicable in. In this paper, we proposed two simple and fast detection algorithms into a cost effective thermal imaging surveillance system. This surveillance system not only used in monitoring the functioning of different machinery and electrical equipments in a factory site, it can also used for detecting the trespassers in poor lighting condition. Experimental results show that the proposed surveillance system achieves high accuracy in monitoring machines conditions and detecting trespassers. <s> BIB014 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> An FPGA-based implementation of a pedestrian detection system using thermal infrared imaging is presented. The main feature of the processing platform is its ability to detect pedestrians at frame-rate from an infrared video stream. It is designed as a stand alone processing unit, supporting security systems in low visibility conditions. The processing unit digitizes a video source from a thermal infrared camera and after preprocessing, provides features of the detected pedestrians such as their position, height and width to a host computer where visualization of tracking and statistical operations can be applied to the received data. The data is transmitted to the host PC via a Bluetooth module interfaced to the development board. The wireless communication allows also for remote control of the processing system's configuration registers. At the present stage of evaluation, the processing unit is limited to detect and transmit the coordinates of one pedestrian in a video frame. The system was implemented using a Xilinx Virtex-II Pro XC2VP30 FPGA chip, no additional memory is required. <s> BIB015 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Perceiving the environment is crucial in any application related to mobile robotics research. In this paper, a new approach to real-time human detection through processing video captured by a thermal infrared camera mounted on the indoor autonomous mobile platform mSecurit TM is introduced. The approach starts with a phase of static analysis for the detection of human candidates through some classical image processing techniques such as image normalization and thresholding. Then, the proposal uses Lukas and Kanade optical flow without pyramids algorithm for filtering moving foreground objects from moving scene background. The results of both phases are compared to enhance the human segmentation by infrared camera. Indeed, optical flow will emphasize the foreground moving areas gotten at the initial human candidates detection. <s> BIB016 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> We present a new method for detecting pedestrians in thermal images. The method is based on the Shape Context Descriptor (SCD) with the Adaboost cascade classifier framework. Compared with standard optical images, thermal imaging cameras offer a clear advantage for night-time video surveillance. It is robust on the light changes in day-time. Experiments show that shape context features with boosting classification provide a significant improvement on human detection in thermal images. In this work, we have also compared our proposed method with rectangle features on the public dataset of thermal imagery [1]. Results show that shape context features are much better than the conventional rectangular features on this task. <s> BIB017 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This paper presents a novel human detection algorithm embedded into a thermal imaging system to form a fast and effective trespasser detection system for smart surveillance purpose. To identify a human object, pattern recognization technique is employed where human head is detected and analyzed for its shape, dimension and position. The experimental result shows that the proposed trespasser detection algorithm is fast and accurate with accuracy of 98.62% in its optimum setting with short routine time of 112 milliseconds. <s> BIB018 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Fainting is the unwanted event occur amongst the senior citizens, patients or pregnant women. This event can cause physical injuries or even mental problems. In this paper, we proposed a simple faint detection algorithm apply into a thermal imaging camera to monitor the fainting event. In addition, this system is to allow the immediate treatment can be carried out for life saving purposes when there are people fainting. Experimental results on sample images show that the proposed surveillance system achieved high accuracy of 96.15% for poor lighting condition and 86.19% in indoor in detecting the human faint event. <s> BIB019 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> Rescue robotics is gaining increasing attention as an application of human assistance machines. Robots (or unmanned ground vehicles) for searching victims in disaster sites require a high level of reliability and robustness. This paper proposes a mobile vision system able to search for victims using integration of video and thermal camera inputs. A time efficient scanning strategy is designed for the pan-tilt camera head with a biologically inspired robust search mechanism to improve the performance of the overall system. Results show success of the proposed methodology in enhancing robustness in autonomous survivor detection. <s> BIB020 </s> Thermal cameras and applications: a survey <s> Detection and tracking of humans <s> This article presents a pedestrian tracking methodology using an infrared sensor for surveillance applications. A distinctive feature of this study compared to the existing pedestrian tracking approaches is that the road network information is utilized for performance enhancement. A multiple model particle filter, which uses two different motion models, is designed for enabling the tracking of both road-constrained (on-road) and unconstrained (off-road) targets. The lateral position of the pedestrians on the walkways are taken into account by a specific on-road target model. The overall framework seamlessly integrates the negative information of occlusion events into the algorithm for which the required modifications are discussed. The resulting algorithm is illustrated on real data from a field trial for different scenarios. <s> BIB021
Detection of humans is the first step in many surveillance applications. General purpose systems should be robust and independent of the environment. The thermal cameras are here often a better choice than a normal visual camera. Zin et al. BIB009 proposed a system for human detection, based on the extraction of the head region, and Davis and Sharma [37] proposed a detection system that uses background subtraction, gradient information, watershed algorithm and A* search to robustly extract the silhouettes. Similar approaches are presented in [36, 114] , using Contour Saliency Maps and adaptive filters, while Wang et al. BIB017 presented a detection method based on the Shape Context Descriptor and Adaboost cascade classifier. A common detection problem is that the surroundings during summer are hotter than or equal to the human temperature. Jo et al. [82] tried to overcome this problem using Mahalanobis distance between pixel values and edge orientation histograms. Gade et al. [53, 54] used automatic thresholding and a sorting and splitting of blobs to detect and count people in sports arenas (see Fig. 10 ). Thermal cameras are very useful for the surveillance and detection of intruders, because of their ability to 'see' during the night. For trespasser detection, classification is often based on temperature and simple shape cues. Wong et al. BIB014 proposed two trespasser detection systems, one in which they adjust the camera to detect objects in the temperature range of humans, and then classify the objects based on the shape. The other work aims to identify humans using pattern recognition to detect the human head BIB018 . Fernández-Caballero et al. [46] used thresholding, and then a validation of each blob, to determine if it contains one or more persons. If it contains more than one, it will be split into two blobs. Benezeth et al. BIB012 proposed a real-time detection and tracking system with a classification step based on a cascade of boosted classifiers. Thermal sensors can be used in systems for the detection of fall accidents or unusual inactivity, which is an important safety tool for the independent living of especially elderly people. Sixsmith and Johnson BIB004 proposed a system that uses a low resolution thermal sensor. The system gives an alarm in case of a fall detected, or in the case of inactivity over a long time period. Wong et al. BIB019 also proposed a fall detection system for private homes by analysing the shape of the detected object. In [94], a fall detection system for bathrooms was proposed, using a thermal sensor mounted above the toilet. Analysis of more general human activity has also been performed. Han and Bhanu [63] presented a system that distinguishes between walk and run using spatio-temporal information, while Bhanu and Han  estimated the gait parameters by fitting a 3D kinematic model to the 2D silhouette extracted from the thermal images. In [55] different sports types are classified by the detected location of people over time. Pham et al. [143] proposed a system for analysing the posture of people in crowds, to detect people lying down. This could be useful to detect gas attacks or other threats at public places. Iwasawa et al. [76] proposed a system for estimating the human body posture by finding the orientation of the upper body, and locating the major joints of the body. Rescue robots are used during natural disasters or terrorist attacks, and are often equipped with a thermal camera to be able to look for victims in the dark. Mano et al. [118] presented a rescue robot equipped with several sensors, including thermal camera, visual camera and laser range scanner. This robot was able to detect victims in a scene and drive autonomously to the destination. Aziz and Mertsching BIB020 proposed a robot rescue system using thermal and visual camera to identify victims in a scene. For use on unmanned aerial vehicles, Rudol and Doherty BIB013 proposed a human detection system that uses a thermal camera to detect warm objects. The shape of the object is analysed to reject false detections, before the corresponding region in the colour image is processed with a cascade of boosted classifiers. Thermal cameras are very popular in the research of pedestrian detection, due to the cameras' independence of lighting changes, which means that it will also work during the night, when most accidents between cars and pedestrians happen. One of the car-based detection systems is proposed in BIB006 , where they present a tracking system for pedestrians. It works well with both still and moving vehicles, but some problems still remain when a pedestrian enters the scene running. Fang et al. [42] proposed a shape-independent pedestrian detection method. Using a thermal sensor with low spatial resolution, Mahlisch et al. [116] built a robust pedestrian detector by combining three different methods. Kallhammer et al. [89] also proposed a low resolution system for pedestrian detection from vehicles. Olmeda et al. [136] proposed a pedestrian detection system that detects people based on their temperature and dimensions and tracks them using a Kalman filter. In [135] , the authors proposed a detection system based on histogram of oriented phase congruency and an SVM classifier for classification of pedestrians. Bertozzi et al. BIB003 proposed a pedestrian detection system with detection based on symmetric edges, histogram analysis and size of the object. The subsequent work BIB005 adds a validation step, where the detected objects are matched with a pedestrian model. Xu et al. BIB002 proposed a system that uses SVM for detection and a combination of Kalman filter prediction and mean shift for tracking. Wide purpose pedestrian detection includes shape-and appearance-based approaches and local feature-based approaches. Dai et al. [33] used a shape-based detection and an appearance-based localisation of humans. In [34], the foreground was separated from the background, after that shape cues were used to eliminate non-pedestrian objects, and appearance cues helped locate the exact position of pedestrians. A tracking algorithm is also implemented. Zhang et al. BIB010 used combinations of local features and classifiers. HOG features and Edgelets are used for detection, and Adaboost and SVM cascade are used as classifiers. Suard et al. BIB008 and Li et al. [112] also used HOG detectors and SVM classifier for pedestrian detection. Walczyk et al. BIB015 implemented an embedded pedestrian detection system on FPGA. In BIB007  , a car-based stereo-vision system has been tested, detecting warm areas and classify if they are humans, based on distance estimation, size, aspect ratio, and head shape localization. Bertozzi et al. BIB011 and Nanda and Davis [128] used probabilistic template models for pedestrian classification, while Meis et al. [120] used a statistical approach for head detection. For tracking pedestrians, Padole and Alexandre [138,137] used both spatial and temporal data association, the Wigner distribution, and a motion-based particle filter. Skoglar et al. BIB021 used a multiple-model particle filter, and prior information about the walkways to enhance the performance. Treptow et al. BIB001 also used a particle filter, combined with two shape-and feature-based measurement models, to track humans in real time from a mobile robot. Other robot-based systems for detection and tracking are proposed in [45, BIB016 . For the static case, when the robot is still, image differencing and thresholding are applied for human detection. When it moves, the system uses optical flow for filtering the moving foreground objects from moving scene background. Lee et al. [104] proposed a human tracking algorithm for mobile robots that combines a curve matching framework with Kalman filter tracking. Jüngling and Arens [87,88] proposed a local feature (SURF) based method for detection of body parts and tracking of humans. The tracking part uses Kalman-based prediction of object positions to overcome the lack of colour features for distinguishing people. For scenes captured with a moving camera, the Kalman prediction is replaced by a calculation of shift vectors between frames.