Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper presents a method for detecting people on-board a moving vehicle. The perception of the environment is performed through the fusion of an automotive radar sensor and a monocular vision system. The fusion uses a two-step approach for efficient object detection. In the first step, a target-list is generated from the radar sensor. The items in the list are hypotheses for the presence of people. In a second step, hypotheses are proved by the vision system. This method is considerably faster than a sole image processing solution. For the covering abstract see ITRD E123380. <s> BIB001 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper aims to present a review of recent as well as classic image registration methods. Image registration is the process of overlaying images (two or more) of the same scene taken at different times, from different viewpoints, and/or by different sensors. The registration geometrically align two images (the reference and sensed images). The reviewed approaches are classified according to their nature (areabased and feature-based) and according to four basic steps of image registration procedure: feature detection, feature matching, mapping function design, and image transformation and resampling. Main contributions, advantages, and drawbacks of the methods are mentioned in the paper. Problematic issues of image registration and outlook for the future research are discussed too. The major goal of the paper is to provide a comprehensive reference source for the researchers involved in image registration, regardless of particular application areas. q 2003 Elsevier B.V. All rights reserved. <s> BIB002 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This article presents a multi sensor approach for driver assistance systems: the detection and tracking of pedestrians in a road environment. A multi sensor system consisting of a far infrared camera and a laser scanning device is used for the detection and precise localization of pedestrians. Kalman filter based data fusion handles the combination of the sensor information of the infrared camera and of the laser scanner. Arranging a set of Kalman filters in parallel, a multi sensor/multi target tracking system was created. The usage of suitable movement models has a great influence on the performance of the tracking system. Several types of models are discussed focussing on the typical behavior of pedestrians in road environments. The multi sensor/multi target tracking system is installed on a test vehicle to obtain practical results which is discussed in this article too. <s> BIB003 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper describes the development activities leading up to field testing of the transit integrated collision warning system, with special attention to the side component. Two buses, one each in California and Pennsylvania, have been outfitted with sensors, cameras, computers, and driver-vehicle interfaces in order to detect threats and generate appropriate warnings. The overall project goals, integrated concept, side component features, and future plans are documented here. <s> BIB004 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper presents a novel real-time pedestrian detection system utilizing a LIDAR-based object detector and convolutional neural network (CNN)-based image classifier. Our method achieves over 10 frames/second processing speed by constraining the search space using the range information from the LIDAR. The image region candidates detected by the LIDAR are confirmed for the presence of pedestrians by a convolutional neural network classifier. Our CNN classifier achieves high accuracy at a low computational cost thanks to its ability to automatically learn a small number of highly discriminating features. The focus of this paper is the evaluation of the effect of region of interest (ROI) detection on system accuracy and processing speed. The evaluation results indicate that the use of the LIDAR-based ROI detector can reduce the number of false positives by a factor of 2 and reduce the processing time by a factor of 4. The single frame detection accuracy of the system is above 90% when there is 1 false positive per second. <s> BIB005 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This article presents a tetra-vision (4 cameras) system for the detection of pedestrians by the means of the simultaneous use of one far infra-red and one visible cameras stereo pairs. The main idea is to exploit both the advantages of far infra-red and visible cameras trying at the same time to benefit from the use of each system. Initially, the two stereo flows are independently processed, then the results are fused together. The final result of this low-level processing is a list of obstacles that have a shape and a size compatible with the presence of a potential pedestrian. In addition, the system is able to remove the background from the detected obstacles to simplify a possible further high level processing. The developed system has been installed on an experimental vehicle and preliminarily tested in different situations. <s> BIB006 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper presents an approach for the registration of multimodal imagery for pedestrian detection when the significant depth differences of objects in the scene preclude a global alignment assumption. Using maximization-of-mutual-information matching techniques and sliding correspondence windows over calibrated image pairs, we demonstrate successful registration of color and thermal data. We develop a robust method using disparity voting for determining the registration of each object in the scene and provide a statistically based measure for evaluating the match confidence. Testing shows successful registration in complex scenes with multiple people at different depths and levels of occlusion <s> BIB007 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper presents an analysis of color and infrared stereo approaches to pedestrian detection. We design a four camera experimental testbed consisting of two color and two infrared cameras that allows for synchronous capture and direct frame-by-frame comparison of pedestrian detection approaches. We incorporate this four camera system in a test vehicle and conduct comparative experiments of stereo-based approaches to obstacle detection using color and infrared imagery. A detailed analysis of these experiments shows the robustness of both color and infrared stereo imagery to generate the dense stereo maps necessary for robust object detection and motivates investigation of color and infrared features that can be used to further classify detected obstacles into pedestrian regions. The complementary nature of color and infrared features gives rise to a discussion of a feature fusion techniques, including a cross-spectral stereo solution to pedestrian detection. <s> BIB008 </s> Pedestrian Protection Systems: Issues, Survey, and Challenges <s> E. Sensor Fusion <s> This paper presents an analysis of color-, infrared-, and multimodal-stereo approaches to pedestrian detection. We design a four-camera experimental testbed consisting of two color and two infrared cameras for capturing and analyzing various configuration permutations for pedestrian detection. We incorporate this four-camera system in a test vehicle and conduct comparative experiments of stereo-based approaches to obstacle detection using unimodal color and infrared imageries. A detailed analysis of the color and infrared features used to classify detected obstacles into pedestrian regions is used to motivate the development of a multimodal solution to pedestrian detection. We propose a multimodal trifocal framework consisting of a stereo pair of color cameras coupled with an infrared camera. We use this framework to combine multimodal-image features for pedestrian detection and to demonstrate that the detection performance is significantly higher when color, disparity, and infrared features are used together. This result motivates experiments and discussion toward achieving multimodal-feature combination using a single color and a single infrared camera arranged in a cross-spectral stereo pair. We demonstrate an approach to registering multiple objects across modalities and provide an experimental analysis that highlights issues and challenges of pursuing the cross-spectral approach to multimodal and multiperspective pedestrian analysis. <s> BIB009
Fusion of multiple sensors is used in many systems to improve the performance of pedestrian detection (also see Table V ). Sensor fusion can be performed using sequential or parallel configuration. In sequential configuration, one sensor such as RADAR or LASER scanner is used in the attention focusing stage to detect potential candidates. These candidates are projected onto the imaging sensor into areas of interest where the target presence is verified. On the other hand, in parallel configuration, inputs from multiple sensors are processed independently and then fused using a decision mechanism. The sequential and parallel approaches can also be combined in multisensor systems. In BIB001 , a sequential fusion of RADAR and monocular vision sensors is performed. The first step generates a list of potential targets using RADAR. The second step uses the images from the vision sensor to verify the targets using flexible shape models that are trained from manually extracted pedestrians. In BIB005 , a sequential combination of LIDAR-based object detector and an image-based classifier using CNN is used. The former gives a coarse resolution range image over its FOV, which is used to create a 3-D map with candidate object positions. This map is projected into the image in order to identify the locations and scale to search for pedestrians in the image. In BIB003 , a combination of far IR sensors and LASER scanner is used in parallel in order to obtain robust detection and accurate localization of pedestrians. Every object corresponds to a pair of steps in the plot of range against the azimuth angles. These are detected using the first and second derivatives. In far IR images, pedestrians are characterized by high brightness and vertical orientation. Detection is performed using thresholding, grouping of valid pixels, and orientation checking. The outputs from LASER scanner and far IR sensors are integrated in the tracking module using Kalman filter. In BIB006 , the "tetravision" system that combines stereo pairs of visible light and IR cameras is described. Disparity analysis is performed separately for each stereo pair, and the results are fused together. This processing removes the background and gives a list of candidates that are compatible with the size and shape of pedestrians. In BIB004 , a side collision warning system for transit buses is described, illustrating the use of multiple sensors in order to cover the surroundings of the vehicle. The sensors used include a pair of front-mounted cameras, RADAR, a LASER scanner, and a curb detector, as well as LASER scanners and blind spot cameras on sides. Multiple levels of warnings are generated for both front and side components. The system handles threats that span multiple views, such as passing and cutting in of a vehicle. The system architecture is designed to be flexible, which is composed of modules such as object tracker, warning generator, and data logger, which can be developed individually and plugged in to the overall system. Detection and tracking is performed from the LASER scanner. The curb detector consists of a LASER line generator and a camera separated by a known distance. The distance to the sidewalk edges can be computed by triangulation. An important issue in multisensor fusion is the registration of images from each sensor. A survey of registration methods is presented in BIB002 . Recently, in BIB007 , a novel approach is proposed for registration of thermal IR with visible light imagery for pedestrian detection using maximization of mutual entropy (Fig. 5) . Unlike many other registration approaches that assume global model for alignment, this approach uses stereo disparities of individual persons, which are detected and tracked in both images in order to obtain the 3-D locations of the persons. In BIB008 and BIB009 , a detailed comparative analysis of stereo approaches using color, IR, and their combination is described. A four-camera testbed consisting of two color and two IR cameras is used to analyze various configurations including as follows: 1) visible light binocular stereo; 2) thermal IR stereo; 3) multimodal trifocal stereo using two color cameras and one thermal camera, and 4) cross-spectral stereo with one color and one thermal camera. It is concluded that the multimodal trifocal stereo approach gives benefits of multimodality and robustness. On the other hand, the cross-spectral stereo is potentially the more cost-effective solution, but obtaining robust registration between two modalities is still a challenging problem.