A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> More and more vehicles are being equipped with Automatic Emergency Braking (AEB) systems. These systems intend to help the driver avoid or mitigate accidents by automatically applying the brakes prior to an accident. Initially only rear-end collision were addressed but over time more accident types are incorporated and brakes are applied earlier and stronger, in order to increase the velocity reduction before the accident occurs. This paper describes one of the latest AEB systems called Collision Warning with Full Auto Brake and Pedestrian Detection (CWAB-PD). It helps the driver with avoiding both rear-end and pedestrian accidents by providing a warning and, if necessary, automatic braking using full braking power. A limited set of accident scenarios is selected to illustrate the theoretical and practical performance of this system. It is shown that the CWAB-PD system can avoid accidents up to 35 km/h and can mitigate accidents achieving an impact speed reduction of 35 km/h. To the best of the authors knowledge CWAB-PD is the only system on the market that automatically can avoid accidents with pedestrians. <s> BIB001 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Using multiple sensors in the context of environment perception for autonomous vehicles is quite common these days. Perceived data from these sensors can be fused at different levels like: before object detection, after object detection and finally after tracking the moving objects. In this paper we detail our object detection level fusion between laser and stereo vision sensors as opposed to pre-detection or track level fusion. We use the output of our laser processing to get a list of objects with position and dynamic properties for each object. Similarly we use the stereo vision output of another team which consists of a list of detected objects with position and classification properties for each object. We use Bayesian fusion technique on objects of these two lists to get a new list of fused objects. This fused list of objects is further used in tracking phase to track moving objects in an intersection like scenario. The results obtained on data sets of INTERSAFE-2 demonstrator vehicle show that this fusion has improved data association and track management steps. <s> BIB002 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Pedestrian detection systems constitute an important field of research and development in computer vision, specially when applied in protection/safety systems in urban scenarios due to their direct impact in the society, specifically in terms of traffic casualties. In order to face such challenge, this work exploits some developments on statistical machine learning theory, in particular structural risk minimization (SRM) in a cascade ensemble. Namely, the ensemble applies the principle of SRM on a set of linear support vector machines (SVM). The linear SVM complexity, in the Vapnik sense, is controlled by choosing the dimension of the feature space in each cascade stage. To support experimental analysis, a multi-sensor dataset constituted by data from a LIDAR, a monocular camera, an IMU, encoder and a DGPS is introduced in this paper. The dataset, named Laser and Image Pedestrian Detection (LIPD) dataset, was collected in an urban environment, at day light conditions, using an electrical vehicle driven at low speed. Labeled pedestrians and non-pedestrians samples are also available for benchmarking purpose. The cascade of SVMs, trained with image-based features (HOG and COV descriptors), is used to detect pedestrian evidences on regions of interest (ROI) generated by a LIDAR-based processing system. Finally, the paper presents experimental results comparing the performance of a Boosting-SVM cascade and the proposed SRM-SVM cascade classifiers, in terms of detection errors. <s> BIB003 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> In this work, a context-based multisensor system, applied to pedestrian detection in urban environments, is presented. The proposed system comprises three main processing modules: (i) a LIDAR-based module acting as the primary object detector, (ii) a module which supplies the system with contextual information obtained from a semantic map of the roads, and (iii) an image-based detection module, using sliding window detectors, with the role of validating the presence of pedestrians in the regions of interest generated by the LIDAR module. A Bayesian strategy is used to combine information from sensors onboard the vehicle ('local' information) with information contained in a digital map of the roads ('global' information). To support experimental analysis, a multisensor dataset, named the Laser and Image Pedestrian Detection dataset (LIPD), is used. The LIPD dataset was collected in an urban environment, under daylight conditions, using an electrical vehicle driven at low speed. A down-sampling method, using support vectors extracted from multiple linear SVMs, was used to reduce the cardinality of the training set and, as a consequence, to decrease the CPU time during the training process of the image-based classifiers. The performance of the system is evaluated, in terms of detection rate and the number of false positives per frame, using three image-detectors: a linear SVM, a SVM-cascade, and a benchmark method. Additionally, experiments are performed to assess the impact of contextual information on the performance of the detection system. <s> BIB004 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> This paper presents a new approach to fuse 3D and 2D information in a driver assistance setup, in particular to perform obstacle detection and tracking. We propose a new cooperative fusion method between two exteroceptive sensors: it is able to address highly non linear dynamic configuration without any assumption on the driving maneuver. Information are provided by a mono-layer laser scanner and a monocular camera which are unsynchronized. The initial detection stage is performed using the 1D laser data, which computes clusters of points which might correspond to vehicles present on the road. These clusters are projected to the image to define targets, which will be tracked using image registration techniques. This multi-object association and tracking scheme is implemented using belief theory integrating temporal and spatial information, which allows the estimation of the dynamic state of the tracks and to monitor appearance and disappearance of obstacles. Accuracy of the method is evaluated on a database made publicly available, focus is cast on the relative localization of the vehicle ahead: estimations of its longitudinal and lateral distances are analysed. <s> BIB005 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> One geometry segmentation algorithm is proposed to parse scanner pointclouds.One efficient multilayer perception classifier is trained to parse camera images.We propose one fuzzy logic based fusion method to integrate results of two sensors.We propose one Markov random field based temporal fusion method.The fused results are more reliable than those of individual sensors. Fusion of information gathered from multiple sources is essential to build a comprehensive situation picture for autonomous ground vehicles. In this paper, an approach which performs scene parsing and data fusion for a 3D-LIDAR scanner (Velodyne HDL-64E) and a video camera is described. First of all, a geometry segmentation algorithm is proposed for detection of obstacles and ground areas from data collected by the Velodyne scanner. Then, corresponding image collected by the video camera is classified patch by patch into more detailed categories. After that, parsing result of each frame is obtained by fusing result of Velodyne data and that of image using the fuzzy logic inference framework. Finally, parsing results of consecutive frames are smoothed by the Markov random field based temporal fusion method. The proposed approach has been evaluated with datasets collected by our autonomous ground vehicle testbed in both rural and urban areas. The fused results are more reliable than that acquired via analysis of only images or Velodyne data. <s> BIB006 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Autonomous lane change maneuver was developed using cooperative strategy.Proposed system can be potential to prevent lane change crashes and thus reducing injuries and fatalities.A trajectory planning method based on polynomial was developed.A trajectory tracking controller with global convergence ability was designed.Simulations and experimental results were presented to validate the method. Lane change maneuver is one of the most conventional behaviors in driving. Unsafe lane change maneuvers are key factor for traffic accidents and traffic congestion. For drivers' safety, comfort and convenience, advanced driver assistance systems (ADAS) are presented. The main problem discussed in this paper is the development of an autonomous lane change system. The system can be extended applied in intelligent vehicles in future. It solves two crucial issues - trajectory planning and trajectory tracking. Polynomial method was adopted for describing the trajectory planning issue. Movement of a host vehicle was abstracted into time functions. Moreover, collision detection was mapped into a parameter space by adopting infinite dynamic circles. The second issue was described by backstepping principle. According to the Lyapunov function, a tracking controller with global convergence property was verified. Both the simulations and the experimental results demonstrate the feasibility and effectiveness of the designed method for autonomous lane change. <s> BIB007 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> The localization of a vehicle is a central task of autonomous driving. Most of the time, it is solved by considering a single algorithm with a few sensors. In this paper, we propose a cooperative fusion architecture based on two main algorithms: a laser-based Simultaneous Localization And Mapping (SLAM) process and a lane detection and tracking approach using a single camera. Both algorithms are designed individually as cooperative fusion processes where other sensors (GPS and proprioceptive information) and dedicated maps are integrated to strengthen the advantages of each system. The whole architecture is formalized around key components (ego-vehicle, roadway, obstacle and environment). A final decision layer, that takes into account the state of each algorithm, allows the system to choose the most appropriate ego-vehicle localization mean based on the current road situation and the environmental context. <s> BIB008 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> In this paper, we propose a multi-object tracking and reconstruction approach through measurement-level fusion of LiDAR and camera. The proposed method, regardless of object class, estimates 3D motion and structure for all rigid obstacles. Using an intermediate surface representation, measurements from both sensors are processed within a joint framework. We combine optical flow, surface reconstruction, and point-to-surface terms in a tightly-coupled non-linear energy function, which is minimized using Iterative Reweighted Least Squares (IRLS). We demonstrate the performance of our model on different datasets (KITTI with Velodyne HDL-64E and our collected data with 4-layer ScaLa Ibeo), and show an improvement in velocity error and crispness over state-of-the-art trackers. <s> BIB009 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Two Vulnerable Road User Detection Approaches presented.Comparison tests of two state of the art technologies for ADAS application: V2V and sensor based data fusion.V2V technology able to be implemented in any vehicle, with mobile phone interface.Data fusion based on computer vision and laser scanner detection algorithm.Real tests for both subsystem proves the real time performance and the accuracy of the detections. Motorcycles are one of the most dangerous means of transportation. Its death toll is higher than in others, due to the inherent vulnerability of motorcycle drivers. The latest strategies in Advanced Driving Assistance Systems (ADAS) are trying to mitigate this problem by applying the advances of modern technologies to the road transport. This paper presents two different approaches on motorcycle protection, based on two of the most modern available technologies in ADAS, i.e. Computer Vision and Vehicle to Vehicle Communication (V2V). The first approach is based on data fusion of Laser Scanner and Computer Vision, providing accurate obstacle detection and localization based on laser scanner, and obstacle classification using computer vision and laser. The second approach is based on ad-hoc V2V technology and provides detection in case of occlusion for visual sensors. Both technologies have been tested in the presented work, and a performance comparison is given. Tests performed in different driving situations allows to measure the performance of every algorithm and the limitations of each of them based on empirical and scientific foundations. The conclusions of the presented work help foster of expert systems in the automotive sector by providing further discussion of the viability and impact from each of these systems in real scenarios. <s> BIB010 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Abstract Perception system design is a vital step in the development of an autonomous vehicle (AV). With the vast selection of available off-the-shelf schemes and seemingly endless options of sensor systems implemented in research and commercial vehicles, it can be difficult to identify the optimal system for one’s AV application. This article presents a comprehensive review of the state-of-the-art AV perception technology available today. It provides up-to-date information about the advantages, disadvantages, limits, and ideal applications of specific AV sensors; the most prevalent sensors in current research and commercial AVs; autonomous features currently on the market; and localization and mapping methods currently implemented in AV research. This information is useful for newcomers to the AV field to gain a greater understanding of the current AV solution landscape and to guide experienced researchers towards research areas requiring further development. Furthermore, this paper highlights future research areas and draws conclusions about the most effective methods for AV perception and its effect on localization and mapping. Topics discussed in the Perception and Automotive Sensors section focus on the sensors themselves, whereas topics discussed in the Localization and Mapping section focus on how the vehicle perceives where it is on the road, providing context for the use of the automotive sensors. By improving on current state-of-the-art perception systems, AVs will become more robust, reliable, safe, and accessible, ultimately providing greater efficiency, mobility, and safety benefits to the public. <s> BIB011 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Abstract Most of the current successful object detection approaches are based on a class of deep learning models called Convolutional Neural Networks (ConvNets). While most existing object detection researches are focused on using ConvNets with color image data, emerging fields of application such as Autonomous Vehicles (AVs) which integrates a diverse set of sensors, require the processing for multisensor and multimodal information to provide a more comprehensive understanding of real-world environment. This paper proposes a multimodal vehicle detection system integrating data from a 3D-LIDAR and a color camera. Data from LIDAR and camera, in the form of three modalities, are the inputs of ConvNet-based detectors which are later combined to improve vehicle detection. The modalities are: (i) up-sampled representation of the sparse LIDAR’s range data called dense-Depth Map (DM), (ii) high-resolution map from LIDAR’s reflectance data hereinafter called Reflectance Map (RM), and (iii) RGB image from a monocular color camera calibrated wrt the LIDAR. Bounding Box (BB) detections in each one of these modalities are jointly learned and fused by an Artificial Neural Network (ANN) late-fusion strategy to improve the detection performance of each modality. The contribution of this paper is two-fold: (1) probing and evaluating 3D-LIDAR modalities for vehicle detection (specifically the depth and reflectance map modalities), and (2) joint learning and fusion of the independent ConvNet-based vehicle detectors (in each modality) using an ANN to obtain more accurate vehicle detection. The obtained results demonstrate that (1) DM and RM are very promising modalities for vehicle detection, and (2) experiments show that the proposed fusion strategy achieves higher accuracy compared to each modality alone in all the levels of increasing difficulty (easy, moderate, hard) in KITTI object detection dataset. <s> BIB012 </s> A Review of Research on Traffic Conflicts Based on Intelligent Vehicles <s> C. RESEARCH ON TRAFFIC CONFLICT BASED ON MULTI-SENSOR FUSION <s> Abstract In this work, a deep learning approach has been developed to carry out road detection by fusing LIDAR point clouds and camera images. An unstructured and sparse point cloud is first projected onto the camera image plane and then upsampled to obtain a set of dense 2D images encoding spatial information. Several fully convolutional neural networks (FCNs) are then trained to carry out road detection, either by using data from a single sensor, or by using three fusion strategies: early, late, and the newly proposed cross fusion. Whereas in the former two fusion approaches, the integration of multimodal information is carried out at a predefined depth level, the cross fusion FCN is designed to directly learn from data where to integrate information; this is accomplished by using trainable cross connections between the LIDAR and the camera processing branches. To further highlight the benefits of using a multimodal system for road detection, a data set consisting of visually challenging scenes was extracted from driving sequences of the KITTI raw data set. It was then demonstrated that, as expected, a purely camera-based FCN severely underperforms on this data set. A multimodal system, on the other hand, is still able to provide high accuracy. Finally, the proposed cross fusion FCN was evaluated on the KITTI road benchmark where it achieved excellent performance, with a MaxF score of 96.03%, ranking it among the top-performing approaches. <s> BIB013
Multi-sensor Data Fusion (MSDF) technology has been rapidly developed since the 1970s and has been widely used in many modern military and civil fields  . The traffic environment at urban road intersections is complex and variable. Intelligent vehicles need to use a variety of sensors to detect the surrounding environment including the lane lines, pedestrians, vehicles, etc.  , and to compensate for the deficiencies of individual sensors through multi-sensor fusion. The advantages include improved perceptual accuracy, reliability and robustness BIB011 . The fusion framework of multi-sensor sensing classification is shown in Fig. 7 . At present, the Advanced Driver Assistance Systems (ADAS) equipped in the high-end cars have been able to provide a variety of security functions by detecting various entities through radar sensors and detecting lane markings through video sensors. Thus it can actively assists the human drivers and prepare for ''small-scale, short-term upcoming events''  . For example, the Volvo Collision Warning with fully automatic braking and pedestrian detection (CWAB-PD) system uses radar to increase the range and monocular camera to identify the type of object so that it can handle a wider range of forward collisions BIB001 . The examples of multi-sensor fusion have been widely recognized and applied in the research of intelligent vehicles BIB008 . You et al. BIB007 combined the vehicle sensor information to carry out research on environmental information collection and analysis, trajectory generation, collision detection and conflict processing during the lane change processes. Daraei et al. BIB009 proposed that laser scanners can help to reduce the reliability and accuracy of computer vision detection and positioning. Data fusion based on laser scanner and computer vision can improve the accuracy of obstacle detection and positioning. Premebida and Nunes BIB004 combined the data collected by the lidar with the data collected by the monocular camera. An active pedestrian detection system is designed using the sliding window detector and image-based detection module to verify whether there are pedestrians in the recognition area generated by the lidar module. Further, Zhao et al. BIB006 proposed a sensor fusion method for scene analysis using laser scanner and camera. The data collected by Velodyne laser scanner is used to detect obstacles and ground areas. The effect of pedestrian detection is significantly improved. In addition, Anaya et al. BIB010 accurately positioned and classified motorcycles, and Baig et al. BIB002 conducted a moving target tracking study at intersections. Ludwig et al. BIB003 used the support vector machine cascade classifier to implement pedestrian detection and tracking based on laser and vision. Gruyer et al. BIB005 combines stereo vision and laser scanners to detect multiple obstacles for vehicle detection and tracking. Asvadi et al. BIB012 proposed a multi-modal vehicle detection system combining 3D lidar and color camera data, using convolutional neural network fusion strategy to make vehicle detection more accurate. Caltagirone et al. BIB013 combines lidar and camera images for road segmentation and detection via selecting a full convolutional neural network (FCN) for laser radar and camera fusion. For the intelligent vehicles operating in the urban roads, the surrounding traffic environment is an important factor. Intelligent vehicle technology is inseparable from the basic environment-aware data. Thus, it is difficult for a single sensor to cope with the complex and varied driving environment. The advantages and disadvantages of visual and radar environment perception are compared in Table 1 . Therefore, multi-sensor fusion is a research hotspot at this stage. However, how to realize the integration of environmental data is the key to the research of environmental vehicle perception. In addition, multi-sensor fusion system and data fusion algorithms have great difficulties in improving the sensing accuracy. This is also the bottleneck of the development of the multi-sensor fusion technology of intelligent vehicles. At the same time, some of the sensor functions may be replaced by communication between the vehicle and the road infrastructure.