Data fusion has become an active research topic in recent years. Growing computational performance has allowed the use of redundant sensors to measure a single phenomenon. While Bayesian fusion approaches are common in general applications, the computer vision field has largely relegated this approach. Most object following algorithms have gone towards pure machine learning fusion techniques that tend to lack flexibility. Consequently, a more general data fusion scheme is needed. Within this work, a hierarchical Bayesian fusion approach is proposed, which outperforms individual trackers by using redundant measurements. The adaptive framework is achieved by relying on each measurement's local statistics and a global softened majority voting. The proposed approach was validated in a simulated application and two robotic platforms.
This section describes the different sensor fusion and adaptive sensor fusion approaches, from general algorithms tailored for fusing sensor measurements to more specific algorithms used in computer vision available in the literature. This overview covers some of the latest sensor fusion mechanisms mentioned in @cite_32 , computer vision benchmarks such as @cite_38 and the performance evaluation of some vision-based trackers @cite_7 .
@cite_38 @cite_32 @cite_7
2146364557 2074602185 2588362028 2003683977 2514340250 2953373759 2141184637 2080910314 2082023026 2543366260 2016802777 2133380705 2079150870 2124802015 2775609985 2138728841 2127752689 2571773278 2077405386 2607626645
In this paper, we address the problem of optimizing the detection performance of sensor networks under communication constraints on the common access channel. Our work helps understanding tradeoffs between sensor network para-meters like number of sensors, degree of quantization at each local sensor, and SNR. Traditionally, this problem is tack-led using asymptotic assumptions on the number of sensors, an approach that leads to the abstraction of important details such as the structure of the fusion center. We adopt a non-asymptotic approach and optimize both, the sensing and the fusion sides with respect to the probability of detection error. We show that the optimal fusion rule has an interesting structure similar to themajority-voting rule. In addition, we study the convergence with respect to the number of sensors of the performance of the fusion rule. We show that convergence is SNR dependent and that, in low-SNR environments, asymptotics may require a large number of sensors. A perception system for pedestrian detection in urban scenarios using information from a LIDAR and a single camera is presented. Two sensor fusion architectures are described, a centralized and a decentralized one. In the former, the fusion process occurs at the feature level, i.e., features from LIDAR and vision spaces are combined in a single vector for posterior classification using a single classifier. In the latter, two classifiers are employed, one per sensor-feature space, which were offline selected based on information theory and fused by a trainable fusion method applied over the likelihoods provided by the component classifiers. The proposed schemes for sensor combination, and more specifically the trainable fusion method, lead to enhanced detection performance and, in addition, maintenance of false-alarms under tolerable values in comparison with single-based classifiers. Experimental results highlight the performance and effectiveness of the proposed pedestrian detection system and the related sensor data combination strategies. © 2009 Wiley Periodicals, Inc. This paper presents an approach to enable general tracking of extended objects of multiple sensors. Expert information in each input providing sensor module is mapped into simple model parameters and allows the fusion center to use a generalized version of such information. The model type and parameters are presented and a classical Kalman based fusion is extended with a method for integrated extent handling. By this approach the central fusion node can take into account both the track level information and extent estimate from each sensor. The proposed method is compared with a classic method of fusing the object center and the extent estimate separately. Simulated data is used to show that our proposed approach is general in the sense that it can be used on various setups without adaption. Detailed performance results are given based on estimation errors of the extended object space vector. The findings based on simulated data are completed by real world data from a front facing sensor setup. It is shown that the proposed method offers a benefit in position accuracy, especially when the measurement information does not contain a complete extent information in all directions. In this work, we propose a new integrated framework that addresses the problems of thermal-visible video registration, sensor fusion, and people tracking for far-range videos. The video registration is based on a RANSAC trajectory-to-trajectory matching, which estimates an affine transformation matrix that maximizes the overlapping of thermal and visible foreground pixels. Sensor fusion uses the aligned images to compute sum-rule silhouettes, and then constructs thermal-visible object models. Finally, multiple object tracking uses blobs constructed in sensor fusion to output the trajectories. Results demonstrate the advantage of our proposed framework in obtaining better results for both image registration and tracking than separate image registration and tracking methods. Remote sensing satellite sensors feature a tradeoff between the spatial, temporal, and spectral resolutions. In this paper, we propose an integrated framework for the spatio–temporal–spectral fusion of remote sensing images. There are two main advantages of the proposed integrated fusion framework: it can accomplish different kinds of fusion tasks, such as multiview spatial fusion, spatio–spectral fusion, and spatio–temporal fusion, based on a single unified model, and it can achieve the integrated fusion of multisource observations to obtain high spatio–temporal–spectral resolution images, without limitations on the number of remote sensing sensors. The proposed integrated fusion framework was comprehensively tested and verified in a variety of image fusion experiments. In the experiments, a number of different remote sensing satellites were utilized, including IKONOS, the Enhanced Thematic Mapper Plus (ETM+), the Moderate Resolution Imaging Spectroradiometer (MODIS), the Hyperspectral Digital Imagery Collection Experiment (HYDICE), and Systeme Pour l' Observation de la Terre-5 (SPOT-5). The experimental results confirm the effectiveness of the proposed method. This paper presents a compact and accurate representation of 3D scenes that are observed by a LiDAR sensor and a monocular camera. The proposed method is based on the well-established Stixel model originally developed for stereo vision applications. We extend this Stixel concept to incorporate data from multiple sensor modalities. The resulting mid-level fusion scheme takes full advantage of the geometric accuracy of LiDAR measurements as well as the high resolution and semantic detail of RGB images. The obtained environment model provides a geometrically and semantically consistent representation of the 3D scene at a significantly reduced amount of data while minimizing information loss at the same time. Since the different sensor modalities are considered as input to a joint optimization problem, the solution is obtained with only minor computational overhead. We demonstrate the effectiveness of the proposed multimodal Stixel algorithm on a manually annotated ground truth dataset. Our results indicate that the proposed mid-level fusion of LiDAR and camera data improves both the geometric and semantic accuracy of the Stixel model significantly while reducing the computational overhead as well as the amount of generated data in comparison to using a single modality on its own. A scalable feature-level sensor fusion architecture combining the data of a multi-layer laserscanner and a monocular video has been developed. The approach aims at a maximization of synergetic effects by combining low-level measurement features and at the same time trying to keep the fusion architecture as general as possible. A new concept for the geometric modeling of diverse object shapes found in real traffic scenes, including free form models, enhances the precision of the object tracking. Results from real sensor data demonstrate the performance of the new algorithms compared to robust algorithms known from the literature. The fusion of multi-sensor information for state estimation is a well studied problem in robotics. However, the classical methods may fail to take into account the measurements validity, therefore ruining the benefits of sensor redundancy. This work addresses this problem by learning context-dependent knowledge about sensor reliability. This knowledge is later used as a decision rule in the fusion task in order to dynamically select the most appropriate subset of sensors. For this purpose we use the Mixture of Experts framework. In our application, each expert is a Kalman filter fed by a subset of sensors, and a gating network serves as a mediator between individual filters, basing its decision on sensor inputs and contextual information to reason about the operation context. The performance of this model is evaluated for altitude estimation of a UAV. In this paper, we propose a heterogeneous multisensor fusion algorithm for mapping in dynamic environments. The algorithm synergistically integrates the information obtained from an uncalibrated camera and sonar sensors to facilitate mapping and tracking. The sonar data is mainly used to build a weighted line-based map via the fuzzy clustering technique. The line weight, with confidence corresponding to the moving object, is determined by both sonar and vision data. The motion tracking is primarily accomplished by vision data using particle filtering and the sonar vectors originated from moving objects are used to modulate the sample weighting. A fuzzy system is implemented to fuse the two sensor data features. Additionally, in order to build a consistent global map and maintain reliable tracking of moving objects, the well-known extended Kalman filter is applied to estimate the states of robot pose and map features. Thus, more robust performance in mapping as well as tracking are achieved. The empirical ... In this paper, we introduce a novel fusion method that can enhance object detection performance by fusing decisions from two different types of computer vision tasks: object detection and image classification. In the proposed work, the class label of an image obtained from image classification is viewed as prior knowledge about existence or non-existence of certain objects. The prior knowledge is then fused with the decisions of object detection to improve detection accuracy by mitigating false positives of an object detector that are strongly contradicted with the prior knowledge. A recently introduced novel fusion approach called dynamic belief fusion (DBF) is used to fuse the detector output with the classification prior. Experimental results show that the detection performance of all the detection algorithms used in the proposed work is improved on benchmark datasets via the proposed fusion framework. Currently sparse signal reconstruction gains considerable interest and is applied in many fields. In this paper, a similarity induced by joint sparse representation is designed to construct the likelihood function of particle filter tracker so that the color visual spectrum and thermal spectrum images can be fused for object tracking. The proposed fusion scheme performs joint sparse representation calculation on both modalities and the resultant tracking results are fused using min operation on the sparse representation coefficients. In addition, a co-learning approach is proposed to update the reference templates of both modality and enhance the tracking robustness. The proposed fusion scheme outperforms state-of-the-art approaches, and its effectiveness is verified using OTCBVS database. In this paper, we investigate a binary decentralized detection problem in which a network of wireless sensors provides relevant information about the state of nature to a fusion center. Each sensor transmits its data over a multiple access channel. Upon reception of the information, the fusion center attempts to accurately reconstruct the state of nature. We consider the scenario where the sensor network is constrained by the capacity of the wireless channel over which the sensors are transmitting, and we study the structure of an optimal sensor configuration. For the problem of detecting deterministic signals in additive Gaussian noise, we show that having a set of identical binary sensors is asymptotically optimal, as the number of observations per sensor goes to infinity. Thus, the gain offered by having more sensors exceeds the benefits of getting detailed information from each sensor. A thorough analysis of the Gaussian case is presented along with some extensions to other observation distributions. The article deals with the analysis and interpretation of dynamic scenes typical of urban driving. The key objective is to assess risks of collision for the ego-vehicle. We describe our concept and methods, which we have integrated and tested on our experimental platform on a Lexus car and a driving simulator. The on-board sensors deliver visual, telemetric and inertial data for environment monitoring. The sensor fusion uses our Bayesian Occupancy Filter for a spatio-temporal grid representation of the traffic scene. The underlying probabilistic approach is capable of dealing with uncertainties when modeling the environment as well as detecting and tracking dynamic objects. The collision risks are estimated as stochastic variables and are predicted for a short period ahead with the use of Hidden Markov Models and Gaussian processes. The software implementation takes advantage of our methods, which allow for parallel computation. Our tests have proven the relevance and feasibility of our approach for improving the safety of car driving. Treats a problem arising in the design of intelligent vehicles: automated detection of lane and pavement boundaries using forward-looking optical and radar imaging sensors mounted on an automobile. In previous work, lane and pavement boundaries have always been located separately. This separate detection strategy is problematic in situations when either the optical or the radar image is too noisy. We propose a Bayesian multisensor image fusion method to solve our boundary detection problem. This method makes use of a deformable template model to globally describe the boundaries of interest. The optical and radar imaging processes are described with random field likelihoods. The multisensor fusion boundary detection problem is reformulated as a joint MAP estimation problem. However, the joint MAP estimate is intractable, as it involves the computation of a notoriously difficult normalization constant, also known as the partition function. Therefore, we settle for the so-called empirical MAP estimate, as an approximation to the true MAP estimate. Several experimental results are provided to demonstrate the efficacy of the empirical MAP estimation method in simultaneously detecting lane and pavement boundaries. Fusion of multi-modal images is not only of interest to the intelligent vehicles community, but to others as well, such as biomedicine, remote sensing, target recognition. The method presented in the paper is also applicable to image fusion problems in these other areas. This paper investigates how to integrate the complementary information from RGB and thermal (RGB-T) sources for object tracking. We propose a novel Convolutional Neural Network (ConvNet) architecture, including a two-stream ConvNet and a FusionNet, to achieve adaptive fusion of different source data for robust RGB-T tracking. Both RGB and thermal streams extract generic semantic information of the target object. In particular, the thermal stream is pre-trained on the ImageNet dataset to encode rich semantic information, and then fine-tuned using thermal images to capture the specific properties of thermal information. For adaptive fusion of different modalities while avoiding redundant noises, the FusionNet is employed to select most discriminative feature maps from the outputs of the two-stream ConvNet, and updated online to adapt to appearance variations of the target object. Finally, the object locations are efficiently predicted by applying the multi-channel correlation filter on the fused feature maps. Extensive experiments on the recently public benchmark GTOT verify the effectiveness of the proposed approach against other state-of-the-art RGB-T trackers. This work establishes an architecture for Urban Search and Rescue and a methodology for mixing real-world and simulation-based testing. A sensor suite and sensor fusion algorithm for robust victim detection permits aggregation of sensor readings from various sensors on multiple robots. We have embarked on a research program focusing on the enabling technologies of effective USAR robotic rescue devices. The program is also researching system-level design, evaluation, and refinement of USAR rescue architectures that include teams of sensor-laden robots and human rescuers. In this paper, we present highlights from our research, which include our multiagent system (MAS) infrastructure, our simulation environment, and our approach to sensor fusion and interface design for effective robotic control. It has been long known that fusing information from multiple sensors for robot navigation results in increased robustness and accuracy. However, accurate calibration of the sensor ensemble prior to deployment in the field as well as coping with sensor outages, different measurement rates and delays, render multi-sensor fusion a challenge. As a result, most often, systems do not exploit all the sensor information available in exchange for simplicity. For example, on a mission requiring transition of the robot from indoors to outdoors, it is the norm to ignore the Global Positioning System (GPS) signals which become freely available once outdoors and instead, rely only on sensor feeds (e.g., vision and laser) continuously available throughout the mission. Naturally, this comes at the expense of robustness and accuracy in real deployment. This paper presents a generic framework, dubbed MultiSensor-Fusion Extended Kalman Filter (MSF-EKF), able to process delayed, relative and absolute measurements from a theoretically unlimited number of different sensors and sensor types, while allowing self-calibration of the sensor-suite online. The modularity of MSF-EKF allows seamless handling of additional lost sensor signals during operation while employing a state buffering scheme augmented with Iterated EKF (IEKF) updates to allow for efficient re-linearization of the prediction to get near optimal linearization points for both absolute and relative state updates. We demonstrate our approach in outdoor navigation experiments using a Micro Aerial Vehicle (MAV) equipped with a GPS receiver as well as visual, inertial, and pressure sensors. A novel sensor fusion methodology is presented, which provides intelligent vehicles with augmented environment information and knowledge, enabled by vision-based system, laser sensor and global positioning system. The presented approach achieves safer roads by data fusion techniques, especially in single-lane carriageways where casualties are higher than in other road classes, and focuses on the interplay between vehicle drivers and intelligent vehicles. The system is based on the reliability of laser scanner for obstacle detection, the use of camera based identification techniques and advanced tracking and data association algorithms i.e. Unscented Kalman Filter and Joint Probabilistic Data Association. The achieved results foster the implementation of the sensor fusion methodology in forthcoming Intelligent Transportation Systems. An efficient approach for deriving accurate pose and heading values through multi-sensor fusion of data from several inexpensive sensors (such as multiple GPS (Global Positioning Systems), EC (electronic compass), rate gyro) is presented. The proposed multisensor fusion approach is composed of several sub-methods namely initial heading calculation, classification and weighing (CnW), extended Kalman filter (EKF) and then covariance intersection (CI) algorithms. The consecutive implementation of the sub-methods gives an accurate heading value with lesser RMSE (root mean square error) compared to the original GPS COG (course over ground) and EC. Several experimental tests were done to confirm the good performance of the proposed process. Abstract Road detection is one of the key challenges for autonomous vehicles. Two kinds of sensors are commonly used for road detection: cameras and LIDARs. However, each of them suffers from some inherent drawbacks. Thus, sensor fusion is commonly used to combine the merits of these two kinds of sensors. Nevertheless, current sensor fusion methods are dominated by either cameras or LIDARs rather than making the best of both. In this paper, we extend the conditional random field (CRF) model and propose a novel hybrid CRF model to fuse the information from camera and LIDAR. After aligning the LIDAR points and pixels, we take the labels (either road or background) of the pixels and LIDAR points as random variables and infer the labels via minimization of a hybrid energy function. Boosted decision tree classifiers are learned to predict the unary potentials of both the pixels and LIDAR points. The pairwise potentials in the hybrid model encode (i) the contextual consistency in the image, (ii) the contextual consistency in the point cloud, and (iii) the cross-modal consistency between the aligned pixels and LIDAR points. This model integrates the information from the two sensors in a probabilistic way and makes good use of both sensors. The hybrid CRF model can be optimized efficiently with graph cuts to get road areas. Extensive experiments have been conducted on the KITTI-ROAD benchmark dataset and the experimental results show that the proposed method outperforms the current methods.