RGB-Thermal object tracking attempt to locate target object using complementary visual and thermal infrared data. Existing RGB-T trackers fuse different modalities by robust feature representation learning or adaptive modal weighting. However, how to integrate dual attention mechanism for visual tracking is still a subject that has not been studied yet. In this paper, we propose two visual attention mechanisms for robust RGB-T object tracking. Specifically, the local attention is implemented by exploiting the common visual attention of RGB and thermal data to train deep classifiers. We also introduce the global attention, which is a multi-modal target-driven attention estimation network. It can provide global proposals for the classifier together with local proposals extracted from previous tracking result. Extensive experiments on two RGB-T benchmark datasets validated the effectiveness of our proposed algorithm.
RGB-T tracking receives more and more attention in computer vision community with the popularity of thermal infrared sensors. Wu @cite_15 concatenate the image patches from RGB and thermal sources, and then sparsely represent each sample in the target template space for tracking. Modal weights are introduced for each source to represent the image quality, and combine with the sparse representation in Bayesian filtering framework to perform object tracking @cite_0 . Zhu @cite_17 propose a quality-aware feature aggregation network to fuse multi-layer deep feature and multimodal information adaptively for RGB-T tracking. Although these works also achieve good performance on RGB-T benchmarks, however, they still adopt local search strategy for target localization and seldom of them consider explore long-term visual attentions for their tracker.
@cite_0 @cite_15 @cite_17
2765667535 2106944077 2901716381
In this paper, we propose a novel graph model, called weighted sparse representation regularized graph, to learn a robust object representation using multispectral (RGB and thermal) data for visual tracking. In particular, the tracked object is represented with a graph with image patches as nodes. This graph is dynamically learned from two aspects. First, the graph affinity (i.e., graph structure and edge weights) that indicates the appearance compatibility of two neighboring nodes is optimized based on the weighted sparse representation, in which the modality weight is introduced to leverage RGB and thermal information adaptively. Second, each node weight that indicates how likely it belongs to the foreground is propagated from others along with graph affinity. The optimized patch weights are then imposed on the extracted RGB and thermal features, and the target object is finally located by adopting the structured SVM algorithm. Moreover, we also contribute a comprehensive dataset for RGB-T tracking purpose. Comparing with existing ones, the new dataset has the following advantages: 1) Its size is sufficiently large for large-scale performance evaluation (total frame number: 210K, maximum frames per video pair: 8K). 2) The alignment between RGB-T video pairs is highly accurate, which does not need pre- and post-processing. 3) The occlusion levels are annotated for analyzing the occlusion-sensitive performance of different methods. Extensive experiments on both public and newly created datasets demonstrate the effectiveness of the proposed tracker against several state-of-the-art tracking methods. Information from multiple heterogenous data sources (e.g. visible and infrared) or representations (e.g. intensity and edge) have become increasingly important in many video-based applications. Fusion of information from these sources is critical to improve the robustness of related visual information processing systems. In this paper we propose a data fusion approach via sparse representation with applications to robust visual tracking. Specifically, the image patches from different sources of each target candidate are concatenated into a one-dimensional vector that is then sparsely represented in the target template space. The template space representation, which naturally fuses information from different sources, brings several benefits to visual tracking. First, it inherits robustness to appearance contaminations from the previously proposed sparse trackers. Second, it provides a flexible framework that can easily integrate information from different data sources. Third, it can be used for handling various number of data sources, which is very useful for situations where the data inputs arrive at different frequencies. The sparsity in the representation is achieved by solving an l1-regularized least squares problem. The tracking result is then determined by finding the candidate with the smallest approximation error. To propagate the results over time, the sparse solution is combined with the Bayesian state inference framework using the particle filter algorithm. We conducted experiments on several real videos with heterogeneous information sources. The results show that the proposed approach can track the target more robustly than several state-of-the-art tracking algorithms. This paper investigates how to perform robust visual tracking in adverse and challenging conditions using complementary visual and thermal infrared data (RGB-T tracking). We propose a novel deep network architecture "quality-aware Feature Aggregation Network (FANet)" to achieve quality-aware aggregations of both hierarchical features and multimodal information for robust online RGB-T tracking. Unlike existing works that directly concatenate hierarchical deep features, our FANet learns the layer weights to adaptively aggregate them to handle the challenge of significant appearance changes caused by deformation, abrupt motion, background clutter and occlusion within each modality. Moreover, we employ the operations of max pooling, interpolation upsampling and convolution to transform these hierarchical and multi-resolution features into a uniform space at the same resolution for more effective feature aggregation. In different modalities, we elaborately design a multimodal aggregation sub-network to integrate all modalities collaboratively based on the predicted reliability degrees. Extensive experiments on large-scale benchmark datasets demonstrate that our FANet significantly outperforms other state-of-the-art RGB-T tracking methods.