Deformable Siamese Attention Networks for Visual Object Tracking

Siamese-based trackers have achieved excellent performance on visual object tracking. However, the target template is not updated online, and the features of the target template and search image are computed independently in a Siamese architecture. In this paper, we propose Deformable Siamese Attention Networks, referred to as SiamAttn, by introducing a new Siamese attention mechanism that computes deformable self-attention and cross-attention. The self attention learns strong context information via spatial attention, and selectively emphasizes interdependent channel-wise features with channel attention. The cross-attention is capable of aggregating rich contextual inter-dependencies between the target template and the search image, providing an implicit manner to adaptively update the target template. In addition, we design a region refinement module that computes depth-wise cross correlations between the attentional features for more accurate tracking. We conduct experiments on six benchmarks, where our method achieves new state of-the-art results, outperforming the strong baseline, SiamRPN++ [24], by 0.464->0.537 and 0.415->0.470 EAO on VOT 2016 and 2018.


Introduction
Visual object tracking aims to track a given target object at each frame over a video sequence. It is a fundamental task in computer vision [17,16,20], and has numerous practical applications, such as automatic driving [23], human-computer interaction [28], robot sensing, etc. Recent efforts have been devoted to improving the performance of visual object trackers. However, developing a fast, accurate and robust tracker is still highly challenging due to the vast amount of deformations, motions and occlusions that often occur on video objects with complex background [38,22,10].
Deep learning technologies have significantly advanced the task of visual object tracking, by providing the strong Figure 1: Tracking results of our deformable Siamese attention networks (SiamAttn) with three state-of-the-art trackers. Our results are more accurate, and are robust to appearance changes, complex background and close distractors with occlusions. Fig. 4 further shows the strong discriminative features learned by our Siamese attention module. capacity of learning powerful deep features. For example, Bertinetto et al. [1] first introduced Siamese networks for visual tracking. Since then, object trackers built on Siamese networks and object detection frameworks have achieved the state-of-the-art performance, such as SiamRPN [25], SiamRPN++ [24], and SiamMask [36]. The Siamese-based trackers formulate the problem of visual object tracking as a matching problem by computing cross-correlation similarities between a target template and a search region, which transforms the tracking problem into finding the target object from an image region by computing the highest visual similarity [1,25,24,36,44]. Therefore, it casts the tracking problem into a Region Proposal Network (RPN) [13] based detection framework by leveraging Siamese networks, which is the key to boost the performance of recent deep trackers.
Siamese-based trackers are trained completely offline by using massive frame pairs collected from videos, and thus the target template can not be updated online. This makes it difficult to precisely track the targets with large appearance variations, significant deformations, or occlusions, which inevitably increase the risk of tracking drift. Furthermore, the convolutional features of the target object and the search image are computed independently in Siamese architecture, where background context information is completely discarded in target features, but is of great importance to distinguish the target from close distractors and complex backgrounds. Recent work attempted to enhance the target representation by integrating the features of previous targets [41,14], but the discriminative context information from the background is ignored. Alternatively, we introduce a new Siamese attention mechanism that encodes rich background context into the target representation by computing cross-attention in the Siamese networks.
Recently, the attention mechanism was introduced to visual object tracking in [35,45], which inspired the current work. However, the attentions and deep features of the target template and the search image are computed separately in [35,45], which limits the potential performance of the Siamese architecture. In this work, we propose Deformable Siamese Attention Networks, referred as SiamAttn, to improve the feature learning capability of Siamese-based trackers. We present a new deformable Siamese attention which can improve the target representation with strong robustness to large appearance variations, and also enhance the target discriminability against distractors and complex backgrounds, resulting in more accurate and stable tracking, as shown in Fig. 1

Related Work
Correlation filter based trackers have been widely used since MOSSE [3], due to their efficiency and expansibility. However, the tracking object can be continuously improved online, which inevitably limits the representation ability of such trackers. Deep learning technologies provide a powerful tool with the ability to learn strong deep representations, and recent work attempted to incorporate the correlation filter framework with such features learning capability, such as MDNet [31], C-COT [8], ECO [7] and GFS-DCF [40].
There is another trend to build trackers on Siamese networks, by learning from massive data offline. Bertinetto et al. [1] first introduced SiamFC for visual tracking, by using Siamese networks to measure the similarity between target and search image. Then Li et al. [25] applied a region proposal network (RPN) [13] into Siamese networks, referred as SiamRPN. Zhu et al. [44] extended the SiamRPN by developing distractor-aware training. Recently, SiamDW-RPN [43] and SiamRPN++ [24] were proposed, which allow the Siamese-based trackers to explore deeper networks, while Wang et al. [36] developed a SiamMask that incorporates instance segmentation into tracking. Our work is related to that of [11] where a C-RPN was developed to progressively refine the location of target with a sequence of RPNs, but we design a new module that only refines a single output region, which is particularly lightweight and can be integrated into very deep Siamese networks.
However, Siamese-based trackers can be affected by distractors with complex backgrounds. Recent work attempted to design various strategies to update the template online, in an effort to improve the target-discriminability of Siamesebased trackers, such as MLT [4], UpdateNet [42] and Grad-Net [26]. An alternative solution is to extend existing online discriminative framework with deep networks for endto-end learning, e.g., ATOM [6] and DiMP [2]. In addition, Zhu et al. [45] exploited motion information in Siamese networks to improve the feature representation.
Recently, the attention mechanism has been widely applied in various tasks. Hu et al. [18] proposed a SENet to enhance the representational power of the networks by modeling channel-wise relationships via attentions. Wang et al. [37] developed a non-local operation in the space-time dimension to guide the aggregation of contextual information. In [12], a self-attention mechanism was introduced to harvest the contextual information for semantic segmentation. Particularly, Wang et al. [35] proposed a RASNet by developing an attention mechanism for Siamese trackers, but it only utilizes the template information, which might limit its representation ability. To better explore the potentials of feature attentions in Siamese networks, we compute both self-attention and cross-branch attention jointly with deformable operations to enhance the discriminative representation of target.

Deformable Siamese Attention Networks
We describe the details of our Deformable Siamese Attention Networks (SiamAttn). As shown in Fig. 2, it consists of three main components: a deformable Siamese attention (DSA) module, Siamese region proposal networks (Siamese RPN), and a region refinement module.
Overview. We use a five-stage ResNet-50 as the backbone of Siamese networks, which computes increasingly highlevel features as the layers become deeper. The features of the last three stages on both Siamese branches can be modulated and enhanced by the proposed DSA module, generating two-stream attentional features. Then we apply three Siamese RPN blocks described in [24] to the attentional features, generating dense response maps, which are further processed by a classification head and a bounding box regression head to predict a single tracking region. Finally, the generated tracking region is further refined by a region refinement module, where depth-wise cross correlations are computed on the two-stream attentional features. The correlated features are further fused and enhanced, and then are used for refining the tracking region via joint bounding box regression and target mask prediction.

Siamese-based Trackers
Bertinetto et al. [1] introduced Siamese networks for visual object tracking, which formulates visual object tracking as a similarity learning problem. Siamese networks consist of a pair of CNN branches with sharing parameters φ, which are used to project the target image (z) and the search image (x) into a common embedding space, where a similarity metric g can be computed to measure the similarity between them, g(φ(x), φ(z)). Li et al. [25] applied region proposal networks (RPN) [13] with Siamese networks for visual object tracking (referred as SiamRPN), where the computed features φ(x) and φ(z) are fed into the RPN framework using an up-channel cross-correlation operation. This generates dense response maps where RPN-based detection can be implemented, leading to significant performance improvements.
SiamRPN++. In [24], SiamRPN++ was introduced to improve the performance of SiamRPN, by exploring the power of deeper networks. A spatial aware sampling strategy was developed to address a key limitation of the Siamese-based trackers, allowing them to benefit from a deeper backbone likes ResNet-50. Furthermore, SiamRPN++ adopts a depthwise cross correlation to replace the up-channel cross correlation, which reduces the number of parameters and accelerates the training process. Moreover, it aggregates multilayer features to predict the target more accurately. Similarly, we use ResNet-50 as backbone, with depth-wise cross correlation and multi-layer aggregation strategy, by following SiamRPN++ [24]. But we introduce a new Siamese attention module that enhances the learned discriminative representations of the target object and the search image, which is the key to improve the tracking performance on both accuracy and robustness.

Deformable Siamese Attention Module
As illustrated in Fig. 3, the proposed DSA module takes a pair of convolutional features computed from Siamese networks as inputs, and outputs the modulated features by applying the Siamese attention mechanism. The DSA module consists of two sub-modules: a self-attention sub-module and a cross-attention sub-module. We denote the feature maps of the target and the search image as Z and X, with feature shapes of C × h × w and C × H × W . Self-Attention. Inspired by [12], our self-attention submodule attends to two aspects, namely channels and special positions. Unlike the classification or detection task where object classes are pre-defined, visual object tracking is a class-agnostic task and the class of object is fixed during the whole tracking process. As observed in [24], each channel map of the high-level convolutional features usually responses for a specific object class. Equally treating the features across all channels will hinder the representation ability. Similarly, as limited by receptive fields, the features computed at each spatial position of the maps can only capture the information from a local patch. Therefore, it is crucial to learn the global context from the whole image.
Specifically, self-attention is computed separately on the target branch and the search branch, and both channel selfattention and spatial self-attention are calculated at each branch. Taking the spatial self-attention for example. Sup-pose the input features are X ∈ R C×H×W , we first apply two separate convolution layers with 1 × 1 kernels on X to generate query features Q and key features K respectively, where Q, K ∈ R C ×H×W and C = 1 8 C is the reduced channel number. The two features are then reshaped toQ,K ∈ R C ×N where N = H × W . We can generate a spatial self-attention map A s s ∈ R N ×N via matrix multiplication and column-wise softmax operations as, Meanwhile, a 1×1 convolution layer with a reshape operation is applied to the features X to generate value features V ∈ R C×N , which are multiplied with the attention map and then are added to the reshaped featuresX ∈ R C×N with a residual connection as, where α is a scalar parameter. The outputs are then reshaped back to the original size as X s s ∈ R C×H×W . We can compute channel self-attention A s c and the channel-wise attentional features X s c in a similar manner. Notice that on computing the channel self-attention and the corresponding attentional features, the query, key and value features are the original convolutional features computed directly from the Siamese networks, without implementing 1 × 1 convolutions. The final self-attentional features X s are generated by simply combining the spatial and channel-wise attentional features using element-wise sum.
Cross-Attention. Siamese networks usually make predictions in the last stage, while the features from two branches are computed separately, but may compensate each other. It is common that multiple objects appear at the same time, even with occlusions during tracking. Therefore, it is of great importance for the search branch to learn the target information, which enables it to generate a more discriminative representation that helps identify the target more accurately. Meanwhile, the target representation can be more meaningful when the contextual information from the search image is encoded. To this end, we propose a cross-attention sub-module to learn such mutual information from two Siamese branches, which in turn allows the two branches to work more collaboratively.
Specifically, we use Z ∈ R C×h×w and X ∈ R C×H×W to denote template features and search features, respectively. Taking the search branch for example, we first reshape the target features Z toZ ∈ R C×n where n = h × w. Then we compute the cross-attention from the target branch by performing similar operations as channel self-attention, where row-wise softmax is implemented on the computed matrix. Then the cross-attention computed from the target branch is encoded into the search features X as, where γ is a scalar parameter, and the reshaped features X c ∈ R C×H×W are the output of the sub-module. Finally, the self-attentional features X s and the crossattentional features X c are simply combined with an element-wise sum, generating the attentional features for the search image. The attentional features for target image can be computed in a similar manner.
Deformable Attention. The building units in CNNs, such as convolution or pooling units, often have fixed geometric structures, by assuming the objects are rigid. For object tracking, it is of importance to model complex geometric transformations because the tracking objects usually have large deformations due to various factors, such as viewpoint, pose, occlusion and so on. The proposed attention mechanism can handle such challenges to some extent. We further introduce deformable attention to enhance the capability for handling such geometric transformations.
The deformable attention can sample the input feature maps at variable locations instead of the fixed ones, making them attend to the content of objects with deformations. Therefore, it is particularly suitable for object tracking, where the visual appearance of a target can be changed significantly over time. Specifically, a 3 × 3 deformable convolution [5] is further applied to the computed attentional features, generating the final attentional features which are more accurate, discriminative and robust. As shown in Fig. 4, with our DSA module, the confidence maps of the attentional features focus more accurately on the interested objects, making the objects more discriminative against distractors and background.
Region Proposals. The DSA module outputs Siamese attentional features for both target image and search image. Then we apply three Siamese RPN blocks on the attentional features for generating a set of target proposals, with corresponding bounding boxes and class scores, as shown in Fig. 2. Specifically, a Siamese RPN block is a combination of multiple fully convolutional layers, depth-wise cross correlation, with a regression head and a classification head on top, as described in [25]. It takes a pair of convolutional features computed from the two branches of Siamese networks, and outputs dense prediction maps. By following [24], we apply three Siamese RPN blocks for the Siamese features computed from the last three stages, generating three prediction maps which are further combined by a weighted sum. Each spatial position of the combined maps predicts a set of region proposals, corresponding to the pre-defined anchors. Then the predicted proposal with the highest classification score is selected as the output tracking region.

Region Refinement Module
We further develop a region refinement module to improve the localization accuracy of the predicted target region. We first apply depth-wise cross correlations between the two attentional features across multiple stages, generating multiple correlation maps. Then the correlation maps are fed into a fusion block, where the feature maps with different sizes are aligned in both spatial and channel domains, e.g., by using up-sampling or down-sampling, with 1 × 1 convolution. Then the aligned features are further fused (with element-wise sum) for predicting both bounding box and mask of the target. Besides, we further perform two additional operations: (1) we combine the convolutional features of the first two-stages into the fused features, which encodes richer local detailed information for mask prediction; (2) a deformable RoI pooling [5] is applied to compute target features more accurately. Bounding box regression and mask prediction often require different levels of convolutional features. Thus we generate the convolutional features with spatial resolutions of 64 × 64 for mask prediction and 25 × 25 for bounding box regression.
Notice that the classification head is not applied since visual object tracking is a class-agnostic task. The input resolution for the bounding box head is 4 × 4. By using two fully-connected layers with 512 dimensions, the bounding box head predicts a 4-tuple t = (t x , t y , t w , t h ). Similarly, the input of mask prediction head has a spatial resolution of 16 × 16. By using four convolutional layers and a de-convolutional layer, the mask head predicts a classagnostic binary mask with 64 × 64 for the tracking object. Compared to ATOM [6] and SiamMask [36] which predict bounding boxes and masks densely, our refinement module uses light-weight convolutional heads to predict a bounding box and a mask for a single tracking region, which is much more computationally efficient.

Training Loss
Our model is trained in an end-to-end fashion, where the training loss is a weighted combination of multiple functions from Siamese RPN and region refinement module: where L rpn-cls and L rpn-reg refer to a classification loss and a regression loss in Siamese RPN. We employ a negative log-likelihood loss and a smooth L1 loss correspondingly. Similarly, L refine-box and L refine-mask indicate a smooth L1 loss for bounding box regression and a binary cross-entropy loss for mask segmentation in region refinement module. The weight parameters λ 1 , λ 2 and λ 3 are used to balance different tasks, which are empirically set to 0.2, 0.2 and 0.1 in our implementation.

Datasets
OTB-2015 [38]. OTB-2015 is one of the most commonly used benchmarks for visual object tracking. It has 100 fully annotated video sequences using two evaluation metrics, a precision score and an area under curve (AUC) of success plot. The precision score is the percentage of frames in which the distance between the center of tracking results and ground-truth is under 20 pixels. The success plot shows the ratios of successfully tracked frames with various thresholds. VOT2016 [21] & VOT2018 [22]. VOT2016 and VOT2018 are widely-used benchmarks for visual object tracking. VOT2016 contains 60 sequences with various challenging factors while VOT2018 has 10 different sequences with VOT2016. The two datasets are annotated with the rotated bounding boxes, and a reset-based methodology is applied for evaluation. For both benchmarks, trackers are measured in terms of accuracy (A), robustness (R), and expected average overlap (EAO). UAV123 [29]. UAV123 contains 123 sequences captured from low-altitude UAVs. Unlike other tracking datasets, the viewpoint of UAV123 is aerial and the targets to be tracked are usually small. LaSOT [10]. LaSOT is a large-scale dataset with 1400 sequences in total, and 280 sequences in test set. high-quality dense annotations are provided, and deformation and occlusion are very common in LaSOT. The average sequence length of LaSOT is 2500 frames, demonstrating long-term performance of the evaluated trackers. TrackingNet [30]. TrackingNet contains 30000 sequences with 14 million dense annotations and 511 sequences in the test set. It covers diverse object classes and scenes, requiring trackers to have both discriminative and generative capabilities.

Implementation Details
We use ResNet-50, pre-trained on ImageNet [9], as the backbone, and the whole networks are then fine-tuned on the training sets of COCO [27], YouTube-VOS [39], La-SOT [10] and TrackingNet [30]. We apply stochastic gradient descent (SGD) with a momentum of 0.9 and a weight decay of 10 −5 for optimization. By following SiamFC [1], we adopt an exemplar image of 127 × 127 and a search image of 255 × 255 for training and testing.
Our model is trained for 20 epochs. By following SiamRPN++ [24], we use a warm-up learning rate of 10 −3     for the first 5 epochs which decays exponentially from 5 × 10 −3 to 5 × 10 −4 for the last 15 epochs. The weights of backbone are frozen for the first 10 epochs, then the whole networks are trained end-to-end for the last 10 epochs. In particular, the learning rate of backbone is 20 times smaller than the other parts. Batch size is set to 12. Our anchor boxes have 5 aspect ratios, [0.33, 0.5, 1, 2, 3]. In Siamese-RPN blocks, an anchor box is labelled as positive when it has an IoU> 0.6, or as negative when IoU< 0.3. Other patches whose IoU overlap falls in between are ignored.
In addition, we sample 16 regions from each image with IoU> 0.5 for training our region refinement module. For the backbone networks, we employ dilated convolutions for the last two blocks to increase the receptive fields, and the effective strides at these two blocks are reduced from 16 or 32 pixels to 8 pixels. We also reduce the number of feature channels to 256 for the last three blocks of the backbone networks via a 1 × 1 convolution layer. During inference, cosine window penalty, scale change penalty and linear interpolation update strategy [25] are applied. Only one single region with the highest score predicted by Siamese RPN blocks is fed into our region refinement module. Our method is implemented using PyTorch, and we use NVIDIA GeForce RTX 2080Ti GPU.

Ablation Study
We study the impact of individual components in SiamAttn, and conduct ablation study on VOT2016.

Model Architecture.
We use SiamRPN++ [24] as baseline. As shown in Tab. 5, SiamRPN++ achieves an EAO of 0.464. By adding mask learning layers to SiamRPN++, the EAO can be improved to 0.477. With our region refinement module, the EAO score is further increased by +2.2%. Compared with the baseline, the accuracy score improves from 0.64 to 0.67, with comparable robustness. Our Siamese attention consists of both self-attention and cross-attention, each of which can further improve the EAO by +4.7% or +4.9% respectively. This suggests that the proposed cross-attention is critical to the tracking results, and even has a more significant impact than the self-attention. Jointly exploring both selfattention and cross-attention makes our method not only robust, but also more accurate. This results in a high EAO of 0.537, surpassing the baseline by a large margin of 7.3%.

Deformable Layers.
In this study, we evaluate the impact of deformable operations by replacing them with a regular convolution. As shown in Tab. 6, this results in slight performance drops, e.g., 0.537→0.520 EAO with deformable convolution and 0.537→0.531 EAO with   On Training Data. In this study, we investigate the impact of training with different training sets. Our current results are achieved by using a combination of multiple training sets from recent LaSOT [10] and TrackingNet [30], with COCO [27] and YouTube-VOS [39], mainly following [6] with an additional YouTube-VOS [39] for providing mask annotations. We also report the results on a different training combination applied by [36], including COCO [27], YouTube-VOS [39], YouTube-BoundingBox [32], ImageNet-VID, and ImageNet-Det [33]. Results are reported in Tab. 7. Using the recent large-scale tracking datasets can improve the results with 1.2% EAO on VOT2016, while our approach can still achieve the state-of-the-art performance using a different choice of the training sets.
Speed Analysis. On OTB-2015, UAV, LaSOT and Track-ingNet benchmarks, our model predicts axis-aligned bounding box, without mask head. It can achieve an inference speed of 45 fps. On VOT benchmarks, our model generates the rotated boxes from the predicted masks, which reduces the inference speed to 33 fps.

Conclusion
We have presented new Deformable Siamese Attention Networks for visual object tracking. We introduce a deformable Siamese attention mechanism consisting of both self-attention and cross-attention. The new Siamese attention can strongly enhance target discriminability, and at the same time, improve the robustness against large appearance variations, complex backgrounds and distractors. Additionally, a region refinement module is designed to further increase the tracking accuracy. Extensive experiments are conducted on six benchmarks, where our method obtains new state-of-the-art results, with real-time running speed.