Reference-based Image Super-Resolution with Deformable Attention Transformer

Reference-based image super-resolution (RefSR) aims to exploit auxiliary reference (Ref) images to super-resolve low-resolution (LR) images. Recently, RefSR has been attracting great attention as it provides an alternative way to surpass single image SR. However, addressing the RefSR problem has two critical challenges: (i) It is difficult to match the correspondence between LR and Ref images when they are significantly different; (ii) How to transfer the relevant texture from Ref images to compensate the details for LR images is very challenging. To address these issues of RefSR, this paper proposes a deformable attention Transformer, namely DATSR, with multiple scales, each of which consists of a texture feature encoder (TFE) module, a reference-based deformable attention (RDA) module and a residual feature aggregation (RFA) module. Specifically, TFE first extracts image transformation (e.g., brightness) insensitive features for LR and Ref images, RDA then can exploit multiple relevant textures to compensate more information for LR features, and RFA lastly aggregates LR features and relevant textures to get a more visually pleasant result. Extensive experiments demonstrate that our DATSR achieves state-of-the-art performance on benchmark datasets quantitatively and qualitatively.


Introduction
Single image super-resolution (SISR), which aims at recovering a high-resolution (HR) image from a low-resolution (LR) input, is an active research topic due to its high practical values [13,51,18,49,41,15,46,21,16,14,9,20]. However, SISR is a highly ill-posed problem since there exist multiple HR images that can degrade to the same LR image [38,8]. While real LR images usually have no corresponding HR ground-truth (GT) images, one can easily find a high-quality image as a reference (Ref) image with high-frequency details from various sources, such  [12]. When the brightness of LR and Ref image is different, our method performs better than C 2 -Matching [12] in transferring relevant textures from the Ref image to the SR image, which is closer to the ground-truth image.
as photo albums, video frames, and web image search, which has similar semantic information (such as content and texture) to the LR image. Such an alternative SISR method is referred to as reference-based super-resolution (RefSR), which aims to transfer HR textures from the Ref images to super-resolved images and has shown promising results over SISR. Although various RefSR methods [12,27,47,45] have been recently proposed, two challenges remain unsolved for SR performance improvement.
First, it is difficult to match the correspondence between the LR and Ref images especially when their distributions are different. For example, the brightness of the Ref images is different from that of the LR images. Existing methods [56,48] mostly match the correspondence by estimating the pixel or patch similarity of texture features between LR and Ref images. However, such similarity metric is sensitive to image transformations, such as brightness and color of images. Recently, the state-of-the-art (SOTA) method C 2 -Matching [12] trains a feature extractor, which demonstrates strong robustness to scale and rotation. However, it neglects to explore the effects of brightness, contrast, and color of images. As a result, this method may transfer inaccurate textures from the Ref image, when the Ref images have different brightness from the LR image, as shown in Fig. 1. Based on the observation and analyses, we can see that the quality of correspondence is affected by the similarity metric and the distribution gap between the LR and Ref images.
On the other hand, some methods [57,34] adopt optical flow or deformable convolutions [4,59,3,42] to align spatial features between the Ref and LR images. However, these methods may find an inaccurate correspondence when the distance between the LR and Ref images is relatively large. With the inaccurate correspondence, their performance would deteriorate seriously since the irrelevant texture cannot provide meaningful details. Therefore, how to accurately match the correspondence between the Ref and LR images is a challenging problem as it affects the quality of super-resolved results.
Second, it is also challenging to transfer textures of the high-quality Ref images to restore the HR images. One representative work CrossNet [57] estimates the flow from the Ref image to the LR image and then warp the features based on the optical flow. However, the optical flow may be inaccurate, since the Ref and LR images could be significantly different. In addition, most existing methods [56,48,27] search the most similar textures and the corresponding position, and then swap the texture features from the Ref image. As a result, these methods may transfer irrelevant textures to the output and have poor SR performance, when the original estimated flow or position is inaccurate. Hence, it is important and necessary to explore a new architecture to adaptively transfer texture and mitigate the impact of inaccurate correspondence in the Ref image.
To address the above two challenges, we propose a novel deformable attention Transformer, namely DATSR, for reference-based image super-resolution. DATSR is built on the U-Net and consists of three basic modules, including texture feature encoders, deformable attention, and residual feature aggregation. Specifically, we first use texture feature encoders to extract multi-scale features with different image transformations. Then, we propose a reference-based deformable attention to discover the multiple relevant correspondences and adaptively transfer the textures. Last, we fuse features and reconstruct the SR images using residual feature aggregation. We conduct extensive comparisons with recent representative SOTA methods on benchmark datasets. The quantitative and visual results demonstrate that our DATSR achieves the SOTA performance.
The main contributions are summarized as follows: -We propose a novel reference-based image super-resolution with deformable attention transformer (DATSR), which is end-to-end trainable by incorporating Transformer into RefSR. Compared with existing RefSR methods, our DATSR performs more robust correspondence matching and texture transfer and subsequently achieves SOTA performance quantitatively and visually. -

Related Work
We will briefly introduce two related super-resolution paradigms, including single image super-resolution and reference-based image super-resolution.
Single image super-resolution (SISR). The goal of SISR is to recover highresolution (HR) images from the low-resolution (LR) images. Recent years have witnessed significant achievements of using deep neural networks to solve SISR [6,55]. SRCNN [6] is the pioneer work of exploiting deep convolutional networks to map LR image into HR image. To further improve SR performance, researchers resort to employing deeper neural networks with attention mechanisms and residual blocks [22,33,55,19,31,20,54,23,21,50,36,5]. However, it is difficult for traditional SISR methods to produce realistic images when the HR textures are highly degraded. To relieve this, some SR methods [17,44,53,11,40,43,58] adopt generative adversarial networks (GANs) to further improve the perceptual quality of the super-resolved outputs.
Reference-based image super-resolution (RefSR  [34] proposes a similarity search and extraction network and it is aware of the best matching position and the relevancy of the best match. To improve the performance, TTSR [48] proposes a hard and soft attention for texture transfer and synthesis. Instead of using the features of a classifier, E2ENT 2 [45] transfers texture features by using a SR task-specific features. To improve the efficiency of matching, MASA [27] proposes a coarse-to-fine correspondence matching module and a spatial adaptation module to map the distribution of the Ref features to that of the LR features. Recently, a strong RefSR method C 2 -Matching [12] first proposes a contrastive correspondence network to learn correspondence, and then adopts a teacher-student correlation distillation to improve LR-HR matching, and last uses a residual feature aggregation to synthesize HR images. It should be noted that RefSR can be extended to the case of multiple reference images, called Multi-RefSR, which aims to transfer the texture features from multiple Ref images to the SR image. Recently, a content independent multi-reference super-resolution model CIMR-SR [47] is proposed to transfer the HR textures from multiple reference images. To improve the performance, AMRSR [32] proposes an attention-based multi-reference super-resolution network to match the most similar textures from multiple reference images. Different from RefSR, Multi-RefSR can exploit more training information as it has multiple Ref images. In this paper, we mainly study RefSR and train the model with single Ref image. Nevertheless, we still compare our model with the above Multi-RefSR methods to further demonstrate the effectiveness of our DATSR.

Proposed Method
Due to the the intrinsic complexity of RefSR, we divide the problem into two main sub-tasks: correspondence matching and texture transfer. To address these, we propose a multi-scale reference-based image SR with deformable Transformer, as shown in Fig. 2 where E q l , E k l and E v l are feature encoders at the l-th scale. In our architecture, we use three scales in the texture feature encoders. With the help of the multiscale features in U-Net, we are able to alleviate the resolution gap between the Ref and LR images since they contain the complementary scale information.
For the distribution gap, we augment images with different image transformations (e.g., brightness, contrast and hue) in the training to improve the robustness of our model. In addition to data augmentation, we use contrastive learning to train the encoder be less sensitive to different image transformations, inspired by [12]. To estimate the stable correspondence between X LR↑ and X Ref , the feature encoders E q l and E k l are the same, and the feature encoder E r l is pretrained and fixed in the training. In contrast, TTSR [48] directly uses a learnable feature encoder, resulting in limited performance since the textures are changing during training and the correspondence matching is unstable. For C 2 -Matching [12], it neglects to improve the robustness to brightness, contrast and hue. To address these, we propose to learn robust multi-scale features Q l , K l , V l , which can be regraded as Query, Key, and Value, and can be used in our attention mechanism conditioned on the LR features.

Reference-based Deformable Attention
Existing attention-based RefSR methods (e.g., [48]) tend to suffer from limited performance when the most relevant features between LR and Ref images are inaccurate, i.e., the learned LR features may not well match the Ref features. To address this, we propose a new reference-based attention mechanism, called RefAttention, as shown in Fig. 3. Formally, given Query Q l , Key K l , Value V l , and LR features F l , the attention feature A l is defined as follows: Different from existing attention mechanism [39], our attention is conditioned on the LR features and designed for the RefSR task. In Fig. 3, we denoted by A l and F l in the downscaling process, andÃ l andF l in the upscaling process. σ(·) is a correspondence matching function to calculate the relevance between the Ref and LR images. Based on the relevance, we propose a texture transfer function T (·) to transfer the textures from the Ref to the LR image.
Correspondence matching. The first important sub-task in RefSR is to match correspondences between LR and Ref images. Most existing methods [56,48] are sensitive to different image transformations (e.g., brightness, contrast and hue) and may match inaccurate correspondences. To relieve this issue, we propose a correspondence matching module in our RefAttention, as shown in Fig. 3. Specifically, we estimate the relevance between X LR↑ and X Ref by calculating similarity between Q l ∈ R C×H1×W1 and K l ∈ R C×H2×W2 . First, we unfold Q l and K l into patches Then, for the given query q i in Q ′ , the top K relevant positions in K ′ can be calculated by normalized inner product, where q i = q i /∥q i ∥ and k j = k j /∥k j ∥ are normalized features, and TopK(·) is a function and returns top K relevant positions the i-th element of P l , and the position p 1 i is the most relevant position in the Ref image to the i-th position in LR. When K > 1, it helps discover multiple correspondences, motivated by KNN [24]. For fair comparisons with other RefSR methods, we set K = 1 and exploit the most relevant position in the experiments.
Similarity-aware texture transfer. The second important sub-task in RefSR is to transfer textures from Ref images to LR images based on the matched correspondence. Most existing RefSR methods [56,48] directly swap the most relevant texture from Ref image. However, it may degrade the performance when the most relevant texture is inaccurate. To address this, we propose to improve the deformable convolution (DCN) [4,59] to transfer the texture around every position p k i of Ref images. Specifically, let ∆p k i be the spatial difference between the position p i and the k-th relevant position p k i , i.e., ∆p k i = p k i − p i . Then, we calculate a feature at the position p using modified DCN, i.e., where p j ∈ {(−1, 1), (−1, 0), . . . , (1, 1)}, s k i is the cooperative weight to aggregate the K textures from the Ref image, i.e., s k i = exp( q i · k p k i )/ j∈Pi exp( q i · k j ), w j is the convolution kernel weight, ∆p j is the j-th learnable offset of ∆P l , and m j is the j-th learnable mask of M l , which can be calculated as follows, where ω is a warping function, [; ] is a concatenation operation, Conv is convolutional layers. Sigmoid and Tanh are activation functions, r is the max magnitude which is set as 10 in default, and F l is the feature of upsampled LR images at the l-th scale.

Residual Feature Aggregation
To aggregate the multi-scale LR features at different layers and the transferred texture features, we propose a residual feature aggregation module (RFA) to perform feature fusion and extraction. As shown in Fig. 4, RFA consists of CNNs and Swin Transformer layers (STL) [25] which gain much attention in many tasks [19,2,26]. Specifically, we first use a convolution layer to fuse the LR feature F l and attention features A l , i.e., F ′ l+1 = Conv(F l , A l ), where Conv is convolutional layers. Then, we use Swin Transformer and a residual connection to extract deeper features of the LR and transferred features, where the details of STL are put in the supplementary materials. At the end of RFA, we use another convolutional layer to extract the features of STL, F l+1 = Conv(F ′ l+1 ). Based on the aggregated features F L at the last scale, we synthesize SR images with a skip connection as

Loss Function
In the training, we aim to i) preserve the spatial structure and semantic information of LR images; ii) discover more texture information of Ref images; iii) synthesize realistic SR images with high quality. To this end, we use a reconstruction loss, a perceptual loss and an adversarial loss, which is the same as [48,12]. The overall loss with the hype-parameters λ 1 and λ 2 is written as: Reconstruction loss. In order to make the SR image X SR to be close to the HR ground-truth image X HR , we adopt the following reconstruction loss where ∥ · ∥ 1 is the ℓ 1 -norm. Perceptual loss. To enhance the visual quality of SR images, the perceptual loss is widely used in SR models [56,12]. The perceptual loss is defined as: where ∥·∥ F is the Frobenius norm, and V and C are the volume and channel number of the feature maps, respectively. The function ϕ i is the i-th intermediate layer in VGG19 [35], and we use the relu5 1 layer of VGG19 in the experiment. Adversarial loss. To improve the visual quality of SR images, many SR methods [17,44] introduce GANs [7,1] which have achieved good performance for SR. Specifically, we use WGAN [1] loss as follows, where D(·) is a discriminator, P SR is the distribution of the generated SR images, and P HR is the distribution of the real data.

Experiments
Datasets. In the experiment, we consider the RefSR dataset, i.e., CUFED5 [56], which consists of a training set and a testing set. Implementation details. The input LR images are generated by bicubicly downsampling the HR images with scale factor 4. For the encoders and discriminator, we adopt the same architectures as [12]. We use a pre-trained relu1 1, relu2 1 and relu3 1 of VGG19 to extract multi-scale features. we augment the training data with randomly horizontal and vertical flipping or different random rotations of 90 • , 180 • and 270 • . Besides, we also augment the training data by randomly changing different brightness, contrast and hue of an image by using ColorJitter in pytorch. In the training, we set the batch size as 9, i.e., each batch has 9 LR, HR and Ref patches. The size of LR images is 40×40, and the size of HR and Ref images is 160×160. Following the training of [12], we set the hypeparameters λ 1 and λ 2 as 1×10 −4 and 1×10 −6 , respectively. We set the learning rate of the SR model and discriminator as 1×10 −4 . For the Adam optimizer, we set β 1 =0.9 and β 2 =0.999. We provide more detailed network architectures and training details in the supplementary material.

Comparison with State-of-the-art Methods
We compare with the SISR methods (SRCNN [6], EDSR [22], RCAN [55], SwinIR [19], SRGAN [17], ENet [33], ESRGAN [44], and RankSR-GAN [53]) and RefSR methods (CrossNet [57], SRNTT [56], SSEN [34], TTSR [48], E2ENT2 [45], and MASA [27]). For fair comparisons, the above models are trained on CUFED5 training set, and tested on CUFED5 testing set, Urban100, Manga109, Sun80 and WR-SR. In this experiment, we train our model on two cases only with reconstruction loss (denoted as '-rec'), and with all loss functions.  Quantitative comparison. We provide quantitative comparisons of SR models trained with only reconstruction loss and all losses in Tables 1 and 2, respectively. In Table 1, our model has the best PSNR and SSIM on all testing sets and significantly outperforms all SISR and RefSR models. It implies that our Transformer achieves the state-of-the-arts and good generalization performance. For the SISR setting, our method performs better than the state-of-the-art SISR method [19]. It is difficult for these SISR methods to synthesize since the highfrequency information is degraded. In contrast, our model is able to adaptively discover the useful information from a reference image on the Urban100 and Manga109 datasets even if it is a random image. For the RefSR setting, our proposed DATSR significantly outperforms all methods with the help of the cooperative transfer with deformable convolution module. In Table 2, our DATSR also achieves the much higher PSNR/SSIM values than other RefSR methods with a large margin. Our DATSR trained with adversarial loss reduces PSNR and SSIM but increases the visual quality. Still, it has the best performance over all compared methods. The above quantitative comparison results on different SR paradigms demonstrate the superiority of our Transformer over state-of-the-art SISR and RefSR methods.  Qualitative comparison. The visual results of our method are shown in Figs. 5 and 6. In these figures, our model also achieves the best performance on visual quality when trained with the reconstruction loss and all loss. These results demonstrate that our proposed method is able to transfer more accurate textures from the Ref images to generate SR images with higher quality. When trained with the reconstruction loss, our medel can synthesize SR images with sharp structure. Moreover, our method is able to search and transfer meaningful texture in a local regions even if the Ref image is not globally relevant to the input image. When trained with the adversarial loss, our model is able to restore the realistic details in the output images which are very close to the HR ground-truth images with the help of the given Ref images. In contrast, it is hard for ESRGAN and RankSRGAN to generate realistic images without the Ref images since the degradation is severely destroyed and high frequency details of images are lost. For RefSR methods, our model is able to synthesize more realistic texture from the Ref images than SRNTT [56], TTSR [48], MASA [27], and C 2 -Matching [12]. For example, in the top of Fig. 6, our model is able to recover the "window" with sharper edge and higher quality than C 2 -Matching, but other methods fail to restore it even if they have a

Further Analyses
Robustness to image transformations. We analyze the robustness of our model to different kinds of image transformations. Specifically, we use ColorJitter to augment the CUFED5 testing set by randomly change the brightness, contrast and hue of Ref images into three group: small, medium and large. The detailed settings are put in the supplementary materials. In Fig. 7, our model is more robust than C 2 -Matching [12] under different image transformations. Note that the medium and large transformations are not included during training but our model still has superior performance.
Effect on type and number of Ref images. We test our model on different Ref images, such as extreme images (i.e., may have only one color or noise without any information) and random images from different testing sets. In Fig.  8, our method has robust performance and high visual quality even if the Ref images have no useful texture information. In addition, our model has better performance when increasing #Ref images in Fig. 9. Table 3 shows the results of four similarity levels ("L1" to "L4") where L1 is the most relevant level. Our method achieves the best performance across all similarity levels.
Comparisons with multi-RefSR methods. We compare our model with multi-RefSR methods, i.e., CIMR-SR [47] and AMRSR [32]. Note that these multi-RefSR methods are trained with a collection of reference images. In Table  4, our model trained with single reference image performs better than CIMR-SR and AMRSR with many reference images, which further demonstrate the superiority of our proposed DATSR.

More Evaluation Results
Perceptual metric. We further use the perceptual metric LPIPS [52] to evaluate the visual quality of the generated SR images on the CUFED5 and WR-SR testing sets. Recently, this metric is also widely used in many methods [28,29]. In general, smaller LPIPS corresponds to the better performance for RefSR. As shown in Table 5, our model achieves smaller LPIPS than C 2 -Matching. Thus, our model generates SR images with better quality than C 2 -Matching.
User study. To further evaluate the visual quality of the SR images, we conduct the user study to compare our proposed method with previous state-of-the-art methods, including SRNTT [56], TTSR [48], MASA [27] and C 2 -Matching [12] on the WR-SR testing set. The user study contains 20 users, and each user is given multiple pairs of SR images where one is our result. Then, each user chooses one image with better visual quality. The final percentage is the average user preference of all images. In Fig. 10, over 80% of the users prefer that our results have better quality than existing RefSR methods.

Discussion on Model Size
To further demonstrate the effectiveness of our model, we also show the comparison of model size (i.e., the number of trainable parameters) with the state-of-theart model (i.e., C 2 -Matching [12]) in Table 6. Our model has a total number of 18.0M parameters and achieves PSNR and SSIM of 28.72 and 0.856, respectively. The results demonstrate that our proposed model outperforms C 2 -Matching with a large margin, although our model size is higher than this method. The part of our model size comes from the Swin Transformer in the RFA module. More discussions of other RefSR models are put in the supplementary materials.

Ablation Study
We first investigate the effectiveness of RDA and RFA in Table 7. Specifically, we replace the texture transfer method in RDA with a feature warping based on the most relevant correspondence, and replace RFA with several convolutional neural networks (CNNs). The model with feature warping or CNNs is worse than original model with RDA or RFA. Therefore, RDA is able to discover more relevant features especially when the correspondence is not inaccurate.
For RFA, our model has better performance than the directly using simple CNNs. Nevertheless, with the help of RDA, training with CNNs still outperforms C 2 -Matching with large margin. Therefore, it verifies that the effectiveness of RFA and it is able to aggregate the features at different scales. More discussions on ablation studies are put in the supplementary materials.

Conclusion
In this work, we propose a novel reference-based image super-resolution with deformable attention Transformer, called DATSR. Specifically, we use texture feature encoders module to extract multi-scale features and alleviate the resolution and transformation gap between LR and Ref images. Then, we propose reference-based deformable attention module to discover relevant textures, adaptively transfer the textures, and relieve the correspondence mismatching issue. Last, we propose a residual feature aggregation module to fuse features and generate SR images. Extensive experiments verify that DATSR achieves the stateof-the-arts performance as it is robust to different brightness, contrast, and color between LR and Ref images, and still shows good robustness even in some extreme cases, when the Ref images have no useful texture information. Moreover, DATSR trained with a single Ref image has better performance than existing Multi-RefSR methods trained with multiple Ref images.
Acknowledgements. This work was partly supported by Huawei Fund and the ETH Zürich Fund (OK).