Meta Transferring for Deblurring

Most previous deblurring methods were built with a generic model trained on blurred images and their sharp counterparts. However, these approaches might have sub-optimal deblurring results due to the domain gap between the training and test sets. This paper proposes a reblur-deblur meta-transferring scheme to realize test-time adaptation without using ground truth for dynamic scene deblurring. Since the ground truth is usually unavailable at inference time in a real-world scenario, we leverage the blurred input video to find and use relatively sharp patches as the pseudo ground truth. Furthermore, we propose a reblurring model to extract the homogenous blur from the blurred input and transfer it to the pseudo-sharps to obtain the corresponding pseudo-blurred patches for meta-learning and test-time adaptation with only a few gradient updates. Extensive experimental results show that our reblur-deblur meta-learning scheme can improve state-of-the-art deblurring models on the DVD, REDS, and RealBlur benchmark datasets.


Introduction
Dynamic scene deblurring aims to recover sharp images from blurred ones caused by camera shakes or moving objects. It is challenging to recover images with such blur since dynamic scene blur is often non-uniform and directional. Using methods with some prior assumptions [4,20,33] or blur uniformity [7] can only achieve limited performance. Existing learning-based methods for single image deblurring [6,12,23,32,34,35,36] or video deblurring [1,8,14,15,18,21,26,29,38,39] could obtain better results by training these models in a supervised manner, learning to deblur images from training with blurred and Figure 1: Illustration of the proposed reblur-deblur meta-transferring scheme. We utilize the reblurring model to generate pseudo-blurred and pseudo-sharp pairs to facilitate metalearning for the test-time adaptation. Note that pseudo-sharp patches are selected from blurred video frames.
sharp image pairs. Although significant progress has been made using these supervisedlearning methods, they did not consider exploiting rich internal information in test data and tend to be sub-optimal during testing. Therefore, we aim to build a domain adaptation strategy that allows test-time adaptation to improve existing deblurring methods. Few deblurring works have been done using meta-learning since ground truth is not accessible during testing. Chi et al. [5] proposed a self-supervised meta-auxiliary learning strategy that meta-trains and meta-tests the model by restoring the input blurred image. However, it could only achieve sub-optimal adaptation since blur self-restoration may not help to deblur. By contrast, we observe that a scene appearing in multiple frames in a video often has different blurring degrees. Some local patches are relatively sharp, whereas some are relatively blurred. Therefore, we propose to leverage these patches to synthesize sharp and blurred pairs to enable meta-learning for dynamic scene deblurring during testing without access to the ground truth like [5] but in a different approach. More specifically, this work proposes a reblur-deblur meta-transferring scheme to generate pseudo-blurred and pseudo-sharp pairs to achieve test-time adaptation in meta-learning, as shown in Fig. 1. Our reblurring model can transfer blurred patterns homogeneous to the task to pseudo-sharp patches selected to synthesize pseudo-blurred patches, which can serve as pseudo-sharp-and-blurred pairs as the support set in meta-learning. Our contributions are three-fold. First, we propose a novel reblur-deblur meta-transferring scheme that can generate pseudo-blurred and pseudo-sharp pairs for meta-learning. Second, the proposed scheme facilitates meta-learning for dynamic scene deblurring without extra training data needed. Third, extensive experimental results show that our method improves the performance of existing deblurring models on various datasets, including DVD [28], REDS [17], and RealBlur-J [ 29,38,39]. The main difference is the number of input frames. Video deblurring utilizes consecutive frames to help deblurring, which can obtain more information than image deblurring. However, most existing methods fail to consider internal information from testing data and ignore the problem of domain gap between training and testing sets, which would be sub-optimal during testing.

Meta learning on low-level vision tasks
Previous low-level vision methods commonly train a generic model using training data without considering the rich internal information in test data. Recently, a few methods developed for low-level vision tasks have been proposed [5,24,27]. Super-resolution (SR) methods [24,27] utilized MAML [9] to efficiently adapt to test images for better SR results, where they used the patch-recurrence property intrinsic to an image itself, downsampling images to generate low-and high-resolution training pairs during testing. Since finding such an intrinsic characteristic from a blurred image is challenging, Chi et al. [5] proposed a meta-auxiliary learning method, where they meta-trained the deblurring model to learn the input blurred image in a self-supervised manner to help deblur. Thus, it can enable the model to adapt to test blurred images without using their ground-truth counterparts. Nevertheless, restoring the input blurred image and deblurring it essentially works in a contradictory fashion, which may only produce sub-optimal results. Unlike meta-auxiliary learning for deblurring proposed in [5] requires attaching additional layers to train the model, our work builds a reblurring model that can transfer blur patterns from relative-blurred to relative-sharp patches. It can synthesize pseudo-blurred and pseudo-sharp pairs for deblurring models without modifying them to facilitate meta-learning.

Reblurring for deblurring.
Some recent deblurring works [3,19,22,37] were proposed to generate additional blurred images to train the models. Zhang et al. [37] used an extra real blurred dataset to produce synthetic blurred images using generative adversarial networks. Park et al. [22] used a recurrent method to produce images with various blurring degrees. However, the generated data for training may not help upon testing. To enable test-time adaptation, Nah et al. [19] utilizes a reblurring loss to constrain deblurring results using a pre-trained reblurring model based on the observation that clean images are hard to reblur. Chen et al. [3] chose to self-supervise a model for deblurring, similar to [5]. In contrast, we leverage the internal information in testing videos to generate domain-aware pseudo-blurred and pseudo-sharp pairs, enabling meta-learning better than [3,5].

Overview
The paper proposes a reblur-deblur meta-transferring scheme for test-time adaptation, composed of meta-training and meta-testing phases.
In meta-training, we consider deblurring each video V i in the training set as a task, where V i contains N blurred frames B n and their sharp counterparts S n (n = {1, 2, ..., N}). Figure 2 depicts the proposed reblur-deblur meta-transferring training scheme, where the inner update trains a deblurring model on the support set generated by the reblurring model, and the Figure 2: Illustration of our reblur-deblur meta-transferring training scheme. In the inner update, we use an adapted reblurring model to generate a domain-aware support set, enabling meta-learning to update the deblurring model. In the outer update, we evaluate the adapted models by query set to obtain meta-learned weights θ RB and θ DB .
outer update trains the deblurring and reblurring models adapted to the query set. Like [5], not needing the ground truth at inference as well, we instead use the proposed reblurring model to generate M pseudo-blurred patchesB Ω m from pseudo-sharp patchesS Ω m selected from {B n } N n=1 . Based on our observation that a scene often has various blurring degrees in multiple frames in a blurred video, we can pick relatively sharp patches in blurred frames to be pseudo-sharp patches. To choose sharp patches, we propose to measure the blurring degree by a self-shift method, and those with the least blur are selected. These M pseudoblurred and pseudo-sharp pairs {B Ω m ,S Ω m } M m=1 are used as the support set. For the query set, we use the whole V i = {B n , S n } N n=1 to train the reblurring and deblurring models. In meta-testing, given any testing video V test i , we generate the support set as in the metatraining phase, not needing the ground truth. Next, we explain how to create the support set for meta-learning in Sec. 3.2 and the architecture of the reblurring model in Sec. 3.3. At last, Sec. 3.4 details the proposed reblur-deblur meta-transferring scheme for test-time adaptation.

Generation of the Support Set
To enable meta-learning for the test-time adaptation, we generate pseudo-blurred and pseudosharp pairs {B Ω m ,S Ω m } M m=1 from each blurred video. These pseudo-sharp patchesS Ω m are fed into a reblurring model to generate pseudo-blurred patchesB Ω m . The support set contains To search for relatively sharp patches in a blurred video, we propose to measure the blurring degree by a self-shift method. Based on our observation, sharp images usually have stronger edges with larger gradients than blurred images. Fig. 3 illustrates the self-shift method, where four images are generated by shifting the input image one pixel to the right, top, top-right, and top-bottom, in four directions. We obtain a self-shift score by averaging four PSNR values calculated by the four shifted images with the input image as the reference. A higher score indicates that the shifted images are similar to their original image ( Fig. 3 (a)), implying it is blurred. In comparison, a lower score represents that the input image is sharp. We randomly select M locations to crop image patches, where the relatively sharp patches are chosen with the lowest shift scores among all the colocated patches in all the input video frames, shown in Fig. 3 (c). Finally, we feed {S Ω m } M m=1 into the reblurring model to generate pseudo-blurred patches {B Ω m } M m=1 to generate the support set. , height, embedded feature height, width, embedded feature width, and channel dimensions, respectively. Inspired by [2], we propose a cross-frame fusion block (CFFB), containing regional self-attention (RSA), cross self-attention (CSA), and local self-attention (LSA) [2]. The CFFB divides F into non-overlapping regional tokens with a window size of r as F ∈ R T ×N×D , where N = H W r 2 and D = C × r 2 denote the number of tokens and dimensions. To reduce the channel dimensions, we use an embedding layer to embed F into F ∈ R T ×N×D , where D = 2C. We then feed F into RSA [2], which performs self-attention on regional tokens to generate F RSA ∈ R T ×N×D , where T is regarded as the number of batches. To utilize cross-frame information, we propose CSA, which considers the center frame patch tokens

Reblurring Model
where W q , W k , and W v ∈ R D ×D denote the embedding weights of the query, key, and value. We concatenate the output features followed by a linear layer with dimensions R 5D ×D to generateF CSA t ∈ R N×D as After CSA, we replace the F RSA t in F RSA withF CSA t to obtain a new F RSA ∈ R T ×N×D , wherẽ F CSA t has fused the neighboring information. Lastly, we feed F RSA and initial input features F into LSA to obtain F LSA ∈ R T ×N×D like [2]. After four CFFBs, the middle frame patch tokens F LSA t ∈ R N×D are fed into a linear layer to increase channel dimensions from D to D and folded into F O ∈ R H ×W ×C followed by a decoder to generate the pseudo-blurred patch B Ω m ∈ R H×W ×3 .
The reblurring model needs to be pre-trained, where we use blurred and sharp image pairs in the training set with the Charbonnier loss [13] and adversarial loss [10] derived from an addional discriminator, i.e., five consecutive sharp frames as input and the middle blurred counterpart as the ground truth. To address the domain gap of the pre-trained reblurring model, we also update the reblurring model and discriminator during testing. Therefore, we generate realistic pseudo pairs {B Ω m ,S Ω m } M m=1 as the support set by transferring blurred patterns at inference to enable meta-learning.

Reblur-Deblur Meta-Transferring
Here, we introduce the meta-learning process for the reblurring and deblurring models. Meta training: We consider deblurring each video V i in the training set as a task, where V i = {B n , S n } N n=1 are available during training. We initialize the reblurring model (parameterized by θ RB ), discriminator (θ D ), and deblurring model (θ DB ) with their pre-trained weights. For each video V i , we first initialize the reblurring, discriminator, and deblurring weightsθ RB i , θ D i , andθ DB i by θ RB , θ D , and θ DB , respectively. As shown in Fig. 2, the meta-process can be split into two stages: one is the inner update to obtain the reblurring and deblurring models using the support set, and the other is the outer update to evaluate the ability of adapted models by the query set.
In the inner update, we first update the reblurring model to generate the support set for the deblurring model. We choose M pseudo-sharp patches {S Ω m } M m=1 from {B n } N n=1 by the self-shift method to generate pseudo-blurred patches {B Ω m } M m=1 from the reblurring model where the five consecutive frame patches from the pseudo-sharp patch are concatenated as the input. To generate realistic and homogenous blurred patterns, we use the discriminator parameterized byθ D i to calculate the adversarial loss L adv . Here, the pseudo-blurred patch B Ω m is considered a faked blurred image, and we select a real blurred patch by the self-shift method with a high score. Additionally, we add a cycle loss based on cycle consistency from the deblurring model f DB (·) parameterized by θ DB as where L is the loss function adopted in the deblurring model. It represents that deblurring the generated pseudo-blurred patch by the reblurring model should obtain its corresponding pseudo-sharp. For updating the reblurring model, we first update the discriminatorθ D i by L adv and then update the reblurring model by the total reblurring loss as L RB = L adv + λ L cycle , where we set λ = 0.01 to focus more on the blur-transferring ability of the disciminator, i.e.,θ where L is the loss function used by the deblurring method, and β is its learning rate.
In the outer update, we verify the adaptability of adapted models for each video V i . We utilize all the blurred and sharp images {B n , S n } N n=1 in V i as the query set to perform metaupdate based on the gradient calculated using the query set as where L char is the Charbonnier loss [13]. Since we regard deblurring on each video V i as a task, the meta-training process is repeated on all the available training videos until the model converges. Through Eq. 7, we can obtain meta-learned weights θ RB and θ DB , more transferable to a new task (video) compared to the original pre-trained weights. Meta testing: Given any testing video V test i = {B n } N n=1 , we take meta-learned reblurring (θ RB ), deblurring (θ DB ) weights and pre-trained discriminator (θ D ) weights as initialization. Similar to the meta-training phase, we choose M pseudo-sharp patches {S Ω m } M m=1 from {B n } N n=1 by the self-shift method and iteratively update the reblurring model using Eq. 5 to obtain the adapted reblurring model (θ RB test ). Next, we use the reblurring model (θ RB test ) to generate the support set to update the deblurring model by Eq. 6. Lastly, we deblur the whole testing video V test i by the adapted deblurring model (θ DB test ). Because our support set is generated by pseudo-blurred and pseudo-sharp patches {B Ω m ,S Ω m } M m=1 , we can enable metalearning on the deblurring task without the ground truth. The detailed training and testing processes are shown in the supplementary. Table 1: Evaluation results on three datasets and four SOTA deblurring models. "Baseline" means the deblurring results obtained using the original models pre-trained on GoPro. "Meta" means the results using "Baseline" with our reblur-deblur meta-transferring scheme.

Experiments
In this section, we evaluate the effectiveness of the proposed method. We first explain datasets and implementation details in Sec. 4.1. We then demonstrate the quantitative and qualitative results in Sec. 4.2. In the end, we conduct ablation studies in Sec. 4.3.

Datasets and Implementation Details
We trained all the compared models on the GoPro [16] training set, containing 22 videos, totally having 2, 103 training blurred and sharp image pairs in both the pre-training and meta-training stages. At the meta-testing time, we use three datasets to demonstrate the effectiveness of the proposed test-time adaptation scheme, including DVD [28] testing set (10 videos, having 1,000 images), REDS [17] validation set (30 videos, having 3,000 images), and RealBlur-J [25] testing set (50 videos, having 980 images). First, we pre-trained the reblurring model on the GoPro training set with a batch size of eight for 1, 000 epochs by Adam optimizer [11], where the initial learning rate is set to 10 −4 , and the decay is 10 −8 with the cosine annealing strategy. We adopted random cropping, flipping, and rotating for data augmentation. Next, for meta-training, we randomly cropped 256 × 256 patches for the support and query sets. Note that the support set contains pseudo-sharp patches selected from these cropped patches and then pseudo-blurred patches generated by the pseudo-sharps using the reblurring model. We set the learning rate α to 10 −6 in the reblurring model and β to 2.5 × 10 −6 in the deblurring model. All methods are tested on an Nvidia A5000 GPU.

Experimental Results
Quantitative Analysis: We employed our scheme on four different state-of-the-art deblurring models, including MIMOUnet+ [6], MPRNet [34], Restormer [35], and CDVD-TSP [21]. The first three models are for image deblurring, and the last is for video deblurring.
In Table 1, we compare the results obtained with or without the proposed reblur-deblur metatransferring scheme. As can be seen, our scheme can improve existing deblurring methods The authors from the universities in Taiwan completed the experiments on the datasets.  by 0.27dB, 0.29dB, and 0.28dB in PSNR on average on the DVD, REDS, and RealBlur-J datasets, respectively, indicating that it achieves the test-time adaptation for better performance.
Qualitative Analysis: Fig. 5 shows visual comparisons of results obtained using the original state-of-the-art methods with or without using the proposed reblur-deblur meta-transferring scheme on public benchmark datasets. As shown, using our meta-transferring scheme restores the blurred content better, indicating that the proposed approach synthesizing images with homogenous blur to each task at inference time for test-time adaptation can effectively boost the deblurring performance.

Ablation Studies
This section analyzes the ablation studies we conducted for the proposed approach. Table 2 shows the effectiveness of different loss terms. Next, we compared meta-testing with finetuning and different numbers of pseudo pairs included in the support set, shown in Fig. 6.
Analyses of the adopted losses in reblurring model: Our approach adopts the adversarial and cycle losses to update the reblurring model, as in Eq. 5. In Table 2, we analyze the effects of using these losses for deblurring on the DVD test set. "Baseline" here denotes the performance using the original models. "Cycle loss" and "GAN loss" mean updating the reblurring model with either the cycle loss or adversarial loss used in the inner update. As observed, using either the cycle loss or GAN loss improves the performance, and adopting both losses leads to more performance gain. Meta-testing and number of pseudo pairs used: Fig. 6 compares our meta-transferring scheme with fine-tuning, which means directly fine-tuning the models with the pseudo pairs instead of meta-learning. It shows that adopting our scheme outperforms merely fine-tuning for all the state-of-the-art methods on the benchmark datasets. We can also observe that including more pseudo pairs in the support set can increase the performance gain but with diminishing marginal benefits since selecting more pseudo-sharp patches may include ones with much blur and harm the performance. In our experiment, M is set to 20 for DVD and RealBlur-J and 10 for REDS. We use more pairs in DVD [28] and RealBlur-J [25] since they have less blur for finding good pseudo-sharp patches to help test-time adaptation. Computation cost and limitation: For meta-testing, each video needs to run the inner update for ten iterations on the REDS dataset and twenty on the DVD and RealBlur-J datasets. Take MIMO-UNet+ as an example. We need additional 0.1 seconds on average for inferencing one image. A limitation of the proposed scheme is that when we deal with a strongly blurred video containing few relative-sharp patches, the reblur-deblur meta-transferring process would be less effective, possibly leading to poor performance.

Conclusion
This paper proposed a novel reblur-deblur meta-transferring scheme to facilitate meta-learning without ground truth at inference time. We combine a reblurring model to generate pseudoblurred patches from selected pseudo-sharp patches as the support set, enabling meta-learning to update deblurring models during testing. Extensive experiments have shown that our method can improve existing deblurring models on benchmark datasets, including DVD, REDS, and RealBlur-J.