Since most inpainting methods are based on sophisticated GANs nowadays, training the inpainting model incrementally is non-trivial. However, benefiting from various auxiliary information [1]}, [2]}, [3]}, incrementally training is flexible to improve the image inpainting.
To improve the pre-trained inpainting model (i.e., FTR in ZITS++) incrementally with holistic structures, we propose to use ZeroRA, which has been leveraged in [4]} to replace the layer normalization in the transformer. Compared to training from scratch with auxiliary information, ZeroRA can save 11 days' computations on \(256\times 256\)  resolution Places2 with 3 V100 16 GB GPUs. The idea of ZeroRA is simple. For the given input feature \(x\) , the output feature \(x^{\prime }\)  is got from adding a skip connection with function \(F\)  to \(x\)  with a zero-initialized trainable residual weight \(\alpha \)  as
\(x^{\prime }=x+\alpha \cdot F(x).\) 
