blackFormally, built upon our conference version [1]}, this paper proposes an improved ZeroRA based Incremental Transformer Structure (ZITS++) inpainting framework.
Our ZITS++ has novel components of Transformer Structure Restorer (TSR), Simple Structure Upsampler (SSU), Masking Positional Encoding (MPE), Structure Feature Encoder (SFE), and Fourier CNN Texture Restoration (FTR).
blackGiven the corrupted image, ZITS++ has this workflow:
TSR first recovers the structural priors in 256\(\times \) 256 of the input image. We upsample these priors by SSU and further take them into SFE for structural features, which are finally leveraged by the CNN-based FTR for restoring textures trained by the ZeroRA strategy.
Specifically,
the TSR aims at learning a structure restorer. It is composed of alternating axial [2]} and standard attention blocks to balance performance and efficiency. Note that our TSR can achieve much better structure recovery compared with CNNs [3]}, [4]}.
blackGiven restored L-Edges, E-NMS is leveraged to eliminate ambiguous blur near the boundaries. Then the output grayscale priors are upsampled by SSU which is consisted of 4-layer CNN to meet the desired resolution.
blackThen, the gated convolutions [5]} based SFE encodes prior features and transfers them to a pre-trained FTR with ZeroRA.
MPE is proposed to express both distances and directions from unmasked regions to masked ones.
