Since the transformer shows an ability to get expressive global structure recoveries [1]}, we leverage the capacity of the transformer to restore holistic structures in a relatively low resolution.
For the input blackmasked priors \(\mathbf {P}\) , and mask \(\textbf {M}\)  in \(256\times 256\) , we firstly downsample them with three convolutions to reduce computation for attention learning. Such simple convolutions can also inject beneficial convolutional inductive bias for vision transformers compared with the patch based MLP embedding [2]}.
Then we add a learnable absolute position embedding to the feature at each spatial position and get \(\mathbf {X}\in \mathbb {R}^{h\times w\times c}\)  for the input to attention layers, where both the height \(h\)  and width \(w\)  are 32, while \(c=256\)  is the feature channel.
