Given a pretrained model and a masked image, we can encode the visible patches using the distributed representations, and invert these partial representations back to the pixel space to reconstruct the masked patches. Specifically, we follow the technique of deep image prior [1]} and minimize the L2 distance on the visible representations between the masked image and the reconstruction.
This reconstruction technique allows us to examine the content of the encoded features without the need to further train a new generative model.
Figure REF  shows the reconstruction result. We vary the masking ratio and compare results with supervised ViT [2]}, DINO [3]} and MAE [4]} using this inversion method. Supervised ViT is unable to produce any meaningful content. Both DINO and MAE fail to inpaint proper colors due to their use of color augmentation and normalized pixels. DINO is additionally inaccurate with spatial localities. Our result is spatially smooth and accurate in color. The inversion technique suffers when the masking ratio is very large, due to limited ability to inpaint unseen semantic areas.
