There has been efforts to adopt such successful ideas in NLP to computer vision, but with limited success. [1]} proposed iGPT, a transformer-based autoregressive model that operates over image pixels, while [2]} trained a ViT model on denoising of images where the noise is applied at pixel level.
More recently, [3]} introduced the Masked Image Modeling loss in computer vision, where image patches are masked, and the goal is to predict the discretized label of the missing patches corresponding to their visual words as defined by a pre-trained discrete VAE [4]}.
