[1]} is designed based on BERT. The images are separated into patches, which are tokenized into different discrete values by training a discrete VAE [2]}. Afterward, the vision transformer are made to predict the tokens of the masked patches. The ground truth of the tokens are generated with the non-mask patches by a discrete VAE trained in the first stage.
