The BEiT method, as proposed by [1]}, relies on the discrete VAE tokenizer from DALL-E, which has been pretrained on a large weakly supervised dataset.
Since we want to study whether it is possible to pre-train models solely on small datasets, or non object-centric ones, we replace the DALL-E tokenizer by a simple alternative.
To this end, we consider different simple alternatives to discretize images at the patch level without any pre-training.
Each of these techniques is applied on each patch independently, making them relatively lightweight and more efficient than the original tokenizer considered in BEiT.
