Our training framework follows the original Vision Transformer as outlined by [1]} with an augmentation that samples image patches before being passed to the ViT. Figure REF  provides an overview of our approach. Each image is first embedded via a patch embedding layer (a single convolution layer) that downsamples the spatial region and increases the number of channels. Then, these image patches (i.e., tokens) are passed into the patch sampling block that determines how many and which patches to keep.
<FIGURE>