Implementation Details. We use the original ViT-base [1]} as the backbone architecture without the layer scale technique [2]}. The class attention follows the original design in [2]} with a default of two transformer blocks and a layer scale hyper-parameter of 0.1. We train our model using the AdamW optimizer [4]} with a batch size of 2048, an initial base learning rate of 1.5e-4, and a weight decay of 0.1. The exponential averaging weight for the momentum encoder is initialized to 0.996 and increased to 1.0 following a cosine schedule. The default augmentation is random resized cropping and random flipping. All models are trained for 300 epochs.
