We use open source VISSL library [1]} for our model training and implement FSDP and activation checkpointing integration for RegNet-Y model architecture. We generate a wider RegNetY-10B parameters architecture with the configuration: w_0 = 1744, w_a = \(620.83\) , w_m = \(2.52\) , depth = (2, 17, 7, 1), group_width = 1010. We use a 3-layer multi-layer perceptron (MLP) projection head of dimensions \(20280\times 8192\) , \(8192\times 8192\)  and \(8192\times 256\) . We do not use BatchNorm layers in the head. We use SyncBatchNorm in the model trunk and synchronize BatchNorm stats globally across all GPU workers. Following  [2]}, we use SwAV algorithm with same data augmentations and 6 crops per image of resolutions \(2\times 160+4\times 96\)We use lower resolution 160 instead of 224 for the bigger crop for better training speed. Our experiments (for smaller model sizes) yielded marginal difference in performance on downstream task between the two crop sizes. For the SwAV objective, we use \(16,000\)  prototypes, temperature \(\tau \)  set to 0.1, sinkhorn regularization parameter (epsilon) to \(0.03\)  and perform 10 iterations of sinkhorn algorithm. We train our model with stochastic gradient descent (SGD) momentum of \(0.9\)  using a large batch size of \(7,936\)  different images distributed over 496 NVIDIA A100 GPUs results in 16 different images per GPU. We use a weight decay of \(1e-5\) , LARS optimizer [3]}, activation checkpointing [4]} and FSDP for training the model. We use learning rate warmup [5]} and linearly ramp up learning rate from \(0.15\)  to \(9.3\)  for the first \(5,500\)  iterations. After warmup, we use cosine learning rate schedule and decay the learning rate to final value \(0.0093\) . We train on 1 billion images in total leading to 126K training iterations. We share details about other smaller variants of SEER model in Appendix Table REF .
