Implementation details:  For all experiments, we use PyTorch with SGD optimizer (momentum = \(0.9\) , weight decay = \(1e-4\) , batch size = 256) except when stated otherwise. Details about the specific architecture, epochs for training, and learning rate are described for each experiment in its respective section. We follow the evaluation protocols in [1]} for nearest neighbor (NN) and linear layer (Linear) evaluation. We use the ImageNet labels only in the setting of evaluating the learned features. To evaluate how SSL features transfer to new tasks, we perform image retrieval on different datasets like CUB200, CARS196, and Stanford Online Product. Unlike linear evaluation, the task of image retrieval does not have any hyper-parameters which allows faster and more consistent evaluation of the learned features.
