Following CompRess [1]}, we train our self-supervised ResNet-18 model with the initial learning rate set to \(0.01\)  and multiplied by \(0.2\)  at epochs 140 and 180. We follow [2]} and add a prediction layer for the student. The hidden and output dimensions of the prediction MLP layer are set to 512. Unlike in [2]}, we do not have any projection layer since it increases the complexity of the model. We use the same set of augmentations used in [4]}, [5]}, [2]}. The results are shown in Tables REF  and REF . Our model outperforms both baselines on ImageNet nearest neighbor (NN) evaluation, ImageNet linear evaluation, and also transfer learning settings on CUB200, CARS196, and Stanford products datasets.
