ResNet-18 experiments: We train a ResNet-18 model from scratch for 200 epochs using our ISD method. This experiment takes at least 4 days on two RTX 2080-Ti GPUs. For baselines, we re-implement BYOL in PyTorch from the details provided in [1]} and use the official implementation of MoCo-v2 [2]}, [3]}. For a fair comparison with our method and MoCo, for BYOL, we use a non-symmetric loss, a small batch size (256), and train for 200 epochs. We use following details for our re-implementation of BYOL: momentum for the teacher (\(0.99\) ), hidden units in the MLP for projection and prediction layers (1024), and output embedding dimension (128).
