Practical issues in recent SSL benchmarks: Although it is common to use ResNet-50 model for self-supervised learning, it is an expensive model to train and experiment with. For instance, training ResNet-50 for 200 epochs of MoCo takes at least 6 days on four RTX 2080-Ti GPUs. Most recent papers [1]}, [2]}, [3]}, [4]}, [5]} report results for 800 epochs or more which is very expensive without specialized hardware.
Further, some methods [3]}, [5]} are even slower as they forward the mini-batch through the model more than once. For instance, BYOL method forwards the images twice to calculate the symmetric loss, so 800 epochs of BYOL is equivalent to almost \(1,600\)  epochs of MoCo in terms of running time. We believe such demanding benchmarks makes this research area less accessible to smaller research labs and is not aligned with the goals of democratizing AI.
