Many works have shown the benefits of training large models on the quality of the resulting features [1]}, [2]}, [3]}.
Training large models is especially important when pretraining on a large dataset, where a model with limited capacity will underfit [4]}.
This becomes even more important when training with unsupervised learning.
In that case, the network has to learn features that capture many aspects of the data, without being guided by a narrow output space defined by the manual annotation.
To scale architecture size, all combinations of increasing width and depth have been explored in the self-supervised learning literature.
Kolesnikov  [5]} demonstrated the importance of wider networks for learning high-quality visual features with self-supervision,
Further, Chen  [6]} achieved impressive performance with deeper and wider configurations.
The largest models trained for each algorithm vary a lot, with architecture that are both deeper and wider, such as ResNet-50-w5, ResNet-200-w2 or ResNet-152-w3.
More generally, a large body of work is dedicated to building efficient models with large capacity [7]}, [2]}, [9]}, [3]}.
Of particular interest, the RegNet model family [1]} achieves competitive performance on standard image benchmarks.
while offering an efficient runtime and memory usage making them a good candidate for training at scale.
In our work, we build up on the existing work and explore all 4 dimensions (depth, width, input resolution, compound) of scaling the RegNet architecture in Sec. REF .
