Towards our goal of scaling model size and pushing the limits further in self-supervised learning, we target training a 10B parameters dense model, which, to the best of our knowledge, is the largest densewhere every input is processed by every parameter, as defined in  [1]} computer vision model (contrary to the model in  [1]} which is a “sparse” model). Following studies [3]}, we explore RegNet  [4]} (a ConvNet) architecture which has demonstrated promising model size scaling without any signs of saturation in performance. Further, since the largest model defined in RegNet family is a \(1.5\) B parameters model, we explore several strategies to increasing the architecture size to 10B parameters.
