Self-supervised learning requires no annotations/labels for training models which means we can train large models from trillions of images at internet scale. Following previous works [1]}, [2]} which demonstrated the possibility to train high-quality visual features from billions of internet images using self-supervised learning, we consider 3 axes of scaling: 1) data size, 2) model size, and 3) data and model size.
