A difficulty with training models on images in the wild is the absence of control on the distribution of images, e.g., the data likely has concepts that are dis-proportionally represented compared to others.
This means that an under-parameterized network may underfit and only learn the most predominant concepts.
For instance, studies [1]} show that even a 1 billion parameter model saturates after 32M images, and do not extract more information when trained on billion of images.
Even without these difficulties, learning the diversity of concepts in images from billions of people around the world requires significantly larger models than what is deployed for training on ImageNet scale.
