Modern computer vision neural networks are heavily parametrized: they routinely have tens or hundreds of millions of parameters [1]}, [2]}, [3]}, [4]}. This has been the key to their success for leveraging large-scale image collections such as ImageNet.
However these high capacity models tend to overfit on small, or even medium sized datasets consisting of hundreds of thousands of images.
This problem was pointed out by Oquab et al. [5]} in 2014:
