Deep learning models are always over-parameterized, that is, they have far more model parameters than the optimal solutions and the number of training samples. However, it is often found that large over-parameterized neural networks exhibit good generalization capability [1]}, [2]}, [3]}. Some studies even find that larger and more complex networks usually generalize better [4]}. This observation is in contradiction with the classical notion of function complexity [4]} and the well-known Occam's razor [6]}, which prefer simple models. What leads to the good generalization capability of over-parameterized deep learning models?
