Based on this phenomenon, Ba and Caruana [1]} conjecture that the strength of deep learning may arise in part from a good match between deep architectures and current training algorithms. That is, compared with shallow architectures, deep architectures may be easier to train by the current optimization techniques. Moreover, they propose that when it is able to mimic the function learned by a deep complex model using a shallow model, the function learned by the deep model is not really too complicated to learn. This study suggests that there may be a big gap between the practical effective complexity of a deep learning model and the theoretical bound of its expressive capacity. We call this the high-capacity low-reality phenomenon.
