Ba and Caruana [1]} show that shallow fully connected neural networks can learn complex functions previously learned by deep neural networks, sometimes even only requiring the same number of parameters as the deep networks.
Specifically, given a well-trained deep model, they propose to train a shallow model based on the outputs of the deep model, to mimic the deep model. They show that, the shallow mimic model can achieve an accuracy as high as the deep model. However, the shallow model cannot be trained directly on the original labeled training data to achieve the same accuracy. This is also well recognized as knowledge distillation [2]}.
