Montufar et al. [1]} provide an explanation for the depth efficiency. They suggest that the intermediary layer of a deep model is able to map several pieces of the inputs into the same output. As the number of layers increases, the layer-wise compositions of the functions re-use lower-level computation exponentially. This allows deep models to compute highly complex functions, even with relatively fewer parameters.
