The study of model complexity inspires explorations on the effectiveness of optimization approaches.
Hanin and Rolnick [1]} use the boundary volumes of linear regions as the complexity measure of ReLU neural networks, and find that during the training, the average boundary volume is always linearly proportional to the number of neurons, irrelevant to the depth of the model. This demonstrates that deep models do not always learn more complex functions than shallow ones, the success of deep learning may be related to optimization algorithms.
Ba and Caruana [2]} suggest that the great performance of deep learning may be due to the fact that deep models are easier to train than shallow architectures using the current optimization algorithms [3]}, [1]}. This calls for further exploration of the effectiveness of optimization approaches and the relationship with model structures.
