Whether gradient descent methods find  global optimal solutions in the training of neural networks and how trained neural networks  generalize are two major open questions in the theory of deep learning. The non-convexity of the loss functions for neural network training may lead to sub-optimal solutions when applying gradient descent methods. It is thus relevant to understand from a theoretical point of view whether specific neural network architectures with a proper choice of learning rates for gradient descent methods can improve the optimization landscape and/or eliminate sub-optimal solutions [1]}. There is some empirical evidence that gradient descent seems to select solutions that generalize well [2]} even without any explicit regularization. Hence, it is believed that gradient descent induces an implicit regularization [3]} and characterizing the nature of this regularization is an interesting research question.
