Deep learning techniques have been very successful in several domains, like computer vision [1]}, speech recognition [2]} and natural language processing [3]}. In practice, deep neural networks (DNN) learned by some optimization algorithms such as gradient descent (GD) and its variants can generalize well to unseen data [4]}. However, deep neural networks are non-convex. The non-convex deep neural networks have been found to have large amount of global minima [5]}, while only few of them can guarantee satisfactory generalization property [6]}. Explaining why the highly non-convex model trained by a specific algorithm can generalize has become an important open question in deep learning.
