Regarding the above question, one plausible explanation is that optimization algorithms implicitly bias training [1]}. That is, the optimization algorithm tends to drive parameters to certain kinds of global minima which generalize well, although no explicit regularization is enforced. Recently, exciting results have been shown for vanilla gradient descent. For linear classifier, GD is proved to converge to minimum \(L_2\)  norm solution for squared loss and max-margin solution for logistic loss [2]}.
[3]} extend the result for GD to deep linear neural networks. A remarkable progress is the work [4]}, which proves GD maximizes the margin of homogeneous deep neural networks.
