Interface between optimization and generalization.  Implicit bias builds the bridge between optimization and generalization, which has been widely studied to explain the generalization ability of machine learning models. Recent works [1]}, [2]}, [3]}, [4]}, [5]} showed that linear classifiers or deep neural networks trained by GD/SGD maximizes the margin of the separating hyperplanes and therefore generalizes well. Other works [6]}, [7]}, [8]}, [9]}, [10]} considered overparameterized neural networks in the lazy training regime where the minimizer has good generalization due to the low “complexity” of the parameter space.
In this work, we focus on specific conditions on loss functions under which we can connect optimization and generalization based on the lengths of optimization paths.
