We start with the simple observation - as illustrated in Figure REF  - that under a generic random initialization, the generalization error for both linear models and neural networks is positively correlated with the length of the optimization pathFor discrete iterations, we use the sum of the distance between every two consecutive iterations to represent the length of the optimization path (See details in Appendix REF ). after convergence. More precisely, short optimization paths are associated with low generalization error. Here, by length we refer to the trajectory or path length of the parameter evolution during trainingThis differs from the notion of “length of time” used in early-stopping type of algorithms.
These empirical findings suggest that the trajectory length could be used to connect optimization and generalization.
Intuitively, the length of the optimization path can be viewed as a kind of capacity control, and a short path indicates low “complexity”.
In other words, a general condition that guarantees a short optimization path can be used to induce good generalization properties, provided that the training error is low.
For this purpose, inspired by the theory in [1]} that Łojasiewicz gradient inequality (LGI) induces an explicit bound for the gradient flow path, we consider Uniform-LGI (Definition REF ) which is a modified version of LGI and plays a critical role in obtaining a length estimate for the gradient flow trajectory. We use this estimate to derive length-based generalization bounds for loss functions satisfying the Uniform-LGI.
<FIGURE>