On Gradient Descent Convergence beyond the Edge of Stability
Gradient Descent (GD) is a powerful workhorse of modern machine learning
thanks to its scalability and efficiency in high-dimensional spaces. Its
ability to find local minimisers is only guaranteed for losses with Lipschitz
gradients, where it can be seen as a 'bona-fide' discretisation of an
underlying gradient flow. Yet, many ML setups involving overparametrised models
do not fall into this problem class, which has motivated research beyond the
so-called "Edge of Stability", where the step-size crosses the admissibility
threshold inversely proportional to the Lipschitz constant above. Perhaps
surprisingly, GD has been empirically observed to still converge regardless of
local instability. In this work, we study a local condition for such an
unstable convergence around a local minima in a low dimensional setting. We
then leverage these insights to establish global convergence of a two-layer
single-neuron ReLU student network aligning with the teacher neuron in a large
learning rate beyond the Edge of Stability under population loss. Meanwhile,
while the difference of norms of the two layers is preserved by gradient flow,
we show that GD above the edge of stability induces a balancing effect, leading
to the same norms across the layers.