Another line of works [1]}, [2]} focus on the implicit bias introduced by large learning rate. [1]} first proposed “catapult phase”, a regime similar to the EoS, where loss does not diverge even if sharpness is larger than \(2/\eta \) . [2]} theoretically gives a convergence analysis on the matrix factorization problem for large learning rate beyond \(2/\lambda \)  where \(\lambda \)  is the sharpness. Their results include two stages: in the first stage the loss may oscillate but never diverge, and the sharpness decreases to enter the second stage where the loss can decrease monotonically. Recently [5]} followed [1]} and provided a more detailed theoretical analysis on the sharpness along the gradient descent trajectory in an over-parameterized two-layer linear network setting under some assumptions during the training process. They also provided some heuristic explanation on EoS in more general models: gradient descent leaves and re-enters the regime where sharpness \(\lambda >2/\eta \)  repeatedly. These works mostly focus on the degree-2 setting which does not have the shaprness adaptivity and sharpness concentration properties.
