The phenomenon of gradient descent on Edge of Stability (EoS) was first formalized and empirically demonstrated in [1]}. They show that the loss can non-monotonically decrease even when the sharpness \(\lambda > 2/\eta \) . The non-monotone property of the loss has also been observed in many other settings [2]}, [3]}, [4]}, [5]}, [6]}, [7]}.
