Many works tried to understand how simple gradient-based methods can optimize complicated neural network objectives. However, recently some empirical observations show that optimization for deep neural networks may operate in a more surprising regime. In particular, [1]} observed that when running gradient descent on neural networks with a fixed step-size \(\eta \) , the sharpness (largest eigenvalue of the Hessian) of the training trajectory often oscillate around the stability threshold \(2/\eta \)The value \(2/\eta \)  is called the stability threshold, because if the objective has a fixed Hessian, the trajectory will become unstable if the largest eigenvalue of the Hessian is larger than \(2/\eta \) ., while the loss still continues to decrease in the long run. This phenomenon is called “edge-of-stability” and has received a lot of attentions.
