It is well known in statistics that generalization error has a "U" shape against model capacity, which is usually attributed to the bias-variance trade-off. Although this common wisdom was challenged by the observation of double descent [1]}, the "U" curve can be recovered from a double descent simply by changing the x-axis from the number of model parameters \(N\)  to the 2-norm of model parameters \(w\equiv ||\mathbf {w}||_2\)  [2]}. Although the LU mechanism may remind readers of related phenomena [3]}, [4]}, [1]}, their setups are not exactly the same as ours. More importantly, our focus and contribution is to understand grokking, a brand new generalization puzzle.
