shown in Figure REF  (a)(b).
The reduced loss landscape reveals three things: (1) Larger initializations lead to grokking. Point A in Figure REF  corresponds to the standard initialization \((\alpha =1)\) , which has low training and test errors, hence no grokking. When increasing the weight norm from A to B, training error is seen to remain low while test error rises. To generalize, implicit or explicit regularization such as weight decay then brings the weight norm down, leading to delayed generalization (grokking) if regularization is small. (2) Larger datasets lead to de-grokking. Comparing B and C in Figure REF , C is seen to have
larger training size than B and lower test error. Larger data size \(N\)  makes the Goldilocks zone broader, reducing or eliminating grokking even for large weight initializations. (3) Critical data size can be defined. As reported in  [1]}, [2]}, we see that there exists a critical training set size below which generalization is impossible. The effective theory analysis in  [2]} only applies to algorithmic datasets, but not to other datasets with unknown optimal representations. The loss landscape analysis presented is this work can apply to all supervised-learning tasks. As shown in Figure REF  (b), the contours of constant test error are thumb-like, and the tip of the thumb determines the minimum amount of data required for generalization.
