Dependence of grokking on training data size Another important observation in  [1]} is that grokking happens faster for larger training size. Our landscape analysis can also explain the data size dependence. In Figure REF , we show the contours (training loss = 0.02) for different training sizes (25, 35, 45, 55). The contours of training size 45 and 55 both connect to the green star, meaning that generalization will eventually happen. However, the slopes of the contours are different, i.e., \(\theta _{55}<\theta _{45}\) . Since \(t=(L+h{\rm tan}\theta )/(\eta _D\gamma )\)  increases as \(\theta \)  increases, we have \(t_{55}<t_{45}\) , i.e, more training data leads to faster grokking. For training size 35 and 25, the contours do not connect to the green star, so generalization will not happen, no matter how long the training will be run.
