Sentiment analysis of text We look for grokking using LSTMs [1]} for IMDb dataset [2]}. Similar to Eq. (REF ), we reduce training and test losses to depend on only the weight norm \(w\)  and data size \(N\) . We show the reduced training and test error in Figure REF  (a)(b). For large data size say the full dataset, training and test errors have similar "U" shapes In principle, reduced training losses should be non-increasing ("L"), but optimization issues may occur for too large initializations [3]}., so one cannot create grokking via the "LU" mechanism. For small data size, say 1k, however, the mismatch between training and test errors makes it possible to create grokking via large initializations. In Figure REF  (c), we initialize weights larger (\(\alpha =6\) ) with weight decay 1, overfitting is complete within \(10^2\)  steps, but generalization does not start until around \(10^3\)  steps. Note that the generalization "jump" is not as sharp as on algorithmic datasets [4]} or MNIST, but at least generalization is delayed here. By contrast, if we use the standard initialization (\(\alpha =1\) ) with no weight decay, generalization happens early on during training, and does not improve much after overfitting.
