The slow dynamics from C to E is the origin of grokking. During this period, the model first moves in the \(-w\)  direction with a velocity \(v\)  over the distance \(L_1=L-h{\rm cot}\theta \) , and then moves along the boundary with a velocity \(v^{\prime }\)  over the distance \(L_2=h/{\rm sin}\theta \) . So the total time is \(t=L_1/v+L_2/v^{\prime }=(L+h{\rm tan}\theta )/(\eta _D\gamma )\) . This formula agrees with the observation that large weight decays \(\gamma \)  and/or larger decoder learning rates \(\eta _D\)  can make generalization happen faster [1]}, [2]}. Besides, the path manifests intriguing multiple descent of test loss, shown in Figure REF .
