While these results are promising, we currently lack theoretical guarantees for early stopping in this setting. It is possible that
our theoretical results could lead to new insights into early stopping
for assessing VI due to the intimate connection between kernel ridge regression and early stopping algorithms [1]}.
In fact, if the eigenvalues of the NTK matrix at the full model initialization decay in a sufficiently fast rate, early stopping of the
reduced model training should give an as good estimate of the reduced model. However, analyzing early stopping in this setting requires characterizing the spectrum of the NTK with the full model initialization,
whereas most spectral properties of the NTK have been developed under the assumption of a random initialization
[2]}, [3]}.
Better understanding the NTK spectrum after full-model initialization in the future could provide new insights into fast algorithms for VI estimation.
