Target alignment in truncated kernel ridge regression
Kernel ridge regression (KRR) has recently attracted renewed interest due to
its potential for explaining the transient effects, such as double descent,
that emerge during neural network training. In this work, we study how the
alignment between the target function and the kernel affects the performance of
the KRR. We focus on the truncated KRR (TKRR) which utilizes an additional
parameter that controls the spectral truncation of the kernel matrix. We show
that for polynomial alignment, there is an \emph{over-aligned} regime, in which
TKRR can achieve a faster rate than what is achievable by full KRR. The rate of
TKRR can improve all the way to the parametric rate, while that of full KRR is
capped at a sub-optimal value. This shows that target alignemnt can be better
leveraged by utilizing spectral truncation in kernel methods. We also consider
the bandlimited alignment setting and show that the regularization surface of
TKRR can exhibit transient effects including multiple descent and non-monotonic
behavior. Our results show that there is a strong and quantifable relation
between the shape of the \emph{alignment spectrum} and the generalization
performance of kernel methods, both in terms of rates and in finite samples.