Early stopping and non-parametric regression: An optimal data-dependent stopping rule

Abstract: The strategy of early stopping is a regularization technique based on choosing a stopping time for an iterative algorithm. Focusing on non-parametric regression in a reproducing kernel Hilbert space, we analyze the early stopping strategy for a form of gradient-descent applied to the least-squares loss function. We propose a data-dependent stopping rule that does not involve hold-out or cross-validation data, and we prove upper bounds on the squared error of the resulting function estimate, measured in either the $L^2(P)$ and $L^2(P_n)$ norm. These upper bounds lead to minimax-optimal rates for various kernel classes, including Sobolev smoothness classes and other forms of reproducing kernel Hilbert spaces. We show through simulation that our stopping rule compares favorably to two other stopping rules, one based on hold-out data and the other based on Stein's unbiased risk estimate. We also establish a tight connection between our early stopping strategy and the solution path of a kernel ridge regression estimator.

\section{Introduction}

The phenomenon of overfitting is ubiquitous throughout statistics.  It
is especially problematic in nonparametric problems, where some form
of regularization is essential to prevent overfitting. In the problem
of nonparametric regression, the most classical form of regularization
is that of Tikhonov regularization, where a quadratic smoothness
penalty is added to the least-squares loss. An alternative and
algorithmic approach to regularization is based on early stopping of
an iterative algorithm, such as gradient descent applied to the
unregularized loss function. The main advantage of early stopping for
regularization, as compared to penalized forms, is lower computational
complexity.

The idea of early stopping has a fairly lengthy history, dating back
to the 1970's in the context of the Landweber iteration.  (For
instance, see the paper by Strand~\cite{Strand74}, with follow-up work
by Anderssen and Prenter~\cite{AndrerssenPrenter81} as well as
Wahba~\cite{Wahba87}.) Early stopping has also been widely used in
neural networks (e.g.,~\cite{MorganBourlard90}), in which stochastic
gradient descent is used to estimate the network parameters. Past work
provided intuitive arguments for the benefits of early stopping. It
was argued that each step of an iterative algorithm will reduce bias
but increase variance, so early stopping ensures the variance of the
estimator is not too high. However, prior to the 1990s, there had been
little theoretical justification for these claims.  A more recent line
of work has provided theoretical justification for various types of
early stopping, including boosting algorithms
(e.g.,~\cite{BartlettTraskin07, BuhlmannYu03, Freund97, Jiang04,
  Mason99, Yao05,ZhangYu05}), greedy methods~\cite{Barron08}, gradient
descent over reproducing kernel Hilbert spaces
(e.g.~\cite{Bauer07,Caponnetto06,CaponnettoYao06,DeVito10, Yao05}),
the conjugate gradient algorithm~\cite{BlanchKram10}, and the power
method for eigenvalue computation~\cite{Orecchia11}. Most relevant to
our work is the work of B\"{u}hlmann and Yu~\cite{BuhlmannYu03}, who
derived optimal mean-squared error bounds for $L^2$-boosting with
early stopping in the case of fixed design regression.  However, these
optimal rates are based on an ``oracle'' stopping rule, one that
cannot be computed based on the data.  Thus, their work left open the
following natural question: is there a data-dependent and easily
computable stopping rule that produces a minimax-optimal estimator?

The main contribution of this paper is to answer this question in the
affirmative for a certain class of non-parametric regression problems,
in which the underlying regression function belongs to a reproducing
kernel Hilbert space (RKHS).  In this setting, a standard estimator is
the method of kernel ridge regression (e.g.,~\cite{Wahba}), which
minimizes a weighted sum of the least-squares loss with a squared
Hilbert norm penalty as a regularizer.  Instead of a penalized form of
regression, we analyze early stopping of an iterative update that is
equivalent to gradient descent on the least-squares loss in an
appropriately chosen coordinate system.  By analyzing the mean-squared
error of our iterative update, we derive a data-dependent stopping
rule that provides the optimal trade-off between the estimated bias
and variance at each iteration. In particular, our stopping rule is
based on the first time that a running sum of step-sizes after $t$
steps increases above the critical trade-off between bias and
variance.  For Sobolev spaces and other types of kernel classes, we
show that the function estimate obtained by this stopping rule
achieves minimax-optimal estimation rates in both the empirical and
population norms.  Importantly, our stopping rule does not require the
use of cross-validation or hold-out data.

In more detail, our first main result (Theorem~\ref{ThmMain}) provides
bounds on the squared prediction error for all iterates prior to the
stopping time, and a lower bound on the squared error for all
iterations after the stopping time.  These bounds are applicable to
the case of fixed design, where as our second main result
(Theorem~\ref{ThmRandDesign}) provides similar types of upper bounds
for randomly sampled covariates.  These bounds are stated in terms of
the squared $\ensuremath{L^2(\mathbb{P})}$ norm, as opposed to the prediction error or $\ensuremath{{L^2(\mathbb{P}_n)}}$
(semi)norm defined by the data.  Both of these theorems apply to any
reproducing kernel, and lead to specific predictions for different
kernel classes, depending on their eigendecay.  For the case of low
rank kernel classes and Sobolev spaces, we prove that our stopping
rule yields a function estimate that achieves the minimax optimal rate
(up to a constant pre-factor), so that the bounds from our analysis
are essentially unimprovable. Our proof is based on a combination of
analytic techniques~\cite{BuhlmannYu03} with techniques from empirical
process theory~\cite{vandeGeer}.  We complement these theoretical
results with simulation studies that compare its performance to other
rules, in particular a method using hold-out data to estimate the
risk, as well as a second method based on Stein's Unbiased Risk
Estimate (SURE).  In our experiments for first-order Sobolev kernels,
we find that our stopping rule performs favorably compared to these
alternatives, especially as the sample size grows.  Finally, in
Section~\ref{SecRidgeCompare}, we provide an explicit link between our
early stopping strategy and the kernel ridge regression estimator.

\section{Background and problem formulation}
\label{SecProbSetup}

We begin by introducing some background on non-parametric regression
and reproducing kernel Hilbert spaces, before turning to a precise
formulation of the problem studied in this paper.

\subsection{Non-parametric regression and kernel classes}
\label{SecRKHS}

Suppose that our goal is to use a covariate $X \in \ensuremath{\mathcal{X}}$ to predict
a real-valued response $Y \in \ensuremath{\mathbb{R}}$.  We do so by using a function
$f: \ensuremath{\mathcal{X}} \rightarrow \ensuremath{\mathbb{R}}$, where the value $f(x)$ represents our
prediction of $Y$ based on the realization $X = x$. In terms of
mean-squared error, the optimal choice is the \emph{regression
  function} defined by \mbox{$\ensuremath{f^*}(x) \ensuremath{: \, = } \ensuremath{\mathbb{E}}[Y \mid x]$.} In the
problem of non-parametric regression with random design, we observe
$\ensuremath{n}$ samples of the form \mbox{$\{ (\ensuremath{x_i}, \ensuremath{\y_i}), i = 1, \ldots,
  \ensuremath{n} \}$,} each drawn independently from some joint distribution
on the Cartesian product $\ensuremath{\mathcal{X}} \times \ensuremath{\mathbb{R}}$, and our goal is to
estimate the regression function $\ensuremath{f^*}$.  Equivalently, we observe
samples of the form
\begin{align}
\label{EqnLinObs}
\ensuremath{\y_i} & = \ensuremath{f^*}(\ensuremath{x_i}) + \ensuremath{w_i}, \quad \mbox{for $i = 1,2, \ldots,
  \ensuremath{n}$,}
\end{align}
where $\ensuremath{w_i} \ensuremath{: \, = } \ensuremath{\y_i} - \ensuremath{f^*}(\ensuremath{x_i})$ is a zero-mean noise random variable.
Throughout this paper, we assume that the random variables $\ensuremath{w_i}$ are
\emph{sub-Gaussian} with parameter $\sigma$, meaning that
\begin{align}
\label{EqnDefnSubGaussian}
\ensuremath{\mathbb{E}}[e^{t w_i}] & \leq e^{t^2 \sigma^2/2} \quad \mbox{for all $t \in
  \ensuremath{\mathbb{R}}$.}
\end{align}
For instance, this sub-Gaussian condition is satisfied for normal
variates $w_i \sim N(0, \sigma^2)$, but it also holds for various
non-Gaussian random variables.  Parts of our analysis also apply to
the fixed design setting, in which we condition on a particular
realization $\{x_i\}_{i=1}^\ensuremath{n}$ of the covariates.

In order to estimate
the regression function, we make use of the machinery of reproducing
kernel Hilbert spaces~\cite{Aron50,Wahba,Gu01}.  Using $\ensuremath{\mathbb{P}}$ to
denote the marginal distribution of the covariates, we consider a
Hilbert space \mbox{$\ensuremath{\mathcal{H}} \subset \ensuremath{L^2(\mathbb{P})}$,} meaning a family of
functions $g: \ensuremath{\mathcal{X}} \rightarrow \ensuremath{\mathbb{R}}$, with $\|g\|_{\ensuremath{L^2(\mathbb{P})}} < \infty$,
and an associated inner product $\inprod{\cdot}{\cdot}_\ensuremath{\mathcal{H}}$ under
which $\ensuremath{\mathcal{H}}$ is complete. The space $\ensuremath{\mathcal{H}}$ is a reproducing kernel
Hilbert space (RKHS) if there exists a symmetric function $\ensuremath{\mathbb{K}}: \ensuremath{\mathcal{X}}
\times \ensuremath{\mathcal{X}} \rightarrow \ensuremath{\mathbb{R}}_+$ such that: (a) for each $x
\in \ensuremath{\mathcal{X}}$, the function $\ensuremath{\mathbb{K}}(\cdot, x)$ belongs to the Hilbert
space $\ensuremath{\mathcal{H}}$, and (b) we have the reproducing relation $f(x) =
\inprod{f}{\ensuremath{\mathbb{K}}(\cdot, x)}_{\ensuremath{\mathcal{H}}}$ for all $f \in \ensuremath{\mathcal{H}}$.  Any such
kernel function must be positive semidefinite; under suitable
regularity conditions, Mercer's theorem~\cite{Mercer09} guarantees
that the kernel has an eigen-expansion of the form
\begin{align}
\label{EqnMercer}
\ensuremath{\mathbb{K}}(x, x') & = \sum_{k=1}^\infty \ensuremath{\lambda}_k \phi_k(x)
\phi_k(x'),
\end{align}
where $\lambda_1 \geq \lambda_2 \geq \lambda_3 \geq \ldots \geq 0$ are a non-negative sequence of eigenvalues, and $\{\phi_k\}_{k=1}^\infty$
are the associated eigenfunctions, taken to be orthonormal in $\ensuremath{L^2(\mathbb{P})}$.
The decay rate of the eigenvalues will play a crucial role in our
analysis.

Since the eigenfunctions $\{\phi_k\}_{k=1}^\infty$ form an orthonormal
basis, any function $f \in \ensuremath{\mathcal{H}}$ has an expansion of the form $f(x) =
\sum_{k=1}^{\infty} \sqrt{\ensuremath{\lambda}_k} a_{k} \phi_k(x)$, where
for all $k$ such that $\ensuremath{\lambda}_k > 0$, the coefficients
\begin{align*}
a_k & \ensuremath{: \, = } \frac{1}{\sqrt{\ensuremath{\lambda}_k}} \inprod{f}{\phi_k}_{\ensuremath{L^2(\mathbb{P})}} =
\int_\ensuremath{\mathcal{X}} f(x) \phi_k(x) \, d\, \ensuremath{\mathbb{P}}(x)
\end{align*}
are rescaled versions of the generalized Fourier
coefficients. Associated with any two functions in $\ensuremath{\mathcal{H}}$---where
\mbox{$f = \sum_{k = 1}^\infty \sqrt{\ensuremath{\lambda}_k} a_k \phi_k$} and
\mbox{$g = \sum_{k =1}^\infty \sqrt{\ensuremath{\lambda}_k} b_k \phi_k$}---are two
distinct inner products. The first is the usual inner product in the
space $\ensuremath{L^2(\mathbb{P})}$---namely, \mbox{$\inprod{f}{g}_{\ensuremath{L^2(\mathbb{P})}} \ensuremath{: \, = } \int_\ensuremath{\mathcal{X}}
  f(x) g(x) \, d\, \ensuremath{\mathbb{P}}(x)$.} By Parseval's theorem, it has an
equivalent representation in terms of the expansion coefficients and
kernel eigenvalues---that is,
\begin{align*}
\inprod{f}{g}_{\ensuremath{L^2(\mathbb{P})}} & = \sum_{k=1}^\infty \ensuremath{\lambda}_k a_k b_k.
\end{align*}
The second inner product, denoted $\inprod{f}{g}_{\ensuremath{\mathcal{H}}}$, is the one
that defines the Hilbert space; it can be written in terms of the
expansion coefficients as
\begin{align*}
\inprod{f}{g}_\ensuremath{\mathcal{H}} & = \sum_{k=1}^\infty {a_k
  b_k}.
\end{align*}
Using this definition, the Hilbert ball of radius $1$ for the Hilbert
space $\ensuremath{\mathcal{H}}$ with eigenvalues $\{\ensuremath{\lambda}_k\}_{k=1}^\infty$ and
eigenfunctions $\{\phi_k\}_{k=1}^\infty$ takes the form
\begin{align}
\ensuremath{\mathbb{B}}_\ensuremath{\mathcal{H}}(1) & \ensuremath{: \, = } \big \{ f = \sum_{k=1}^{\infty} \sqrt{\ensuremath{\lambda}_k}
b_{k} \phi_k \quad \mbox{for some} \quad \sum_{k=1}^{\infty} {b_k^2}
\leq 1 \big\}.
\end{align}
The class of reproducing kernel Hilbert spaces contains many
interesting classes that are widely used in practice, including
polynomials of degree $d$, Sobolev spaces with smoothness $\ensuremath{\nu}$,
and Gaussian kernels. For more background and examples on reproducing
kernel Hilbert spaces, we refer the reader to various standard
references~\cite{Aron50,Saitoh88,Scholkopf02,Wahba,Weinert82}.

Throughout this paper, we assume that any function $f$ in the unit
ball of the Hilbert space is uniformly bounded, meaning that there is
some constant $\ensuremath{B} < \infty$ such that
\begin{align}
\label{EqnCond}
\|f\|_{\infty} & \ensuremath{: \, = } \sup_{x \in \ensuremath{\mathcal{X}}} |f(x)| \leq \ensuremath{B} \qquad
\mbox{for all $f \in \ensuremath{\mathbb{B}}_\ensuremath{\mathcal{H}}(1)$.}
\end{align}
This boundedness condition~\eqref{EqnCond} is satisfied for any RKHS
with a kernel such that $\sup_{x \in \ensuremath{\mathcal{X}}} \ensuremath{\mathbb{K}}(x,x) \leq \ensuremath{B}$.
Kernels of this type include the Gaussian and Laplacian kernels, the
kernels underlying Sobolev and other spline classes, as well as as
well as any trace class kernel with trignometric eigenfunctions.  The
boundedness condition~\eqref{EqnCond} is quite standard in
non-asymptotic analysis of non-parametric regression
procedures~\cite[e.g.]{vandeGeer}.

\subsection{Gradient update equation}
\label{SecGradStep}

We now turn to the form of the gradient update that we study in this
paper. Given the samples \mbox{$\{(x_i, y_i)\}_{i=1}^\ensuremath{n}$},
consider minimizing the least-squares loss function
\begin{align}
\ensuremath{\mathcal{L}}(f) & \ensuremath{: \, = } \frac{1}{2 \ensuremath{n}} \sum_{i=1}^\ensuremath{n} {\big (\ensuremath{\y_i}
  - f(x_i) \big)^2}
\end{align}
over some subset of the Hilbert space $\ensuremath{\mathcal{H}}$.  By the representer
theorem~\cite{Kimeldorf71}, it suffices to restrict attention to
functions $f$ belonging to the span of the kernel functions $\{
\ensuremath{\mathbb{K}}(\cdot, x_i), i = 1, \ldots, \ensuremath{n} \}$.  Accordingly, we adopt
the parameterization
\begin{align}
\label{EqnRepresenter}
f(\cdot) & = \frac{1}{\sqrt{n}} \sum_{i=1}^{n}{\ensuremath{\omega}_i
  \ensuremath{\mathbb{K}}(\cdot ,x_i)},
\end{align}
for some coefficient vector $\ensuremath{\omega} \in \ensuremath{\mathbb{R}}^\ensuremath{n}$.  Here the
rescaling by $1/\sqrt{\ensuremath{n}}$ is for later theoretical convenience.

Our gradient descent procedure is based on a parameterization of the
least-squares loss that involves the \emph{empirical kernel matrix}
\mbox{$\ensuremath{K} \in \mathbb{R}^{n \times n}$} with entries
\begin{equation}
\label{EqnDefnEmpiricalKernel}
[\ensuremath{K}]_{ij} = \frac{1}{n}\ensuremath{\mathbb{K}}(x_i, x_j) \qquad \mbox{for $i, j = 1,
  2, \ldots, \ensuremath{n}$.}
\end{equation}
For any positive semidefinite kernel function, this matrix must be
positive semidefinite, and so has a unique symmetric square root
denoted by $\ensuremath{\sqrt{\EmpKer}}$.  Introducing the convenient shorthand
\mbox{$\ensuremath{y_1^\numobs} \ensuremath{: \, = } \big(y_1 \; y_2 \; \cdots y_\ensuremath{n} \big) \in
  \ensuremath{\mathbb{R}}^\ensuremath{n}$,} we can then write the least-squares loss in the
form
\begin{align*}
\ensuremath{\mathcal{L}}(\ensuremath{\omega}) & = \frac{1}{2 \ensuremath{n}} \| \ensuremath{y_1^\numobs} -
\sqrt{\ensuremath{n}} \ensuremath{K} \ensuremath{\omega} \|_2^2.
\end{align*}
A direct approach would be to perform gradient descent on this form of
the least-squares loss.  For our purposes, it turns out to be more
natural to perform gradient descent in the transformed co-ordinate
system $\BVEC{} = \ensuremath{\sqrt{\EmpKer}} \, \ensuremath{\omega}$.  Some straightforward
calculations (see Appendix~\ref{AppGrad} for details) yield that the
gradient descent algorithm in this new co-ordinate system generates a
sequence of vectors $\{\BVEC{\ensuremath{t}}\}_{\ensuremath{t}=0}^\infty$ via the
recursion
\begin{align}
\label{EqnGradBvec}
\BVEC{\ensuremath{t}+1} & = \BVEC{\ensuremath{t}} - \Step{\ensuremath{t}} \Big( \ensuremath{K} \,
\BVEC{\ensuremath{t}} - \frac{1}{\sqrt{\ensuremath{n}}} \ensuremath{\sqrt{\EmpKer}} \, \ensuremath{y_1^\numobs} \Big),
\end{align}
where $\{\Step{\ensuremath{t}}\}_{\ensuremath{t}=0}^\infty$ is a sequence of positive
step sizes (to be chosen by the user).  We assume throughout that the
gradient descent procedure is initialized with $\BVEC{0} = 0$.

The parameter estimate $\BVEC{\ensuremath{t}}$ at iteration $t$ defines a
function estimate $\FUNIT{\ensuremath{t}}$ in the following way.  We first
compute\footnote{If the empirical matrix $\ensuremath{K}$ is not invertible,
  then we use the pseudoinverse. Note that it may appear as though a
  matrix inversion is required to estimate $\ensuremath{\omega}^{\ensuremath{t}}$ for
  each $\ensuremath{t}$ which is computationally intensive.  However, the
  weights $\ensuremath{\omega}^\ensuremath{t}$ may be computed directly via the
  iteration $\ensuremath{\omega}^{\ensuremath{t}+1} = \ensuremath{\omega}^{\ensuremath{t}} -
  \Step{\ensuremath{t}} \ensuremath{K} (\ensuremath{\omega}^{\ensuremath{t}} -
  \frac{\ensuremath{y_1^\numobs}}{\sqrt{\ensuremath{n}}} )$.  However, the equivalent
  update~\eqref{EqnGradBvec} is more convenient for our analysis.} the
weight vector $\ensuremath{\omega}^\ensuremath{t} = \sqrt{\ensuremath{K}^{-1}} \;
\BVEC{\ensuremath{t}}$, which then defines the function estimate
$\FUNIT{\ensuremath{t}}(\cdot) = \frac{1}{\sqrt{\ensuremath{n}}} \sum_{i=1}^\ensuremath{n}
\ensuremath{\omega}^\ensuremath{t}_i \ensuremath{\mathbb{K}}(\cdot, x_i)$ as before.  In this paper, our
goal is to study how the sequence $\{\FUNIT{\ensuremath{t}}\}_{\ensuremath{t}=0}^\infty$
evolves as an approximation to the true regression function $\ensuremath{f^*}$.
We measure the error in two different ways: the $L^2(\ensuremath{\mathbb{P}}_\ensuremath{n})$
norm
\begin{align}
\label{EqnDefnEllTwoEmp}
\|f^\ensuremath{t} - \ensuremath{f^*}\|_\ensuremath{n}^2 & \ensuremath{: \, = } \frac{1}{\ensuremath{n}}
\sum_{i=1}^\ensuremath{n} \big( f^\ensuremath{t}(x_i) - \ensuremath{f^*}(x_i) \big)^2
\end{align}
compares the functions only at the observed design points, whereas the
$L^2(\ensuremath{\mathbb{P}})$-norm
\begin{align}
\label{EqnDefnEllTwoPop}
\|f^\ensuremath{t} - \ensuremath{f^*}\|_2^2 & \ensuremath{: \, = } \ensuremath{\mathbb{E}} \Big[ \big(f^\ensuremath{t}(X) -
  \ensuremath{f^*}(X) \big)^2 \Big]
\end{align}
corresponds to the usual mean-squared error.

\subsection{Overfitting and early stopping}

In order to illustrate the phenomenon of interest in this paper, we
performed some simulations on a simple problem.  In particular, we
formed $\ensuremath{n} = 100$ i.i.d. observations of the form $y =
\ensuremath{f^*}(x_i) + w_i$, where $w_i \sim N(0, 1)$, and using the fixed
design $x_i = i/\ensuremath{n}$ for $i= 1, \ldots, \ensuremath{n}$.  We then
implemented the gradient descent update~\eqref{EqnGradBvec} with
initialization $\BVEC{0} = 0$ and constant step sizes $\STEP{\ensuremath{t}} =
0l25$.  We performed this experiment with the regression function
$\ensuremath{f^*}(x) = |x-1/2| - 1/2$, and two different choices of kernel
functions.  The kernel $\ensuremath{\mathbb{K}}(x, x') = \min \{x, x'\}$ on the unit
square $[0,1] \times [0,1]$ generates an RKHS of Lipschitz functions,
whereas the Gaussian kernel $\ensuremath{\mathbb{K}}(x, x') = \exp(-\frac{1}{2} (x -
x')^2)$ generates a smoother class of infinitely differentiable
functions.

Figure~\ref{FigPath} provides plots of the squared prediction error
$\|\FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_\ensuremath{n}^2$ as a function of the iteration
number $\ensuremath{t}$.  For both kernels, the prediction error decreases
fairly rapidly, reaching a minimum before or around $T \approx 20$
iterations, before then beginning to increase.
\begin{figure}[h]
\begin{center}
\begin{tabular}{ccc}
\widgraph{.45\textwidth}{fig_sobolevone_earlypath.eps} & &
\widgraph{.45\textwidth}{fig_gauss_earlypath.eps} \\
(a) & & (b)
\end{tabular}
\caption{Behavior of gradient descent update~\eqref{EqnGradBvec} with
  constant step size \mbox{$\STEP{} = 0.25$} applied to least-squares
  loss with $\ensuremath{n} = 100$ with equi-distant design points $x_i =
  i/\ensuremath{n}$ for $i = 1, \ldots, \ensuremath{n}$, and regression function
  $\ensuremath{f^*}(x) = |x-1/2| - 1/2$.  Each panel gives plots the
  $L^2(\ensuremath{\mathbb{P}}_\ensuremath{n})$ error $\|\FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_\ensuremath{n}^2$
  as a function of the iteration number $\ensuremath{t} = 1, 2, \ldots, 100$.
  (a) For the first-order Sobolev kernel $\ensuremath{\mathbb{K}}(x, x') = \min\{x,
  x'\}$.  (b) For the Gaussian kernel $\ensuremath{\mathbb{K}}(x, x') = \exp(-\frac{1}{2}
  (x-x')^2)$. }
\label{FigPath}
\end{center}
\end{figure}
As the analysis of this paper will clarify, too many iterations lead
to fitting the noise in the data (i.e., the additive perturbations
$w_i$), as opposed to the underlying function $\ensuremath{f^*}$.  In a
nutshell, the goal of this paper is to quantify precisely the meaning
of ``too many'' iterations, and in a data-dependent and easily
computable manner.

\section{Main results and their consequences}
\label{SecMain}

In more detail, our main contribution is to formulate a data-dependent
stopping rule, meaning a mapping from the data $\{(x_i,
y_i)\}_{i=1}^\ensuremath{n}$ to a positive integer $\ensuremath{\widehat{T}}$, such that the two
forms of prediction error $\|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_\ensuremath{n}$ and
$\|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_2$ are minimal.  In our formulation of
such a stopping rule, two quantities play an important role: first,
the \emph{running sum} of the step sizes
\begin{align}
\RUNSUM{t} & \ensuremath{: \, = } \sum_{\tau=0}^{t-1}{\Step{\tau}},
\end{align}
and secondly, the eigenvalues $\ensuremath{\widehat{\ensuremath{\lambda}}}_1 \geq \ensuremath{\widehat{\ensuremath{\lambda}}}_2 \geq
\cdots \geq \ensuremath{\widehat{\ensuremath{\lambda}}}_\ensuremath{n} \geq 0$ of the empirical kernel matrix
$\ensuremath{K}$ previously defined~\eqref{EqnDefnEmpiricalKernel}.  The
kernel matrix and hence these eigenvalues are computable from the
data. We also note that there is a large body of work on fast
computation of kernel eigenvalues (e.g., see the
paper~\cite{DrinMah05} and references therein).

\subsection{Stopping rules and general error bounds}

Our stopping rule involves the use of a model complexity measure,
familiar from past work on uniform laws over kernel
classes~\cite{Bartlett02, Mendelson02}, known as the local empirical
Rademacher complexity.  For the kernel classes studied in this paper,
it takes the form
\begin{align}
\label{EqnDefnKernCompEmp}
\ensuremath{\widehat{\Rad}}_{\ensuremath{K}}(\varepsilon) & \ensuremath{: \, = } \biggr[ \frac{1}{n}
  \sum_{i=1}^n \min \big \{ \ensuremath{\widehat{\ensuremath{\lambda}}}_i, \varepsilon^2 \big \}
  \biggr]^{1/2}.
\end{align}
For a given noise variance $\sigma > 0$, a closely related
quantity---one of central importance to our analysis---is
\emph{critical empirical radius} $\ensuremath{\widehat{\varepsilon}_\numobs} > 0$, defined to be the
smallest positive solution to the inequality
\begin{align}
\label{EqnDefnEmpCrit}
\ensuremath{\widehat{\Rad}}_{\ensuremath{K}}(\varepsilon ) & \leq \varepsilon^2/(2 e \sigma).
\end{align}
The existence and uniqueness of $\ensuremath{\widehat{\varepsilon}_\numobs}$ is guaranteed for any
reproducing kernel Hilbert space; see Appendix~\ref{AppRade} for
details.  As clarified in our proof, this inequality plays a key role
in trading off the bias and variance in a kernel regression estimate.

Our stopping rule is defined in terms of an analogous inequality that
involves the running sum \mbox{$\RUNSUM{\ensuremath{t}} = \sum_{\tau =
    0}^{t-1}{\Step{\tau}}$} of the step sizes. Throughout this paper,
we assume that the step sizes are chosen to satisfy the following
properties:
\begin{itemize}
\item Boundedness: $0 \; \leq \; \Step{\tau} \; \leq \; \min \{1,
  1/\ensuremath{\widehat{\ensuremath{\lambda}}}_1 \}$ for all $\tau = 0, 1, 2, \ldots$.
\item Non-increasing: $\Step{\tau+1} \leq \Step{\tau}$ for all $\tau =
  0, 1, 2, \ldots$.
\item Infinite travel: the running sum $\RUNSUM{\ensuremath{t}} =
  \sum_{\tau=0}^{\ensuremath{t}-1} \STEP{\tau}$ diverges as $\ensuremath{t} \rightarrow
  +\infty$.
\end{itemize}
We refer to any sequence $\{\STEP{\tau}\}_{\tau=0}^\infty$ that
satisfies these conditions as a \emph{valid stepsize sequence}.  We
then define the \emph{stopping time}
\begin{align}
\label{EqnStoppingRule}
\ensuremath{\widehat{T}} & \ensuremath{: \, = } \arg \min \biggr \{ \ensuremath{t} \in \ensuremath{\mathbb{N}} \, \mid
\ensuremath{\widehat{\Rad}}_{\ensuremath{K}} \big(1/\sqrt{\RUNSUM{\ensuremath{t}}}\big) > (2 e \sigma
\RUNSUM{\ensuremath{t}})^{-1} \biggr \} - 1.
\end{align}
As discussed in Appendix~\ref{AppRade}, the integer $\ensuremath{\widehat{T}}$ belongs to
the interval $[0, \infty)$ and is unique for any valid stepsize
  sequence.  As will be clarified in our proof, the intuition
  underlying the stopping rule~\eqref{EqnStoppingRule} is that the sum
  of the step-sizes $\RUNSUM{\ensuremath{t}}$ acts as a tuning parameter that
  controls the bias-variance tradeoff.  The stated choice of
  $\widehat{T}$ optimizes this trade-off.

The following result applies to any sequence $\{ \FUNIT{\ensuremath{t}}
\}_{\ensuremath{t}=0}^\infty$ of function estimates generated by the gradient
iteration~\eqref{EqnGradBvec} with a valid stepsize sequence.

\btheos
\label{ThmMain}
Given the stopping time $\ensuremath{\widehat{T}}$ defined by the
rule~\eqref{EqnStoppingRule}, there are universal positive constants
$(\ensuremath{c}_1, \ensuremath{c}_2)$ such that the following events both hold with
probability at least $1 -\ensuremath{c}_1 \exp(-\ensuremath{c}_2 \ensuremath{n}
\ensuremath{\widehat{\varepsilon}_\numobs}^2)$:
\begin{enumerate}
\item[(a)] For all iterations $\ensuremath{t} = 1, 2,...,\ensuremath{\widehat{T}}$:
\begin{align}
\label{EqnGeneralBound}
 \| \FUNIT{\ensuremath{t}} - f^*\|_\ensuremath{n}^2 \; \leq \;
 \frac{4}{e \, \RUNSUM{\ensuremath{t}} }.
\end{align}
\item[(b)] At the iteration $\ensuremath{\widehat{T}}$ chosen according to the stopping
  rule~\eqref{EqnStoppingRule}, we have
\begin{align}
\label{BoundAtOpt}
\| \FUNIT{\ensuremath{\widehat{T}}}- \ensuremath{f^*}\|_\ensuremath{n}^2 & \leq 12 \, \ensuremath{\widehat{\varepsilon}_\numobs}^2.
\end{align}
\item[(c)] Moreover, for all $t > \ensuremath{\widehat{T}}$,
\begin{align}
\label{BoundAfterOpt}
\mathbb{E}[\| \FUNIT{\ensuremath{t}}- \ensuremath{f^*}\|_\ensuremath{n}^2] & \geq
\frac{\sigma^2}{4} \RUNSUM{\ensuremath{t}}
\ensuremath{\widehat{\Rad}}_{\ensuremath{K}}(\RUNSUM{\ensuremath{t}}^{-1/2}).
\end{align}
\end{enumerate}
\etheos

\noindent \paragraph{Remarks:} Although the bounds (a) and (b) are
stated as high probability claims, a simple integration argument can
be used to show that the expected mean-squared error (over the noise
variables, with the design fixed) satisfies a bound of the form
\begin{align}
\label{EqnIterationsEbound}
\ensuremath{\mathbb{E}} \big[ \|\FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_\ensuremath{n}^2 \big] \leq \frac{4}{e
  \, \RUNSUM{\ensuremath{t}} }.
\end{align}
Moreover, as will be clarified in corollaries to follow,
Theorem~\ref{ThmMain} can be used to show that our stopping rule
provides minimax-optimal rates for various function classes.  The
interpretation of Theorem~\ref{ThmMain} is as follows: if the sum of
the step-sizes $\RUNSUM{\ensuremath{t}}$ remains below the threshold defined
by~\eqref{EqnStoppingRule}, applying the gradient
update~\eqref{EqnGradBvec} reduces the prediction error.  Moreover,
note that for Hilbert spaces with a larger kernel complexity, the
stopping time $\ensuremath{\widehat{T}}$ is smaller, since fitting functions in a larger
class incurs a greater risk of overfitting.\\

In the case of random design $x_i \sim \ensuremath{\mathbb{P}}$, we can also provide
bounds on the $L^2(\ensuremath{\mathbb{P}})$-error $\|\FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_2$.  In
this setting, for the purposes of comparing to minimax lower bounds,
it is also useful to state some results in terms of the population
analog of the local empirical Rademacher
complexity~\eqref{EqnDefnKernCompEmp}, namely the quantity
\begin{align}
\label{EqnDefnPopKernComp}
\ensuremath{\mathcal{R}}_{\ensuremath{\mathbb{K}}}(\ensuremath{\varepsilon}) & \ensuremath{: \, = } \biggr[\frac{1}{n}\sum_{j = 1}^{\infty}
  \min \big \{ \ensuremath{\lambda}_j, \ensuremath{\varepsilon}^2 \big \} \biggr]^{1/2}.
\end{align}
Using this complexity measure, we define the \emph{critical population
  rate} $\ensuremath{\varepsilon_\numobs}$ to be the smallest positive solution to the
inequality
\begin{align}
\label{EqnDefnPopCrit}
40 \: \ensuremath{\mathcal{R}}_{\ensuremath{\mathbb{K}}}(\ensuremath{\varepsilon}) & \leq \frac{\ensuremath{\varepsilon}^2}{\sigma}.
\end{align}
(Our choice of the pre-factor $40$ is for later theoretical
convenience.)  In contrast to the critical empirical rate $\ensuremath{\widehat{\varepsilon}_\numobs}$,
this quantity is not data-dependent, since it depends on the
population eigenvalues of the RKHS $\ensuremath{\mathcal{H}}$.

\btheos[Random design]
\label{ThmRandDesign}
Suppose that the design variables $\{x_i\}_{i=1}^\ensuremath{n}$ are sampled
i.i.d. according to $\ensuremath{\mathbb{P}}$. Then under the conditions of
Theorem~\ref{ThmMain}, there are universal positive constants
$\ensuremath{c}_j, j = 1, 2, 3$ such that
\begin{align}
\|\FUNIT{\ensuremath{\widehat{T}}} - f^*\|_2^2 \leq \ensuremath{c}_3 \ensuremath{\varepsilon_\numobs}^2
\end{align}
with probability at least $1 - \ensuremath{c}_1 \exp(-\ensuremath{c}_2 \ensuremath{n}
\ensuremath{\widehat{\varepsilon}_\numobs}^2)$.
\etheos Theorems~\ref{ThmMain} and~\ref{ThmRandDesign} are general
results that apply to any reproducing kernel Hilbert space. Their
proofs involve combination of direct analysis of our iterative
update~\eqref{EqnGradBvec} combined with techniques from empirical
process theory and concentration of measure~\cite{vandeGeer,Ledoux01};
see Section~\ref{SecProofs} for the details.

To compare with the past work of B\"{u}hlmann and
Yu~\cite{BuhlmannYu03}, they also provide a theoretical analysis for
gradient descent (referred to as $L^2$-boosting in their paper),
focusing exclusively on the fixed design case.  Our theory applies to
random as well as fixed design, and a broader set of step-size
choices. The most significant difference between Theorem~\ref{ThmMain}
in our paper and Theorem 3 in the paper~\cite{BuhlmannYu03} is that we
provide a data-dependent stopping rule where as their analysis does
not lead to a computable stopping rule.

\subsection{Some consequences for specific kernel classes}

Let us now illustrate some consequences of our general theory for
special choices of kernels that are of interest in practice.

\paragraph{Kernels with polynomial eigendecay:}
We begin with the class of RKHSs whose eigenvalues satisfy a
\emph{polynomial decay condition}, meaning that
\begin{align}
\label{EqnPolyDecay}
\ensuremath{\lambda}_k & \leq C \big(\frac{1}{k}\big)^{2 \ensuremath{\nu}} \qquad \mbox{for
  some $\ensuremath{\nu} > 1/2$ and constant $C$.}
\end{align}
Among other examples, this type of scaling covers various types of
Sobolev spaces, consisting of functions with $\nu$ derivatives
(e.g.,~\cite{BirSol67,Gu02}).  For instance, the first-order Sobolev
kernel \mbox{$\ensuremath{\mathbb{K}}(x, x') = \min \{x, x'\}$} on the unit square
\mbox{$[0,1] \times [0,1]$} generates an RKHS of functions that are
differentiable almost \mbox{everywhere}, given by
\begin{align}
\label{EqnFirstOrderSobolev}
\ensuremath{\mathcal{H}} & \ensuremath{: \, = } \big \{ f: [0,1] \rightarrow \ensuremath{\mathbb{R}} \mid \, f(0) = 0,
\quad \int_0^1 (f'(x))^2 dx < \infty \big \},
\end{align}
For the uniform measure on $[0,1]$, this class exhibits polynomial
eigendecay~\eqref{EqnPolyDecay} with $\ensuremath{\nu} = 1$.  For any class
that satisfies the polynomial decay condition, we have the following
corollary: \\
\bcors
\label{CorAchieveSmooth}
Suppose that in addition to the assumptions of
Theorem~\ref{ThmRandDesign}, the kernel class $\ensuremath{\mathcal{H}}$ satisfies the
polynomial eigenvalue decay~\eqref{EqnPolyDecay} for some parameter
$\ensuremath{\nu} > 1/2$.  Then there is a universal constant $\ensuremath{c}_5$ such
that
\begin{align}
\label{EqnSobolevUpper}
\ensuremath{\mathbb{E}} \big[\|\FUNIT{\ensuremath{\widehat{T}}}- f^*\|_2^2] & \leq \ensuremath{c}_5 \,
\big(\frac{\sigma^2}{\ensuremath{n}} \big)^{\frac{2 \ensuremath{\nu}}{2 \ensuremath{\nu}+1}}.
\end{align}
Moreover, if $\ensuremath{\lambda}_k \geq c \, (1/k)^{2 \ensuremath{\nu}}$ for all $k = 1,
2, \ldots$, then
\begin{align}
\ensuremath{\mathbb{E}} \big[\|\FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_2^2 \big] & \geq \frac{1}{4} \min
\big \{ 1, \; \sigma^2 \frac{(\RUNSUM{\ensuremath{t}})^{\frac{1}{2
      \ensuremath{\nu}}}}{\ensuremath{n}} \big \} \quad \mbox{for all iterations
  $\ensuremath{t} = 1, 2, \ldots$.}
\end{align}
\ecors

\noindent The proof, provided in
Section~\ref{SecProofCorAchieveSmooth}, involves showing that the
population critical rate~\eqref{EqnDefnPopKernComp} is of the order
$\ensuremath{\mathcal{O}}(\ensuremath{n}^{- \frac{2 \ensuremath{\nu}}{2 \ensuremath{\nu}+1}})$.  By known results
on non-parametric regression~\cite{Sto85, YanBar99}, the error
bound~\eqref{EqnSobolevUpper} is minimax-optimal.

In the special case of the first-order spline
family~\eqref{EqnFirstOrderSobolev}, Corollary~\ref{CorAchieveSmooth}
guarantees that
\begin{align}
\label{EqnOptimalSobone}
\ensuremath{\mathbb{E}}[ \|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_2^2] & \precsim \big(
\frac{\sigma^2}{\ensuremath{n}} \big)^{2/3}.
\end{align}
In order to test the accuracy of this prediction, we performed the
following set of simulations.  First, we generated samples from the
observation model
\begin{align}
\label{EqnStandard}
y_i & = \ensuremath{f^*}(x_i) + w_i, \qquad \mbox{for $i = 1, 2, \ldots,
  \ensuremath{n}$},
\end{align}
where $x_i = i/\ensuremath{n}$, and $w_i \sim N(0,\sigma^2)$ are i.i.d. noise
terms.  We present results for the function $\ensuremath{f^*}(x) = |x - 1/2| -
1/2$, a piecewise linear function belonging to the first-order Sobelev
class.  For all our experiments, the noise variance $\sigma^2$ was set
to one, but so as to have a data-dependent method, this knowledge was
not provided to the estimator.  There is a large body of work on
estimating the noise variance $\sigma^2$ in non-parametric regression
(see e.g. Hall and Marron~\cite{HallMarron90}). For our simulations,
we use the simple estimator based on Hall and
Marron~\cite{HallMarron90}. They proved that their estimator is ratio
consistent, which is sufficient for our purposes.

\begin{figure}[h]
\begin{center}
\begin{tabular}{ccc}
\widgraph{.45\textwidth}{Fig2a.eps} & &
\widgraph{.45\textwidth}{Fig2b.eps} \\
(a) & & (b)
\end{tabular}
\caption{Prediction error obtained from the stopping rule applied to a
  regression with $\ensuremath{n}$ samples of the form $\ensuremath{f^*}(x_i) + w_i$ at
  equi-distant design points $x_i = i/\ensuremath{n}$ for $i = 0, 1, \ldots
  99$, and i.i.d. Gaussian noise $w_i \sim N(0,)$.  For these
  simulations, the true regression function is given by $\ensuremath{f^*}(x) =
  |x - 1/2| - 1/2$. Panel (a): Mean-squared error (MSE) using the
  stopping rule~\eqref{EqnStoppingRule} versus the sample size
  $\ensuremath{n}$.  Each point is based on $10,000$ independent realizations
  of the noise variables $\{w_i\}_{i=1}^\ensuremath{n}$.  $10,000$
  randomizations of $(w_i)_{i=1}^\ensuremath{n}$ against the sample size
  $\ensuremath{n}$. Panel (b): Plots of the quantity $MSE^{-3/2}$ versus
  sample size $\ensuremath{n}$.  As predicted by the theory, this
  representation yields a straight line.}
\label{FigRates}
\end{center}
\end{figure}

 For a range of sample sizes $\ensuremath{n}$ between $10$ and $300$, we
 performed the updates~\eqref{EqnGradBvec} with constant stepsize
 $\STEP{} = 0.25$, stopping at the specified time $\ensuremath{\widehat{T}}$.  For each
 sample size, we performed $10,000$ independent trials, and averaged
 the resulting prediction errors.  In panel (a) of
 Figure~\ref{FigRates}, we plot the mean-squared error versus the
 sample size, which shows consistency of the method.  We also plotted
 the mean-squared error raised to the power $-3/2$ versus the sample
 size.  After this rescaling, the bound~\eqref{EqnOptimalSobone}
 predicts a linear relation, as is observed in panel (b) of
 Figure~\ref{FigRates}.  We also performed the same experiments for
 the case of randomly drawn designs $x_i \sim \mbox{Unif}(0,1)$.  In
 this case, we observed similar results but with more trials required
 to average out the additional randomness in the design.

\paragraph{Finite rank kernels:}  We
now turn to the class of RKHSs based on finite-rank kernels, meaning
that there is some finite integer $\ensuremath{m} < \infty$ such that
$\ensuremath{\lambda}_j = 0$ for all $j \geq \ensuremath{m} + 1$.  For instance, the
kernel function $\ensuremath{\mathbb{K}}(x, x') = (1 + x x')^2$ is a finite rank kernel
(with $\ensuremath{m} = 2$) that generates the RKHS of all quadratic
functions.  More generally, for any integer $d \geq 2$, the kernel
$\ensuremath{\mathbb{K}}(x, x') = (1 + x x')^d$ generates the RKHS of all polynomials
with degree at most $d$.  For any such kernel, we have the following
corollary:

\bcors
\label{CorAchieveFinite}
If, in addition to the conditions of Theorem~\ref{ThmRandDesign}, the
kernel has finite rank $\ensuremath{m}$, then
\begin{align}
\ensuremath{\mathbb{E}} \big [\|\widehat{f}_{\widehat{T}}- f^*\|_2^2 \big] & \leq
\ensuremath{c}_5 \, \sigma^2 \frac{\ensuremath{m}}{\ensuremath{n}}.
\end{align}

\ecors
\noindent Importantly, for a rank $\ensuremath{m}$-kernel, the rate
$\frac{\ensuremath{m}}{n}$ is minimax optimal in terms of squared $\ensuremath{L^2(\mathbb{P})}$
error~\cite[e.g.,]{RasWaiYu10b}. \\

\subsection{Comparison with other stopping rules}

\label{SecRuleComp}

In this section, we provide a comparison of our stopping rule to two
other stopping rules, as well as a oracle method (that involves
knowledge of $\ensuremath{f^*}$, and so cannot be computed in practice).

\paragraph{Hold-out method:}

First, we consider a simple hold-out method: it performs gradient
descent using $50\%$ of the data, and uses the other $50 \%$ of the
data to estimate the risk (e.g.~\cite{DevroyeWagner79}).  Assuming
that the sample size is even for simplicity, we split the full data
set $\{x_i\}_{i=1}^\ensuremath{n}$ into two equally sized subsets $\ensuremath{S_{\mbox{\tiny{tr}}}}$
and $S_{te}$.  The data indexed by the training set $\ensuremath{S_{\mbox{\tiny{tr}}}}$ is used
to estimate the function $\ensuremath{f_{\mbox{\tiny{tr}}}}^t$ using the gradient descent
update~\eqref{EqnGradBvec}.  At each iteration $\ensuremath{t} = 0,1,2,
\ldots$, the data indexed by $S_{te}$ is used to estimate the risk via
$\ensuremath{R_{\mbox{\tiny{HO}}}} (f_\ensuremath{t}) = \frac{1}{n} \sum_{i \in S_{te}} \big(y_i -
\ensuremath{f_{\mbox{\tiny{tr}}}}^\ensuremath{t}(x_i) \big)^2$, which defines the stopping rule
\begin{align}
\label{EqnStoppingRuleHO}
\ensuremath{\STOP_{\mbox{\tiny{HO}}}} & \ensuremath{: \, = } \arg \min \biggr \{ \ensuremath{t} \in \ensuremath{\mathbb{N}} \, \mid
\ensuremath{R_{\mbox{\tiny{HO}}}}(\ensuremath{f_{\mbox{\tiny{tr}}}}^{\ensuremath{t}+1}) > \ensuremath{R_{\mbox{\tiny{HO}}}}(\ensuremath{f_{\mbox{\tiny{tr}}}}^\ensuremath{t}) \biggr \} - 1.
\end{align}
A line of past work~\cite{Yao05,
  Bauer07,Caponnetto06,CaponnettoYao06,CaponnettoYaoJournal,
  DeVito10}) has analyzed stopping rules based on this type of
hold-out rule.  For instance, Caponetto~\cite{Caponnetto06} analyzes a
hold-out method, and shows that it yields rates that are optimal for
Sobolev spaces with $\ensuremath{\nu} \leq 1$ but not in general. The major
drawback of using hold-out as that a percentage of the data is lost
which increases the risk.

\paragraph{SURE method:} Stein's Unbiased Risk estimate (SURE)
can be used to define an alternative stopping rule.  If we define the
shrinkage matrix $\tilde{S}^\ensuremath{t} = \prod_{\tau=0}^{\ensuremath{t}-1}{(I -
  \STEP{\tau}\ensuremath{K})}$, then it can be shown that the SURE
estimator~\cite{Stein81} takes the form
\begin{align}
\ensuremath{R_{\mbox{\tiny{SU}}}}(f^\ensuremath{t}) & = \frac{1}{\ensuremath{n}} \{ \ensuremath{n} \sigma^2 + Y^T
(\tilde{S}^\ensuremath{t})^2 Y - 2 \sigma^2 \ensuremath{\operatorname{trace}}(\tilde{S}^\ensuremath{t}) \},
\end{align}
which is easy to compute.  This risk estimate defines the associated
stopping rule
\begin{align}
\label{EqnStoppingRuleSURE}
\ensuremath{\STOP_{\mbox{\tiny{SU}}}} & \ensuremath{: \, = } \arg \min \biggr \{ \ensuremath{t} \in \ensuremath{\mathbb{N}} \, \mid
\ensuremath{R_{\mbox{\tiny{SU}}}}(f^{\ensuremath{t}+1}) > \ensuremath{R_{\mbox{\tiny{SU}}}}(f^\ensuremath{t}) \biggr \} - 1.
\end{align}
In contrast with hold-out, this approach makes use of all the data.
However, we are not aware of any theoretical guarantees for early
stopping using the stopping rule~\eqref{EqnStoppingRuleSURE}.

It can be shown for both stopping rules~\eqref{EqnStoppingRuleHO} and
~\eqref{EqnStoppingRuleSURE}, a valid sequence of step-sizes
guarantees existence and uniqueness of the stopping point. Note that
our stopping rule $\ensuremath{\widehat{T}}$ based on~\eqref{EqnStoppingRule} requires
estimation of both the empirical eigenvalues, and the noise variance
$\sigma^2$.  In contrast, the SURE-based rule requires estimation of
$\sigma^2$ but not the empirical eigenvalues, whereas the hold-out
rule requires no parameters to be estimated, but a percentage of the
data is used to estimate the risk.

\paragraph{Oracle method:}  As a third
point of reference, we also plot the mean-squared error for an
``oracle'' method.  It is allowed to base its stopping time on the
exact prediction error $\ensuremath{R_{\mbox{\tiny{OR}}}}(f^\ensuremath{t}) = \|f^{\ensuremath{t}} -
\ensuremath{f^*}\|_\ensuremath{n}^2$, which defines the oracle stopping rule
\begin{align}
\label{EqnStoppingRuleOracle}
\ensuremath{\STOP_{\mbox{\tiny{OR}}}} & \ensuremath{: \, = } \arg \min \biggr \{ \ensuremath{t} \in \ensuremath{\mathbb{N}} \, \mid
\ensuremath{R_{\mbox{\tiny{OR}}}}(f^{\ensuremath{t}+1}) > \ensuremath{R_{\mbox{\tiny{OR}}}}(f^\ensuremath{t}) \biggr \} - 1.
\end{align}
Note that this stopping rule is not computable from the data, since it
assumes exact knowledge of the function $\ensuremath{f^*}$ that we are trying to
estimate.

In order to compare our stopping rule~\eqref{EqnStoppingRule} with
these alternatives, we generated i.i.d. samples from the previously
described model (see equation~\eqref{EqnStandard} and the following
discussion).  We varied the sample size $\ensuremath{n}$ from $10$ to $300$,
and for each sample size, we performed $10,000$ independent trials
(randomizations of the noise variables $\{w_i\}_{i=1}^\ensuremath{n}$), and
computed the average of squared prediction error.

\begin{figure}[h]
\begin{center}
\begin{tabular}{ccc}
\widgraph{.45\textwidth}{Fig3a.eps} & &
\widgraph{.45\textwidth}{Fig3b.eps} \\
(a) & & (b)
\end{tabular}
\caption{The non-parametric function is $f^*(x) = |x - 1/2| - 1/2$
  with kernel $\ensuremath{\mathbb{K}}(x,y) = \min(|x|,|y|)$. We apply the gradient
  descent update~\eqref{EqnGradBvec} with $\Step{\ensuremath{t}}=1$ for all
  $t$, and plot the average mean-squared error over $10,000$
  randomizations against the sample size for $n
  =10,20,30,40,50,60,70,80,90,100,200,300$. Mean-squared error is
  plotted for $4$ stopping rules: (i) our stopping
  rule~\eqref{EqnStoppingRule}; (ii) holding out $50 \%$ of the data
  and using~\eqref{EqnStoppingRuleHO}; (iii)
  SURE~\eqref{EqnStoppingRuleSURE}; and (iv) oracle stopping
  rule~\eqref{EqnStoppingRuleHO}. For panel (a) results are plotted on
  a normal scale and for panel (b), curves are plotted using a log-log
  scale.}
\label{FigHO}
\end{center}
\end{figure}
Figure~\ref{FigHO} plots the resulting mean-squared errors of our
stopping rule, the hold-out stopping rule~\eqref{EqnStoppingRuleHO},
the SURE-based stopping rule~\eqref{EqnStoppingRuleSURE}, and the
oracle rule~\eqref{EqnStoppingRuleOracle}.  Panel (a) shows the
mean-squared error versus sample size, whereas panel (b) shows the
same curves in terms of logarithm of mean-squared error.  Our proposed
rule exhibits better performance than the hold-out and SURE-based
rules for sample sizes $\ensuremath{n}$ larger than $50$.  On the flip side,
since the construction of our stopping rule is based on the assumption
that $\ensuremath{f^*}$ belongs to a known RKHS, it is unclear how robust it
would be to model mis-specification.  In contrast, the hold-out and
SURE-based stopping rules are generic methods, not based directly on
the RKHS structure, so might be more robust to model mis-specification.
Thus, one interesting direction is to explore the robustness of our
stopping rule.  On the theoretical front, it would be interesting to
determine whether the hold-out and/or SURE-based stopping rules can be
proven to achieve minimax optimal rates for general kernels, as we have
established for our stopping rule.

\subsection{Connections to  kernel ridge regression}
\label{SecRidgeCompare}

We conclude by presenting an interesting link between our early
stopping procedure and kernel ridge regression.  The kernel ridge
regression (KRR) estimate is defined as
\begin{align}
\label{EqnKernelRidge}
\ensuremath{\widehat{f}}_\nu & \ensuremath{: \, = } \arg \min_{f \in \ensuremath{\mathcal{H}}} \big\{ \frac{1}{2
  \ensuremath{n}} \sum_{i=1}^\ensuremath{n} (y_i - f(x_i))^2 + \frac{1}{2 \nu}
\|f\|_\ensuremath{\mathcal{H}}^2 \big \},
\end{align}
where $\nu$ is the (inverse) regularization parameter.  For any
$\nu < \infty$, the objective is strongly convex, so that the KRR
solution is unique.

Friedman and Popescu~\cite{Friedman04} observed through simulations
that the regularization paths for early stopping of gradient descent
and ridge regression are similar, but did not provide any theoretical
explanation of this fact.  As an illustration of this empirical
phenomenon, Figure~\ref{FigRidgePath} compares the prediction error
$\|\ensuremath{\widehat{f}}_{\nu} - \ensuremath{f^*}\|_\ensuremath{n}^2$ of the kernel ridge
regression estimate over the interval $\nu \in [1, 100]$ versus
that of the gradient update~\eqref{EqnGradBvec} over the first $100$
iterations.  Note that the curves, while not identical, are
qualitatively very similar.

\begin{figure}[h]
\begin{center}
\begin{tabular}{ccc}
\widgraph{0.45\textwidth}{fig_sobone_bothcurves.eps} & &
\widgraph{0.45\textwidth}{fig_sobone_bothcurves2.eps} \\
(a) & & (b)
\end{tabular}
\caption{Comparison of the prediction error of the path of kernel
  ridge regression estimates~\eqref{EqnKernelRidge} obtained by
  varying $\nu \in [1, 100]$ to those of the gradient
  updates~\eqref{EqnGradBvec} over $100$ iterations with constant step
  size.  All simulations were performed with the kernel $\ensuremath{\mathbb{K}}(x, x') =
  \min \{x, x' \}$ based on $\ensuremath{n} = 100$ samples at the design
  points $x_i = i/\ensuremath{n}$ with $\ensuremath{f^*}(x) = |x - 1/2| - 1/2$.  (a)
  Noise variance $\sigma^2 = 1$.  (b) Noise variance $\sigma^2 = 2$.}
\label{FigRidgePath}
\end{center}
\end{figure}
From past theoretical work~\cite[e.g.,]{vandeGeer,Mendelson02}, kernel
ridge regression, with the appropriate setting of the penalty
parameter $\nu$, is known to achieve minimax-optimal error for
various kernel classes, among them the Sobolev and finite-rank kernels
for which stopping rule is provably optimal.  In this section, we
provide a theoretical basis for these connections, in particular by
showing that if the inverse penalty parameter $\nu$ is chosen
using the same criterion as our stopping rule, then the prediction
error satisfies the same type of bounds (with $\nu$ now playing
the role of the running sum $\RUNSUM{\ensuremath{t}}$).

More precisely, suppose that we choose $\ensuremath{\widehat{\PenPar}}$ to be the smallest
positive solution to the inequality
\begin{align}
\label{EqnParChoice}
\big(4 \sigma \nu \big)^{-1} < \ensuremath{\widehat{\Rad}}_{\ensuremath{K}}(1/\sqrt{\nu}
\big).
\end{align}
Note that this criterion is identical to the one underlying our
stopping rule, except that the continuous parameter $\nu$ replaces
the discrete parameter $\RUNSUM{\ensuremath{t}} =
\sum_{\tau=0}^{t-1}{\Step{\tau}}$.

\bprops
\label{PropMainRidge}
Consider the kernel ridge regression estimator~\eqref{EqnKernelRidge}
applied to $\ensuremath{n}$ i.i.d. samples $\{(x_i, y_i)\}$ with $\sigma$-sub
Gaussian noise.  Then there are universal constants $(\ensuremath{c}_1,
\ensuremath{c}_2, \ensuremath{c}_3)$ such that for all $\ensuremath{\delta} > 0$, the following
claims hold with probability at least $1 - \ensuremath{c}_1 \exp(-\ensuremath{c}_2
\, \ensuremath{n} \, \ensuremath{\widehat{\varepsilon}_\numobs}^2 )$:
\begin{enumerate}
\item[(a)] For all $0 < \nu \leq \ensuremath{\widehat{\PenPar}}$, we have
\begin{align}
\label{EqnRidgeBoundBefore}
\|\ensuremath{\widehat{f}}_{\nu}- f^*\|_\ensuremath{n}^2 & \leq \frac{2}{\nu}
\end{align}
\item[(b)] With $\ensuremath{\widehat{\PenPar}}$ chosen according to the
  rule~\eqref{EqnParChoice}, we have
\begin{align}
\label{EqnRidgeBoundOpt}
\|\ensuremath{\widehat{f}}_{\ensuremath{\widehat{\PenPar}}} - \ensuremath{f^*}\|_\ensuremath{n}^2 & \leq \ensuremath{c}_3 \:
\ensuremath{\widehat{\varepsilon}_\numobs}^2.
\end{align}
\item[(c)] Moreover, for all $\nu > \ensuremath{\widehat{\PenPar}}$, we have
\begin{align}
\label{EqnRidgeBoundOptAfter}
\mathbb{E}[\| \ensuremath{\widehat{f}}_{\nu}- \ensuremath{f^*}\|_\ensuremath{n}^2] & \geq
\frac{\sigma^2}{4} \nu \ensuremath{\widehat{\Rad}}_{\ensuremath{K}}(\nu^{-1/2}).
\end{align}
\end{enumerate}
\eprops

Note that (apart from a slightly different leading constant) the upper
bound~\eqref{EqnRidgeBoundBefore} is \emph{identical} to the upper
bound in equation~\eqref{EqnGeneralBound} in Theorem~\ref{ThmMain}.
The only difference is that the inverse regularization parameter
$\nu$ replaces the running sum $\RUNSUM{\ensuremath{t}} = \sum_{\tau =
  0}^{\ensuremath{t}-1}{\Step{\tau}}$.  Similarly, part (b) of
Proposition~\ref{PropMainRidge} guarantees that the kernel ridge
regression~\eqref{EqnKernelRidge} has prediction error that is upper
bounded by the empirical critical rate $\ensuremath{\widehat{\varepsilon}_\numobs}^2$, as in part (b) of
Theorem~\ref{ThmMain}.  Let us emphasize that bounds of this type on
kernel ridge regression have been derived in past
work~\cite[e.g.,]{Mendelson02,vandeGeer}.  The novelty here is that
the structure of our result reveals the intimate connection to early
stopping, and in fact, the proofs follow a parallel thread.

In conjunction, Proposition~\ref{PropMainRidge} and
Theorem~\ref{ThmMain} provide a theoretical explanation for why, as
shown in Figure~\ref{FigRidgePath}, the paths of the gradient descent
update~\eqref{EqnGradBvec} and kernel ridge regression
estimate~\eqref{EqnKernelRidge} are so similar. However, it is
important to emphasize that from a computational point of view, early
stopping has certain advantages over kernel ridge regression.  In
general, solving a quadratic program of the
form~\eqref{EqnKernelRidge} requires on the order of
$\ensuremath{\mathcal{O}}(\ensuremath{n}^3)$ basic operations, and this must be done repeatedly
at each new choice of $\nu$.  On the other hand, by its very
construction, the iterates of the gradient algorithm correspond to the
desired path of solutions, and each gradient update involves
multiplication by the kernel matrix, incurring $\ensuremath{\mathcal{O}}(\ensuremath{n}^2)$
operations.

\section{Proofs}
\label{SecProofs}

We now turn to the proofs of our main results.  The main steps in each
proof are provided in the main text, with some of the more technical
results deferred to the appendix.

\subsection{Proof of Theorem~\ref{ThmMain}}

In order to derive upper bounds on the $\ensuremath{{L^2(\mathbb{P}_n)}}$-error in
Theorem~\ref{ThmMain}, we first rewrite the gradient
update~\eqref{EqnGradBvec} in an alternative form.  For each iteration
$\ensuremath{t} = 0, 1, 2, \ldots$, let us introduce the shorthand
\begin{align}
\FUN{\ensuremath{t}} & \ensuremath{: \, = } \begin{bmatrix} f^\ensuremath{t}(x_1) & f^\ensuremath{t}(x_2) &
  \cdots & f^\ensuremath{t}(x_\ensuremath{n})
\end{bmatrix} \in \ensuremath{\mathbb{R}}^\ensuremath{n},
\end{align}
corresponding to the $\ensuremath{n}$-vector obtained by evaluating the
function $f^\ensuremath{t}$ at all design points, and the short-hand
\begin{align}
\ensuremath{w} & \ensuremath{: \, = } \begin{bmatrix} w_1, w_2,...,w_n\end{bmatrix} \in \ensuremath{\mathbb{R}}^\ensuremath{n},
\end{align}
corresponding to the vector of zero mean sub-Gaussian noise random variables. From
equation~\eqref{EqnRepresenter}, we have the relation
\begin{align*}
\FUN{\ensuremath{t}} = \frac{1}{\sqrt{\ensuremath{n}}} \ensuremath{K} \, \ensuremath{\omega}^\ensuremath{t} \;
= \; \frac{1}{\sqrt{\ensuremath{n}}} \ensuremath{\sqrt{\EmpKer}} \, \BVEC{\ensuremath{t}}.
\end{align*}
Consequently, by multiplying both sides of the gradient
update~\eqref{EqnGradBvec} by $\ensuremath{\sqrt{\EmpKer}}$, we find that the sequence
$\{\FUN{\ensuremath{t}}\}_{\ensuremath{t}=0}^\infty$ evolves according to the recursion
\begin{align}
\label{EqnAltGrad}
\FUN{\ensuremath{t}+1} & = \FUN{\ensuremath{t}} - \Step{\ensuremath{t}} \ensuremath{K} \, ( \FUN{\ensuremath{t}}
- \ensuremath{y_1^\numobs}) \; = \; \Big( I_{\ensuremath{n} \times \ensuremath{n}} - \Step{\ensuremath{t}}
\ensuremath{K} \Big) \FUN{\ensuremath{t}} - \Step{\ensuremath{t}} \ensuremath{K} \, \ensuremath{y_1^\numobs}.
\end{align}
Since $\BVEC{0} = 0$, the sequence is initialized with $\FUN{0} = 0$.
The recursion~\eqref{EqnAltGrad} lies at the heart of our analysis.

Letting $r = \mbox{rank}(\ensuremath{K})$, the empirical kernel matrix has
the eigendecomposition $\ensuremath{K} = U \Dmat U^T$, where $U \in
\mathbb{R}^{\ensuremath{n} \times \ensuremath{n}}$ is an orthonormal matrix
(satisfying $U U^T = U^T U = I_{n \times n}$) and
\begin{align*}
\Dmat \ensuremath{: \, = } \mbox{diag}(\ensuremath{\widehat{\ensuremath{\lambda}}}_1, \ensuremath{\widehat{\ensuremath{\lambda}}}_2, \ldots, \ensuremath{\widehat{\ensuremath{\lambda}}}_r,
0,0, \cdots, 0 )
\end{align*}
is the diagonal matrix of eigenvalues, augmented with $\ensuremath{n} - r$
zero eigenvalues as needed.  We then define a sequence of diagonal
\emph{shrinkage matrices} $\SHRINK{\ensuremath{t}}$ as follows:
\begin{align*}
\SHRINK{\ensuremath{t}} & \ensuremath{: \, = } \prod_{\tau = 0}^{t-1} {(I_{n \times n} -
  \Step{\tau} \Dmat)} \in \ensuremath{\mathbb{R}}^{n \times n}.
\end{align*}
The matrix $\SHRINK{\ensuremath{t}}$ indicates the extent of shrinkage towards
the origin; since $0 \; \leq \; \Step{\ensuremath{t}} \; \leq \; \min
\{1,1/\ensuremath{\widehat{\ensuremath{\lambda}}}_1 \}$ for all iterations $\ensuremath{t}$, in the positive
semodefinite ordering, we have the sandwich relation
\begin{align*}
0 \preceq \SHRINK{\ensuremath{t} + 1} \preceq \SHRINK{\ensuremath{t}} \preceq I_{\ensuremath{n}
  \times \ensuremath{n}}.
\end{align*}
Moreover, the following lemma shows that the
$L^2(\ensuremath{\mathbb{P}}_\ensuremath{n})$-error at each iteration can be bounded in terms
of the eigendecomposition and these shrinkage matrices:

\blems[Bias/variance decomposition]
\label{LemMain}
At each iteration $\ensuremath{t} = 0, 1, 2, \ldots$,
\begin{align}
\label{EqnMainUpper}
\| \FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_\ensuremath{n}^2
 & \leq \underbrace{\frac{2}{n}
  \sum_{j=1}^{r}{\SHRINKSQ{\ensuremath{t}}_{jj} [U^T \ensuremath{f^*(x_1^\numobs)}]_j^2} +
  \frac{2}{n} \sum_{j=r+1}^{n}{[U^T \ensuremath{f^*(x_1^\numobs)}]_j^2}}_{\mbox{Squared
    Bias $\ensuremath{B_\iter^2}$}} + \underbrace{\frac{2}{n} \sum_{j = 1}^{r} {(1 -
    \SHRINK{\ensuremath{t}}_{jj})^{2} [U^T \ensuremath{w}]_j^2}}_{\mbox{Variance
    $\ensuremath{V_\iter}$}}.
\end{align}
Moreover, we have the lower bound $ \mathbb{E}[\| \FUNIT{\ensuremath{t}} -
\ensuremath{f^*}\|_\ensuremath{n}^2] \geq \mathbb{E}[\ensuremath{V_\iter}]$.
\elems
\noindent See Appendix~\ref{AppLemMain} for the proof of this
intermediate claim. \\

In order to complete the proof of the upper bound in
Theorem~\ref{ThmMain}, our next step is to obtain high probability
upper bounds on these two terms.  We summarize our conclusions in an
additional lemma, and use it to complete the proof of
Theorem~\ref{ThmMain}(a) before returning to prove it.

\blems[Bounds on the bias and variance]
\label{LemBiasVarianceBound}
For all iterations $\ensuremath{t} = 1, 2, \ldots$, the squared bias is upper
bounded as
\begin{align}
\label{EqnBiasBound}
\ensuremath{B_\iter^2} & \leq \frac{1}{e \, \RUNSUM{\ensuremath{t}} },
\end{align}
Moreover, there is a universal constant $\ensuremath{c}_1 > 0$ such that, for
any iteration $\ensuremath{t} = 1, 2, \ldots, \ensuremath{\widehat{T}}$,
\begin{align}
\label{EqnVarBound}
\ensuremath{V_\iter} \; \leq \; 5 \sigma^2 \, \RUNSUM{\ensuremath{t}} \ensuremath{\mathcal{R}}^2_{\ensuremath{K}}
\big(1/\sqrt{\eta_\ensuremath{t}} \big)
\end{align}
with probability at least $1 - \exp \big(- \ensuremath{c}_1 \, \ensuremath{n}
\ensuremath{\widehat{\varepsilon}_\numobs}^2 \big)$. Moreover for all $t$, we have $\mathbb{E}[\ensuremath{V_\iter}]
\geq \frac{\sigma^2}{4} \, \RUNSUM{\ensuremath{t}} \ensuremath{\mathcal{R}}^2_{\ensuremath{K}}\big(1/\sqrt{\eta_\ensuremath{t}} \big)$.
\elems

We can now complete the proof of Theorem~\ref{ThmMain}(a).  The
bound~\eqref{EqnGeneralBound} follows quickly: conditioned on the
event $\ensuremath{V_\iter} \leq 5 \sigma^2 \RUNSUM{\ensuremath{t}} \ensuremath{\mathcal{R}}^2_{\ensuremath{K}}
\big(1/\sqrt{\eta_\ensuremath{t}} \big)$, we have
\begin{align*}
\|\FUNIT{\ensuremath{t}} - \ensuremath{f^*}\|_\ensuremath{n}^2 & \stackrel{(i)}{\leq} \ensuremath{B_\iter^2} +
\ensuremath{V_\iter} \; \stackrel{(ii)}{\leq} \; \frac{1}{e \, \RUNSUM{\ensuremath{t}}} + 5
\sigma^2 \, \RUNSUM{\ensuremath{t}} \ensuremath{\mathcal{R}}^2_{\ensuremath{K}}
\big(1/\sqrt{\RUNSUM{\ensuremath{t}}} \big) \stackrel{(iii)}{\leq} \frac{4}{e
  \, \RUNSUM{\ensuremath{t}}},
\end{align*}
where inequality (i) follows from~\eqref{EqnMainUpper} in
Lemma~\ref{LemMain}, and inequality (ii) follows from the bounds in
Lemma~\ref{LemBiasVarianceBound} and (iii) follows since $\ensuremath{t} \leq
\ensuremath{\widehat{T}}$. The lower bound (c) in equation~\eqref{BoundAfterOpt} follows
from ~\eqref{EqnVarBound}.

Turning to the proof of part (b), using the upper bound from (a)
\begin{align*}
\|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_\ensuremath{n}^2 & \leq \frac{1}{e \,
  \RUNSUM{\ensuremath{\widehat{T}}}} + \frac{5}{\RUNSUM{\ensuremath{\widehat{T}}}} \; \leq \;
\frac{4}{e \RUNSUM{\ensuremath{\widehat{T}}}}.
\end{align*}
Based on the definition of $\ensuremath{\widehat{T}}$ and $\ensuremath{\widehat{\varepsilon}_\numobs}$,
we are guaranteed that $\frac{1}{\RUNSUM{\ensuremath{\widehat{T}}+1}} \leq \ensuremath{\widehat{\varepsilon}_\numobs}^2$,
Moreover, by the non-decreasing nature of our step sizes, we have
$\STEP{\ensuremath{\widehat{T}}+1} \leq \STEP{\ensuremath{\widehat{T}}}$, which implies that
$\RUNSUM{\ensuremath{\widehat{T}}+1} \leq 2 \RUNSUM{\ensuremath{\widehat{T}}}$, and hence
\begin{align*}
\frac{1}{\RUNSUM{\ensuremath{\widehat{T}}}} \leq \frac{2}{\RUNSUM{\ensuremath{\widehat{T}}+1}} \; \leq \; 2
\ensuremath{\widehat{\varepsilon}_\numobs}^2.
\end{align*}
Putting together the pieces establishes the bound claimed in part (b).

It remains to establish the bias and variance bounds stated in
Lemma~\ref{LemBiasVarianceBound}, and we do so in the following
subsections.  The following auxiliary lemma plays a role in both
proofs:

\blems[Properties of shrinkage matrices]
\label{LemMatrices}
For all indices $j \in \{1, 2, \ldots, r\}$, the shrinkage matrices
$\SHRINK{\ensuremath{t}}$ satisfy the bounds
\begin{subequations}
\begin{align}
\label{EqnUpperBias}
0 \leq \; \SHRINKSQ{\ensuremath{t}}_{jj} & \leq \frac{1}{2 e \RUNSUM{\ensuremath{t}}
  \ensuremath{\widehat{\ensuremath{\lambda}}}_j}, \quad \mbox{and} \\
\label{EqnLowerBias}
\frac{1}{2} \min \{1, \eta_\ensuremath{t} \ensuremath{\widehat{\ensuremath{\lambda}}}_j \} & \leq
1-\SHRINK{\ensuremath{t}}_{jj} \; \leq \min \{ 1, \RUNSUM{\ensuremath{t}} \ensuremath{\widehat{\ensuremath{\lambda}}}_j
\}.
\end{align}
\end{subequations}
\elems
\noindent See Appendix~\ref{AppLemMatrices}  for the proof of this result.

\subsubsection{Bounding the squared bias}
\label{SecBias}

Let us now prove the upper bound~\eqref{EqnBiasBound} on the squared
bias.  We bound each of the two terms in the
definition~\eqref{EqnMainUpper} of $\ensuremath{B_\iter^2}$ in term.  Applying the
upper bound ~\eqref{EqnUpperBias} from Lemma~\ref{LemMatrices}, we see
that
\begin{align*}
\frac{2}{n} \sum_{j=1}^{r} {\SHRINKSQ{\ensuremath{t}}_{jj} [U^T
    \ensuremath{f^*}(x_1^\ensuremath{n})]_j^2} & \leq \frac{1}{e \; \ensuremath{n} \;
  \RUNSUM{\ensuremath{t}}} \; \sum_{j=1}^{r} { \frac{[U^T \ensuremath{f^*}(x_1^\ensuremath{n})
    ]_j^2}{\ensuremath{\widehat{\ensuremath{\lambda}}}_j}}.
\end{align*}
Now consider the linear operator \mbox{$\Phi_X: \ell^2(\ensuremath{\mathbb{N}})
  \rightarrow \ensuremath{\mathbb{R}}^\ensuremath{n}$} defined element-wise via $[\Phi_X]_{jk}
= \phi_j(x_k)$.  Similarly, we define a (diagonal) linear operator
\mbox{$\DiagOpt: \ell^2(\ensuremath{\mathbb{N}}) \rightarrow \ell^2(\ensuremath{\mathbb{N}})$} with entries
$[\DiagOpt]_{jj} = \lambda_j$ and $[\DiagOpt]_{jk} = 0$ for $j \neq
k$.  With these definitions, the vector $\FUN{} \in \ensuremath{\mathbb{R}}^\ensuremath{n}$ can
be expressed in terms of some sequence $a \in \ell^2(\ensuremath{\mathbb{N}})$ in the
form
\begin{align*}
\FUN{} & = \Phi_X \DiagOpt^{1/2} a.
\end{align*}
In terms of these quantities, we can write $\ensuremath{K} =
\frac{1}{n}\Phi_X \DiagOpt \Phi_X^T$.  Moreover, as previously noted,
we also have $\ensuremath{K} = U \Dmat U^T$ where $\Dmat = \mbox{diag}
\{\ensuremath{\widehat{\ensuremath{\lambda}}}_1, \ensuremath{\widehat{\ensuremath{\lambda}}}_2, \ldots, \ensuremath{\widehat{\ensuremath{\lambda}}}_\ensuremath{n} \}$, and $U
\in \ensuremath{\mathbb{R}}^{\ensuremath{n} \times \ensuremath{n}}$ is orthonormal.  Combining the two
representations, we conclude that
\begin{equation*}
\frac{\Phi_X \DiagOpt^{1/2}}{\sqrt{n}} = U \Dmat^{1/2} \Psi^*,
\end{equation*}
for some linear operator $\Psi: \ensuremath{\mathbb{R}}^n \rightarrow \ell^2(\ensuremath{\mathbb{N}})$
(with adjoint $\Psi^*$) such that $\Psi^* \Psi = I_{\ensuremath{n} \times
  \ensuremath{n}}$.  Using this equality, we have
\begin{align}
\frac{1}{e \, \RUNSUM{\ensuremath{t}} \, n} \sum_{j=1}^{r}{\frac{[U^T
      f^*(X)]_j^2}{\widehat{\ensuremath{\lambda}_j}}} & = \frac{1}{e \, \RUNSUM{\ensuremath{t}}
  \, \ensuremath{n}} \; \sum_{j=1}^{r} {\frac{[U^T \Phi_X \DiagOpt^{1/2}
      a]_j^2}{\ensuremath{\widehat{\ensuremath{\lambda}}}_j}} \nonumber \\
& = \frac{1}{e \, \RUNSUM{\ensuremath{t}}} \; \sum_{j = 1}^{r} { \frac{[U^T U
           \Dmat^{1/2} V^* a ]_j^2}{\ensuremath{\widehat{\ensuremath{\lambda}}}_j}} \nonumber \\
& = \frac{1}{e \, \RUNSUM{\ensuremath{t}}} \; \sum_{j = 1}^{r}
     {\frac{\ensuremath{\widehat{\ensuremath{\lambda}}}_j \, [\Psi^* a]_j^2}{\ensuremath{\widehat{\ensuremath{\lambda}}}_j}} \nonumber \\
& \leq \frac{1}{e \, \RUNSUM{\ensuremath{t}}} \; \|\Psi^* a\|_2^2 \nonumber \\
\label{EqnBoundOne}
& \leq \frac{1}{e \, \RUNSUM{\ensuremath{t}}},
\end{align}
Here the final step follows from the fact that $\Psi$ is a unitary
operator, so that \mbox{$\|\Psi^* a\|_2^2 \leq \|a\|_2^2 =
  \|f^*\|_{\ensuremath{\mathcal{H}}}^2 \leq 1$.}

Turning to the second term in the definition~\eqref{EqnMainUpper}, we
have
\begin{align}
\sum_{j=r+1}^{n} {[U^T \ensuremath{f^*}(x_1^\ensuremath{n}) ]_j^2} & = \frac{2}{n}
\sum_{j=r+1}^{n}{[U^T \Phi_X \DiagOpt^{1/2} a]_j^2} \nonumber \\
& = \sum_{j=r+1}^\ensuremath{n} {[U^T U \Dmat^{1/2} \Psi^* a]_j^2} \nonumber \\
& = \sum_{j = r+1}^{n} [\Dmat^{1/2} \Psi^* a]_j^2 \nonumber \\
\label{EqnBoundTwo}
& = 0,
\end{align}
where the final step uses the fact that $\Dmat^{1/2}_{jj} = 0$ for all
$j \in \{r +1, \ldots, \ensuremath{n} \}$ by construction.  Combining the
upper bounds~\eqref{EqnBoundOne} and~\eqref{EqnBoundTwo} with the
definition~\eqref{EqnMainUpper} of $\ensuremath{B_\iter^2}$ yields the
claim~\eqref{EqnBiasBound}.

\subsubsection{Controlling the variance}
\label{SecVar}

Let us now prove the bounds~\eqref{EqnVarBound} on the variance term
$\ensuremath{V_\iter}$.  (To simplify the proof, we assume throughout that $\sigma =
1$; the general case can be recovered by a simple rescaling argument).
By the definition of $\ensuremath{V_\iter}$, we have
\begin{align*}
\ensuremath{V_\iter} & = \frac{2}{n} \sum_{j = 1}^{r}{(1-\SHRINK{\ensuremath{t}}_{jj})^{2}
  [U^T w]_j^2} \; = \; \frac{2}{\ensuremath{n}} \ensuremath{\operatorname{trace}}(U Q U^T \, w w^T),
\end{align*}
where $Q = \mbox{diag} \{ (1 -\SHRINK{\ensuremath{t}}_{jj})^2, \; j = 1, \ldots,
\ensuremath{n} \}$ is a diagonal matrix.  Since $\ensuremath{\mathbb{E}}[w w^T] \leq I_{\ensuremath{n}
  \times \ensuremath{n}}$ by assumption, we have $\ensuremath{\mathbb{E}}[\ensuremath{V_\iter}] =
\frac{2}{\ensuremath{n}} \ensuremath{\operatorname{trace}}(Q)$.  Using the upper bound in
equation~\eqref{EqnLowerBias} from Lemma~\ref{LemMatrices}, we have
\begin{align*}
\frac{1}{\ensuremath{n}} \ensuremath{\operatorname{trace}}(Q) & \leq \frac{1}{\ensuremath{n}} \sum_{j=1}^r \min
\{1, (\RUNSUM{\ensuremath{t}} \ensuremath{\widehat{\ensuremath{\lambda}}}_j)^2 \} \; = \; \RUNSUM{\ensuremath{t}} \;
\biggr( \ensuremath{\mathcal{R}}_{\ensuremath{K}}(1/\sqrt{\RUNSUM{\ensuremath{t}}}) \biggr)^2,
\end{align*}
where the final equality uses the definition of $\ensuremath{\mathcal{R}}_\ensuremath{K}$.
Putting together the pieces, we see that
\begin{subequations}
\begin{align}
\label{EqnUpperEmyvar}
\ensuremath{\mathbb{E}}[\ensuremath{V_\iter}] & \leq 2 \; \RUNSUM{\ensuremath{t}} \biggr(
\ensuremath{\mathcal{R}}_{\ensuremath{K}}(1/\sqrt{\RUNSUM{\ensuremath{t}}}) \biggr)^2.
\end{align}
Similarly, using the lower bound in equation~\eqref{EqnLowerBias}, we
can show that
\begin{align}
\label{EqnLowerEmyvar}
\ensuremath{\mathbb{E}}[\ensuremath{V_\iter}] & \geq \frac{\sigma^2}{4} \RUNSUM{\ensuremath{t}} \biggr(
\ensuremath{\mathcal{R}}_{\ensuremath{K}}(1/\sqrt{\RUNSUM{\ensuremath{t}}}) \biggr)^2.
\end{align}
\end{subequations}

Our next step is to obtain a bound on the two-sided tail probability
$\ensuremath{\mathbb{P}}[|\ensuremath{V_\iter} - \ensuremath{\mathbb{E}}[\ensuremath{V_\iter}]| \geq \delta]$, for which we make use of
a result on two-sided deviations for quadratic forms in sub-Gaussian
variables.  In particular, consider a random variable of the form
$Q_\ensuremath{n} = \sum_{i,j=1}^\ensuremath{n} a_{ij} (Z_i Z_j - \ensuremath{\mathbb{E}}[Z_i Z_j])$
where $\{Z_i\}_{i=1}^\ensuremath{n}$ are i.i.d. zero-mean and sub-Gaussian
variables (with parameter $1$).  Wright~\cite{Wri73} proves that there
is a constant $c$ such that
\begin{align}
\label{EqnWrightBound}
\ensuremath{\mathbb{P}} \big[|Q - \ensuremath{\mathbb{E}}[Q]| \geq \ensuremath{\delta} \big] \leq \exp \Big(-c \;
\min \big \{ \frac{\ensuremath{\delta}}{\opnorm{A}},
\frac{\ensuremath{\delta}^2}{\fronorm{A}^2} \big \} \Big) \quad \mbox{for all $u>
  0$,}
\end{align}
where $(\opnorm{A}, \fronorm{A})$ are (respectively) the operator and
Frobenius norms of the matrix \mbox{$A = \{a_{ij}\}_{i,j=1}^\ensuremath{n}$.}

If we apply this result with $A= \frac{2}{\ensuremath{n}} U Q U^T$ and $Z_i =
w_i$, then we have $Q = \ensuremath{V_\iter}$, and moreover
\begin{align*}
\opnorm{A} & \leq \frac{2 }{\ensuremath{n}}, \quad \mbox{and} \\
\fronorm{A}^2 \; = \; \frac{4}{\ensuremath{n}^2} \ensuremath{\operatorname{trace}}(U^T Q U^T U Q U^T) &
= \frac{4}{\ensuremath{n}^2} \ensuremath{\operatorname{trace}}(Q^2) \leq \frac{4}{\ensuremath{n}^2} \ensuremath{\operatorname{trace}}(Q)
\leq \frac{4}{\ensuremath{n}} \RUNSUM{\ensuremath{t}} \; \biggr(
\ensuremath{\mathcal{R}}_{\ensuremath{K}}(1/\sqrt{\RUNSUM{\ensuremath{t}}}) \biggr).
\end{align*}
Consequently, the bound~\eqref{EqnWrightBound} implies that
\begin{align}
\label{EqnTwoMyvar}
\ensuremath{\mathbb{P}} \big[ |\ensuremath{V_\iter} - \ensuremath{\mathbb{E}}[\ensuremath{V_\iter}] | \geq \ensuremath{\delta} \big] & \leq \exp
\big( - 4 c \, \ensuremath{n} \, \delta \min \{1, \delta \biggr(\RUNSUM{\ensuremath{t}} \ensuremath{\mathcal{R}}_{\ensuremath{K}}(1/\sqrt{\RUNSUM{\ensuremath{t}}}) \biggr)^{-1}\} \big).
\end{align}
Since $\ensuremath{t} \leq \ensuremath{\widehat{T}}$ setting $\delta = 3 \sigma^2 \RUNSUM{\ensuremath{t}} \; \biggr(\ensuremath{\mathcal{R}}_{\ensuremath{K}}(1/\sqrt{\RUNSUM{\ensuremath{t}}}) \biggr)$, the claim~\eqref{EqnVarBound} follows.

\subsection{Proof of Theorem~\ref{ThmRandDesign}}

This proof is based on the following two steps:
\begin{itemize}
\item first, proving that the error $\|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_2$ in
  the $L^2(\ensuremath{\mathbb{P}})$ norm is, with high probability, close to the error
  in the $L^2(\ensuremath{\mathbb{P}}_\ensuremath{n})$ norm, and
\item second, showing the empirical critical radius $\ensuremath{\widehat{\varepsilon}_\numobs}$ defined
  in equation~\eqref{EqnDefnEmpCrit} is upper bounded by the
  population critical radius $\ensuremath{\varepsilon_\numobs}$ defined in
  equation~\eqref{EqnDefnPopCrit}.
\end{itemize}

\noindent Our proof is based on a number of more technical auxiliary
lemmas, proved in the appendices.  The first lemma provides a high
probability bound on the Hilbert norm of the estimate $\FUNIT{\ensuremath{\widehat{T}}}$.
\blems
\label{LemHilbBound}
There exist universal constants $c_1$ and $c_2 > 0$ such that
$\|\FUNIT{\ensuremath{t}}\|_{\ensuremath{\mathcal{H}}} \leq 2$ for all $\ensuremath{t} \leq \ensuremath{\widehat{T}}$ with
probability greater than or equal to $1 - c_1 \exp(-c_2 \ensuremath{n}
\ensuremath{\widehat{\varepsilon}_\numobs}^2)$.
\elems
See Appendix~\ref{AppLemHilbBound} for the proof of this claim.  Our
second lemma shows in any bounded RKHS, the $\ensuremath{L^2(\mathbb{P})}$ and $\ensuremath{{L^2(\mathbb{P}_n)}}$ norms
are uniformly close up to the population critical radius $\ensuremath{\varepsilon_\numobs}$
over a Hilbert ball of constant radius:
\blems

\label{LemGeneral}
Consider a Hilbert space such that $\|g\|_\infty \leq \ensuremath{B}$ for all
$g \in \ensuremath{\mathbb{B}}_\ensuremath{\mathcal{H}}(3)$.  Then there exist universal constants
$(\ensuremath{c}_1, \ensuremath{c}_2, \ensuremath{c}_3)$ such that for any $t \geq
\ensuremath{\varepsilon_\numobs}$, we have
\begin{equation}
\label{EqnSandwich}
| \|g\|_\ensuremath{n}^2 - \|g\|_2^2 | \; \leq \; \ensuremath{c}_1 t^2,
\end{equation}
with probability at least $1 - \ensuremath{c}_2 \exp(-\ensuremath{c}_3 \ensuremath{n}
t^2)$.
\elems
\noindent This claim follows from known results on reproducing kernel
Hilbert spaces (e.g., Lemma 5.16 in the paper~\cite{vandeGeer} and
Theorem 2.1 in the paper~\cite{Bartlett05}).  Our final lemma, proved
in Appendix~\ref{AppLemEmpTru}, relates the critical empirical radius
$\ensuremath{\widehat{\varepsilon}_\numobs}$ to the population radius $\ensuremath{\varepsilon_\numobs}$:
\blems
\label{LemEmpTru}
The inequality $\ensuremath{\widehat{\varepsilon}_\numobs} \leq \ensuremath{\varepsilon_\numobs}$ holds with probability at
least $1 - \ensuremath{c}_1 \exp( - \ensuremath{c}_2 \ensuremath{n} \ensuremath{\widehat{\varepsilon}_\numobs}^2)$.
\elems

With these lemmas in hand, the proof of the theorem is
straightforward.  First, from Lemma~\ref{LemHilbBound}, we have
$\|\FUNIT{\ensuremath{\widehat{T}}}\|_{\ensuremath{\mathcal{H}}} \leq 2$ and hence by triangle inequality,
$\|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_{\ensuremath{\mathcal{H}}} \leq 3$ with high probability as
well.  Next, applying Lemma~\ref{LemGeneral} with $t = \ensuremath{\varepsilon_\numobs}$, we
find that
\begin{align*}
\| \FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*} \|_2^2 & \leq \| \FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}
\|_\ensuremath{n}^2 + \ensuremath{c}_1 \ensuremath{\varepsilon_\numobs}^2 \leq \ensuremath{c}_4(\ensuremath{\widehat{\varepsilon}_\numobs}^2 +
\ensuremath{\varepsilon_\numobs}^2),
\end{align*}
with probability greater than $1 -\ensuremath{c}_2 \exp(-\ensuremath{c}_3
\ensuremath{n} \ensuremath{\varepsilon_\numobs}^2)$.  Finally, applying Lemma~\ref{LemEmpTru} yields
that the bound \mbox{$\|\FUNIT{\ensuremath{\widehat{T}}} - \ensuremath{f^*}\|_2^2 \leq \ensuremath{c}
  \ensuremath{\varepsilon_\numobs}^2$} holds with the claimed probability.

\subsection{Proof of Corollaries}
\label{SecProofCorAchieveSmooth}

In each case, it suffices to upper bound the population critical rate
$\ensuremath{\varepsilon_\numobs}^2$ previously defined.

\paragraph{Proof of Corollary~\ref{CorAchieveFinite}:}

In this case, we have
\begin{align*}
\ensuremath{\mathcal{R}}_{\ensuremath{\mathbb{K}}}(\epsilon) & = \frac{1}{\sqrt{\ensuremath{n}}} \sqrt{\sum_{j=1}^m
  \min \{ \ensuremath{\lambda}_j, \epsilon^2 \}} \; \leq \;
\sqrt{\frac{m}{\ensuremath{n}}} \, \epsilon
\end{align*}
so that $\ensuremath{\varepsilon_\numobs}^2 = c' \sigma^2 \frac{m}{\ensuremath{n}}$.

\paragraph{Proof of Corollary~\ref{CorAchieveSmooth}:}

For any $M \geq 1$, we have
\begin{align*}
\ensuremath{\mathcal{R}}_{\ensuremath{\mathbb{K}}}(\epsilon) \; = \; \frac{1}{\sqrt{\ensuremath{n}}} \,
\sqrt{\sum_{j=1}^\infty \min \{C \, j^{-2 \ensuremath{\nu}}, \epsilon^2 \}} &
\leq \sqrt{\frac{M}{\ensuremath{n}}} \epsilon + \sqrt{\frac{C}{\ensuremath{n}}} \,
\sqrt{\sum_{j=\lceil M \rceil}^\infty j^{-2 \ensuremath{\nu}}} \\ & \leq
\sqrt{\frac{M}{\ensuremath{n}}} \epsilon + \sqrt{\frac{C'}{\ensuremath{n}}}
\sqrt{\int_{M}^\infty t^{-2 \ensuremath{\nu}} dt} \\
& \leq \sqrt{\frac{M}{\ensuremath{n}}} \epsilon + C''
\frac{1}{\sqrt{\ensuremath{n}}} (1/M)^{\ensuremath{\nu} -\frac{1}{2}}
\end{align*}
Setting $M = \epsilon^{-1/\ensuremath{\nu}}$ yields $\ensuremath{\mathcal{R}}_\ensuremath{\mathbb{K}}(\epsilon) \leq
C^* \epsilon^{1 - \frac{1}{2 \ensuremath{\nu}}}$.  Consequently, the critical
inequality $\ensuremath{\mathcal{R}}_\ensuremath{\mathbb{K}}(\epsilon) \leq 40 \epsilon^2/\sigma$ is
satisfied for $\ensuremath{\varepsilon_\numobs} \asymp (\sigma^2/\ensuremath{n})^{\frac{2 \ensuremath{\nu}}{2
    \ensuremath{\nu}+1}}$, as claimed.

\subsection{Proof of Proposition~\ref{PropMainRidge}}

We now turn to the proof of our results on the kernel ridge regression
estimate~\eqref{EqnKernelRidge}.  The proof follows a very similar
structure to that of Theorem~\ref{ThmMain}.  Recall the
eigendecomposition $\ensuremath{K} = U \Dmat U^T$ of the empirical kernel
matrix, and that we use $r$ to denote its rank.  For each $\nu >
0$, we define the \emph{ridge shrinkage matrix}
\begin{align}
\label{EqnDefnKERSHRINK}
\ensuremath{R^\PenPar} & \ensuremath{: \, = } \big(I_{\ensuremath{n} \times \ensuremath{n}} + \nu \Dmat
\big)^{-1}.
\end{align}
We then have the following analog of Lemma~\ref{LemBiasVarianceBound}
from the proof of Theorem~\ref{ThmMain}:
\blems[Bias/variance decomposition for kernel ridge regression]
\label{LemRidgeOne}
For any $\nu > 0$, the prediction error for the estimate
$\ensuremath{\widehat{f}}_{\nu}$ is bounded as
\begin{align}
\| \ensuremath{\widehat{f}}_{\nu} - \ensuremath{f^*}\|_\ensuremath{n}^2 & \leq \frac{2}{\ensuremath{n}}
\sum_{j=1}^{r} [\ensuremath{R^\PenPar}]_{jj}^{2} [U^T \ensuremath{f^*}(x_1^\ensuremath{n})]_j^2 +
\frac{2}{\ensuremath{n}} \sum_{j = r+1}^{n}{[U^T \ensuremath{f^*}(x_1^\ensuremath{n})]_j^2} +
\frac{2}{\ensuremath{n}} \sum_{j = 1}^{r} \big(1- \ensuremath{R^\PenPar}_{jj}\big)^{2}
     [U^T w]_j^2.
\end{align}
\elems
\noindent Note that Lemma~\ref{LemRidgeOne} is identical to
Lemma~\ref{LemBiasVarianceBound} with the shrinkage matrices
$\SHRINK{\ensuremath{t}}$ replaced by their analogues $\ensuremath{R^\PenPar}$.  See
Appendix~\ref{AppRidgeOne} for the proof of this claim. \\

\noindent Our next step is to show that the diagonal elements of the
shrinkage matrices $\ensuremath{R^\PenPar}$ are bounded:
\blems[Properties of kernel ridge shrinkage]
\label{LemRidgeTwo}
For all indices $j \in
\{1, 2, \ldots, r\}$, the diagonal entries $\ensuremath{R^\PenPar}$ satisfy the
bounds
\begin{subequations}
\begin{align}
\label{EqnUpperRidgeBias}
0 \; \leq \; (\ensuremath{R^\PenPar}_{jj})^2 \leq \frac{1}{4 \nu
  \ensuremath{\widehat{\ensuremath{\lambda}}}_j}, \quad \mbox{and} \\
\label{EqnLowerRidgeBias}
\frac{1}{2} \min \big \{ 1, \nu \ensuremath{\widehat{\ensuremath{\lambda}}}_j \big \} \; \leq \;
1-\ensuremath{R^\PenPar}_{jj} \; \leq \; \min \big \{1, \nu \ensuremath{\widehat{\ensuremath{\lambda}}}_j \big
\}.
\end{align}
\end{subequations}
\elems
\noindent Note that this is the analog of Lemma~\ref{LemMatrices} from
Theorem~\ref{ThmMain}, albeit with the constant $\frac{1}{4}$ in the
bound~\eqref{EqnUpperRidgeBias} instead of $\frac{1}{2 e}$.  See
Appendix~\ref{AppRidgeTwo} for the proof of this claim.  With these
lemmas in place, the remainder of the proof follows as in the proof of
Theorem~\ref{ThmMain}.

\section{Discussion}

In this paper, we have analyzed the early stopping strategy as applied
to gradient descent on the non-parametric least squares loss.  Our
main contribution was to propose an easily computable and
data-dependent stopping rule, and to provide upper bounds on the
empirical $\ensuremath{{L^2(\mathbb{P}_n)}}$ error (Theorem~\ref{ThmMain}) and population $\ensuremath{L^2(\mathbb{P})}$
error (Theorem~\ref{ThmRandDesign}). We demonstrate in
Corollaries~\ref{CorAchieveSmooth} and ~\ref{CorAchieveFinite} that
our stopping rule yields minimax optimal rates for both low rank
kernel classes and Sobolev spaces. Our simulation results confirm that
our stopping rule yields theoretically optimal rates of convergence
for Lipschitz kernels, and performs favorably in comparison to
stopping rules based on hold-out data and Stein's Unbiased Risk
Estimate. We also showed that early stopping with sum of step-sizes
$\RUNSUM{\ensuremath{t}} = \sum_{k=0}^{t-1}{\Step{k}}$ has a regularization
path that satisfies almost identical mean-squared error bounds as
kernel ridge regression indexed by penalty parameter $\nu$.

Our analysis and stopping rule may be improved and extended in a
number of ways. First, it would interesting to see how our stopping
rule can be adapted to mis-specified models. As specified, our method
relies on computation of the eigenvalues of the kernel matrix.  A
stopping rule based on approximate eigenvalue computations, for
instance via some form of sub-sampling~\cite{DrinMah05}, would be
interesting to study as well.

\subsection*{Acknowledgements}
This work was partially supported by NSF grant DMS-1107000 to MJW and BY. In addition, BY was partially supported by the NSF grant SES-0835531 (CDI), ARO-W911NF-11-1-0114 and the Center for Science of
Information (CSoI), an US NSF Science and Technology Center, under grant agreement CCF-0939370, and MJW was also partially supported ONR MURI grant N00014-11-1-086. During this work, GR received partial support from a Berkeley Graduate Fellowship.


Universal Cycles on 3-Multisets

Abstract: Consider the collection of all t-multisets of {1,...,n}. A universal cycle on multisets is a string of numbers, each of which is between 1 and n, such that if these numbers are considered in t-sized windows, every multiset in the collection is present in the string precisely once. The problem of finding necessary and sufficient conditions on n and t for the existence of universal cycles and similar combinatorial structures was first addressed by DeBruijn in 1946 (who considered t-tuples instead of t-multisets). The past 15 years has seen a resurgence of interest in this area, primarily due to Chung, Diaconis, and Graham's 1992 paper on the subject. For the case t=3, we determine necessary and sufficient conditions on n for the existence of universal cycles, and we examine how this technique can be generalized to other values of t.

\section{Introduction and Previous Work}
Consider the collection of all $t$--multisets over the universe
$[n]=\{1,\ldots, n\}$. A universal cycle (ucycle) on multisets is
a cyclic string $X=a_1a_2...a_k$ with $a_i\in[n]$ for which the
collection $\big\{\{a_1,a_2,...,a_t\},$
$\{a_2,a_3,...,a_{t+1}\},...,$ $\{a_{k-t+1},a_{k-t+2},...,a_k\},$
$\{a_{k-t+2},a_{k-t+3},...,a_k,a_1\},...
\{a_k,a_1,a_2,...,a_{t-1}\} \big\}$ is precisely the collection of
all $t$--multisets over $[n]$, i.e. each $t$--multiset over $[n]$
occurs precisely once in the above collection. For the remainder
of this paper, the term universal cycle will refer to universal
cycles on multisets unless noted otherwise.

Universal cycles do not exist for every value of $n$ and $t$.
Indeed, simple symmetry arguments show that each of the numbers
$1,\ldots,n$ must occur an equal number of times in the ucycle.
Since the length of the ucycle is equal to the number of
$t$--multisets over $[n]$, which is $\binom{n+t-1}{t}$, we must
have that $n\!\left.|\binom{n+t-1}{t}\right..$ While this
condition is necessary, it is not sufficient for the existence of
ucycles.

To date, the bulk of research on ucycles has been devoted to
studying ucycles over sets (as opposed to multisets). Ucycles over
sets are constructed in the same fashion as ucycles over
multisets, except that we consider the collection of all $t$--sets
over $[n]$ instead of the collection of all $t$--multisets, and
our divisibility condition becomes $n\!\left|\binom{n}{t}\right.$.
In \cite{Chung}, Chung, Diaconis and Graham conjectured that for
each value of $t$, there exists a number $n_0(t)$ such that
universal cycles exist for $n\left|\binom{n}{t}\right.$ and $n\geq
n_0(t)$. In \cite{Hurlbert}, Hurlbert consolidated and extended
previous work, verifying the conjecture for $t=2$ and $3$ and
developing partial results for $t=4$ and $6$. In \cite{Godbole},
Godbole et al.~considered universal cycles over multisets for the
case $t=2$, and verified the analogous form of the Chung--Diaconis--Graham conjecture (i.e. with the modified divisibility
criterion) for this case. This work is of particular interest
because Godbole et al.~used a new inductive technique to arrive at
their proof, and in this paper we extend this technique to the
case $t=3$. This new work suggests that the inductive method is a
promising way of addressing the Chung--Diaconis--Graham
conjecture. We also consider a second proof of the conjecture for
the $t=3$ multiset case, which builds off universal cycles on sets
and lends itself more easily to generalization.
\section{An Inductive Proof for Universal Cycles on 3--Multisets}
For $t=3$, the condition $n\left|\binom{n+t-1}{t}\right.$ implies
that $t\equiv 1$ or $2$ (mod 3). We will consider the case
$n\equiv 1$ (mod 3), as the other case can be dealt with
similarly. We will show that for $n\geq 4$, universal cycles exist
whenever $n$ satisfies $n\left|\binom{n+t-1}{t}\right.$

Before describing the proof itself, we will define some
terminology that will be useful for describing universal cycles.
We say that a cyclic string $X=a_1a_2...a_k$ \emph{contains} the
multiset collection $\mathcal{I}$ if
$\mathcal{I}=\big\{\{a_1,a_2,a_3\},\{a_2,a_3,a_4\},...,\{a_{k-2},a_{k-1},{a_k}\},\{a_{k-1},a_k,a_1\},\{a_k,a_1,a_2\}
\big\}$, where each of these sets must be distinct. Clearly
$k=\binom{n+2}{3}$, since this is the number of $3$--multisets on
$[n]$.

For a string $X=a_1a_2...a_k$, we call the \emph{lead-in} of $X$
the substring $a_1a_2$ and the \emph{lead-out} the substring
$a_{k-1}a_k$.

Now, consider the collection of all 3--multisets over $[n]$. We
shall partition this collection into four subcollections. Let
$\mathcal{A}$ be the collection of all 3--multisets over $[n-3]$,
and let $\mathcal{B}$ be the collection of all $3$--multisets over
$\{n-2,n-1,n\}$ and $[n-6]$ which contain at least one element
from $\{n-2,n-1,n\}$. Let $\mathcal{C}$ be the collection of all
3--multisets with one or two elements from $\{n-5,n-4,n-3\}$ and
one or two elements from $\{n-2,n-1,n\}$, and let $\mathcal{D}$ be
the collection of all 3--multisets with one element from each of
$[n-6]$, $\{n-5,n-4,n-3\}$, and $\{n-2,n-1,n\}$. We can see that
$\mathcal{A,B,C},$ and $\mathcal{D}$ are disjoint, and that their
union is the collection of all 3--multisets on $[n]$, as desired.

Now, let $S$ be a universal cycle on $[n-6]$, and since $1,1,1$
must occur somewhere in $S$ and the beginning of $S$ is arbitrary,
we shall have $S$ begin with $1,1,1$. We shall also select $S$ so
that its lead-out is $n-6,n-7$. Thus $S$, when considered as a
cyclic string, contains all $3$--multisets over $[n-6]$, and when
considered as a non-cyclic string, contains all $3$--multisets
except $\{1,n-7,n-6\}$ and $\{1,1,n-7\}$. Let $T$ be a string over
$[n-3]$ such that $ST$---the concatenation of $S$ and $T$---is a
universal cycle over $[n-3]$. It is not clear that such a $T$ must
exist, but we shall find a specific example shortly. In the
example we will find, $T$ will begin with $1,1$ and will end with
$n-3,n-4$. Since $T$ begins with $1,1$, the string $ST$ contains
the multisets $\{1,n-7,n-6\},\ \{1,1,n-7\}$. We can see that the
cyclic string $ST$ contains all of the multisets in $\mathcal{A}$,
and that when $ST$ is considered as a non-cyclic string, it
contains $\mathcal{A}\backslash\big\{\{1,n-4,n-3\},\
\{1,1,n-4\}\big\}$. Now, consider the string $T^\prime$ obtained
by taking $T$ and replacing each instance of $n-5$ by $n-2$, $n-4$
by $n-1$, and $n-3$ by $n$. Since $T$ contained all multisets over
$[n-3]$ which contained at least one element from
$\{n-5,n-4,n-3\}$, we have that $T^\prime$ contains all multisets
over $\{n-2,n-1,n\}$ and $[n-6]$ which contain at least one
element from $\{n-2,n-1,n\}$, i.e. $T^\prime$ contains all the
multisets in $\mathcal{B}$. Since the lead-in of $T$ is $1,1$, the
lead-in of $T^\prime$ is also $1,1$, and since $T$ ends with
$n-3,n-4$, $T^\prime$ ends with $n,n-1$. If we consider the cyclic
string $STT^\prime$, we can see that this string contains all the
multisets in $\mathcal{A}\cup \mathcal{B}$, while the non-cyclic
version of this string is missing the multisets $\{1,n-1,n\},\
\{1,1,n-1\}$.

For notational convenience, we will use the following assignments:
$a:=n-5,\ b:=n-4,\ c:=n-3,\ d:=n-2,\ e:=n-1,$ and $f:=n$. Now,
consider the following string:

\begin{eqnarray*}
V=&&\mathrm{be}(n-6)\mathrm{af}(n-7)\mathrm{be}(n-8)\mathrm{af}(n-9)...\mathrm{af}1\mathrm{be}\\
&&\ \ \ \mathrm{ad}(n-6)\mathrm{ce}(n-7)\mathrm{ad}(n-8)\mathrm{ce}(n-9)...\mathrm{ce}1\mathrm{ad}\\\
&&\ \ \ \ \ \
\mathrm{cf}(n-6)\mathrm{bd}(n-7)\mathrm{cf}(n-8)\mathrm{bd}(n-9)...\mathrm{bd}1\mathrm{cfe}.
\end{eqnarray*}
We can see that this string contains every multiset in
$\mathcal{D}$, as well as the multisets $\{a,b,e\}$, $\{a,d,e\}$,
$\{a,c,d\}$, and $\{c,d,f\}$. Now, the following string (found
with the aid of a computer) contains all of the multisets in
$\mathcal{C}\backslash \big\{\{a,b,e\}$, $\{a,d,e\}$, $\{a,c,d\}$,
$\{c,d,f\} \big\}$:
$$U=\mathrm{aaffc\phantom{1}aeebb\phantom{1}decec\phantom{1}bddcc\phantom{1}fbada\phantom{1}dfbf}.$$

Note that while the multisets $\{b,b,f\}$ and $\{b,e,f\}$ are not
present in the above string $U$, they are present in the
concatenation of $U$ with $V$. Similarly, while $U$ does not
contain $\{a,e,f\}$ and $\{a,a,f\}$, these multisets are present
in the concatenation of $T^\prime$ with $U$.

Now, we can see that the string $STT^\prime UV$ is a universal
cycle over $[n]$ because the non-cyclic string $STT^\prime$
contained all the multisets in
$\mathcal{A}\cup\mathcal{B}\backslash\big\{\{1,n-1,n\},\
\{1,1,n-1\} \big\}$, and it is precisely the multisets
$\{1,n-1,n\}$ and $\{1,1,n-1\}$ which are obtained by the
wrap-around of the lead-out of $V$ with the lead-in of $S$. The
lead-in and lead-out of the other strings has been engineered so
as to ensure that each multiset occurs precisely once.

This completes the induction proof, since the string $ST$ is a
universal cycle over $[n-3]$ (taking the place of $S$ in the
previous iteration of the induction), and the string $T^\prime UV$
extends this cycle to $[n]$ (taking the place of $T$ in the
previous iteration of the induction). Also note that $T^\prime UV$
begins with $1,1$ and ends with $n,n-1$, as required for the
induction hypothesis.

Thus, all that remains is the find a base case from which the
induction can proceed. A possible base case (there are many) for
$n-6=4,\ n-3=7$ is
\begin{eqnarray*}S&=&11144\phantom{1}42223\phantom{1}33121\phantom{1}24343\\
T&=&11522\phantom{1}63374\phantom{1}45166\phantom{1}27732\phantom{1}57366\phantom{1}77135\phantom{1}34641\phantom{1}71555\phantom{1}36127\phantom{1}42556\phantom{1}66477\phantom{1}75526\phantom{1}4576,\end{eqnarray*}
which would lead to
\begin{eqnarray*}
T^\prime&=&11822\phantom{1}93304\phantom{1}48199\phantom{1}20032\phantom{1}80399\phantom{1}00138\phantom{1}34941\phantom{1}01888\phantom{1}39120\phantom{1}42889\phantom{1}99400\phantom{1}08829\phantom{1}4809\\
U&=&55007\phantom{1}59966\phantom{1}89797\phantom{1}68877\phantom{1}06585\phantom{1}8060\\
V&=&69450\phantom{1}36925\phantom{1}01695\phantom{1}84793\phantom{1}58279\phantom{1}15870\phantom{1}46837\phantom{1}02681\phantom{1}709,
\end{eqnarray*}
Where ``0'' denotes 10 and the spacings have been added to
increase readability.

All of the work up to this point has dealt with $n \equiv 1 $ (mod
3). The proof for $n\equiv2$ (mod 3) is similar, so it has been
omitted for the sake of brevity.
\section{A Second Proof of the Existence of Ucycles on 3--Multisets}
In this proof, we construct a ucycle on 3--multisets of $[n]$ by modifying a ucycle on 3--subsets of $[n]$.  (We know from \cite{Hurlbert} that ucycles on 3--subsets of $[n]$ exist for all $n \geq 8$ not divisible by $3$.)  Before giving the proof, we introduce two terms.  We call each element of $[n]$ a \emph{letter}, and each $a_i$ in the ucycle $X=a_1\ldots a_k$ a \emph{character}.  To summarize, a ucycle on 3--multisets of $[n]$ is made up of $\binom{n+t-1}{t}$ characters, each of which equals one of $n$ letters.

To demonstrate the proof's technique, we will first use an argument similar to it to create ucycles on 2--multisets from ucycles on 2--subsets.
We start with this ucycle on 2--subsets of $[5]$:
$$
1234513524
$$
Then, we repeat the first instance of every letter to create the following ucycle on 2--multisets:
$$
112233445513524
$$
The technique works because repeating a character $a_i$ as above adds the multiset $\{a_i,a_i\}$ to the ucycle and has no other effect.

To use this technique on ucycles on 3--subsets, we repeat not single characters, but pairs of characters.  For example, changing
$$
\ldots a_{i-1}a_ia_{i+1}a_{i+2}\ldots
$$
to
$$
\ldots a_{i-1}a_ia_{i+1}a_ia_{i+1}a_{i+2} \ldots
$$
has only has the effect of adding the 3--multisets $\{a_i,a_i,a_{i+1}\}$ and $\{a_i,a_{i+1},a_{i+1}\}$ to the cycle.  In order to use this technique, we will need to know which consecutive pairs of letters appear in a ucycle on 3--subsets.  For instance, the following ucycle (generated using methods from \cite{Hurlbert}) on 3--subsets of $[8]$ contains every unordered pair of letters as consecutive characters but $\{1,5\}$, $\{2,6\}$, $\{3,7\}$, and $\{4,8\}$:
$$
1235783\ 6782458\ 3457125\ 8124672\ 5671347\ 2346814\ 7813561\ 4568236
$$
(The spaces in the cycle are added only for readability.)
This ucycle is missing 4 pairs, which happens to be $n/2$.  This is no coincidence: in fact, this is the most pairs that a ucycle on 3--subsets can fail to contain.

\newtheorem*{missingpairs}{Lemma}
\begin{missingpairs}
No two unordered pairs not appearing as consecutive characters in a ucycle on 3--subsets have a letter in common.  A ucycle can hence be missing at most $n/2$ pairs of letters.
\end{missingpairs}
\begin{proof}
Suppose that we have a ucycle on 3--subsets that contains neither $a$ and $b$ as consecutive characters, nor $a$ and $c$ as consecutive characters, where $a,b,c \in [n]$.  Then the ucycle does not contain the 3--subset $abc$, for all permutations of $abc$ contain either $a$ and $b$ consecutively, or $a$ and $c$ consecutively.  But this is a contradiction, as a ucycle by definition contains all 3--subsets.

Hence, no two pairs of characters missing in the ucycle can have a letter in common.  By the pigeonhole principle, the ucycle can be missing at most $n/2$ pairs of letters.
\end{proof}

With this lemma, we can finish our proof, creating a ucycle on 3--multisets of $[n]$ whenever $n$ is not divisible by 3.
First, we consider the case when $n$ is even.  Let $X$ be a ucycle on 3--subsets of $[n]$.
Let $x_1,\ldots,x_n$ be a permutation of $[n]$ such that
\begin{itemize}
\item $x_1$ equals the first character in $X$.
\item $x_n$ equals the last character in $X$.
\item The list $\{x_1,x_2\}, \{x_3,x_4\}, \ldots, \{x_{n-1}x_n\}$ contains all unordered pairs of letters not contained as consecutive characters in $X$, which is possible by our lemma.  (If $X$ is missing exactly $n/2$ pairs of letters, these pairs will be exactly the pairs missing from $X$.  If $X$ is missing fewer than $n/2$ pairs of letters, then the pairs consist of all missing pairs of letters, plus the remaining letters paired arbitrarily.)
\end{itemize}
Make $X'$ by repeating the first instance of every
unordered pair of letters in $X$ except for $\{x_1,x_2\},
\{x_2,x_3\},$ $\ldots,$ $\{x_{n-1},x_n\},\{x_n,x_1\}$.  The cycle $X'$
now contains all multisets except
\setlength\arraycolsep{1pt}
$$
\{x_1,x_1,x_1\},\ldots,\{x_n,x_n,x_n\}\\
$$
$$
\{x_1,x_1,x_2\},\{x_1,x_2,x_2\},\{x_2,x_2,x_3\},\{x_2,x_3,x_3\},\ldots,\{x_n,x_n,x_1\},\{x_n,x_1,x_1\}
$$
Now, add the string $x_1x_1x_1x_2x_2x_2\ldots x_nx_nx_n$ to the
end of $X'$ to create $X''$.  This provides exactly the missing multisets, creating a
ucycle on 3--multisets.

For example, when $n=8$, we start with the following ucycle on 3--subsets:
\begin{eqnarray*}
X & = & 1235783\ 6782458\ 3457125\ 8124672\\
&& 5671347\ 2346814\ 7813561\ 4568236\\
\end{eqnarray*}
The ucycle on 3--subsets $X$ does not contain the pairs $\{1,5\}$, $\{2,6\}$, $\{3,7\}$, and $\{4,8\}$.  Hence, we set
\begin{eqnarray*}
x_1 & = & 1,\ x_2=5,\ x_3=3,\ x_4=7\\
x_5 & = & 4,\ x_6=8,\ x_7=2,\ x_8=6
\end{eqnarray*}
Note that $x_1$ equals the first character of $X$, and $x_8$ equals the last.

Now, we repeat the first instance of every unordered pair except for $\{1,5\}$, $\{5,3\}$, $\{3,7\}$, $\{7,4\}$, $\{4,8\}$, $\{8,2\}$, $\{2,6\}$, and $\{6,1\}$.  (Note that four of these pairs do not appear in $X$.  If some of these pairs actually did appear in $X$, because $X$ was missing fewer than $n/2$ pairs of letters, it would not affect the proof.):
\begin{eqnarray*}
X' & = &12123235757878383\
63676782424545858\
3434571712525\
81812464672\\
&& 56567131347\
2723468681414\
7813561\
4568236
\end{eqnarray*}
Finally, we add the string $x_1x_1x_1\ldots x_nx_nx_n$ to complete the ucycle:
\begin{eqnarray*}
X'' & = &12123235757878383\
63676782424545858\
3434571712525\
81812464672\\
&& 56567131347\
2723468681414\
7813561\
4568236\\
&& 111555333777444888222666
\end{eqnarray*}

The proof is similar when $n$ is odd, and we omit it for
the sake of brevity.
\section{Further Directions and Remarks}
Both of the proofs given above suggest natural extensions to the
$t=4$ and larger cases, and it is simple to use the
techniques described above to create a proof sketch. In personal
correspondence, Glenn Hurlbert indicated that his technique for creating ucycles on sets in \cite{Hurlbert} can also be used to create ucycles on multisets.
Though this provides a more concise proof for the existence of ucycles on 3--multisets, the two proofs presented may prove useful by their introduction of new techniques for approaching ucycles.  The first proof is notable for its use of induction, a technique which has not been used before to create ucycles.  The second, while it is tied to ucycles on sets, is not tied to any particular approach for creating ucycles on sets; it could perhaps be extended to situations to which Hurlbert's technique cannot.

For values of $n$ and $t$ for which ucycles do exist, one
interesting question is how many ucycles exist. Clearly each
ucycles has $n!$ representations, since there are $n!$
permutations of $1,\ldots,n$. However, when searching for ucycles
using a computer, vast numbers of \emph{distinct} (i.e. not
differing merely by a permutation of $1,\ldots,n$) ucycles were
found. Currently, it is not clear whether $N(n,t)$, the number of
distinct ucycles for a given value of $n$ and $t$, is a function
that has a simple description.
\bibliographystyle{amsplain}

Universal Cycles on 3-Multisets