Universal Average-Case Optimality of Polyak Momentum

Abstract: Polyak momentum (PM), also known as the heavy-ball method, is a widely used optimization method that enjoys an asymptotic optimal worst-case complexity on quadratic objectives. However, its remarkable empirical success is not fully explained by this optimality, as the worst-case analysis -- contrary to the average-case -- is not representative of the expected complexity of an algorithm. In this work we establish a novel link between PM and the average-case analysis. Our main contribution is to prove that any optimal average-case method converges in the number of iterations to PM, under mild assumptions. This brings a new perspective on this classical method, showing that PM is asymptotically both worst-case and average-case optimal.

\section{Introduction}

Polyak momentum (PM), also known as the heavy-ball method, is a widely used optimization method. Originally developed to solve linear equations~\citep{frankel1950convergence, rutishauser1959theory}, it was generalized to smooth functions and popularized in the optimization community by Boris Polyak~\citep{polyak1964some,polyak1987introduction}.
This method has seen a renewed interest in recent years, as its stochastic variant which replaces the gradient with a stochastic estimate is effective on deep learning problems~\citep{sutskever2013importance, zhang20202dive}.

PM also enjoys a locally optimal rate of convergence for strongly convex and twice differentiable objectives. As is common within the optimization literature, this optimality is relative to the \emph{worst-case} analysis, that provides complexity bounds for \emph{any} input from a function class, no matter how unlikely.
Despite its widespread use, the worst-case is not representative of the typical behavior of optimization methods. The simplex method, for example, has a worst-case exponential complexity, that becomes polynomial when considering the average-case \citep{spielman2004smoothed}.

A more representative analysis of the typical behavior is given by the \emph{average-case} complexity, which averages the algorithm's complexity over all possible inputs.
The average-case analysis is standard for analyzing sorting~\citep{knuth1997art} and cryptography~\citep{katz2014introduction} algorithms, to name a few.
However, little is known of the average-complexity of optimization algorithms, whose analysis depends on the often unknown probability distribution over the inputs.

The recent work of \citet{pedregosa2020acceleration, lacotte2020optimal}
overcame this dependency on the input probability distribution through the use of random matrix theory techniques.
In the same papers, the authors noticed the convergence of some optimal average-case methods to PM, as the number of iterations grows (see Figure~\ref{fig:convergence_pm}). This is rather surprising given their crucial differences. For instance, average-case optimal methods use knowledge of the full spectral distribution, while PM only requires knowledge of its edges (i.e., smallest and largest eigenvalue).
Since this convergence was only shown on specific methods, it raises the question on whether this is a spurious phenomenon or if this holds more generally:

\begin{myenv}{Conjecture}
\centering As the number of iterations grows, all average-case optimal methods converge to Polyak momentum.

\vspace{-0.8em}\vphantom{1}
\end{myenv}

\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{figures/polyak_convergence.pdf}
    \caption{{\bfseries Convergence of optimal average-case methods to Polyak Momentum}. For the Marchenko-Pastur and uniform distribution of eigenvalues (left), we construct the method that has optimal average-case complexity and plot the momentum (middle) and steps-size (right) parameters. For the two methods considered, the momentum and step-size parameters converge as the number of iterations grows to those of Polyak momentum, displayed here as a straight line.
    }
    \label{fig:convergence_pm}
\end{figure*}

The {\bfseries main contribution} of this paper is to give a positive answer to this conjecture.
The main, but not so restrictive assumption, is that the probability density function of the eigenvalues is non-zero on the interval containing its support.
With this we can show the previously unknown property that PM is \emph{asymptotically} optimal under the average-case analysis, bringing a new perspectiveon the remarkable empirical performance of this classical method.
Furthermore, this statement is \emph{universal}, i.e., independent of the probability distribution over the inputs.

\subsection{Related work}

This work draws from the fields of optimization, complexity analysis and orthogonal polynomials, of which we comment on the most closely related ideas.

\paragraph{Average-case analysis.}
The average-case analysis has a long history in computer science and numerical analysis. Often it is used to justify the superior performances of algorithms such as Quicksort \citep{Hoare1962Quicksort} and the simplex method in linear programming~\citep{spielman2004smoothed}.
Despite this rich history, it's challenging to transfer these ideas into continuous optimization due to
the ill-defined notion of a typical continuous optimization problem.

In the context of optimization, \citet{pedregosa2020acceleration} derived a framework for analyzing the average-case gradient-based methods and developed methods that are non-asymptotic optimal algorithms with respect to the average-case.
Such average-case analysis finds applications in various domains. For instance, \citet{lacotte2020optimal} use this framework to derive optimal average-case algorithms to minimize least-squares with random matrix sketching. Prior to this stream of papers, \citet{berthier2018accelerated} use methods based on Jacobi polynomials to design average-case optimal gossip methods, but without generalizing the framework.

In the numerical analysis literature,~\citet{deift2019conjugate} have recently developed an average-case complexity of conjugate gradient.

\paragraph{Asymptotics or orthonormal polynomials.} A key ingredient of the proof are asymptotics or orthonormal polynomials. This is a vast subject with applications in stochastic processes~\citep{grenander1958toeplitz}, random matrix theory~\citep{deift1999orthogonal} and numerical integration~\citep{mhaskar1997introduction} to name a few. The monograph of \citep{lubinsky2000asymptotics} discusses all results used in this paper.

\paragraph{Notation.}  Throughout the paper we denote vectors in lowercase boldface (${\boldsymbol x}$), matrices in uppercase boldface letters ($\boldsymbol H$), and polynomials in uppercase latin letter ($P, Q$). We will sometimes omit integration variable, with the understanding that $\int \varphi \mathop{}\!\mathrm{d} \mu$ is a shorthand for $\int \varphi(\lambda) \mathop{}\!\mathrm{d}\mu(\lambda)$.

\section{Average-Case Analysis of Gradient-Based Methods}\label{scs:average_analysis}

The goal of the average-case analysis is to quantify the expected error ${\mathbb{E}\,} \|{\boldsymbol x}_t - {\boldsymbol x}^\star\|^2$, where ${\boldsymbol x}_t$ is the $t$-th update of some optimization method and the expectation is taken over all possible problem instances. To make this analysis tractable, and following \citep{pedregosa2020acceleration}, we consider quadratic optimization problems of the form
\begin{empheq}[box=\mybluebox]{equation*}\tag{OPT}\label{eq:quad_optim}
  \vphantom{\sum_0^i}\min_{{\boldsymbol x} \in {\mathbb R}^d} \Big\{ f({\boldsymbol x}) \stackrel{\text{def}}{=}\!\mfrac{1}{2}({\boldsymbol x}\!-\!{\boldsymbol x}^\star)^\top\!{\boldsymbol H}({\boldsymbol x}\!-\!{\boldsymbol x}^\star) \Big\},
\end{empheq}
where ${\boldsymbol H} \in {\mathbb R}^{d \times d}$ is a \textit{random} symmetric positive-definite matrix and ${\boldsymbol x}^\star$ is a \textit{random} $d$-dimensional vector which is a solution of \eqref{eq:quad_optim}.

\begin{remark} Problem \eqref{eq:quad_optim} subsumes the quadratic minimization problem $\min_{{\boldsymbol x}} {\boldsymbol x}^\top{\boldsymbol H}{\boldsymbol x} + {\boldsymbol b}^\top {\boldsymbol x} + c$ but the notation above will be more convenient for our purposes.
\end{remark}

\begin{remark}
The expectation in ${\mathbb{E}\,} \|{\boldsymbol x}_t - {\boldsymbol x}^\star\|^2$ is over the inputs and not over any randomness of the algorithm, as is common in the stochastic literature. In this paper we only consider deterministic algorithms.
\end{remark}

We consider in this paper the class of \textit{first order methods}, which build ${\boldsymbol x}_t$ using a pre-defined linear combination of an initial guess and previous gradients:
\begin{equation}
    {\boldsymbol x}_{t} \in {\boldsymbol x}_0 + \textbf{span}\{\nabla f({\boldsymbol x}_0),\;\ldots,\;\nabla f({\boldsymbol x}_t)\}. \label{eq:first_order_definition}
\end{equation}
This wide class includes most gradient-based optimization methods, such as gradient descent and momentum. However, it excludes  quasi-Newton methods, preconditioned gradient descent or Adam (to cite a few), as the preconditioning allows the iterates to go outside span.

\subsection{Tools of the trade: orthogonal polynomials and spectral densities}

Average-case optimal methods rely on two key concepts that we now introduce: \textit{residual orthogonal polynomials} and the \textit{expected spectral distribution}.

\subsubsection{Orthogonal (residual) polynomials}

This section defines orthogonal polynomials and residual polynomials.

\begin{definition} \label{def:orthogonal_polynomial}
    Let $\alpha$ be a non-decreasing function such that $\int Q\mathop{}\!\mathrm{d}\alpha$ is finite for all polynomials $Q$.
    We will say that the sequence of polynomials $P_0, P_1, \ldots$ is orthogonal with respect to $\mathop{}\!\mathrm{d}\alpha$ if $P_i$ has degree $i$ and
\begin{equation}\label{eq:optimal_orthogonal_polynomials}
    \int_{\mathbb{R}} P_i\,P_j\mathop{}\!\mathrm{d}\alpha \begin{cases}
    = 0 & \text{if } i\neq j \\
    > 0 & \text{if } i = j
    \end{cases}.
\end{equation}
Furthermore, if they verify $P_i(0) = 1$ for all $i$, we call these {\bfseries residual orthogonal} polynomials.
\end{definition}

Residual orthogonal polynomials verify a three-term recurrence \citep[\S 2.4]{fischer1996polynomial}, that is, there exists a sequence of real values $a_t, b_t$ such that
\begin{equation}\label{eq:recurence_orthogonal_polynomials}
    P_{t}(\lambda)
    = (a_t + b_t \lambda) P_{t-1}(\lambda) + (1-a_{t})P_{t-2}(\lambda)\,,
\end{equation}
where $P_0(\lambda) = 1$ and $P_1(\lambda) = 1+b_1\lambda$\,.

\subsubsection{Expected spectral distribution}

The expected spectral distribution and the extreme eigenvalues of the matrix ${\boldsymbol H}$ play similar roles in the case of, respectively, average-case and worst-case optimal methods. They measure the problem's difficulty and define the optimal method's parameters.
\begin{definition}[Empirical/Expected Spectral Measure]
Let ${\boldsymbol H}$ be a random matrix with eigenvalues $\{\lambda_1, \ldots, \lambda_d\}$. The {\bfseries empirical spectral measure} of ${\boldsymbol H}$, called ${\mu}_{{\boldsymbol H}}$, is the probability measure
\begin{equation}\label{eq:empirical_spectral_density}
    \mu_{{\boldsymbol H}}(\lambda) \stackrel{\text{def}}{=} {\textstyle{\frac{1}{d}\sum_{i=1}^d}} \delta_{\lambda_i}(\lambda) ~,
\end{equation}
where $\delta_{\lambda_i}$ is the Dirac delta, a distribution equal to zero everywhere except at $\lambda_i$ and whose integral over the entire real line is equal to one.

Since ${\boldsymbol H}$ is random, the empirical spectral measure $\mu_{{\boldsymbol H}}$ is a random measure. Its expectation over ${\boldsymbol H}$,
\begin{equation}
\mu \stackrel{\text{def}}{=} {\mathbb{E}\,}_{{\boldsymbol H}}[\mu_{{\boldsymbol H}}]\,,
\end{equation}
is called the {\bfseries expected spectral distribution}.
\end{definition}

\begin{example}[Marchenko-Pastur density and large least squares problems]
Consider a matrix ${\boldsymbol A} \in {\mathbb R}^{n \times d}$, where each entry is an iid random variable with mean zero and variance $\sigma^2$. Then it is known that the expected spectral distribution of ${\boldsymbol H} = \frac{1}{n}{\boldsymbol A}^\top\!{\boldsymbol A}$ converges to to the Marchenko-Pastur distribution \citep{marchenko1967distribution} as $n$ and $d \to \infty$ at a rate in which the asymptotic ratio $d / n \rightarrow r$ is finite.
The Marchenko-Pastur distribution $\dif\mu_{\mathrm{MP}}$ is defined as
\begin{equation} \label{eq:mp_distribution}
     \max\{1 - \tfrac{1}{r}, 0\}\delta_0(\lambda) + \frac{\sqrt{(L - \lambda)(\lambda - \ell)}}{2 \pi \sigma^2 r \lambda}1_{\lambda \in [\ell, L]}\,.
\end{equation}
Here $\ell\stackrel{\text{def}}{=} \sigma^2(1 - \sqrt{r})^2$, $L \stackrel{\text{def}}{=} \sigma^2(1 + \sqrt{r})^2$ are the extreme nonzero eigenvalues,  $\delta_0$ is a Dirac delta at zero (which disappears if $r \geq 1$) and $1_{\lambda \in [\ell, L]}$ is a rectangular window function, equal to 1 for $\lambda \in [\ell, L]$ and 0 elsewhere.

\end{example}

\subsection{Average-case optimal methods}

With these two ingredients, we can construct the method with optimal average-case complexity. We rewrite the expected error as an integral with weight function the expected spectral density $\mu$.

\begin{theorem}\label{thm:pedregosa_rate}\citep{pedregosa2020acceleration}
Assume ${\boldsymbol x}_0$, ${\boldsymbol x}^{\star}$ are random variables independent of ${\boldsymbol H}$,  satisfying $\mathbb{E}[({\boldsymbol x}_0-{\boldsymbol x}^\star)({\boldsymbol x}_0-{\boldsymbol x}^\star)^\top] = R^2\boldsymbol{I}$. Let ${\boldsymbol x}_t$ be generated by a first-order method, associated to the polynomial $P_t$. Then the expected error at iteration $t$ reads
\begin{empheq}{equation}\label{eq:error_norm_x}
  \vphantom{\sum_0^i}\mathbb{E}\|{\boldsymbol x}_t-{\boldsymbol x}^\star\|^2 = {\overbrace{R^2\vphantom{R_t}}^{\text{initialization}}}\int_{\mathbb R} {\underbrace{P_t^2}_{\text{algorithm}}} {\overbrace{\mathop{}\!\mathrm{d}\mu}^{\text{problem}}}
  \,.
\end{empheq}
\end{theorem}

The optimal first order method is obtained by minimizing the above identity over the space of residual polynomials of degree~$t$. This turns out to be equivalent to finding a sequence of residual polynomials $\{P_i\}$ orthogonal w.r.t. the weight function $\lambda \mu(\lambda)$, as shown in the following theorem.

\begin{theorem}\label{thm:optimal_polynomial}
\citep{pedregosa2020acceleration}
Let $a_t$ and $b_t$ be the coefficients of the three-term recurrence \eqref{eq:recurence_orthogonal_polynomials} for the sequence of residual polynomials orthogonal w.r.t. $\lambda \mathop{}\!\mathrm{d}\mu(\lambda)$.
Then the following method has optimal average-case complexity over the class of problems \eqref{eq:quad_optim}:\footnote{Throughout the paper, we will color-code {\color{colormomentum} momentum} and {\color{colorstepsize}step-size} parameters.}
\begin{align} \label{eq:optimal_algorithm}
    &{\boldsymbol x}_1 = {\boldsymbol x}_0 + {\color{colorstepsize}b_1} \nabla f({\boldsymbol x}_0),\\
    &{\boldsymbol x}_{t} = {\boldsymbol x}_{t-1} + {\color{colormomentum}(a_t-1)}({\boldsymbol x}_{t-1}-{\boldsymbol x}_{t-2}) + {\color{colorstepsize}b_t} \nabla f({\boldsymbol x}_{t-1})\nonumber\,.
\end{align}
\end{theorem}

Due to the dependency of the coefficients $a_t, b_t$ on the expected spectral distribution, equation \eqref{eq:optimal_algorithm} does not represents a single scheme, but rather a family of algorithms: each different expected spectral distribution generates a different optimal method. Below is an example of such optimal algorithm w.r.t the Marchenko-Pastur expected spectral distribution.

\begin{example}[Marchenko-Pastur acceleration]\label{ex:mp}
Let $\mathop{}\!\mathrm{d}\mu$ be the density associated with the Marchenko-Pastur distribution. Then, the recurrence of the optimal average-case method associated with this distribution is
\begin{equation*}\label{algo:mp_algo}
\begin{split}
    &\rho = \mfrac{1+r}{\sqrt{r}}\,,~\delta_{0} = 0;\\
    & {\boldsymbol x}_1 = {\boldsymbol x}_0-{\color{colorstepsize}\mfrac{1}{(1+r)\sigma^2}}\nabla f({\boldsymbol x}_0)\,;~\\
    &\delta_{t} = -({\rho+\delta_{t-1}})^{-1}\,; \\
    &{\boldsymbol x}_{t} = {\boldsymbol x}_{t-1} + {\color{colormomentum}\left(1 + \rho\delta_t\right)}({\boldsymbol x}_{t-2} - {\boldsymbol x}_{t-1}) +  {\color{colorstepsize}\mfrac{ \delta_t}{\sigma^2\sqrt{r}}}\nabla f({\boldsymbol x}_{t-1})\,.
\end{split}
\end{equation*}
The coefficients come from the orthogonal polynomials w.r.t. $\lambda \mathop{}\!\mathrm{d}\mu(\lambda)$, which is a shifted Chebyshev polynomials of the second kind.
\end{example}

\subsection{Polyak Momentum and worst-case optimality}

The Polyak momentum algorithm~\citep{polyak1964some}
has an optimal worst-case convergence rate over the class of first order methods with constant coefficients~\citep{polyak1987introduction, scieur2017integration}.
The method requires knowledge of the smallest and largest eigenvalue of the Hessian ${\boldsymbol H}$ (denoted $\ell$ and $L$ respectively) and iterates as follows:
\begin{align}\label{algo:pm_algo}\tag{PM}
&{\boldsymbol x}_1 = {\boldsymbol x}_0 - {\color{brown}\mfrac{2}{L + \ell}}\nabla f({\boldsymbol x}_0)\\
&{\boldsymbol x}_{t+1} = {\boldsymbol x}_t + {\color{colormomentum}\textstyle{\Big(\frac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}}\Big)^2}}({\boldsymbol x}_{t} - {\boldsymbol x}_{t-1}) - {\color{colorstepsize} \textstyle\Big(\frac{ 2}{\sqrt{L}+\sqrt{\ell}}\Big)^2}\nabla  \nonumber
    f({\boldsymbol x}_t)
\end{align}

\vspace{0.5em}\begin{remark} Unlike the Marchenko-Pastur accelerated method of Example~\ref{ex:mp}, coefficients of this method are constant in the iterations. Furthermore, these coefficients only depend on the edges  of the spectral distribution and not on the full density.
\end{remark}

\break

\section{All Roads Lead to Polyak Momentum}\label{scs:asymptotic_heavyball}

\begin{myenv}{Main result}
\begin{theorem}\label{thm:conv_hb}
    Assume the density function $\mathop{}\!\mathrm{d}\mu$ is strictly positive in the interval $[\ell, L]$ with $\ell > 0$ and let ${a_t}, b_t$ be the parameters of the optimal average-case method (Theorem~\ref{thm:optimal_polynomial}). Then these parameters converge to those of \eqref{algo:pm_algo}. More precisely, we have the limits:
    \begin{align} \label{eq:limit_method}
        &\lim_{t \to \infty} {\color{colormomentum} a_t-1} = \underbrace{\left(\mfrac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}}\right)^2}_{=\,\text{\eqref{algo:pm_algo} momentum}}\,, \;\;\quad\text{ and }\\
        &\lim_{t \to \infty}{\color{colorstepsize} b_t} = \underbrace{- \left(\mfrac{2}{\sqrt{L} + \sqrt{\ell}}\right)^2}_{=\,\text{\eqref{algo:pm_algo} step-size}}\,.
    \end{align}
~
\end{theorem}
\end{myenv}

The key insight of the proof is to cast the three-term recurrence of residual \textit{orthogonal} polynomials into orthonormal polynomials\footnote{A sequence $Q_1, Q_2, ...$ of orthogonal polynomials with respect to $\mathop{}\!\mathrm{d}\omega$ is orthonormal if $\int Q_i^2 \mathop{}\!\mathrm{d} \omega = 1$.}  in the interval $[-1,\,1]$. Once this is done, we will use asymptotic properties of these polynomials. The proof is split into three steps.
\begin{itemize}[leftmargin=*]
    \item Step 0 introduces notation and some known results.
    \item Step 1 writes the coefficients of optimal average-case methods in terms of properties of a class of orthonormal polynomials in the $[-1, 1]$ interval.
    \item Step 2 computes the limits of the expressions derived in the previous step by using known asymptotic properties of orthonormal polynomials.
\end{itemize}

{\bfseries \underline{Step 0}: Definitions.} In the classical theory of orthogonal polynomials, the weight function associated with orthogonal polynomials is defined in the interval $[-1, 1]$. However, in our case the spectral densities are instead defined in $[\ell, L]$. To translate results from one setting to the other we define the following linear mapping from $[\ell, L]$ to $[-1, 1]$:
\begin{equation}
    m(\lambda) = \mfrac{L+\ell}{L - \ell} - \mfrac{2}{L-\ell}\lambda\,.
\end{equation}
For notational convenience, we will also use the shorthand $m_0 \stackrel{\text{def}}{=} m(0)$.
We now define $Q_i(m(\cdot))$ as the $i$-th degree orthonormal polynomial with respect to the weight function $\lambda\mu(\lambda)$. That is, the sequence $Q_1, Q_2, \ldots$ verifies
\begin{equation}
    \int_{\ell}^L Q_i(m(\lambda))Q_j(m(\lambda)) \lambda \mathop{}\!\mathrm{d}\mu(\lambda) = \begin{cases} 1 \text{ if $i=j$} \\ 0 \text{ otherwise\,.}\end{cases}\,,
\end{equation}
where $\delta_{ij}$ represents Kronecker's delta.
Like residual orthogonal polynomials, orthonormal  polynomials also verify a three-term recurrence relation. This time, the relation is of the form
\begin{equation}\label{eq:recurrence_orthonormal_poly}
    \alpha_t Q_t(\xi) = (\xi - \beta_t)Q_{t-1}(\xi) - \alpha_{t-1}Q_{t-2}(\xi)\,,
\end{equation}
and depends on coefficients $\alpha_t, \beta_t$:

{\bfseries \underline{Step 1}: Parameters of optimal method and orthonormal polynomials}.
In this step we derive the recurrence relation for an \emph{orthonormal} family with respect to the density $\mathop{}\!\mathrm{d}\nu$. This will allow us to use existing results on the asymptotics of orthonormal polynomials.

\begin{lemma}\label{lemma:orthonormal_a_b}
Let $a_t, b_t$ be the parameters associated with optimal average-case method (Theorem~\ref{thm:optimal_polynomial}). These coefficients verify the following identity,
\begin{align}\label{eq:reformation_a_b}
    &{\color{colormomentum}1 - a_t} = -\frac{\alpha_{t-1}}{ \alpha_t}\frac{Q_{t-2}(m_0)}{ Q_t(m_0)} ~, \quad \text{ and }\\
    &{\color{colorstepsize} b_t} =  - \frac{2}{\alpha_t (L - \ell)} \frac{Q_{t-1}(m_0)}{Q_{t}(m_0)}~.
\end{align}
\end{lemma}
\begin{proof}
Since orthogonality is preserved after multiplication by a scalar, the polynomial $P_t(\lambda) \stackrel{\text{def}}{=} Q_t(m(\lambda)) / Q_t(m_0)$ is also orthogonal with respect to the weight function $\lambda \mathop{}\!\mathrm{d}\mu(\lambda)$. The normalization  $1/Q_t(m_0)$ ensures $P_t$ is a residual polynomial. Note that $Q_t(m_0)$ cannot be zero because $m_0$ lies outside of the weight function's support $[-1,\,1]$.

Using Theorem~\ref{thm:optimal_polynomial}, the coefficients of the optimal average-case method can be derived from the three-term recurrence of this polynomial. Indeed, starting from the three-term recurrence of $Q_i$ \eqref{eq:recurrence_orthonormal_poly}, we obtain for $P_t$
\begin{align*}
P_t(\lambda) &= (m(\lambda)  - \beta_{t-1})\mfrac{Q_{t-1}(m(\lambda))}{ \alpha_t \, Q_t(m_0)} - \alpha_{t-1}\mfrac{Q_{t-2}(m(\lambda))}{ \alpha_t \, Q_t(m_0)}\\
& = \underbrace{\mfrac{1}{\alpha_t}\left(\mfrac{L+\ell}{L - \ell}   - \beta_{t-1} - \mfrac{2}{L-\ell}\lambda\right)\mfrac{Q_{t-1}(m_0)}{ \, Q_t(m_0) }}_{= (a_t + b_t\lambda)} P_{t-1}\\
&\qquad - \underbrace{\mfrac{\alpha_{t-1}}{ \alpha_t}\mfrac{Q_{t-2}(m_0)}{ Q_t(m_0)}}_{=-(1 - a_t)} P_{t-2}(\lambda)\,,
\end{align*}
where in the last line we used the definition of $m$ and the identity $P_{i}(\lambda) = Q_i(m(\lambda))/Q_i(m_0)$ for $i=t-1$ and $i=t-2$.
Finally, matching the coefficients of this recurrence with \eqref{eq:recurence_orthogonal_polynomials} yields the identity in the Lemma.
\end{proof}

{\bfseries \underline{Step 2}: Asymptotics of orthonormal polynomials.} This step uses known result on asymptotics of orthonormal polynomials to compute the limit $t \to \infty$ of expressions derived in the previous step.

We use the following theorem on the asymptotic ratio between two successive orthonormal polynomials.

\begin{theorem}[\citep{rakhmanov1983asymptotics};\footnote{The original version of this theorem was stated for monic orthogonal polynomials but is valid for polynomials with other normalizations like orthonormal, see for instance \citep{lubinsky2000asymptotics, denisov2004rakhmanov}. } Ratio Asymptotics] \label{thm:rakhmanov1983asymptotics}
    Let $\{Q_i\}$ be a sequence of orthonormal polynomials with respect to a weight function strictly positive in $]-1, 1[$, and zero elsewhere. Then  we have the following limit for the ratio of polynomials evaluated outside of the support,
    \begin{equation}\label{eq:rakhmanov}
         \lim_{t\rightarrow\infty} \frac{Q_t(\xi)}{Q_{t-1}(\xi)} = \xi + \sqrt{\xi^2-1} \quad \text{for} \;\; \xi > 1\,.
    \end{equation}
\end{theorem}

We can use this result to compute the limit of the ratio $Q_{t-1}(m_0)/Q_t(m_0)$, that appears in \eqref{eq:reformation_a_b}, as $m(0)> 1$ (and thus is not in the interval $[-1,\,1]$):
\begin{align}
    \lim_{t \to \infty} \mfrac{Q_{t-1}(m_0)}{Q_t(m_0)} &\stackrel{\eqref{eq:rakhmanov}}{=} \Big(\mfrac{L + \ell}{L - \ell} + \sqrt{\big(\mfrac{L + \ell}{L - \ell}\big)^2-1}\big)\Big)^{-1} \nonumber\\
    &= \mfrac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}}\,. \label{eq:limit_ratio_q}
\end{align}

The other dependency of Eq. \eqref{eq:reformation_a_b} on the iteration $t$ is through the coefficients $\alpha_t, \beta_t$. To compute the limits of these we use the following known asymptotics:\footnote{It can be shown that the last two theorems are equivalent \citep[Theorem 13]{nevai1979orthogonal}. However, it will be more convenient for our purposes to present them as independent results.}

\begin{theorem}[\citet{mate1985asymptotics}; Limits of recurrence coefficients] Under the same assumptions as Theorem \ref{thm:rakhmanov1983asymptotics}, the limits of the coefficients $\alpha_t, \beta_t$ in the orthonormal three-terms recurrence (Eq. \ref{eq:recurrence_orthonormal_poly}) is
    \begin{equation}\label{eq:mate}
        \lim_{t\rightarrow\infty}\alpha_t = \mfrac{1}{2}~,\qquad \lim_{t\rightarrow\infty}\beta_t = 0 ~.
    \end{equation}
\end{theorem}

Using this last theorem together with \eqref{eq:limit_ratio_q}, we have
\begin{align*}
\lim_{t \to \infty} {\color{colormomentum}(1 - a_t)} &~~\stackrel{\eqref{eq:reformation_a_b}}{=}
-\left(\lim_{t \to \infty} \mfrac{\alpha_{t-1}}{ \alpha_t}\right) \left( \lim_{t \to \infty}\mfrac{Q_{t-2}(m_0)}{ Q_t(m_0)}\right) \nonumber\\
&\!\stackrel{(\ref{eq:limit_ratio_q}, \ref{eq:mate})}{=} -\Big(\mfrac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}}\Big)^2,
\end{align*}
which is the claimed limit.

To conclude the proof, we compute the same limit for the step-size ${\color{colorstepsize}b_t}$:
\begin{align}
    \lim_{t \to \infty} {\color{colorstepsize}b_t} &\stackrel{\eqref{eq:reformation_a_b}}{=}  - \mfrac{2}{ L - \ell} \left(\lim_{t \to \infty} \alpha_t^{-1}\right) \left(\lim_{t \to \infty} \mfrac{Q_{t-1}(m_0)}{Q_{t}(m_0)}\right) \\
    &\!\!\!\!\stackrel{(\ref{eq:limit_ratio_q}, \ref{eq:mate})}{=} - \left(\mfrac{2}{\sqrt{L} + \sqrt{\ell}}\right)^2\,\,,~\widetilde{r}_1 = r_1, \widetilde{r}_0 = r_0.
\end{align}

\section{Asymptotic Expected Convergence Rates} \label{sec:proof}

The previous section showed convergence of the method's parameters to PM, but said nothing about its rate of convergence. This section fills this gap by providing the asymptotic convergence of the expected convergence rate $\mathbb{E}\|{\boldsymbol x}_t-{\boldsymbol x}^\star\|^2$.
More precisely, we show that the expected convergence rate converges to the rate of convergence of Polyak, and that this convergence rate is \textit{independent} of the probability distribution.

\begin{restatable}{thm}{conv_hb_rate} \label{thm:conv_hb_rate}
    Under the same assumptions of Theorem \ref{thm:conv_hb}, the asymptotic expected rate of convergence of the optimal method converges to the worst-case rate of convergence,
    \begin{equation}\label{eq:asympt_rate}
         \limsup_{t\rightarrow \infty}\sqrt[t]{\mathbb{E}\left[\mfrac{\|{\boldsymbol x}_t-{\boldsymbol x}^\star\|^2}{\|{\boldsymbol x}_0-{\boldsymbol x}^\star\|^2}\right]} = \left(\mfrac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}}\right)^2.
    \end{equation}
\end{restatable}

\begin{proof}
Let $P_t$ be the residual orthogonal polynomial w.r.t. $\lambda \mathop{}\!\mathrm{d}\mu(\lambda)$. \citet[Theorem 3.1]{pedregosa2020acceleration} showed that the expected rate of convergence for average-case optimal methods admits the following simple form
\begin{equation}
    \mathbb{E}\|{\boldsymbol x}_t-{\boldsymbol x}^\star\|^2 =  R^{2}\int_{\mathbb{R}} P_t  \mathop{}\!\mathrm{d}\mu~.
\end{equation}
This form is particularly convenient for us, as we can then use the the three-term recurrence to obtain a recurrence of this expression. Let $r_t = \int_{\mathbb{R}} P_t \mathop{}\!\mathrm{d}\mu$. After using the recurrence over $P_t$,
\begin{align}
    r_t &= \int_{\mathbb{R}} (a_t+\lambda b_t)P_{t-1}(\lambda) + (1-a_t) P_{t-2}(\lambda) \mathop{}\!\mathrm{d}\mu(\lambda)\nonumber\\
    &= a_t \underbrace{\int_{\mathbb{R}} P_{t-1}\mathop{}\!\mathrm{d}\mu}_{=r_{t-1}} + (1-a_t) \underbrace{\int_{\mathbb{R}}P_{t-2} \mathop{}\!\mathrm{d}\mu}_{=r_{t-2}}\,,
\end{align}
where in the last identity we have used the orthogonality between $P_t$ and $P_0(\lambda)=1$ w.r.t. $\lambda \mathop{}\!\mathrm{d}\mu(\lambda)$. In all, we have that the convergence rate $r_t$ is described by the recurrence
\begin{equation}
\begin{split}
    r_t &= a_t r_{t-1} + (1-a_t) r_{t-2}\,,\\
    r_1 &= 1 + b_1 \int_{\mathbb{R}} \lambda \mathop{}\!\mathrm{d}\mu\,,~ r_0 = 1 \,.
\end{split}
\end{equation}

A classical result, often referred to as the Poincar{\'e}-Perron theorem (see for example \citet[Theorem C]{pituk2002more} or \citep[Thm. 8.11]{elaydi2005introduction}  ), states that if $a_t$ has a finite limit --guaranteed by the previous theorem and which we denote $a_{\infty}$-- then the recurrence has a fundamental set of solutions $\{r_t^1, r_t^2\}$ such that
\begin{equation}
    \limsup_{t \to \infty} \sqrt[t]{r^i_t} = |\lambda_i|\quad \text{ $i =1, 2$}\,,
\end{equation}
where $\lambda_i$ are the roots of the characteristic equation ${\lambda^2 - a_{\infty} \lambda - (1 - a_{\infty})}$. In our case, these roots are $1$ and $1 - a_{\infty}$. Now, since the method we're considering is average-case optimal, this limit cannot be larger than that of Polyak momentum, known to be $(\tfrac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}})^2 < 1$. Hence, we can eliminate the solution $r^1_t = 1$ and conclude
\begin{equation}
    \limsup_{t \to \infty} \sqrt[t]{r_t} = (1 - a_{\infty}) = \left(\mfrac{\sqrt{L}-\sqrt{\ell}}{\sqrt{L}+\sqrt{\ell}}\right)^2\,.
\end{equation}

\end{proof}

\section{Discussion and Simulations: Speed of Convergence to PM}

The main result (Theorem~\ref{thm:conv_hb}) shows that, asymptotically, any average-case optimal method converge towards Polyak momentum. This could be interpreted as evidence against average-case optimal methods, as average-case optimal methods are not ``essentially different'' from PM. However, simulations show other dynamics at play.

\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{figures/speed_convergence.pdf}
    \includegraphics[width=\linewidth]{figures/speed_convergence_uniform.pdf}
    \caption{{\bfseries Speed of convergence to Polyak Momentum}. For different parametrizations of the Marchenko-Pastur (\textit{top line}) and uniform (\textit{bottom line}) distributions, we plot the absolute difference between the average-case optimal momentum parameter (\textit{middle}) and average-case optimal step-size (\textit{right}) and the momentum and step-size of the Polyak method.
    The plots show a high anti-correlation between the speed of convergence of optimal average-case methods to PM and problem conditioning: for well-conditioned problems (small condition number) the parameters converge faster to PM than for ill-conditioned (large condition number) problems.
    Thus, in a regime were we perform only a few iterations, Polyak momentum may not be the best choice.}
    \label{fig:speed_convergence}
\end{figure*}

In Figure \ref{fig:speed_convergence} we plot the speed of convergence of the parameters of the optimal average-method for the Marchenko-Pastur distribution with different ratios $r = \frac{d}{n}$ (and hence condition number) and for the uniform distribution with different intervals. We see a clear effect of the condition number on the speed of convergence. The more ill-conditioned the problem, the slower the convergence of the optimal method to PM, implying that PM behaves sub-optimally for a larger number of iterations. This observation is consistent with the results of \citep{pedregosa2020acceleration}, who showed important speedups in the ill-conditioned regime.

\section{Conclusion and Perspectives}

In this work, we've shown that optimal average-case methods for minimizing quadratics converge to PM under mild assumptions on the expected spectral distribution. This universality over the probability measure is somewhat surprising, as Polyak momentum method only depends on the edges of the spectrum, while on the other hand optimal average-case methods depend on the whole spectrum.

A potential area for future work is the analysis of the rate of convergence of optimal method to Polyak momentum algorithm. It seems the convergence of the step-size and momentum parameters are bounded polynomially in the number of iterations. This observation indicates the potential benefit of optimal methods over PM in the case where we perform a small number of iteration, typical in machine-learning problems.

A second research direction is the study of optimal polynomials on the \textit{complex} plane. In this case, we are no longer solving the optimization problem \eqref{eq:quad_optim}. Instead, we aim to solve the linear system ${\boldsymbol A} {\boldsymbol x}={\boldsymbol b}$, where the matrix ${\boldsymbol A}$ is non-symmetric, with potentially complex eigenvalues. This has implication in the study of optimal algorithm in game theory \citep{azizian2020accelerating} or in the acceleration of primal-dual algorithms \citep{bollapragada2018nonlinear}.

Finally, our results are only valid in the strongly convex regime ($\ell > 0$), ruling out the important case $r=1$ in the Marchenko-Pastur distribution, which corresponds to large least squares problems with a square matrix. After the first version of this paper appeared, \citet{paquette2020} derived an average-case analysis for gradient descent and showed a gap between the asymptotic average-case and worst-case convergence rate. The development of average-case optimal methods and the study of their asymptotic limits in this regime remains an open problem.

\clearpage
\section*{Acknowledgements}

We would like to thank our colleague Gauthier Gidel for identifying and reporting some gaps in the proof of Theorem \ref{thm:conv_hb_rate}. A note of gratitude also goes to Reza Babanezad, Simon Lacoste-Julien, Remi Lepriol, Nicolas  Loizou, Adam Ibrahim, Nicolas Leroux and Courtney Paquette for their insightful discussions and relevant remarks. We also thank Francis Bach and Raphaël Berthier for their useful remarks and pointers.


Understanding Nesterov's Acceleration via Proximal Point Method

Abstract: The proximal point method (PPM) is a fundamental method in optimization that is often used as a building block for designing optimization algorithms. In this work, we use the PPM method to provide conceptually simple derivations along with convergence analyses of different versions of Nesterov's accelerated gradient method (AGM). The key observation is that AGM is a simple approximation of PPM, which results in an elementary derivation of the update equations and stepsizes of AGM. This view also leads to a transparent and conceptually simple analysis of AGM's convergence by using the analysis of PPM. The derivations also naturally extend to the strongly convex case. Ultimately, the results presented in this paper are of both didactic and conceptual value; they unify and explain existing variants of AGM while motivating other accelerated methods for practically relevant settings.

\section{Introduction}

The \emph{proximal point method (PPM)}~\cite{moreau1965proximite,martinet1970regularisation,rockafellar1976monotone} is a fundamental method in optimization which solves the minimization of the cost function $f:\mathbb{R}^d\to \mathbb{R}$ by iteratively solving the  subproblem
\begin{align}
\label{prox}
  x_{t+1} \leftarrow \argmin_{x\in \mathbb{R}^d} \left\{ f(x) + \frac{1}{2\eta_{t+1}} \norm{x-x_t}^2 \right\}
\end{align}
for a step size $\eta_{t+1}>0$, where the norm is chosen as the $\ell_2$ norm.
The motivation of the method is clear: we add a quadratic regularization to make the cost function well conditioned for faster optimization\footnote{For instance, when $f$ is nonconvex,  the regularization term can  make each subproblem  \eqref{prox}  convex, and even when $f$ is convex, the regularization term will serve to increase the strong convexity parameter, which results in faster optimization.
}.
Nevertheless, solving \eqref{prox} is in general as difficult as solving the original optimization problem, and PPM is largely regarded as a ``conceptual'' guiding principle for accelerating optimization algorithms~\cite{drusvyatskiy2017proximal}.

On the other hand, there is another prevalent accelerated method called \emph{Nesterov's accelerated gradient method} (AGM)~\cite{nesterov1983method}.
In constrast to PPM, AGM is \emph{implementable} and has been applied to a myriad of applications, including sparse linear regression~\cite{beck2009fast}, compressed sensing~\cite{becker2011nesta}, the maximum  flow problem~\cite{lee2013new}, and deep neural networks~\cite{sutskever2013importance}.
Nonetheless, in contrast with the clear motivation of PPM, AGM has an obscure driving principle.
In particular, original construction of AGM relies on an ingenious yet abstruse technique called \emph{estimate sequence}~\cite[Section 2.2.1]{nesterov2018lectures}, which has motivated researchers to investigate numerous alternative explanations (see Section~\ref{sec:related} for details).

Recently, Defazio~\cite{defazio2019curved} established an inspiring connection  between PPM and AGM. The main observation is that  for strongly convex costs, one can derive  a version of AGM from the primal-dual form of PPM with a tweak of geometry.
This observation constitutes an important step toward understanding AGM because PPM is not as difficult to understand.
This inspiring result, nevertheless, leaves open many other important questions.
Most importantly, \cite{defazio2019curved} lacks \emph{quantitative} explanations as to why AGM achieves the accelerated convergence rates of $O(\nicefrac{1}{T^2})$ for smooth (Definition~\ref{def:ell2}) costs and $O(\exp(\nicefrac{-T}{\sqrt{\kappa}}))$ for smooth and strongly convex (Definition~\ref{def:str}) costs, where $T$ is the number of iterations and $\kappa$ is the condition number of the problem.
Moreover, it is not clear whether such connection can be made without assuming strong convexity and can be extended to other  more general and practical versions of AGM.

In this work, we build a \emph{thorough} understanding of Nesterov's acceleration from the proximal point method by strengthening the connection made in \cite{defazio2019curved}.
The main observation in this paper is that the mysterious updates of AGM can be fully understood by viewing it as a simple approximation of PPM.
In particular, this observation leads to a straightforward derivation of  AGM that does not rely on duality unlike \cite{defazio2019curved}.
Moreover, the PPM view of AGM   offers a simple analysis of AGM based on the  standard analysis of PPM~\cite{guler1991convergence}.
We also demonstrate the generality of our view.
More specifically, our view naturally extends to the strongly convex case and obtains a general\footnote{It is general in the sense that it smoothly interpolates between the  strongly convex case and  the non-strongly convex case.} version of AGM~\cite[(2.2.19)]{nesterov2018lectures}, and our view also gives rise to the key idea of the \emph{method of similar triangles}, a version of AGM shown to have simple extensions to practically relevant settings~\cite{tseng2008accelerated,gasnikov2018universal,nesterov2018lectures}.

\section{Baseline: analysis of  the proximal point method}\label{sec:ppm}

The baseline of our discussion is the following convergence rate of PPM for convex costs  proved  in a seminal paper by G{\"u}ler~\cite{guler1991convergence}  (here $\xs$ denotes a global optimum point, i.e., $\xs \in \argmin_x f(x)$):
\begin{align}
    \boxed{\textstyle f(x_{T})- f(\xs) \leq \OO{\big(\sum_{t=1}^{T} \eta_t\big)^{-1}}\quad \text{for any $T\geq 1$.}} \label{conv:prox}
\end{align}
In words, one can achieve an arbitrarily fast convergence rate by choosing step sizes $\eta_t$'s large.
Below, we review  a short Lyapuov function proof of \eqref{conv:prox}, which will serve as a backbone to other analyses.
\begin{proof}[{\bf Proof of \eqref{conv:prox}}]
It turns out that the following Lyapunov function is suitable:
\begin{align}
    \boxed{\textstyle \Phi_t:=  \big(\sum_{i=1}^{t} \eta_i \big)\cdot \big(f(x_t)-f(\xs)\big) + \frac{1}{2}\norm{\xs-x_t}^2,}\label{def:lya}
\end{align}
where $\Phi_0:= \frac{1}{2}\norm{\xs-x_0}^2$ and here and below, $\norm{\cdot}$ is the $\ell_2$ norm unless stated otherwise.
Now,  it suffices to show that  $\Phi_t$  is decreasing, i.e., $\Phi_{t+1}\leq \Phi_t$ for all $t\geq 0$.
Indeed, if $\Phi_t$ is decreasing, we have $\Phi_T\leq \Phi_0$ for any $T\geq 1$, which  precisely recovers \eqref{conv:prox}. To that end, we use a standard result: \vspace{-5pt}
\begin{propbox}[Proximal inequality~(see e.g. {\cite[Proposition 12.26]{bauschke2011convex}})] \label{prop:per} For a convex function $\phi:\mathbb{R}^d\to \mathbb{R}$,  let $x_{t+1}$ be the unique minimizer of the following proximal step:
\vspace{-5pt}
\begin{align}
    \textstyle x_{t+1} \leftarrow \min_{x\in \mathbb{R}^d} \left\{\phi(x) +\frac{1}{2}\norm{x-x_t}^2\right\}\,. \label{prox:cond}
\end{align}
 \vspace{-10pt}

\noindent Then, for any $u\in \mathbb{R}^d$, $\phi(x_{t+1})-\phi(u) +\frac{1}{2}\norm{u-x_{t+1}}^2 +\frac{1}{2}\norm{ x_{t+1}-x_t}^2  -\frac{1}{2}\norm{u-x_t}^2\leq 0$.
\end{propbox}
Now Proposition~\ref{prop:per} completes the proof as follows:  First, we apply Proposition~\ref{prop:per} with $\phi=\eta_{t+1} f$ and $u=\xs$ and drop the term $\frac{1}{2}\norm{x_{t+1}-x_t}^2$ to obtain:
    \begin{align}
    \tag{B1}  \eta_{t+1}\left[f( x_{t+1}) -f( \xs)\right] + \frac{1}{2} \norm{\xs-    x_{t+1}}^2  -\frac{1}{2} \norm{ \xs-x_t}^2\leq  0\,.\label{ineq:1}
\end{align}
Next, from the optimality of $x_{t+1}$ for \eqref{prox:cond}, it readily follows that
    \begin{align}
    \tag{B2} f(x_{t+1}) - f(x_t) \leq 0\,. \label{ineq:2}
    \end{align}
Now,  computing $\eqref{ineq:1}+( \sum_{i=1}^{t} \eta_i )\times $\eqref{ineq:2} yields $\Phi_{t+1}\leq \Phi_t$, which finishes the proof. \end{proof}
 \vspace{-10pt}

\subsection{Our conceptual question}
Although the convergence rate \eqref{conv:prox} seems powerful, it does not have any practical values as PPM is in general not implementable.
Nevertheless, one can ask the following conceptual question:
\begin{center}
    \emph{Can we develop an implementable approximation of PPM for large step sizes $\eta_t$'s?}
\end{center}
Perhaps, the most straightforward approximation would be to replace the cost function $f$ in \eqref{prox} with its lower-order approximations.
We implement this idea in the next section.

\section{Two simple approximations of the proximal point method} \label{sec:warmup}

To analyze approximation errors, let us assume that the cost function $f$ is $L$-smooth.
\begin{definition}[Smoothness] \label{def:ell2}
For $L>0$, we say a differentiable function $f:\mathbb{R}^d\to \mathbb{R}$ is  $L$-smooth if  $f(x) \leq f(y) + \inp{\nabla f(y)}{x-y} +\frac{L}{2}\norm{x-y}^2 $ for any $x,y\in\mathbb{R}^d$.
\end{definition}
\noindent From the convexity and the $L$-smoothness of $f$, we have the following lower and upper  bounds:
\begin{align*}
\textstyle f(y) +\inp{\nabla f(y)}{x-y}    \leq f(x) \leq f(y) +\inp{\nabla f(y)}{x-y}+\frac{L}{2}\norm{x-y}^2\quad \text{for any $x,y\in \mathbb{R}^d$.}
\end{align*}
In this section, we use these bounds to approximate PPM.
\subsection{First approach: using first-order approximation}
\label{sec:app1}
Let us first replace $f$ in the objective \eqref{prox} with its lower approximation:
\begin{align}
   x_{t+1} \leftarrow \argmin_{x} \left\{ f(x_t)+\inp{\nabla f(x_t)}{x-x_t} + \frac{1}{2\eta_{t+1}} \norm{x-x_t}^2 \right\}\,. \label{gd}
\end{align}
Writing the optimality condition, one quickly notices that \eqref{gd} actually leads to  gradient descent:
\begin{align}
x_{t+1} =x_t -\eta_{t+1} \nabla f(x_{t})\,.\label{gd:1}
\end{align}
Let us see how well \eqref{gd} approximates PPM:
\vspace{-7pt}
\begin{proof}[{\bf Analysis of the first approach}] We first establish counterparts  of  \eqref{ineq:1} and \eqref{ineq:2}.
We begin with \eqref{ineq:1}. We first apply Proposition~\ref{prop:per} with $\phi(x)=\eta_{t+1}[f(x_t)+\inp{\nabla f(x_t)}{x-x_t}]$ and $u=\xs$:
    \begin{align*}
        &\phi(x_{t+1})-\phi(\xs)+ \frac{1}{2} \norm{\xs -x_{t+1}}^2 + \frac{1}{2} \norm{ x_{t+1}-x_t}^2 -\frac{1}{2 } \norm{\xs -x_{t}}^2\leq 0\,.
    \end{align*}
    Now using convexity and $L$-smoothness, we have $\phi(x)\leq \eta_{t+1} f(x)\leq \phi(x) +\frac{L\eta_{t+1}}{2}\norm{x-x_t}^2$, and hence the above inequality implies the following approximate version of \eqref{ineq:1}:
    \begin{align*}
 \eta_{t+1} \left[ f(x_{t+1})- f(\xs)\right] + \frac{1}{2} \norm{\xs -x_{t+1}}^2- \frac{1}{2 } \norm{\xs -x_{t}}^2 \leq   \text{($\mathcal{E}_1$)},
    \end{align*}
    where $\text{($\mathcal{E}_1$)}:= (\frac{L\eta_{t+1}}{2}-\frac{1  }{2} )\norm{x_{t+1}-x_t}^2$.
Next, we use the $L$-smoothness of $f$ and the fact $\nabla f(x_t)= -\nicefrac{1}{\eta_{t+1}}(x_{t+1}-x_t)$ (due to \eqref{gd:1}), to obtain the following counterpart of \eqref{ineq:2}:
    \begin{align*}
         f(x_{t+1}) -f(x_t) \leq  \inp{\nabla f(x_t)}{x_{t+1}-x_t} + \frac{L}{2} \norm{x_{t+1}-x_t}^2 =   \text{($\mathcal{E}_2$)},
    \end{align*}
    where $\text{($\mathcal{E}_2$)}:=( \frac{L}{2}- \frac{1}{\eta_{t+1}}) \norm{x_{t+1}-x_t}^2$.

Now paralleling the proof of \eqref{conv:prox},  to show that $\Phi_t$~\eqref{def:lya} is a valid Lyapunov function, we need to find the step sizes $\eta_t$'s that satisfy the following relation: $\text{($\mathcal{E}_1$)} + ( \sum_{i=1}^t \eta_i) \times \text{($\mathcal{E}_2$)} \leq 0$.
On the other hand, note that both \text{($\mathcal{E}_1$)}~and \text{($\mathcal{E}_2$)}~become positive numbers when $\eta_{t+1}>2/L$.
Hence, the admissible choices for $\eta_t$ at each iteration are upper bounded by $2/L$, which together with the PPM convergence rate \eqref{conv:prox} implies that $O(\nicefrac{1}{\sum_{t=1}^T\eta_t}) =O(\nicefrac{1}{T})$ is the best convergence rate  one can prove.
Indeed, choosing  $\eta_t\equiv 1/L$, then we have $\text{($\mathcal{E}_1$)}=0$ and $\text{($\mathcal{E}_2$)}<0$, obtaining the well-known bound of $f(x_T)-f(\xs) \leq \frac{L\norm{x_0-\xs}^2}{2T} =O(\nicefrac{1}{T})$.
\end{proof}\vspace{-7pt}
\noindent To summarize, the first approach only leads to a disappointing result: the approximation is valid only for the small step size regime of $\eta_t = \OO{1}$.
We empirically verify this fact for a  quadratic cost in Figure~\ref{fig:1}.
As one can see from Figure~\ref{fig:1}, the lower approximation approach \eqref{gd}  overshoots for large step sizes like  $\eta_t = \Theta(t)$ and quickly steers away from PPM iterates.\vspace{-7pt}

\begin{figure}
\begin{subfigure}{.5\textwidth}
  \centering

  \includegraphics[width=\linewidth]{p1.eps} \caption{$\eta_t\equiv 1/3$.}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
  \centering

  \includegraphics[width=\linewidth]{p2.eps}  \caption{$\eta_t=t/3$.}
\end{subfigure}
\caption{ Iterates comparison between PPM~\eqref{prox}, the first approach~\eqref{gd}, the second approach~\eqref{gd:2-1}, and the combined approach~\eqref{approx:ppm}. For the setting, we choose $f(x,y) = 0.1x^2+y^2$ and   $x_0=(10,10)$. }
\label{fig:1} \vspace{-7pt}
\end{figure}
\subsection{Second approach: using smoothness} \label{sec:app2} \vspace{-3pt}
After seeing the disappointing outcome of the first approach, our second approach is to replace $f$ with  its upper approximation  due to the $L$-smoothness:
\begin{align}
     x_{t+1} \leftarrow \argmin_{x} \left\{ f(x_t)+\inp{\nabla f(x_t)}{x-x_t} +\frac{L}{2} \norm{x-x_t}^2+ \frac{1}{2\eta_{t+1}} \norm{x-x_t}^2 \right\}\,. \label{gd:2-1}
\end{align}
Writing the optimality condition,  \eqref{gd:2-1} actually leads to a conservative update of gradient descent:
\begin{align}
x_{t+1} = x_t -\frac{1}{L+\eta_{t+1}^{-1}} \nabla f(x_{t})\,. \label{gd:2-2}
\end{align}
Note that regardless of how large $\eta_{t+1}$ we choose, the actual update step size in \eqref{gd:2-2} is always upper bounded by $\nicefrac{1}{L}$.
Although this conservative update  prevents the  overshooting phenomenon of the first approach, as we increase $\eta_{t}$, this conservative update becomes too tardy to be a good approximation of PPM; see Figure~\ref{fig:1}.

\section{Nesterov's acceleration via alternating two approaches}
\label{sec:deriv}
In the previous section, we have seen that the two simple approximations of PPM both have limitations.
Nonetheless, observe that their limitations are opposite to each other: while the first approach is too ``reckless,'' the second approach is too ``conservative.''
This observation motivates us to consider a \emph{combination} of the two approaches which could mitigate each other's limitation.

Let us implement this idea by alternating between the two approximations \eqref{gd} and \eqref{gd:2-1} of PPM.
The key modification is that for both approximations, we introduce an additional sequence of points $\{y_t\}$ for cost function approximation; i.e., we use the following approximations for the $t$-th iteration:
\begin{align*}
  f(y_t) +\inp{\nabla f(y_t)}{x-y_t}    \leq f(x) \leq f(y_t) +\inp{\nabla f(y_t)}{x-y_t}+\frac{L}{2}\norm{x-y_t}^2\,.
\end{align*}
Indeed, this modification is crucial: if we just use approximations at $x_t$, the resulting alternation merely concatenates  \eqref{gd} and \eqref{gd:2-1} during each iteration, and the two limitations we discussed in Section~\ref{sec:warmup} will remain in the combined approach.

Having introduced a separate sequence $\{y_t\}$ for cost approximations,  we consider the following alternation  where during each iteration, we update $x_{t}$ with  \eqref{gd} and $y_{t}$ with \eqref{gd:2-1}:
\begin{mdframed}[backgroundcolor=gray!20]
{\bf Approximate PPM with alternating two approaches.} Given $x_0\in \mathbb{R}^d$, let $y_0=x_0$ and run:
\begin{subequations} \label{approx:ppm}
  \begin{align}
    \label{ppm:a} &{\textstyle x_{t+1} \leftarrow  \argmin_{x } \left\{ f(y_t)+\inp{\nabla f(y_t)}{x-y_t} + \frac{1}{2\eta_{t+1}} \norm{x-x_t}^2 \right\}}, \\
    \label{ppm:b}
    &{\textstyle y_{t+1}\leftarrow   \argmin_{x} \left\{ f(y_t)+\inp{\nabla f(y_t)}{x-y_t} +\frac{L}{2} \norm{x-y_t}^2+ \frac{1}{2\eta_{t+1}} \norm{x-x_{t+1}}^2 \right\} }.
  \end{align}
\end{subequations}
\end{mdframed}
In  Figure~\ref{fig:1}, we empirically verify that \eqref{approx:ppm} indeed gets the best of both worlds: this combined approach successfully approximates PPM even for the regime $\eta_t=\Theta(t)$.
More remarkably, \eqref{approx:ppm} exactly recovers Nesterov's AGM.
More specifically, turning \eqref{approx:ppm} into the equational form by writing the optimality conditions, and introducing an auxiliary iterate $z_{t+1}:= y_t -\nicefrac{1}{L}\nabla f(y_t)$,  we obtain the following ($z_0:=x_0=y_0$):
 \begin{mdframed}
 \begin{minipage}{0.50\textwidth}
 {\bf An equivalent representation of \eqref{approx:ppm}:}
 \begin{subequations} \label{agm}
  \begin{align}
 & \textstyle y_{t} =\frac{\nicefrac{1}{L}}{ \nicefrac{1}{L}+\eta_t} x_t+\frac{ \eta_t }{  \nicefrac{1}{L}+\eta_t} z_t \,, \label{agm:a}\\
& \textstyle x_{t+1} = x_t -\eta_{t+1} \nabla f(y_{t})\,,\label{agm:b}\\
& \textstyle z_{t+1} = y_{t}-\frac{1}{L}\cdot \nabla f(y_{t})\,. \label{agm:c}
  \end{align}
\end{subequations}
\end{minipage}
\begin{minipage}{0.50\textwidth}
\vspace*{-12pt}
\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.35]
\coordinate   (a1) at (-2,0);
\coordinate   (a2) at (-2,-4.5*0.6);
\coordinate   (a3) at (-2,-4.5);
\coordinate [label=left:{ $\boldsymbol{x_t}$}] (xt) at (0,0);
\coordinate [label=right:{ $\boldsymbol{x_{t+1}}$}] (xt1) at (11,0);
\coordinate [label=left:{  $\boldsymbol{z_{t}}$}] (zt) at (2,-4.5);
\coordinate [label=left: {$\boldsymbol{y_{t}}$}] (yt1) at (2*0.6,-4.5*0.6);
\coordinate [label=right: {  $\boldsymbol{z_{t+1}}$}] (zt1) at (6.0,-4.5*0.6);
\draw [->,>=stealth,dotted, line width=1pt,red] (xt) -- node[below] {\tiny $\textcolor{red}{\boldsymbol{-\eta_{t+1} \nabla f(y_{t})}}$} (xt1);
\draw [->,>=stealth, dotted, line width=1pt,red] (yt1) -- node[below] {\tiny $\textcolor{red}{\boldsymbol{-\frac{1}{L}\nabla f(y_{t})}}$} (zt1);

\draw [<->,dotted] (a1) -- node[left] {\small $  {\eta_t}$}  (a2);
\draw [<->,dotted] (a2) -- node[left] {\small $  {\nicefrac{1}{L}}$}  (a3);
\draw [dashed] (zt) --   (zt1);
\draw [dashed] (zt1) --   (xt1);
\draw [dashed] (xt) --   (zt);
\end{tikzpicture}
\vspace*{-4pt}
    \caption{The  iterates of \eqref{agm}.}
    \label{fig:2}
\end{figure}
\end{minipage}
 \end{mdframed}
Hence, we arrive at AGM without relying on any non-trivial derivations in the literature such as estimate sequence~\cite{nesterov2018lectures} or linear coupling~\cite{allen2014linear}.
To summarize, we have demonstrated:
\begin{center}
    \emph{Nesterov's accelerated method is a simple approximation of the proximal point method!}
\end{center}
\begin{remark}
Our derivation  is inspired by the one in the recent work by Defazio~\cite[Sections 5 and 6]{defazio2019curved}.
However, unlike the approach in \cite{defazio2019curved}, our derivation does not rely on duality, which could be advantageous in the settings where duality fails.
\end{remark}
\begin{remark}[Understanding mysterious parameters of AGM] \label{rmk:par}
It is often the case in the literature that the interpolation step \eqref{agm:a} is written as an abstract form $y_t = \tau_t x_t + (1-\tau_t) z_t$ with a weight parameter $\tau_t>0$ to be chosen~\cite{allen2014linear,lessard2016analysis,wilson2016lyapunov,bansal2019potential, ahn2020nesterov}.
That said, in the previous works, $\tau_t$ is carefully chosen according to the analysis without conveying  much intuition.
One important aspect of our PPM view is that it reveals a close relation between the weight parameter $\tau_t$ and the step size $\eta_t$.
More specifically, $\tau_t$ is chosen so that the ratio of the distances  $\norm{y_t-x_t}:\norm{y_t-z_t}$ is equal to $\eta_t:\nicefrac{1}{L}$ (see Figure~\ref{fig:2}).
\end{remark}

\subsection{Understanding the accelerated convergence rate}
\label{subsec:conv}
In order to determine $\eta_t$'s in \eqref{agm},  we revisit the analysis of PPM from Section~\ref{sec:warmup}.
In turns out that following Section~\ref{sec:app1}, one can derive from first principles the following inequalities  using Proposition~\ref{prop:per}  (we defer the derivations to Appendix~\ref{app:agm}):
\begin{align*}
    \text{Counterpart of }\eqref{ineq:1}: &\quad\eta_{t+1}(f(z_{t+1})- f(\xs)) + \frac{1}{2} \norm{\xs -x_{t+1}}^2-
   \frac{1}{2} \norm{\xs -x_{t}}^2\leq \text{($\mathcal{F}_1$)}\quad \text{and}\\
   \text{Counterpart of }\eqref{ineq:2}: &\quad  f(z_{t+1}) -f(z_t) \leq \text{($\mathcal{F}_2$)}\,,
\end{align*}
where $\text{($\mathcal{F}_1$)}:=(\frac{\eta_{t+1}^2}{2}-\frac{\eta_{t+1}}{2L}) \norm{\nabla f(y_{t})}^2 + L\eta_t\eta_{t+1}\inp{\nabla f(y_{t})}{ z_{t}-y_t}$ and $\text{($\mathcal{F}_2$)}:= -\frac{1}{2L} \norm{\nabla f(y_{t})}^2$ $- \inp{\nabla f(y_{t})}{z_{t}-y_t}$.
Hence, we modify the Lyapunov function \eqref{def:lya} by replacing  the first $x_t$ with $z_t$:
\begin{align}
    \textstyle \Phi_t:= (\sum_{i=1}^{t} \eta_i)\cdot (f(z_t)-f(\xs)) + \frac{1}{2}\norm{\xs-x_t}^2\,. \label{lya2}
\end{align}
Then as before, to prove the validity of the chosen Lyapunov function, it suffices to verify $\text{($\mathcal{F}_1$)} + ( \sum_{i=1}^t \eta_i)\cdot \text{($\mathcal{F}_2$)} \leq 0$, which is equivalent to
\begin{align}
    \label{cond:1} \textstyle \frac{1}{2L}\left( L\eta_{t+1}^2 - \sum_{i=1}^{t+1} \eta_i\right)   \norm{\nabla f(y_t)}^2 + \left( L\eta_t\eta_{t+1} - \sum_{i=1}^t \eta_i\right)\inp{\nabla f(y_{t})}{z_{t}-y_t} \leq 0
\end{align}
From \eqref{cond:1}, it suffices to choose $\{\eta_t\}$ so that
$L\eta_t\eta_{t+1}=\sum_{i=1}^t \eta_i$.
Indeed, with such a choice, the coefficient of the inner product term in \eqref{cond:1} becomes zero and the coefficient of the squared norm term becomes $\nicefrac{1}{2L}(L\eta_{t+1}^2 - L\eta_{t+1}\eta_{t+2})\leq 0$ (if $\{\eta_t\}$ is increasing).
Indeed, one can quickly notice that choosing $\eta_{t} =\nicefrac{t}{2L}$ satisfies  the desired relation.
Therefore,  we obtain the well known accelerated convergence rate of $f(z_T)-f(\xs)\leq \frac{2L \norm{x_0-\xs}^2}{T(T+1)}= O(\nicefrac{1}{T^2})$~\cite{nesterov1983method}.

\subsection{Separating step sizes for flexibility} \label{subsec:two}

Since \eqref{approx:ppm} is an approximation of PPM, it is helpful to give \eqref{approx:ppm} more flexibility when we try to extend it to other settings.
In particular, we relax \eqref{approx:ppm} by separating the two step sizes:
\begin{mdframed}
{\bf Approximate PPM with two separate step sizes $\{\eta_t\}$ and $\{\widetilde{\eta}_t\}$.} Given $x_0=y_0\in \mathbb{R}^d$,
\begin{subequations}\label{approx:ppm2}
 \begin{align}
    \label{ppm2:a} &{\textstyle x_{t+1} \leftarrow  \argmin_{x } \left\{ f(y_t)+\inp{\nabla f(y_t)}{x-y_t} + \frac{1}{2\eta_{t+1}} \norm{x-x_t}^2 \right\}}, \\
    \label{ppm2:b}
    &{\textstyle y_{t+1}\leftarrow   \argmin_{x} \left\{ f(y_t)+\inp{\nabla f(y_t)}{x-y_t} +\frac{L}{2} \norm{x-y_t}^2+ \frac{1}{2\widetilde{\eta}_{t+1}} \norm{x-x_{t+1}}^2 \right\} }.
  \end{align}
\end{subequations}
\end{mdframed}
As we shall see in the next subsection, this simple relaxation allows us to recover a well known general version of AGM~\cite[Section 2.2]{nesterov2018lectures}.

\subsection{Acceleration for strongly convex costs} \label{sec:str}
Let us  apply our PPM view  to the strongly convex cost case. We begin with the definition:
\begin{definition}[Strong convexity] \label{def:str}
For $\mu>0$, we say a differentiable function $f:\mathbb{R}^d\to \mathbb{R}$ is  $\mu$-strongly convex if   $f(x) \geq f(y) + \inp{\nabla f(y)}{x-y} +\frac{\mu}{2}\norm{x-y}^2 $ for any $x,y\in\mathbb{R}^d$.
\end{definition}
\noindent Since $f$ is additionally assumed to be strongly convex, one can strengthen the step \eqref{ppm2:a} by
\begin{align}
      \label{ppm:str}
  x_{t+1} \leftarrow \argmin_{x\in \mathbb{R}^d} \left\{ f(y_t)+\inp{\nabla f(y_t)}{x-y_t} + \frac{\mu}{2}\norm{x-y_t}^2 + \frac{1}{2\eta_{t+1}} \norm{x-x_t}^2 \right\}\,.
\end{align}
Writing the optimality condition of \eqref{ppm:str}, it is straightforward to check that \eqref{approx:ppm2} is equivalent to the following form (again, we introduce another auxiliary iterate $w_t$, and let $z_0:=x_0=y_0$):
 \begin{mdframed}
 \begin{minipage}{0.5\textwidth}
 {\bf Approximate PPM  for strongly convex costs:}
 \vspace*{-10pt}
 \begin{subequations}\label{agm2}
  \begin{align}
  & \textstyle y_{t} =\frac{\nicefrac{1}{L}}{  \nicefrac{1}{L}+\widetilde{\eta}_t} x_t +\frac{ \widetilde{\eta}_t }{   \nicefrac{1}{L}+\widetilde{\eta}_t} z_t\,, \label{agm2:a}\\
 & \textstyle w_t = \frac{\nicefrac{1}{\mu}}{ \nicefrac{1}{\mu}+\eta_{t+1}}x_t +\frac{\eta_{t+1}}{ \nicefrac{1}{\mu} +\eta_{t+1}}y_t\\
& \textstyle x_{t+1} = w_t- \frac{\nicefrac{1}{\mu}\cdot \eta_{t+1}}{ \nicefrac{1}{\mu}+\eta_{t+1}}\nabla f(y_{t})\,,\label{agm2:b}\\
& \textstyle z_{t+1} = y_{t}-\frac{1}{L}\nabla f(y_{t})\,. \label{agm2:c}
  \end{align}
\end{subequations}
\end{minipage}
\begin{minipage}{0.5\textwidth}
\vspace{-15pt}
\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.35]
\coordinate   (a1) at (-3,0);
\coordinate   (a2) at (-3,-7*0.7);
\coordinate   (a3) at (-3,-7);

\coordinate   (b1) at (-2.5,0);
\coordinate   (b2) at (-2.5,-7*0.3);
\coordinate   (b3) at (-2.5,-7*0.7);
\coordinate [label=left:{ $\boldsymbol{x_t}$}] (xt) at (0,0);
\coordinate [label=right:{$\boldsymbol{x_{t+1}}$}] (xt1) at (11,0);
\coordinate  [label=left:{ $\boldsymbol{w_t}$}]   (wt) at (1*0.3,-7*0.3);
\coordinate [label=left:{  $\boldsymbol{z_{t}}$}] (zt) at (1,-7);
\coordinate [label=left: {$\boldsymbol{y_{t}}$}] (yt1) at (1*0.7,-7*0.7);
\coordinate [label=below: {$\boldsymbol{z_{t+1}}$}] (zt1) at (1+10*0.3/0.7,-7+7*0.3/0.7);
\draw [->,>=stealth,dotted, line width=1pt,red] (wt) -- node[above] {\tiny $\textcolor{red}{\boldsymbol{-\frac{\eta_{t+1}\cdot \nicefrac{1}{\mu}}{ \eta_{t+1}+\nicefrac{1}{\mu}} \nabla f(y_{t})}}$} (xt1);
\draw [->,>=stealth, dotted, line width=1pt,red] (yt1) -- node[above] {\tiny $\textcolor{red}{\boldsymbol{-\frac{1}{L}\nabla f(y_{t})}}$} (zt1);

\draw [<->,dotted] (a1) -- node[left] {\small $  {\widetilde{\eta}_t}$}  (a2);
\draw [<->,dotted] (a2) -- node[left] {\small $  {\nicefrac{1}{L}}$}  (a3);
\draw [dashed] (zt) --   (zt1);
\draw [dashed] (zt1) --   (xt1);
\draw [dashed] (xt) --   (zt);

\draw [<->,dotted] (b1) -- node[right] {\scriptsize $  {\eta_{t+1}}$}  (b2);
\draw [<->,dotted] (b2) -- node[right] {\small $  {\nicefrac{1}{\mu}}$}  (b3);
\end{tikzpicture}
\vspace*{-4pt}
    \caption{The  iterates of \eqref{agm2}.}
    \label{fig:2-2}
\end{figure}
\end{minipage}
 \end{mdframed}
Paralleling Remark~\ref{rmk:par}, our derivation provides new insights into the choices of the AGM step sizes by expressing them in terms of the PPM step sizes $\eta_t$'s and $\widetilde{\eta}_t$'s.
Furthermore, our derivation actually demystifies the mysterious parameter choices made in the Nesterov's book~\cite[Section 2.2]{nesterov2018lectures}.
To see this, let us recall the well known convergence rate of PPM for strongly convex costs due to Rockafellar~\cite[(1.14)]{rockafellar1976monotone}:
\begin{align}
    \boxed{\textstyle f(x_{T})- f(\xs) \leq \OO{\prod_{t=1}^{T} (1+\mu\eta_t)^{-1}}\quad \text{for any $T\geq 1$.}} \label{conv:prox:str}
\end{align}
In light of \eqref{conv:prox:str}, it turns out that  for the approximate PPM \eqref{agm2}, choosing the following step sizes
\begin{center}
    $\eta_t \equiv \eta:= \mu^{-1}\cdot  (\sqrt{\kappa}-1)^{-1}$ and $\widetilde{\eta}_t \equiv \widetilde{\eta}:= \mu^{-1}\cdot  (\sqrt{\kappa})^{-1}$  (where $\kappa:=\nicefrac{L}{\mu}$)
\end{center}
 actually recovers the well known parameters choice~\cite[(2.2.22)]{nesterov2018lectures} which leads to the convergence rate of $\OO{(1+\eta \mu)^{-T}}=\OO{(1+\nicefrac{1}{(\sqrt{\kappa}-1)})^{-T}}$. See \cite[Section 5.5]{bansal2019potential} for a simple Lyapunov function proof  of this convergence rate.

 \begin{remark}[Nesterov's general method]
Remarkably, \eqref{agm2} also exactly recovers a general version of AGM~\cite[(2.2.19)]{nesterov2018lectures} which smoothly interpolates between the strongly convex case and the non-strongly convex case.
To see this, we follow a simple equivalent representation of Nesterov's general step sizes  given in \cite{ahn2020nesterov}.
More specifically, given a sequence $\{\xi_t\}$ of positive numbers defined as per the nonlinear recursion $\frac{\xi_{t+1}(\xi_{t+1}-\kappa^{-1})}{1-\xi_{t+1}}=\xi_t^2$ with an initial value $\xi_0>0$,  choosing $\eta_t=\mu^{-1}\cdot  (\kappa \xi_t-1)^{-1}$  and $\widetilde{\eta}_t =  \mu^{-1}\cdot (\kappa \xi_{t+1}-1)^{-1}\cdot  (1-\xi_{t+1})$ exactly recovers the choices in~\cite[Section 2]{ahn2020nesterov} which are shown to be equivalent to Nesterov's choices~\cite[(2.2.19)]{nesterov2018lectures}.
\end{remark}

 \section{Simple generalizations with similar triangles}
\label{sec:tri}
In the previous section, we have demonstrated that Nesterov's method is nothing but an approximation of PPM.
This view point has not only provided  simple derivations of versions of  AGM, but also offered clear explanations of the step sizes. In this section, we demonstrate that these interpretations offered by  PPM  actually lead to a great simplification of Nesterov's AGM in the form of the \emph{method of similar triangles}~\cite{nesterov2018lectures,gasnikov2018universal} which admits simple generalizations to practically relevant settings.

Our starting point is the observations made in the previous section:  (i) from Remark~\ref{rmk:par}, we have seen $\norm{y_t-x_t}:\norm{y_t-z_t}=\eta_t:\nicefrac{1}{L}$; (ii) from Section~\ref{subsec:conv}, we have seen that we need to choose $\eta_t=\Theta(t)$, and hence, $\eta_{t+1}\approx\eta_t \gg 1$.
 From these observations, one can readily see that the triangle $\triangle x_tx_{t+1}z_t$ is approximately similar to $\triangle y_tz_{t+1} z_t$.
 Therefore, one can simplify AGM by further exploiting this fact: we  modify the updates so that the two triangles are indeed \emph{similar}:
\begin{mdframed}
 \begin{minipage}{0.50\textwidth}
 {\bf Similar triangle approximation of PPM:}
 \begin{subequations}\label{sim}
  \begin{align}
  & \textstyle y_{t} = \frac{\nicefrac{1}{L}}{ \nicefrac{1}{L}+\eta_t} x_t+\frac{ \eta_t }{ \nicefrac{1}{L}+\eta_t} z_t\,, \label{sim:a}\\
& \textstyle x_{t+1} = x_t -\eta_{t+1} \nabla f(y_{t})\,,\label{sim:b}\\
 & \textstyle z_{t+1} = \frac{\nicefrac{1}{L}}{ \nicefrac{1}{L}+\eta_t} x_{t+1}+\frac{ \eta_t }{ \nicefrac{1}{L}+\eta_t } z_t\,. \label{sim:c}
  \end{align}
\end{subequations}
\end{minipage}
\begin{minipage}{0.50\textwidth}
\vspace*{-12pt}
\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.35]
\coordinate   (a1) at (-2,0);
\coordinate   (a2) at (-2,-4.5*0.6);
\coordinate   (a3) at (-2,-4.5);
\coordinate [label=left:{ $\boldsymbol{x_t}$}] (xt) at (0,0);
\coordinate [label=right:{ $\boldsymbol{x_{t+1}}$}] (xt1) at (11,0);
\coordinate [label=left:{  $\boldsymbol{z_{t}}$}] (zt) at (2,-4.5);
\coordinate [label=left: {$\boldsymbol{y_{t}}$}] (yt) at (2*0.6,-4.5*0.6);
\coordinate [label=right: {  $\boldsymbol{z_{t+1}}$}] (zt1) at (2+9*0.4,-4.5*0.6);
\draw [->,>=stealth, dotted, line width=0.8pt,red] (xt) -- node[below] {\tiny $\textcolor{red}{\boldsymbol{-\eta_{t+1} \nabla f(y_{t})}}$} (xt1);
\draw [->,>=stealth,dotted, line width=0.8pt,red] (yt) --  (zt1);

\draw [<->,dotted] (a1) -- node[left] {\small $  {\eta_t}$}  (a2);
\draw [<->,dotted] (a2) -- node[left] {\small $  {\nicefrac{1}{L}}$}  (a3);
\draw [dashed] (zt) --   (xt1);
\draw [dashed] (xt) --   (zt);
 \draw   pic[draw=black, fill=gray!10, -, angle eccentricity=1.2, angle radius=0.5cm, line width=1pt]
    {angle=xt--xt1--zt};
 \draw   pic[draw=black, fill=gray!10,  -, angle eccentricity=1.2, angle radius=0.5cm, line width=1pt]
    {angle=yt--zt1--zt};
 \draw   pic[draw=black,  fill=gray!10,-, densely dotted, angle eccentricity=1.2, angle radius=0.3cm, line width=1pt]
    {angle=zt--yt--zt1};
 \draw   pic[draw=black, fill=gray!10, -, densely dotted, angle eccentricity=1.2, angle radius=0.3cm, line width=1pt]
    {angle=zt--xt--xt1};
\end{tikzpicture}
\vspace*{-5pt}
    \caption{The similar triangle updates \eqref{sim}.}
    \label{fig:sim}
\end{figure}
\end{minipage}
 \end{mdframed}
 We provide a PPM-based analysis of \eqref{sim} for a more general setting in Section~\ref{sec:gen}.
 \begin{remark}
The updates akin to \eqref{sim} can be found in~\cite[Algorithm 1]{tseng2008accelerated} (note that the step sizes are slightly different).
 Our derivation based on the PPM view indeed clarifies why such similar triangles are natural updates to consider.
 We also remark that the updates based on similar triangles is useful in developing universal methods for stochastic composite optimizations~\cite{gasnikov2018universal}.
  \end{remark}
  \begin{remark}[Momentum methods] \label{rmk:momentum}
 An alternative way to approximate PPM with similar triangles is:
\begin{mdframed}
 \begin{minipage}{0.50\textwidth}
 {\bf Alternative similar triangle approximation:}
 \begin{subequations}\label{sim2}
  \begin{align}
  & \textstyle y_{t} = \frac{\nicefrac{1}{L}}{ \nicefrac{1}{L}+\eta_t} x_t+\frac{ \eta_t }{ \nicefrac{1}{L}+\eta_t} z_t\,, \label{sim2:a}\\
& \textstyle z_{t+1} = y_t -\frac{1}{L} \nabla f(y_{t})\,,\label{sim2:b}\\
 & \textstyle x_{t+1} =z_{t+1} + L\eta_t (z_{t+1}-z_t)\,. \label{sim2:c}
  \end{align}
\end{subequations}
\end{minipage}
\begin{minipage}{0.50\textwidth}
\vspace*{-12pt}
\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.35]
\coordinate   (a1) at (-2,0);
\coordinate   (a2) at (-2,-4.5*0.6);
\coordinate   (a3) at (-2,-4.5);
\coordinate [label=left:{ $\boldsymbol{x_t}$}] (xt) at (0,0);
\coordinate [label=right:{ $\boldsymbol{x_{t+1}}$}] (xt1) at (11,0);

\filldraw (2,-4.5) circle (3pt) node[] {};
\coordinate [label=left:{  $\boldsymbol{z_{t}}$}] (zt) at (2,-4.5);

\filldraw (2*0.6,-4.5*0.6) circle (3pt) node[] {};
\coordinate [label=left: {$\boldsymbol{y_{t}}$}] (yt) at (2*0.6,-4.5*0.6);

\filldraw (2+9*0.4,-4.5*0.6) circle (3pt) node[] {};
\coordinate [label=below: {  $\boldsymbol{z_{t+1}}$}] (zt1) at (2+9*0.4,-4.5*0.6);

\filldraw (2+9*0.6,-4.5*0.4) circle (3pt) node[] {};
\coordinate [label=above: {  $\boldsymbol{y_{t+1}}$}] (yt1) at (2+9*0.6,-4.5*0.4);
\draw [->,>=stealth, dotted, line width=0.8pt,red] (xt) -- (xt1);
\draw [->,>=stealth,dotted, line width=0.8pt,red] (yt) --
node[above] {\tiny $\textcolor{red}{\boldsymbol{-\frac{1}{L} \nabla f(y_{t})}}$} (zt1);

\draw [<->,dotted] (a1) -- node[left] {\small $  {\eta_t}$}  (a2);
\draw [<->,dotted] (a2) -- node[left] {\small $  {\nicefrac{1}{L}}$}  (a3);
\draw [dashed] (zt) --   (xt1);
\draw [dashed] (xt) --   (zt);
 \draw   pic[draw=black, fill=gray!10, -, angle eccentricity=1.2, angle radius=0.5cm, line width=1pt]
    {angle=xt--xt1--zt};
 \draw   pic[draw=black, fill=gray!10,  -, angle eccentricity=1.2, angle radius=0.5cm, line width=1pt]
    {angle=yt--zt1--zt};
 \draw   pic[draw=black,  fill=gray!10,-, densely dotted, angle eccentricity=1.2, angle radius=0.3cm, line width=1pt]
    {angle=zt--yt--zt1};
 \draw   pic[draw=black, fill=gray!10, -, densely dotted, angle eccentricity=1.2, angle radius=0.3cm, line width=1pt]
    {angle=zt--xt--xt1};
\end{tikzpicture}
\vspace*{-5pt}
    \caption{The updates of \eqref{sim2}.}
    \label{fig:sim2}
\end{figure}
\end{minipage}
 \end{mdframed}
 In fact, \eqref{sim2} can be equivalently expressed without $\{x_t\}$: at the $t$-iteration, we compute \eqref{sim2:b} and then update $y_{t+1}$ via $y_{t+1}= z_{t+1} + \frac{L\eta_t}{L\eta_{t+1}+1}(z_{t+1}-z_t)$ (these updates are illustrated with dots in Figure~\ref{fig:sim2}).
 Hence, \eqref{sim2} is equal to the well-known momentum based AGM~\cite{nesterov1983method,beck2009fast}.
 Notably, it turns out that our  PPM-based analysis suggests the choice of $\{\eta_t\}$ as per the recursive relation $(L\eta_{t+1}+\frac{1}{2})^2= (L\eta_t+1)^2 +\frac{1}{4}$, which after substitution $L\eta_t+1\leftarrow a_t$ exactly recovers the popular recursive relation $a_{t+1} = \frac{1}{2} (1+\sqrt{1+4a_t^2})$ in \cite{nesterov1983method,beck2009fast}.
 See Appendix~\ref{app:rmk} for precise details.
  \end{remark}
 The main advantage of this similar triangles approximation \eqref{sim} becomes clearer in the constraint  optimization case: when there is a constraint set, the  steps \eqref{agm:b} and \eqref{agm:c} both become projections steps which could be costly when the constraint set does not admit simple projections.
On the other hand, since \eqref{sim} only requires a single projection in each iteration, it minimizes such costly computations.

  \subsection{Extension to  composite costs and general norms}
  \label{sec:gen}
  It turns out  \eqref{sim} admits a simple extension to the practically relevant setting of composite costs and general norms (see e.g.  \cite[Section 6.1.3]{nesterov2018lectures}).
  More specifically, for a closed convex set $Q\subseteq \mathbb{R}^d$ and a closed\footnote{This means that the epigraph of the function is closed. See \cite[Definition 3.1.2]{nesterov2018lectures}. } convex function $\Psi:Q\to \mathbb{R}$, consider \begin{align*}
    \textstyle \min_{x\in Q} f^{\Psi}(x):= f(x)+\Psi(x) \,,
\end{align*}
where $f:Q\to \mathbb{R}$ is a differentiable convex function which is $L$-smooth with respect to a norm $\norm{\cdot}$ that is not necessarily  the $\ell_2$ norm (i.e., we regard the norm in Definition~\ref{def:ell2} to be our chosen norm).
\noindent For the general norm case, we use the Bregman divergence for the regularizer\footnote{The reason why we need the Bregman divergence in place of the norm squared regularization is because for norms other than the $\ell_2$ norm, $\frac{1}{2}\norm{u-v}^2$ is not strongly convex with respect to the chosen norm. }:
\begin{definition}
Given a $1$-strongly convex (w.r.t the chosen norm $\norm{\cdot}$) function $h:Q\to \mathbb{R} \cup \{\infty\}$  that is differentiable on the interior of $Q$, $\breg{h}{u}{v}:=h(u)-h(v)-\inp{\nabla h(v)}{u-v}$ for all $u,v\in Q$.
\end{definition}
\noindent Under the above setting and assumption, \eqref{sim} admits a simple generalization:
\begin{mdframed}
 {\bf Similar triangle approximations of PPM for composite costs and general norms:}
 \begin{subequations}\label{simg}
  \begin{align}
 &\textstyle y_{t} =\frac{\nicefrac{1}{L}}{\nicefrac{1}{L}+\eta_t } x_t+\frac{ \eta_t }{  \nicefrac{1}{L}+\eta_t } z_t\,, \label{simg:a}\\
&\textstyle x_{t+1} \leftarrow \argmin_{x\in Q} \left\{ f(y_t)+\inp{\nabla f(y_t)}{x-y_t} + \frac{1}{\eta_{t+1}} \breg{h}{x}{x_t} +\Psi(x) \right\}\,,\label{simg:b}\\
& \textstyle z_{t+1} =\frac{\nicefrac{1}{L}}{\nicefrac{1}{L}+\eta_t} x_{t+1}+\frac{ \eta_t }{\nicefrac{1}{L}+ \eta_t } z_t \,. \label{simg:c}
  \end{align}
\end{subequations}
\end{mdframed}
Again, the similar triangle approximation \eqref{simg} is computationally advantageous in that it only requires a single projection in each iteration. Now we provide a simple PPM-based analysis of \eqref{simg}:
\begin{proof}[{\bf PPM-based analysis of \eqref{simg}}]
To obtain counterparts of \eqref{ineq:1} and \eqref{ineq:2}, we now use a generalization of Proposition \ref{prop:per} to the Bregman divergence (\cite[Lemma 3.1]{teboulle2018simplified}).
With such a generalization,  we obtain the following inequality for  $\phi^{\Psi}(x):=\eta_{t+1}[f(y_t)+\inp{\nabla f(y_t)}{x-y_t} +\Psi(x)]$:
\begin{align}
    \phi^{\Psi}(x_{t+1}) -\phi^{\Psi}(\xs) +  \breg{h}{\xs}{x_{t+1}}+\breg{h}{x_{t+1}}{x_{t}}-
  \breg{h}{\xs}{x_{t}} \leq 0\,, \label{gen:1}
\end{align}
where $\xs\in \argmin_{x\in Q}f^{\Psi}(x)$.
Now using \eqref{gen:1}, one can derive from first principles the following inequalities   (we defer the derivations to Appendix~\ref{app:sim}):
\begin{align*}
    \text{Counterpart of }\eqref{ineq:1}: &\quad \eta_{t+1}(f^{\Psi}(z_{t+1})- f^{\Psi}(\xs) ) + \breg{h}{\xs}{x_{t+1}}-
  \breg{h}{\xs}{x_{t}} \leq \text{($\mathcal{G}_1$)} \quad \text{and}\\
   \text{Counterpart of }\eqref{ineq:2}: &\quad  f^{\Psi}(z_{t+1}) -f^{\Psi}(z_t) \leq \text{($\mathcal{G}_2$)}\,,
\end{align*}
where $\text{($\mathcal{G}_1$)}:=  \textstyle-\frac{1}{2}\norm{ x_{t+1}-x_t}^2  +\eta_{t+1}[\frac{L}{2}\norm{z_{t+1}-y_{t}}^2 +\inp{\nabla f(y_{t})}{z_{t+1}-x_{t+1}}   +\Psi(z_{t+1}) -\Psi(x_{t+1})]$ and $\text{($\mathcal{G}_2$)}:= \frac{L}{2} \norm{z_{t+1}-y_t}^2 +  \inp{\nabla f(y_{t})}{z_{t+1}-z_t} +\Psi(z_{t+1})-\Psi(z_{t})$.
    Similarly to Section~\ref{subsec:conv}, yet replacing the norm squared term with the Bregman divergence, we choose
    \begin{align*}
\textstyle\Phi_t:= (\sum_{i=1}^{t} \eta_i)\cdot (f^{\Psi}(z_t)-f^{\Psi}(\xs)) +\breg{h}{\xs}{x_t}.
    \end{align*}
    Then, it suffices to show  $\text{($\mathcal{G}_1$)} + ( \sum_{i=1}^t \eta_i)\cdot \text{($\mathcal{G}_2$)} \leq 0$.
    Using   the facts (i) $z_{t+1}-x_{t+1} = L\eta_t (z_t-z_{t+1})$ and (ii) $\norm{x_{t+1}-x_t}= (L\eta_t+1)\norm{z_{t+1}-y_t}$  (both are immediate consequences of the similar triangles) and rearranging, one can easily check that  $\text{($\mathcal{G}_1$)} + ( \sum_{i=1}^t \eta_i)\cdot \text{($\mathcal{G}_2$)}$ is equal to
    \begin{align}
\textstyle     \frac{1}{2}\left(-(L\eta_t+1)^2 +L \eta_{t+1}+  L\sum_{i=1}^{t}\eta_{i} \right)\norm{z_{t+1}-y_{t}}^2 \label{a1}\\
\textstyle +\left(L\eta_t \eta_{t+1} - \sum_{i=1}^t\eta_i\right)\inp{\nabla f(y_{t})}{z_{t}-z_{t+1}} \label{a2}\\
\textstyle  +\eta_{t+1}[ \Psi(z_{t+1}) -\Psi(x_{t+1}) ] +   \left( \sum_{i=1}^t \eta_i\right)\cdot [\Psi(z_{t+1})-\Psi(z_{t})].
   \label{a3}
\end{align}
    Now choosing $\eta_t= \nicefrac{t}{2L}$ analogously to Section~\ref{subsec:conv}, one can easily verify  $\eqref{a1}+\eqref{a2}+\eqref{a3} \leq 0$.
    Indeed, for \eqref{a1}, since  $L  \eta_t \eta_{t+1}=\sum_{i=1}^t\eta_i$, the coefficient becomes $\nicefrac{1}{2}(L\eta_t+1)(L\eta_{t+1}-L\eta_t-1)$ which is a negative number since $L\eta_{t+1}-L\eta_t-1=-\nicefrac{1}{2}$;
    for \eqref{a2}, the coefficient becomes zero due to the relation $L  \eta_t \eta_{t+1}=\sum_{i=1}^t\eta_i$;
    lastly, for \eqref{a3}, we have
    \begin{align}
\eqref{a3}=    \eta_{t+1}\left[ (1+L\eta_t) \Psi(z_{t+1}) -\Psi(x_{t+1}) -L\eta_{t}\Psi(z_{t})\right] \leq 0\,,
    \end{align}
    where the equality is due to the relation $L  \eta_t \eta_{t+1} =\sum_{i=1}^t\eta_i$, and the inequality is due to the update \eqref{simg:c} (which can be equivalently written as $(1+L\eta_t) z_{t+1} = x_{t+1}+ L\eta_t z_t$) and the convexity of $\Psi$.
    Hence, we obtain the accelerated rate of  $f^{\Psi}(z_T)-f^{\Psi}(\xs)\leq \frac{4L\breg{h}{\xs}{x_0}}{T(T+1)} = O(\nicefrac{1}{T^2})$.
\end{proof}

\section{Related work} \label{sec:related}

Motivated by the obscure scope of Nesterov's estimate sequence technique, there have been a flurry of works on developing alternative approaches to Nesterov's acceleration.
The most contributions are made based on understanding the continuous limit dynamics of Nesterov's AGM~\cite{su2014differential,krichene2015accelerated,wibisono2016variational}.
These continuous dynamics approaches have brought about new intuitions about Nesterov's acceleration, and follow-up works have developed analytical techniques for such dynamics~\cite{wilson2016lyapunov,diakonikolas2019approximate}.
However, these approaches share a limitation that when applying discretization techniques to obtain optimization algorithms, some auxiliary modifications are required to recover/obtain analyzable algorithms.
In contrast, our PPM approach directly yields accelerated methods and does not require additional adjustments.

Another notable contribution is made based on the  linear coupling framework~\cite{allen2014linear}.
The main observation is that the two most popular first-order methods, namely gradient descent and mirror descent, have complementary performances, and hence, one can come up with a faster method by linearly coupling  the two methods.
This view indeed offers a general framework of developing fast optimization algorithms; however, for understanding Nesterov's acceleration, this view has less expressive power compared to  our PPM view.
More specifically, it is \emph{a priori}  not clear why one needs to couple two methods \emph{linearly}.
Moreover, with the linear coupling view, one cannot interpret the interpolation weight as we did in Remark~\ref{rmk:par}.

 It is also important to note that  PPM has been given new attention as a building block for designing and analyzing fast optimization methods~\cite{drusvyatskiy2017proximal}.
 To list few instances, PPM has given rise to methods  for weakly convex problems~\cite{davis2019proximally}, the prox-linear methods for composite optimizations~\cite{burke1995gauss,nesterov2007modified,lewis2016proximal}, accelerated methods for stochastic optimizations~\cite{lin2015universal}, and methods for saddle-point problems~\cite{mokhtari2019unified}.
 Moreover, using Proposition~\ref{prop:per} (and its generalization to the Bregman divergence) as a unified tool for analyzing  first-order methods has been discussed in~\cite{teboulle2018simplified}.

\section{Conclusion}

	 This work provides a complete understanding of Nesterov's acceleration by making analytical and quantitative connections to the proximal point method.
The key observation is that an alternation of two simple approximations of the PPM exatly recovers Nesterov's AGM.
Through this connection, we are able to explain all the step sizes of AGM in terms of the PPM step sizes, demystifying the mysterious choices made in the literature.
This view naturally extends to the strongly convex case and recovers Nesterov's general accelerated method from his book.
Moreover, our PPM view  motivates a simplification of AGM using similar triangles, which admits a simple PPM-based analysis as well as a simple extension to the general norm and composite optimization case.
For future directions, it would be interesting to connect our PPM view to accelerated stochastic methods
~\cite{lin2015universal,lan2018optimal} and other accelerated methods, including geometric descent~\cite{bubeck2015geometric}.

\section*{Acknowledgement} The author thanks Suvrit Sra, Alp Yurtsever and   Jingzhao Zhang  for detailed comments and stimulating discussions, Aaron Defazio for clarifications that help the author develop Section~\ref{sec:gen}, and  Heinz Bauschke for constructive suggestions on the presentation of Section~\ref{sec:deriv} as well as Remark~\ref{rmk:momentum}.

	\bibliographystyle{alpha}

Optimization and Control (math.OC); Machine Learning (cs.LG)