Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

Abstract: The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

\section{Conclusion} \label{sec:conclu}
In this work, we studied the effect of the initialization parameter values of deep linear neural networks on the convergence time of gradient descent. We found that when the initial weights are iid Gaussian, the convergence time grows exponentially in the depth unless the width is at least as large as the depth. In contrast, when the initial weight matrices are drawn from the orthogonal group, the width needed to guarantee efficient convergence is in fact independent of the depth. These results establish for the first time a concrete proof that orthogonal initialization is superior to Gaussian initialization in terms of convergence time.

\section{Experiments} \label{sec:experiment}
In this section, we provide empirical evidence to support the results in Sections \ref{sec:ortho} and \ref{sec:gaussian}.
To study how depth and width affect convergence speed of gradient descent under orthogonal and Gaussian initialization schemes, we train a family of linear networks with their widths ranging from 10 to 1000 and depths from 1 to 700, on a fixed synthetic dataset $(X, Y)$.\footnote{We choose $X \in \mathbb{R}^{1024\times16}$ and $W^* \in \mathbb{R}^{10\times 1024}$, and set $Y = W^*X$. Entries in $X$ and $W^*$ are drawn i.i.d. from ${\mathcal{N}}(0, 1)$.}
Each network is trained using gradient descent staring from both Gaussian and orthogonal initializations.
In Figure~\ref{fig:heat-maps}, We lay out the logarithm of the relative training loss $\frac{\ell(t)}{\ell(0)}$, using heap-maps, at steps $t=1258$ and $t=10000$. In each heat-map, each point represents the relative training loss of one experiment; the darker the color, the smaller the loss.
Figure~\ref{fig:heat-maps} clearly demonstrates a sharp transition from untrainable to trainable (i.e., from red to black) when we increase the width of the network:
\begin{itemize}
    \item for Gaussian initialization, this transition occurs across a contour characterized by a linear relation between width and depth;
    \item for orthogonal initialization, the transition occurs at a width that is approximately independent of the depth.
\end{itemize}
These observations excellently verify our theory developed in Sections~\ref{sec:ortho} and \ref{sec:gaussian}.

\begin{figure}[t]
\begin{center}
         \begin{subfigure}[b]{0.4\textwidth}
                \centering
\includegraphics[height=4.5cm]{graphs/gaussian-dataset-16-learning_rate-01-steps-1258.pdf}
                \caption{Gaussian, steps=1258}
                \label{fig:tanhphasediag}
        \end{subfigure}%
        \begin{subfigure}[b]{0.4\textwidth}
                \centering
               \includegraphics[height=4.5cm]{graphs/gaussian-dataset-16-learning_rate-01-steps-10000.pdf}
                \caption{Gaussian, steps=10000}
                \label{fig:tanhphasediag}
        \end{subfigure}%
        \\
 \begin{subfigure}[b]{0.4\textwidth}
                \centering
     \includegraphics[height=4.5cm]{graphs/orthogonal-dataset-16-learning_rate-01-steps-1258.pdf}
                \caption{Orthogonal, steps=1258}
                \label{fig:tanhphasediag}
        \end{subfigure}%
        \begin{subfigure}[b]{0.4\textwidth}
                \centering
                \includegraphics[height=4.5cm]{graphs/orthogonal-dataset-16-learning_rate-01-steps-10000.pdf}
                \caption{Orthogonal, steps=10000}
                \label{fig:tanhphasediag}
        \end{subfigure}%
\end{center}
\caption{$\log \frac{\ell(t)}{\ell(0)}$ at $t=1258$ and $t=10000$, for different depth-width configurations and different initialization schemes. Darker color means smaller loss.}
\label{fig:heat-maps}
\end{figure}

To have a closer look into the training dynamics, we also plot ``relative loss v.s. training time'' for a variety of depth-width configurations. See Figure~\ref{fig:loss}. There again we can clearly see that orthogonal initialization enables fast training at small width (independent of depth), and that the required width for Gaussian initialization depends on depth.

\begin{figure}[t]
\begin{center}
         \begin{subfigure}[b]{0.33\textwidth}
                \centering
\includegraphics[height=3.5cm]{graphs/dataset-16-learning_rate-01-depth-50.pdf}
                \caption{Depth=50}
                \label{fig:tanhphasediag}
        \end{subfigure}%
        \begin{subfigure}[b]{0.33\textwidth}
                \centering
               \includegraphics[height=3.5cm]{graphs/dataset-16-learning_rate-01-depth-200.pdf}
                \caption{Depth=200}
                \label{fig:tanhphasediag}
        \end{subfigure}%
        \begin{subfigure}[b]{0.33
        \textwidth}
                \centering
                \includegraphics[height=3.5cm]{graphs/dataset-16-learning_rate-01-depth-400.pdf}
                \caption{Depth=400}
                \label{fig:tanhphasediag}
        \end{subfigure}%
\end{center}
\caption{Relative loss v.s. training time. For each plot, we vary width from 50 (yellow) to 1200 (purple). Solid and dashed lines represent Gaussian (GS) and orthogonal (OT) initializations.}
\label{fig:loss}
\end{figure}
\section{Exponential Curse of Gaussian Initialization} \label{sec:gaussian}

In this section, we show that gradient descent with Gaussian random initialization necessarily suffers from a running time that scales exponentially with the depth of the network, unless the width becomes nearly linear in the depth.
Since we mostly focus on the dependence of width and running time on depth, we will assume the depth $L$ to be sufficiently large.

Recall that we want to minimize the objective $\ell(W_1, \ldots, W_L) = \frac{1}{2} \norm{\alpha W_{L:1} X - Y}_F^2$ by gradient descent.
We assume $Y=W^* X$ for some $W^* \in \mathbb{R}^{d_y\times d_x}$, so that the optimal objective value is $0$.
For convenience, we assume $\norm{X}_F = \Theta(1)$ and $\norm{Y}_F=\Theta(1)$.

Suppose that at layer $i\in[L]$, every entry of $W_i(0)$ is sampled from ${\mathcal{N}}(0, \sigma_i^2)$, and all weights in the network are independent.
We set the scaling factor $\alpha$ such that the initial output of the network does not blow up exponentially (in expectation):
\begin{equation} \label{eqn:scaling-not-blow-up}
\expect{\norm{f(x; W_1(0), \ldots, W_L(0))}^2} \le L^{O(1)} \cdot \norm{x}^2, \quad \forall x\in \mathbb{R}^{d_x}.
\end{equation}
Note that $\expect{\norm{f(x; W_1(0), \ldots, W_L(0))}^2} = \alpha^2 \prod_{i=1}^L (d_i\sigma_i^2) \norm{x}^2$.
Thus \eqref{eqn:scaling-not-blow-up} means
$$
\alpha^2 \prod_{i=1}^L (d_i\sigma_i^2) \le L^{O(1)}.
$$
We also assume that the magnitude of initialization at each layer cannot vanish with depth:
\begin{equation} \label{eqn:init-not-vanish}
    d_i\sigma_i^2 \ge \frac{1}{L^{O(1)}}, \quad \forall i\in[L].
\end{equation}
Note that the assumptions \eqref{eqn:scaling-not-blow-up} and \eqref{eqn:init-not-vanish} are just sanity checks to rule out the obvious pathological cases -- they are easily satisfied by all the commonly used initialization schemes in practice.

Now we formally state our main theorem in this section.
\begin{thm} \label{thm:gaussian}
	Suppose $\max\{d_0, d_1, \ldots, d_L\} \le O(L^{1-\gamma})$ for some universal constant $0< \gamma \le 1$.
	Then there exists a universal constant $c>0$ such that, if gradient descent is run with learning rate $\eta \le e^{cL^\gamma}$, then with probability at least $0.9$ over the random initialization, for the first $e^{\Omega(L^\gamma)}$ iterations, the objective value is stuck between $0.4\norm{Y}_F^2$ and $0.6\norm{Y}_F^2$.

\end{thm}

Theorem~\ref{thm:gaussian} establishes that efficient convergence from Gaussian initialization is impossible for large depth unless the width becomes nearly linear in depth.
This nearly linear dependence is the best we can hope for, since \cite{du2019width} proved a positive result when the width is larger than linear in depth.
Therefore, a phase transition from untrainable to trainable happens at the point when the width and depth has a nearly linear relation.
Furthermore, Theorem~\ref{thm:gaussian} generalizes the result of \cite{shamir2018exponential}, which only treats the special case of $d_0=\cdots=d_L=1$.

\subsection{Proof of \Theoremref{thm:gaussian}}

For convenience, we define a scaled version of $W_i$:
let $A_i = W_i / (\sqrt{d_i} \sigma_i)$ and $\beta = \alpha \prod_{i=1}^L (\sqrt{d_i} \sigma_i)$.
Then we know $\beta \le L^{O(1)}$ and $\alpha W_{L:1} = \beta A_{L:1}$, where $A_{j:i} = A_j\cdots A_i$.

We first give a simple upper bound on $\norm{A_{j:i}(0)}$ for all $1\le i \le j \le L$.
\begin{lem} \label{lem:init-product-bound}
	With probability at least $1-\delta$, we have $\norm{A_{j:i}(0)} \le O\left( \frac{L^3}{\delta} \right)$ for all $1\le i \le j \le L$.
\end{lem}
The proof of Lemma~\ref{lem:init-product-bound} is given in Appendix~\ref{app:proof-gaussian-init-product-bound}. It simply uses Markov inequality and union bound.

Furthermore, a key property at initialization is that if $j-i$ is large enough, $\norm{A_{j:i}(0)}$ will become exponentially small.
\begin{lem} \label{lem:init-long-product-bound}
	With probability at least $1-e^{-\Omega(L^\gamma)}$, for all $1\le i \le j \le L$ such that $j-i \ge \frac{L}{10}$, we have $\norm{A_{j:i}(0)} \le e^{-\Omega(L^\gamma)}$.
\end{lem}
\begin{proof}
	We first consider a fixed pair $(i, j)$ such that $j-i \ge \frac{L}{10}$.
	In order to bound $\norm{A_{j:i}(0)}$, we first take an arbitrary unit vector $v \in \mathbb{R}^{d_{i-1}}$ and bound $\norm{A_{j:i}(0) v}$.
	We can write $\norm{A_{j:i}(0) v}^2 = \prod_{k=i}^j Z_k$, where $Z_k = \frac{\norm{A_{k:i}(0) v}^2}{\norm{A_{k-1:i}(0)v}^2}$.
	Note that for any nonzero $v' \in \mathbb{R}^{d_{k-1}}$ independent of $A_k(0)$, the distribution of $ d_k \cdot \frac{\norm{A_k(0) v'}^2}{\norm{v'}^2} $ is $\chi^2_{d_k}$.
	Therefore, $Z_i, \ldots, Z_j$ are independent, and $d_k Z_k \sim \chi^2_{d_k}$ ($ k =i, i+1, \ldots, j$).
	Recall the expression for the moments of chi-squared random variables: $\expect{Z_k^\lambda} = \frac{2^\lambda \Gamma(d_k/2 + \lambda)}{d_k^\lambda \Gamma(d_k/2)}$ ($\forall \lambda>0$).
	Taking $\lambda=\frac12$ and using the bound $\frac{\Gamma(a+\frac12)}{\Gamma(a)} \le \sqrt{a-0.1}\, (\forall a \ge \frac12) $ \citep{qi2012bounds}, we get $\expect{\sqrt{Z_k}} \le \sqrt{\frac{2(d_k/2-0.1)}{d_k}} = \sqrt{1 - \frac{0.2}{d_k}} \le 1 - \frac{0.1}{d_k}$.
	Therefore we have
	\begin{align*}
	\expect{\sqrt{\prod\nolimits_{k=i}^j Z_k}} \le \prod\nolimits_{k=i}^j \left( 1 - \frac{0.1}{d_k} \right) \le \left( 1 - \frac{0.1}{O(L^{1-\gamma})} \right)^{j-i+1}
	\le \left( 1 - \Omega(L^{\gamma-1}) \right)^{\frac{L}{10}} = e^{-\Omega(L^\gamma)}.
	\end{align*}
	Choose a sufficiently small constant $c'>0$.
	By Markov inequality we have $\Pr\left[ \sqrt{\prod\nolimits_{k=i}^j Z_k} > e^{-c'L^\gamma} \right] \le e^{c'L^\gamma} \expect{\sqrt{\prod\nolimits_{k=i}^j Z_k}} \le e^{c'L^\gamma} e^{-\Omega(L^\gamma)} = e^{-\Omega(L^\gamma)}$.
	Therefore we have shown that for any fixed unit vector $v \in \mathbb{R}^{d_{i-1}}$, with probability at least $1- e^{-\Omega(L^\gamma)}$ we have $\norm{A_{j:i}(0)v} \le e^{-\Omega(L^\gamma)}$.

	Next, we use this to bound $\norm{A_{j:i}(0)}$ via an $\epsilon$-net argument.
	We partition the index set $[d_{i-1}]$ into $[d_{i-1}] = S_1 \cup S_2 \cup \cdots \cup S_q$ such that $|S_l| \le L^{\gamma/2} \, (\forall l\in[q])$ and $q = O(\frac{d_{i-1}}{L^{\gamma/2}})$.
	For each $l\in[q]$, let ${\mathcal{N}}_l$ be a $\frac{1}{2}$-net for all the unit vectors in $\mathbb{R}^{d_{i-1}}$ with support in $S_l$. Note that we can choose ${\mathcal{N}}_l$ such that $|{\mathcal{N}}_l| = e^{O(|S_l|)} = e^{O(L^{\gamma/2})}$.
	Taking a union bound over $ \cup_{l=1}^q {\mathcal{N}}_l $, we know that $\norm{A_{j:i}(0)v} \le e^{-\Omega(L^\gamma)} \norm{v}$ simultaneously for all $v \in \cup_{l=1}^q {\mathcal{N}}_l $ with probability at least $1 - \left( \sum_{l=1}^q |{\mathcal{N}}_l| \right) e^{-\Omega(L^\gamma)} \ge 1 - q \cdot e^{O(L^{\gamma/2})} e^{-\Omega(L^\gamma)} = 1 - e^{-\Omega(L^\gamma)}$.

	Now, for any $u\in \mathbb{R}^{d_{i-1}}$, we write it as $u = \sum_{l=1}^q a_l u_l$ where $a_l $ is a scalar and $u_l$ is a unit vector supported on $S_l$.
	By the definition of $\frac{1}{2}$-net, for each $l\in[q]$ there exists $v_l \in {\mathcal{N}}_l$ such that $\norm{v_l-u_l}\le \frac{1}{2}$.
	We know that $\norm{A_{j:i}(0)v_l} \le e^{-\Omega(L^\gamma)} \norm{v_l}$ for all $l\in[q]$.
	Let $v = \sum_{l=1}^q a_l v_l$. We have
	\begin{align*}
	\norm{A_{j:i}(0)v} &\le \sum\nolimits_{l=1}^q |a_l| \cdot \norm{A_{j:i}(0)v_l} \le \sum\nolimits_{l=1}^q |a_l| \cdot e^{-\Omega(L^\gamma)} \norm{v_l}
	\le e^{-\Omega(L^\gamma)} \sqrt{q \cdot \sum\nolimits_{l=1}^q a_l^2 \norm{v_l}^2} \\
	&= \sqrt{q} e^{-\Omega(L^\gamma)} \norm{v} = e^{-\Omega(L^\gamma)} \norm{v}.
	\end{align*}
	Note that $\norm{u-v} = \norm{\sum_{l=1}^q a_l (u_l-v_l)} = \sqrt{\sum_{l=1}^q a_l^2 \norm{u_l-v_l}^2} \le \sqrt{\frac14 \sum_{l=1}^q a_l^2 } = \frac12 \norm{u}$, which implies $\norm{v} \le \frac32\norm{u}$. Therefore we have
	\begin{align*}
	\norm{A_{j:i}(0)u} &\le \norm{A_{j:i}(0)v} + \norm{A_{j:i}(0)(u-v)}
	\le e^{-\Omega(L^\gamma)} \norm{v} + \norm{A_{j:i}(0)} \cdot \norm{u-v} \\
	&\le e^{-\Omega(L^\gamma)}\cdot \frac32 \norm{u} + \norm{A_{j:i}(0)} \cdot \frac12 \norm{u}
	= e^{-\Omega(L^\gamma)} \norm{u} + \norm{A_{j:i}(0)} \cdot \frac12 \norm{u}.
	\end{align*}
	The above inequality is valid for any $u\in \mathbb{R}^{d_{i-1}}$. Thus we can take the unit vector $u$ that maximizes $\norm{A_{j:i}(0)u} $. This gives us
	$\norm{A_{j:i}(0)} \le  e^{-\Omega(L^\gamma)} + \frac12 \norm{A_{j:i}(0)}$, which implies $\norm{A_{j:i}(0)} \le e^{-\Omega(L^\gamma)} $.

	Finally, we take a union bound over all possible $(i, j)$. The failure probaility is at most $L^2 e^{-\Omega(L^\gamma)} = e^{-\Omega(L^\gamma)}$.
\end{proof}

The following lemma shows that the properties in Lemmas~\ref{lem:init-product-bound} and~\ref{lem:init-long-product-bound} are still to some extent preserved after applying small perturbations on all the weight matrices.
\begin{lem} \label{lem:perturbation-bound}
	Suppose that the initial weights satisfy $\norm{A_{j:i}(0)} \le O(L^3)$ for all $1\le i \le j \le L$, and $\norm{A_{j:i}(0)} \le e^{-c_1 L^\gamma}$ if $j-i\ge \frac{L}{10}$, where $c_1>0$ is a universal constant.
	Then for another set of matrices $A_1, \ldots, A_L$ satisfying $\norm{A_i-A_i(0)} \le e^{-0.6c_1 L^\gamma}$ for all $i\in[L]$, we must have
	\begin{equation} \label{eqn:product-bound}
	\begin{aligned}
	&\norm{A_{j:i}} \le O(L^3), \quad \forall 1\le i \le j \le L, \\
	&\norm{A_{j:i}} \le O\left( e^{- c_1 L^\gamma}\right), \quad \forall 1\le i \le j \le L, j-i\ge \frac{L}{4}.
	\end{aligned}
	\end{equation}
\end{lem}
\begin{proof}
	It suffices to show that the difference $A_{j:i} - A_{j:i}(0)$ is tiny.
	Let $\Delta_i = A_i - A_i(0)$.
	We have $A_{j:i} = (A_j(0) + \Delta_j) \cdots (A_{i+1}(0) + \Delta_{i+1}) (A_i(0) + \Delta_i)$.
	Expanding this product, except for the one term corresponding to $A_{j:i}(0)$, every other term has the form $A_{j:(k_s+1)}(0)	\cdot \Delta_{k_s} \cdot A_{(k_s-1):(k_{s-1}+1)}(0) \cdot \Delta_{k_{s-1}} \cdots  \Delta_{k_1} \cdot A_{(k_1-1):i}(0) $, where $i \le k_1 < \cdots < k_s \le j$. By assumption, each $\Delta_k$ has spectral norm $e^{-0.6c_1 L^\gamma}$, and each $A_{j':i'}(0)$ has spectral norm $O(L^3)$, so we have $\norm{A_{j:(k_s+1)}(0)	\cdot \Delta_{k_s} \cdot A_{(k_s-1):(k_{s-1}+1)}(0) \cdot \Delta_{k_{s-1}} \cdots  \Delta_{k_1} \cdot A_{(k_1-1):i}(0) } \le \left(e^{-0.6c_1 L^\gamma}\right)^s \left( O(L^3) \right)^{s+1}$.
	Therefore we have
	\begin{align*}
	&\norm{A_{j:i} - A_{j:i}(0)} \le \sum_{s=1}^{j-i+1} \binom{j-i+1}{s} \left(e^{-0.6c_1 L^\gamma}\right)^s \left( O(L^3) \right)^{s+1} \\
	\le\,& \sum_{s=1}^{j-i+1} L^s \left(e^{-0.6c_1 L^\gamma}\right)^s \left( O(L^3) \right)^{s+1} \le O(L^3) \sum_{s=1}^\infty \left( O(L^4) e^{-0.6c_1 L^\gamma} \right)^s
	\le O(L^3) \sum_{s=1}^\infty (1/2)^s = O(L^3),
	\end{align*}
	which implies $\norm{A_{j:i}} \le O(L^3)$ for all $1\le i \le j \le L$.

	The proof of the second part of the lemma
	is postponed to Appendix~\ref{app:proof-gaussian-perturbation-bound}.
\end{proof}

As a consequence of \Lemref{lem:perturbation-bound}, we can control the objective value and the gradient at any point sufficiently close to the random initialization.

\begin{lem} \label{lem:loss-gradient-bound}
	For a set of weight matrices $W_1, \ldots, W_L$ with $A_i = W_i/(\sqrt{d_i}\sigma_i)$ that satisfy~\eqref{eqn:product-bound}, the objective and the gradient satisfy
	\begin{align*}
	& 0.4\norm{Y}_F^2 < \ell(W_1, \ldots, W_L) < 0.6\norm{Y}_F^2,\\
	 & \norm{\nabla_{W_i}\ell(W_1, \ldots, W_L)} \le (\sqrt{d_i}\sigma_i )^{-1} e^{-0.9c_1L^\gamma}, \quad \forall i\in[L].
	\end{align*}
\end{lem}
The proof of Lemma~\ref{lem:loss-gradient-bound} is given in Appendix~\ref{app:proof-gaussian-loss-gradient-bound}.

Finally, we can finish the proof of \Theoremref{thm:gaussian} using the above lemmas.

\begin{proof}[Proof of \Theoremref{thm:gaussian}]
	From Lemmas~\ref{lem:init-product-bound} and \ref{lem:init-long-product-bound}, we know that with probability at least $0.9$, we have (i) $\norm{A_{j:i}(0)} \le O(L^3)$ for all $1\le i \le j \le L$, and (ii) $\norm{A_{j:i}(0)} \le e^{-c_1 L^\gamma}$ if $(i, j)$ further satisfies $j-i\ge \frac{L}{10}$.
	Here $c_1>0$ is a universal constant.
	From now on we are conditioned on these properties being satisfied.
	We suppose that the learning rate $\eta$ is at most $e^{0.2c_1L^\gamma}$.

	We say that a set of weight matrices $W_1, \ldots, W_L$ are in the ``initial neighborhood'' if $\norm{A_i - A_i(0)} \le e^{-0.6c_1L^\gamma}$ for all $i\in[L]$.
	From Lemmas~\ref{lem:perturbation-bound} and \ref{lem:loss-gradient-bound} we know that in the ``initial neighborhood'' the objective value is always between $0.4\norm{Y}_F^2$ and $0.6\norm{Y}_F^2$.
	Therefore we have to escape the ``initial neighborhood'' in order to get the objective value out of this interval.

	Now we calculate how many iterations are necessary to escape the ``initial neighborhood.''
	According to \Lemref{lem:loss-gradient-bound}, inside the ``initial neighborhood'' each $W_i$ can move at most $\eta (\sqrt{d_i}\sigma_i )^{-1} e^{-0.9c_1L^\gamma}$ in one iteration by definition of the gradient descent algorithm.
	In order to leave the ``initial neighborhood,'' some $W_i$ must satisfy $\norm{W_i - W_i(0)} = \sqrt{d_i}\sigma_i \norm{A_i - A_i(0)} > \sqrt{d_i}\sigma_i e^{-0.6c_1L^\gamma}$.
	In order to move this amount, the number of iterations has to be at least
	\begin{equation*}
	\frac{\sqrt{d_i}\sigma_i e^{-0.6c_1L^\gamma}}{ \eta (\sqrt{d_i}\sigma_i )^{-1} e^{-0.9c_1L^\gamma} }
	= \frac{d_i\sigma_i^2 e^{0.3c_1L^\gamma}}{ \eta }
	\ge \frac{1}{L^{O(1)}} \cdot \frac{e^{0.3c_1L^\gamma}}{e^{0.2c_1L^\gamma}}
	\ge e^{\Omega(L^\gamma)}.
	\end{equation*}
	This finishes the proof.
\end{proof}

\section{Introduction} \label{sec:intro}

Through their myriad successful applications across a wide range of disciplines, it is now well established that deep neural networks possess an unprecedented ability to model complex real-world datasets, and in many cases they can do so with minimal overfitting. Indeed, the list of practical achievements of deep learning has grown at an astonishing rate, and includes models capable of human-level performance in tasks such as image recognition~\citep{krizhevsky2012}, speech recognition~\citep{hinton2012}, and machine translation~\citep{wu2016}.

Yet to each of these deep learning triumphs corresponds a large engineering effort to produce such a high-performing model. Part of the practical difficulty in designing good models stems from a proliferation of hyperparameters and a poor understanding of the general guidelines for their selection. Given a candidate network architecture, some of the most impactful hyperparameters are those governing the choice of the model's initial weights. Although considerable study has been devoted to the selection of initial weights, relatively little has been proved about how these choices affect important quantities such as rate of convergence of gradient descent.

In this work, we examine the effect of initialization on the rate of convergence of gradient descent in deep linear networks. We provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. In particular, we show that for deep networks, the width needed for efficient convergence for orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence of Gaussian networks scales linearly in the depth.

Orthogonal weight initializations have been the subject of a significant amount of prior theoretical and empirical investigation. For example, in a line of work focusing on \emph{dynamical isometry}, it was found that orthogonal weights can speed up convergence for deep linear networks~\citep{saxe2014exact,advani2017high} and for deep non-linear networks~\citep{pennington2018emergence,xiao2018dynamical,gilboa2019dynamical,chen2018dynamical,pennington2017resurrecting,tarnowski2019dynamical,ling2019spectrum} when they operate in the linear regime. In the context of recurrent neural networks, orthogonality can help improve the system's stability. A main limitation of prior work is that it has focused almost exclusively on model's properties at initialization. In contrast, our analysis focuses on the benefit of orthogonal initialization on the entire training process, thereby establishing a provable benefit for optimization.

The paper is organized as follows. After reviewing related work in Section~\ref{sec:related} and establishing some preliminaries in Section~\ref{sec:prelim}, we present our main positive result on efficient convergence from orthogonal initialization in Section~\ref{sec:ortho}. In Section~\ref{sec:gaussian}, we show that Gaussian initialization leads to exponentially long convergence time if the width is too small compared with the depth.
In Section~\ref{sec:experiment}, we perform experiments to support our theoretical results.

\section{Efficient Convergence using Orthogonal Initialization}
\label{sec:ortho}

In this section we present our main positive result for orthogonal initialization.
We show that orthogonal initialization enables efficient convergence of gradient descent to a global minimum provided that the hidden width is not too small.

In order to properly define orthogonal weights, we let the widths of all hidden layers be equal: $d_1=d_2=\cdots=d_{L-1}=m$, and let $m \ge \max\{d_x, d_y\}$.
Note that all intermediate matrices $W_2, \ldots, W_{L-1}$ are $m\times m$ square matrices, and $W_1 \in  \mathbb{R}^{m\times d_x}, W_L\in\mathbb{R}^{d_y\times m}$.
We sample each initial weight matrix $W_i(0)$ independently from a uniform distribution over scaled orthogonal matrices satisfying
\begin{equation} \label{eqn:ortho-init}
\begin{aligned}
&W_1^\top(0) W_1(0) = mI_{d_x},\\
&W_i^\top(0) W_i(0) = W_i(0)W_i^\top(0) = mI_m, \qquad 2\le i \le L-1, \\
&W_L(0)W_L^\top(0)=mI_{d_y}.
\end{aligned}
\end{equation}
In accordance with such initialization, the scaling factor $\alpha$ in~\eqref{eqn:linear-net} is set as $\alpha = \frac{1}{\sqrt{m^{L-1}d_y}}$, which ensures $\expect{\norm{f(x; W_L(0), \ldots, W_1(0))}^2} = \norm{x}^2$ for any $x\in \mathbb{R}^{d_x}$.\footnote{We have $\expect{\norm{f(x; W_L(0), \ldots, W_1(0))}^2} = \alpha^2 \expect{x^\top W_1^\top(0) \cdots W_L^\top(0) W_L(0) \cdots W_1(0) x }$. Note that by our choice \eqref{eqn:ortho-init} we have $\expect{W_L^\top(0) W_L(0)}=d_yI_m$ and $W_i^\top(0) W_i(0)=mI \,(1\le i \le L-1)$, so we have $\expect{\norm{f(x; W_L(0), \ldots, W_1(0))}^2} = \alpha^2 m^{L-1} d_y \norm{x}^2 = \norm{x}^2 $.}
The same scaling factor was adopted in \cite{du2019width}, which preserves the expectation of the squared $\ell_2$ norm of any input.

Let $W^* \in \argmin_{W \in \mathbb{R}^{d_y \times d_x}} \norm{WX-Y}_F$ and $\ell^* = \frac12 \norm{W^* X - Y}_F^2$. Then $\ell^*$ is the minimum value for the objective~\eqref{eqn:loss-func}.
Denote $r = \mathrm{rank}(X)$, $\kappa = \frac{\lambda_{\max}(X^\top X)}{\lambda_r(X^\top X)}$, and $\tilde{r} = \frac{\norm{X}_F^2}{\norm{X}^2}$.\footnote{$\tilde{r}$ is known as the \emph{stable rank} of $X$, which is always no more than the rank.} Our main theorem in this section is the following:

\begin{thm}\label{thm:ortho}
	Suppose
	\begin{equation} \label{eqn:m-bound-for-ortho}
	m\ge C \cdot \tilde{r} \kappa^2 \left( d_y(1+\norm{W^*}^2) + \log(r/\delta) \right) \text{ and } m \ge d_x,
	\end{equation}
	for some $\delta\in(0, 1)$ and a sufficiently large universal constant $C>0$.
	Set the learning rate $\eta \le \frac{d_y}{2L \norm{X}^2}$.
	Then with probability at least $1-\delta$ over the random initialization, we have
	\begin{align*}
	&\ell(0) - \ell^* \le O\left( 1 + \frac{\log(r/\delta)}{d_y} + \norm{W^*}^2 \right) \norm{X}_F^2, \\
	&\ell(t) - \ell^* \le \left( 1 - \frac{1}{2} \eta L \lambda_r(X^\top X) / d_y \right)^t (\ell(0)-\ell^*), \quad t = 0, 1, 2, \ldots,
	\end{align*}
	where $\ell(t)$ is the objective value at iteration $t$.
\end{thm}

Notably, in \Theoremref{thm:ortho}, the width $m$ need not depend on the depth $L$. This is in sharp contrast with the result of \cite{du2019width} for Gaussian initialization, which requires $m\ge \tilde{\Omega}(L r \kappa^3 d_y)$.
It turns out that a near-linear dependence between $m$ and $L$ is necessary for Gaussian initialization to have efficient convergence, as we will show in \Secref{sec:gaussian}.
Therefore the requirement in \cite{du2019width} is nearly tight in terms of the dependence on $L$.
These results together rigorously establish the benefit of orthogonal initialization in optimizing very deep linear networks.

If we set the learning rate optimally according to Theorem~\ref{thm:ortho} to $\eta = \Theta(\frac{d_y}{L\norm{X}^2})$, we obtain that $\ell(t) - \ell^*$ decreases by a ratio of $1 - \Theta(\kappa^{-1})$ after every iteration. This matches the convergence rate of gradient descent on the ($1$-layer) linear regression problem $\min\limits_{W\in\mathbb{R}^{d_y\times d_x}} \frac12 \norm{WX-Y}_F^2$.

\subsection{Proof of \Theoremref{thm:ortho}}

The proof uses the high-level framework from \cite{du2019width}, which tracks the evolution of the network's output during optimization. This evolution is closely related to a time-varying positive semidefinite (PSD) matrix (defined in \eqref{eqn:P-def}), and the proof relies on carefully upper and lower bounding the eigenvalues of this matrix throughout training, which in turn implies the desired convergence result.

First, we can make the following simplifying assumption \emph{without loss of generality}. See Appendix B in \cite{du2019width} for justification.
\begin{asmp} \label{asmp:full-rank-data}
	(Without loss of generality) $X \in \mathbb{R}^{d_x \times r}$, $\mathrm{rank}(X) = r$, $Y = W^* X$, and $\ell^* =0$.
\end{asmp}

Now we briefly review \cite{du2019width}'s framework.
The key idea is to look at the network's output, defined as
\begin{align*}
U = \alpha W_{L:1} X \in \mathbb{R}^{d_y \times n}.
\end{align*}
We also write $U(t) = \alpha W_{L:1}(t) X$ as the output at time $t$. Note that $\ell(t) = \frac{1}{2}\norm{U(t)-Y}_F^2$.
According to the gradient descent update rule, we write
 \begin{align*}
&W_{L:1}(t+1)
= \prod_i \left( W_i(t) - \eta \frac{\partial \ell}{\partial W_i}(t)  \right)
= W_{L:1}(t) - \sum_{i=1}^L \eta W_{L:i+1}(t) \frac{\partial \ell}{\partial W_i}(t) W_{i-1:1}(t) + E(t),
\end{align*}
where $E(t)$ contains all the high-order terms (i.e., those with $\eta^2$ or higher).
With this definition, the evolution of $U(t)$ can be written as the following equation:
 \begin{equation} \label{eqn:U-dynamics}
\begin{aligned}
&\vectorize{U(t+1) - U(t)}
= -\eta P(t) \cdot \vectorize{U(t)-Y} + \alpha \cdot \vectorize{E(t)X},
\end{aligned}
\end{equation}
where
\begin{equation} \label{eqn:P-def}
\begin{aligned}
	P(t) = \alpha^2 \sum_{i=1}^L \Big[ \left( \left( W_{i-1:1}(t) X \right)^\top \left( W_{i-1:1}(t)X \right) \right)  \otimes \left( W_{L:i+1}(t)  W_{L:i+1}^\top(t) \right) \Big] .
	\end{aligned}
\end{equation}
Notice that $P(t)$ is always PSD since it is the sum of $L$  PSD matrices.
Therefore, in order to establish convergence, we only need to (i) show that the higher-order term $E(t)$ is small and (ii) prove upper and lower bounds on $P(t)$'s eigenvalues.
For the second task, it suffices to control the singular values of $W_{i-1:1}(t)$ and $W_{L:i+1}(t)$ ($i\in[L]$).\footnote{Note that for symmetric matrices $A$ and $B$, the set of eigenvalues of $A \otimes B$ is the set of products of an eigenvalue of $A$ and an eigenvalue of $B$.}
Under orthogonal initialization, these matrices are perfectly isometric at initialization, and we will show that they stay close to isometry during training, thus enabling efficient convergence.

The following lemma summarizes some properties at initialization.
\begin{lem} \label{lem:ortho-init}
	At initialization, we have
	\begin{equation} \label{eqn:init-spec}
	\begin{aligned}

	&\sigma_{\max}(W_{j:i}(0)) = \sigma_{\min}(W_{j:i}(0)) = m^{\frac{j-i+1}{2}} , & \forall 1\le i\le j \le L, (i, j)\not=(1,L).
	\end{aligned}
	\end{equation}
	Furthermore, with probability at least $1-\delta$, the loss at initialization satisfies
	\begin{equation} \label{eqn:init-loss}
	\ell(0) \le O\left( 1 + \frac{\log(r/\delta)}{d_y} + \norm{W^*}^2 \right) \norm{X}_F^2.
	\end{equation}
\end{lem}

\begin{proof}[Proof sketch]
	The spectral property~\eqref{eqn:init-spec} follows directly from~\eqref{eqn:ortho-init}.

	To prove~\eqref{eqn:init-loss}, we essentially need to upper bound the magnitude of the network's initial output. This turns out to be equivalent to studying the magnitude of the projection of a vector onto a random low-dimensional subspace, which we can bound using standard concentration inequalities. The details are given in Appendix~\ref{app:proof-ortho-init}.
\end{proof}

Now we proceed to prove \Theoremref{thm:ortho}.
We define $B = O \left( 1 + \frac{\log(r/\delta)}{d_y} + \norm{W^*}^2  \right) \norm{X}_F^2$ which is the upper bound on $\ell(0)$ from~\eqref{eqn:init-loss}.
Conditioned on \eqref{eqn:init-loss} being satisfied, we will use induction on $t$ to prove the following three properties ${\mathcal{A}}(t)$, ${\mathcal{B}}(t)$ and ${\mathcal{C}}(t)$ for all $t=0, 1, \ldots$:
\begin{itemize}
	\item ${\mathcal{A}}(t)$:

	$\ell(t) \le \left(  1 - \frac12 \eta L \sigma_{\min}^2(X) / d_y \right)^t \ell(0)
	  \le \left(  1 - \frac12 \eta L \sigma_{\min}^2(X) /d_y \right)^t   B$.
	\item ${\mathcal{B}}(t)$:

	$

	\sigma_{\max}(W_{j:i}(t)) \le 1.1 m^{\frac{j-i+1}{2}} , \sigma_{\min}(W_{j:i}(t)) \ge 0.9 m^{\frac{j-i+1}{2}} , \quad  \forall 1\le i\le j \le L, (i, j)\not=(1, L).

	$

	\begin{comment}
		\begin{align*}
		\begin{cases}
		&\sigma_{\max}(W_{j:i}(t)) \le \frac54 m^{\frac{L-i+1}{2}}, \, \forall 1<i\le L, \\
		&\sigma_{\min}\left( W_{L:i}(t) \right) \ge \frac34  m^{\frac{L-i+1}{2}} , \, \forall 1<i\le L, \\
		&\sigma_{\max}(W_{i:1}(t) \cdot X) \le \frac54 m^{\frac{i}{2}} \sigma_{\max}(X), \, \forall 1\le i < L, \\
		&\sigma_{\min}\left( W_{i:1}(t) \cdot X \right)  \ge \frac34  m^{\frac{i}{2}}   \sigma_{\min}(X), \, \forall 1\le i < L, \\
		&	\norm{W_{j:i}(t)} \le O\left( \sqrt L m^{\frac{j-i+1}{2}}\right), \, \forall 1<i\le j <L.
		\end{cases}
		\end{align*}
	\end{comment}

	\item ${\mathcal{C}}(t)$:

	$\norm{W_i(t) - W_i(0)}_F \le  \frac{8\sqrt{Bd_y}\norm{X}}{ L \sigma_{\min}^2(X)}, \quad \forall 1\le i\le L$.

\end{itemize}

${\mathcal{A}}(0)$ and ${\mathcal{B}}(0)$ are true according to \Lemref{lem:ortho-init}, and ${\mathcal{C}}(0)$ is trivially true.
In order to prove ${\mathcal{A}}(t)$, ${\mathcal{B}}(t)$ and ${\mathcal{C}}(t)$ for all $t$, we will prove the following claims for all $t\ge0$:
\begin{claim} \label{claim:induction-1}
	${\mathcal{A}}(0), \ldots, {\mathcal{A}}(t), {\mathcal{B}}(0), \ldots, {\mathcal{B}}(t) \Longrightarrow {\mathcal{C}}(t+1)$.
\end{claim}
\begin{claim}\label{claim:induction-2}
	${\mathcal{C}}(t) \Longrightarrow {\mathcal{B}}(t)$.
\end{claim}
\begin{claim}\label{claim:induction-3}
	${\mathcal{A}}(t), {\mathcal{B}}(t) \Longrightarrow {\mathcal{A}}(t+1)$.
\end{claim}

The proofs of these claims are given in Appendix~\ref{app:proof-ortho}.
Notice that we finish the proof of \Theoremref{thm:ortho} once we prove ${\mathcal{A}}(t)$ for all $t\ge 0$.

\section{Preliminaries} \label{sec:prelim}

\subsection{Notation}

Let $[n]=\{1, 2, \ldots, n\}$.
Denote by $I_d$ the $d\times d$ identity matrix, and by $I$ an identity matrix when its dimension is clear from context.
Denote by ${\mathcal{N}}(\mu, \sigma^2)$ the Gaussian distribution with mean $\mu$ and variance $\sigma^2$, and by $\chi^2_k$ the chi-squared distribution with $k$ degrees of freedom.

Denote by $\norm{\cdot}$ the $\ell_2$ norm of a vector or the spectral norm of a matrix. Denote by $\norm{\cdot}_F$ the Frobenius norm of a matrix.
For a symmetric matrix $A$, let $\lambda_{\max}(A)$ and $\lambda_{\min}(A)$ be its maximum and minimum eigenvalues, and let $\lambda_i(A)$ be its $i$-th largest eigenvalue.
For a matrix $B \in \mathbb{R}^{m\times n}$, let $\sigma_i(B)$ be its $i$-th largest singular value ($i=1, 2, \ldots, \min\{m, n\}$), and let $\sigma_{\max}(B) = \sigma_1(B)$, $\sigma_{\min}(B) = \sigma_{\min\{m,n\}}(B)$.
Denote by $\vectorize{A}$ be the vectorization of a matrix $A$ in column-first order.
The Kronecker product between two matrices $A \in \mathbb{R}^{m_1 \times n_1}$ and $B \in \mathbb{R}^{m_2 \times n_2}$ is defined as
$$
A \otimes B = \left ( \begin{matrix}
a_{1,1} B& \cdots &a_{1,n_1} B \\
\vdots & \ddots & \vdots \\
a_{m_1,1} B & \cdots & a_{m_1,n_1} B
\end{matrix} \right) \in \mathbb{R}^{m_1 m_2 \times n_1 n_2},
$$
where $a_{i,j}$ is the element in the $(i, j)$-th entry of $A$.

We use the standard $O(\cdot)$, $\Omega(\cdot)$ and $\Theta(\cdot)$ notation to hide universal constant factors.
We also use $C$ to represent a sufficiently large universal constant whose specific value can differ from line to line.

\subsection{Problem Setup}

Suppose that there are $n$ training examples $\{(x_k,y_k)\}_{k=1}^{n}\subset\mathbb{R}^{d_x}\times\mathbb{R}^{d_y}$.
Denote by $X = \left( x_1, \ldots, x_n \right) \in \mathbb{R}^{d_x \times n}$ the input data matrix and by $Y = \left( y_1, \ldots, y_n \right) \in \mathbb{R}^{d_y \times n}$ the target matrix.
Consider an $L$-layer linear neural network with weight matrices $W_1, \ldots, W_L$, which given an input $x\in \mathbb{R}^{d_x}$ computes
\begin{equation} \label{eqn:linear-net}
f(x; W_1, \ldots, W_L) = \alpha W_L W_{L-1} \cdots  W_1 x ,
\end{equation}
where $W_i \in \mathbb{R}^{d_i\times d_{i-1}} (i=1, \ldots, L)$, $d_0=d_x$, $d_L=d_y$, and $\alpha$ is a normalization constant which will be specified later according to the initialization scheme.
We study the problem of training the deep linear network by minimizing the $\ell_2$ loss over training data:
\begin{equation} \label{eqn:loss-func}
	\begin{aligned}
	\ell(W_1, \ldots, W_L) =
	\frac{1}{2} \sum_{k=1}^n \norm{f(x_k; W_1, \ldots, W_L) - y_k}^2
	= \frac{1}{2} \norm{\alpha W_L \cdots W_1 X - Y}_F^2.
	\end{aligned}
\end{equation}

The algorithm we consider to minimize the objective~\eqref{eqn:loss-func} is gradient descent with random initialization, which first randomly samples the initial weight matrices $\{W_i(0)\}_{i=1}^L$ from a certain distribution, and then updates the weights using gradient descent: for time $t=0, 1, 2, \ldots$,
	\begin{equation} \label{eqn:gd-update}
	W_i(t+1) = W_i(t) - \eta \frac{\partial \ell}{\partial W_i}(W_1(t), \ldots, W_L(t)) , \qquad i\in[L],
	\end{equation}
	where $\eta>0$ is the learning rate.

For convenience, we denote $W_{j:i} = W_jW_{j-1}\cdots W_i\, (1\le i \le j \le L)$ and $W_{i-1:i} = I \, (i\in[L])$.
The time index $t$ is used on any variable that depends on $W_1, \ldots, W_L$ to represent its value at time $t$, e.g., $W_{j:i}(t) = W_j(t) \cdots W_i(t)$, $\ell(t) = \ell(W_1(t), \ldots, W_L(t))$, etc.

\section{Proofs for \Secref{sec:gaussian}}  \label{app:proof-gaussian}

\subsection{Proof of Lemma~\ref{lem:init-product-bound}} \label{app:proof-gaussian-init-product-bound}
\begin{proof}[Proof of Lemma~\ref{lem:init-product-bound}]
	Notice that for any $1\le i \le j \le L$ we have $\expect{\norm{A_{j:i}(0)}_F^2} = d_{i-1}$.
	Then by Markov inequality we have $\Pr\left[ \norm{A_{j:i}(0)}_F^2 \ge \frac{d_{i-1}}{\delta/L^2} \right] \le \delta/L^2$.
	Taking a union bound, we know that with probability at least $1-\delta$, for all $1\le i \le j \le L$ simultaneously we have $\norm{A_{j:i}(0)} \le \norm{A_{j:i}(0)}_F \le \frac{d_{i-1}}{\delta/L^2} \le O(L^3/\delta)$ (note that $d_{i-1} \le O(L^{1-\gamma}) =O(L)$).
\end{proof}

\subsection{Proof of Lemma~\ref{lem:perturbation-bound}} \label{app:proof-gaussian-perturbation-bound}
\begin{proof}[Proof of Lemma~\ref{lem:perturbation-bound} (continued)]
	For the second part of the lemma ($j-i\ge \frac L4$), we need to bound the terms of the form $A_{j:k+1}(0) \cdot \Delta_k \cdot A_{k-1:i}(0)$ more carefully. In fact, if $j-i\ge \frac L4$, then $\max\{ j-k-1, k-1-i \} \ge \frac{L}{10}$, which by assumption means either $A_{j:k+1}(0)$ or $A_{k-1:i}(0)$ has spectral norm bounded by $e^{-c_1 L^\gamma}$.
	This implies $\norm{A_{j:k+1}(0) \cdot \Delta_k \cdot A_{k-1:i}(0)} \le e^{-c_1 L^\gamma} e^{-0.6c_1 L^\gamma} \cdot O(L^3) = e^{-1.6c_1 L^\gamma} \cdot O(L^3)$.
	Therefore we have
	\begin{align*}
	&\norm{A_{j:i} - A_{j:i}(0)} \le (j-i+1) e^{-1.6c_1L^\gamma} \cdot O(L^3) + \sum_{s=2}^{j-i+1} \binom{j-i+1}{s} \left(e^{-0.6c_1 L^\gamma}\right)^s \left( O(L^3) \right)^{s+1}  \\
	\le\,& e^{- c_1 L^\gamma} +  \sum_{s=2}^{\infty} L^s  \left(e^{-0.6c_1 L^\gamma}\right)^s \left( O(L^3) \right)^{s+1}
	\le e^{- c_1 L^\gamma} +  \sum_{s=2}^\infty \left( e^{-0.5c_1L^\gamma} \right)^s
	= O\left( e^{- c_1 L^\gamma} \right).
	\end{align*}
	This implies $\norm{A_{j:i}} \le  O\left( e^{- c_1 L^\gamma} \right) $.
\end{proof}

\subsection{Proof of Lemma~\ref{lem:loss-gradient-bound}} \label{app:proof-gaussian-loss-gradient-bound}

\begin{proof}[Proof of Lemma~\ref{lem:loss-gradient-bound}]
	We can bound the network's output as $$\norm{\alpha W_{L:1}(0)X}_F = \norm{\beta A_{L:1}(0)X}_F \le L^{O(1)} \cdot e^{-\Omega(L^\gamma)} \norm{X}_F = e^{-\Omega(L^\gamma)} .$$
	Thus the objective value $\ell(W_1, \ldots, W_L) = \frac12 \norm{\alpha W_{L:1}(0)X  -Y}_F^2 $ must be extremely close to $\frac{1}{2}\norm{Y}_F^2$ for large $L$, so $0.4 \norm{Y}_F^2 < \ell(W_1, \ldots, W_L) < 0.6 \norm{Y}_F^2$.

	As for the gradient, for any $i\in[L]$ we have
	\begin{align*}
	&\norm{\nabla_{W_i} \ell(W_1, \ldots, W_L)}
	= \norm{ \alpha W_{L:i+1}^\top \left(\alpha W_{L:1}X  -Y\right) X^\top W_{i-1:1}^\top}\\
	=\,& \norm{ \beta / (\sqrt{d_i}\sigma_i) \cdot A_{L:i+1}^\top \left(\alpha W_{L:1}X  -Y\right) X^\top A_{i-1:1}^\top }
	\le \frac{L^{O(1)}}{\sqrt{d_i}\sigma_i} \norm{A_{L:i+1}} \cdot O(1) \cdot \norm{A_{i-1:1}} .
	\end{align*}
	Using~\eqref{eqn:product-bound}, and noting that either $L-i-1$ or $i-1$ is greater than $\frac L4$, we have
	\begin{equation*}
	\norm{\nabla_{W_i} \ell(W_1, \ldots, W_L)} \le \sigma_i^{-1} L^{O(1)} \cdot O\left( e^{-c_1L^\gamma} \right) \cdot O(L^3)
	\le (\sqrt{d_i}\sigma_i )^{-1} e^{-0.9c_1L^\gamma} . \qedhere
	\end{equation*}
\end{proof}

\section{Proofs for \Secref{sec:ortho}}  \label{app:proof-ortho}

\subsection{Proof of Lemma~\ref{lem:ortho-init}} \label{app:proof-ortho-init}

\begin{proof}[Proof of Lemma~\ref{lem:ortho-init}]
We only need to prove~\eqref{eqn:init-loss}. We first upper bound the magnitude of the network's initial output on any given input $x\in\mathbb{R}^{d_x}$.
	Let $z = \frac{1}{\sqrt{m^{L-1}}} W_{L-1:1}(0) \cdot x \in \mathbb{R}^m$. Then we have $\norm{z}=\norm{x}$, and $f(x; W_1(0), \ldots, W_L(0)) = \frac{1}{\sqrt{d_y}} W_L(0) \cdot z = \sqrt{\frac{m}{d_y}} \cdot \frac{1}{\sqrt{m}} W_L(0) \cdot z $.
	Note that $\frac{1}{\sqrt{m}} W_L(0) \cdot z $ is the (signed) projection of $z$ onto a random subspace in $\mathbb{R}^m$ of dimension $d_y$. Therefore $\norm{\frac{1}{\sqrt{m}} W_L(0) \cdot z}^2 / \norm{z}^2$ has the same distribution as $\frac{g_1^2 + \cdots + g_{d_y}^2}{g_1^2 + \cdots + g_m^2}$, where $g_1, \ldots, g_m $ are i.i.d. samples from ${\mathcal{N}}(0, 1)$.
	By the standard tail bounds for chi-squared distributions we have
	\begin{align*}
	&\Pr\left[ g_1^2 + \cdots + g_{d_y}^2 \le d_y + 2\sqrt{d_y \log(1/\delta')} + 2\log(1/\delta') \right] \ge 1-\delta', \\
	&\Pr\left[ g_1^2 + \cdots + g_{m}^2 \ge m - 2\sqrt{m \log(1/\delta') } \right] \ge 1-\delta'.
	\end{align*}
	Let $\delta' = \frac{\delta}{2r}$.
	Note that $m > C\cdot \log(r/\delta)$. We know that with probability at least $1-\frac{\delta}{r}$ we have
	\begin{align*}
	\norm{\frac{1}{\sqrt{m}} W_L(0) \cdot z}^2 / \norm{z}^2
	\le \frac{d_y + 2\sqrt{d_y \log(2r/\delta)} + 2\log(2r/\delta)}{m - 2\sqrt{m \log(2r/\delta) }}
	= \frac{O(d_y + \log(r/\delta))}{\Omega(m)},
	\end{align*}
	which implies
	\begin{equation} \label{eqn:ortho-inproof-1}
	\begin{aligned}
	\norm{ f(x; W_1(0), \ldots, W_L(0)) }^2
	&= \frac{m}{d_y} \norm{\frac{1}{\sqrt{m}} W_L(0) \cdot z}^2
	= \frac{m}{d_y} \cdot O\left( \frac{d_y + \log(r/\delta)}{m} \right) \norm{z}^2 \\
	&= O\left( 1 + \frac{\log(r/\delta)}{d_y} \right) \norm{x}^2.
	\end{aligned}
	\end{equation}
	Finally, taking a union bound, we know that with probability at least $1-\delta$, the inequality \eqref{eqn:ortho-inproof-1} holds for every $x\in\{x_1, \ldots, x_r\}$, which implies
	\begin{align*}
	\ell(0) &= \frac12 \sum_{k=1}^r \norm{ f(x_k; W_1(0), \ldots, W_L(0)) - y_k }^2
	\le \sum_{k=1}^r \left( \norm{ f(x_k; W_1(0), \ldots, W_L(0))}^2 + \norm{ y_k }^2 \right) \\
	&\le O\left( 1 + \frac{\log(r/\delta)}{d_y} \right) \sum_{k=1}^r\norm{x_k}^2 + \sum_{k=1}^r \norm{ y_k }^2
	=  O\left( 1 + \frac{\log(r/\delta)}{d_y} \right) \norm{X}_F^2 +  \norm{ Y }_F^2 \\
	& \le O\left( 1 + \frac{\log(r/\delta)}{d_y} + \norm{W^*}^2 \right) \norm{X}_F^2. \qedhere
	\end{align*}
\end{proof}

\subsection{Proof of Claim~\ref{claim:induction-1}}
\label{app:proof-induction-1}

\begin{proof}[Proof of Claim~\ref{claim:induction-1}]
Let $\gamma = \frac12 L \sigma_{\min}^2(X)/d_y$.
From ${\mathcal{A}}(0), \ldots, {\mathcal{A}}(t)$ we have $\ell(s) \le (1-\eta\gamma)^sB$ for all $0\le s \le t$.
The gradient of the objective function~\eqref{eqn:loss-func} is
$\frac{\partial\ell}{\partial W_i} = \alpha W_{L:i+1}^\top (U - Y) \left( W_{i-1:1} X \right)^\top$.
Thus we can bound the gradient norm as follows for all $0\le s\le t$ and all $i\in[L]$:
\begin{equation} \label{eqn:gradient-norm-bound}
\begin{aligned}
&\norm{\frac{\partial\ell}{\partial W_i}(s)}_F
\le \alpha  \norm{W_{L:i+1}(s)} \norm{U(s) - Y}_F \norm{W_{i-1:1}(s)} \norm{X}\\
\le\,& \frac{1}{\sqrt{m^{L-1}d_y}} \cdot 1.1 m^{\frac{L-i}{2}}  \cdot \sqrt{2\ell(s)} \cdot 1.1 m^{\frac{i-1}{2}} \norm{X}
\le \frac{2\sqrt{(1-\eta\gamma)^sB}}{\sqrt{d_y}}   \norm{X} ,
\end{aligned}
\end{equation}
where we have used ${\mathcal{B}}(s)$.
Then for all $i\in[L]$ we have:
\begin{align*}
&\norm{W_i(t+1) - W_i(0)}_F
\le \sum_{s=0}^{t} \norm{W_i(s+1) - W_i(s)}_F
=\,  \sum_{s=0}^{t} \norm{\eta \frac{\partial\ell}{\partial W_i}(s)}_F\\
\le\,& \eta \sum_{s=0}^{t} \frac{2\sqrt{(1-\eta\gamma)^sB}}{\sqrt{d_y}}   \norm{X}
\le \frac{2\eta \sqrt{B}}{\sqrt{d_y}} \norm{X} \sum_{s=0}^{t-1}(1-\eta\gamma/2)^s
\le \frac{2 \eta \sqrt{B}}{ \sqrt{d_y}} \norm{X} \cdot \frac{2}{\eta\gamma}     \\
=\,& \frac{8 \sqrt{B d_y} \norm{X}}{ L \sigma_{\min}^2(X)} .
\end{align*}
This proves ${\mathcal{C}}(t+1)$.
\end{proof}

\subsection{Proof of Claim~\ref{claim:induction-2}}
\label{app:proof-induction-2}

\begin{proof}[Proof of Claim~\ref{claim:induction-2}]
Let $R = \frac{8\sqrt{Bd_y}\norm{X}}{ L \sigma_{\min}^2(X)}$ and $\Delta_i = W_i(t) - W_i(0)$ $(i\in[L])$.
Then ${\mathcal{C}}(t)$ means $\norm{\Delta_i}_F \le R\, (\forall i\in[L])$.

For $1\le i \le j \le L$, we have
\begin{align*}
W_{j:i}(t) = \left( W_j(0) + \Delta_j \right) \cdots \left( W_i(0) + \Delta_i \right) .
\end{align*}
Expanding this product,
each term except $W_{j:i}(0)$ has the form:
\begin{equation} \label{eqn:perturbation-terms}
\begin{aligned}
W_{j:(k_s+1)}(0)	\cdot \Delta_{k_s} \cdot W_{(k_s-1):(k_{s-1}+1)}(0) \cdot \Delta_{k_{s-1}} \cdots  \Delta_{k_1} \cdot W_{(k_1-1):i}(0)  ,
\end{aligned}
\end{equation}
where $i \le k_1 < \cdots < k_s \le j$ are locations where terms like $\Delta_{k_l}$ are taken out.
Note that every factor in~\eqref{eqn:perturbation-terms} of the form $W_{j':i'}(0)$ satisfies $\norm{W_{j':i'}(0)} = m^{\frac{j'-i'+1}{2}}$ according to~\eqref{eqn:init-spec}.
Thus, we can bound the sum of all terms of the form~\eqref{eqn:perturbation-terms} as
\begin{align*}
&\norm{W_{j:i}(t) - W_{j:i}(0)}
\le \sum_{s=1}^{j-i+1} \binom{j-i+1}{s} R^s  m^{\frac{j-i+1-s}{2}}
= (\sqrt m + R)^{j-i+1} - (\sqrt m)^{j-i+1} \\
=\,& (\sqrt m)^{j-i+1} \left( \left( 1 + {R}/{\sqrt m} \right)^{j-i+1} - 1 \right)
\le (\sqrt m)^{j-i+1} \left( \left( 1 + {R}/{\sqrt m} \right)^{L} - 1 \right)
\le 0.1 (\sqrt m)^{j-i+1}.
\end{align*}
Here the last step uses $m > C (LR)^2$ which is implied by~\eqref{eqn:m-bound-for-ortho}.
Combined with~\eqref{eqn:init-spec}, this proves ${\mathcal{B}}(t)$.
\end{proof}

\begin{comment}
The proof of~\eqref{eqn:perturbation-left} is very similar.
We still look at products of the form~\eqref{eqn:perturbation-terms} with $j=L$.
Still, there are at most $s+1$ factors of the form $W_{b:a}(0)$ each satisfying $\norm{W_{b:a}(0)} \le O\left(\sqrt Lm^{\frac{b-a+1}{2}}\right)$. However, there are at most $s$ (instead of $s+1$) such factors such that $1<a\le b<L$.
Therefore, the product of spectral norms of all such factors is at most $\left(O(\sqrt{L})\right)^{s} m^{\frac{L-i+1-s}{2}} $.
In other words, we can save a factor of $O(\sqrt L)$ compared with~\eqref{eqn:perturbation-bound-middle}. This gives us
\begin{align*}
\norm{W_{L:i}(t) - W_{L:i}(0)} \le 0.05 m^{\frac{L-i+1}{2}},
\end{align*}
proving~\eqref{eqn:perturbation-left}.

Next we prove~\eqref{eqn:perturbation-right}.
For $1\le i<L$, we need to bound the sum of terms of the following form:
\begin{equation*}
\begin{aligned}
W_{i:(k_s+1)}(0)	\cdot \Delta_{k_s} \cdot W_{(k_s-1):(k_{s-1}+1)}(0) \cdot \Delta_{k_{s-1}} \cdots \\ \cdot \Delta_{k_1} \cdot W_{(k_1-1):1}(0) X .
\end{aligned}
\end{equation*}
Again, similar as before and noting $\norm{W_{(k_1-1):1}(0) X} \le \frac54 m^{\frac{k_1-1}{2}}\norm{X}$, we have
\small \begin{equation*}
\begin{aligned}
&\norm{W_{i:1}(t) - W_{i:1}(0)} \\
\le\,& \sum_{s=1}^{i} \binom{i}{s} R^s \left( O\left(\sqrt L\right) \right)^{s} \cdot \frac54  m^{\frac{i-s}{2}} \norm{X} \\
\le\,& \frac54 \sum_{s=1}^{i} L^s R^s \left( O\left(\sqrt L\right) \right)^{s} m^{\frac{i-s}{2}} \norm{X} \\
=\,& \frac54 m^{\frac i2} \sum_{s=1}^{i} \left( \frac{ O\left(L^{3/2}R\right)}{\sqrt m} \right)^s \norm{X}  \\
\le\,& 0.05 m^{\frac{i}{2}} \sigma_{\min}(X) ,
\end{aligned}
\end{equation*} \normalsize
as long as $ m > C L^{3} R^2 \cdot \frac{\norm{X}^2}{\sigma_{\min}^2(X)} = CL^3R^2 \kappa$ which is implied by~\eqref{eqn:m-bound-implication}.
This finishes the proof of~\eqref{eqn:perturbation-right}.

\end{comment}

\subsection{Proof of Claim~\ref{claim:induction-3}}
\label{app:proof-induction-3}

\begin{proof}[Proof of Claim~\ref{claim:induction-3}]
Recall that we have the dynamics~\eqref{eqn:U-dynamics} for $U(t)$.
In order to establish convergence from~\eqref{eqn:U-dynamics} we need to prove upper and lower bounds on the eigenvalues of $P(t)$, as well as show that the high-order term $E(t)$ is small.
We will prove these using ${\mathcal{B}}(t)$.

Using the definition~\eqref{eqn:P-def} and property ${\mathcal{B}}(t)$, we have
 \begin{align*}
\lambda_{\max}(P(t))
&\le \alpha^2  \sum_{i=1}^L  \lambda_{\max}\left( \left( W_{i-1:1}(t) X \right)^\top \left( W_{i-1:1}(t)X \right) \right)   \cdot   \lambda_{\max}\left( W_{L:i+1}(t)  W_{L:i+1}^\top(t) \right)  \\
&\le \frac{1}{m^{L-1}d_y} \sum_{i=1}^L \left( 1.1 m^{\frac{i-1}{2}} \sigma_{\max}(X) \right)^2 \left( 1.1 m^{\frac{L-i}{2}} \right)^2
\le 2 L \sigma_{\max}^2(X) / d_y, \\
\lambda_{\min}(P(t))
&\ge \alpha^2 \sum_{i=1}^L  \lambda_{\min}\left( \left( W_{i-1:1}(t) X \right)^\top \left( W_{i-1:1}(t)X \right) \right)  \cdot   \lambda_{\min}\left( W_{L:i+1}(t)  W_{L:i+1}^\top(t) \right)  \\
&\ge \frac{1}{m^{L-1}d_y} \sum_{i=1}^L \left( 0.9 m^{\frac{i-1}{2}} \sigma_{\min}(X) \right)^2 \left( 0.9 m^{\frac{L-i}{2}} \right)^2
\ge \frac{3}{5} L \sigma_{\min}^2(X) / d_y.
\end{align*}
In the lower bound above, we make use of the following relation on dimensions: $m\ge d_x \ge r$, which enables the inequality $\lambda_{\min}\left( \left( W_{i-1:1}(t) X \right)^\top \left( W_{i-1:1}(t)X \right) \right) = \sigma_{\min}^2\left( W_{i-1:1}(t) X \right) \ge \sigma_{\min}^2\left( W_{i-1:1}(t) \right) \cdot \sigma_{\min}^2(X)$.

Next, we will prove the following bound on the high-order term $E(t)$:
\[
\frac{1}{\sqrt{m^{L-1}d_y}} \norm{E(t) X}_F \le  \frac16 \eta \lambda_{\min}(P_t) \norm{U(t)-Y}_F .
\]
Recall that $E(t)$ is the sum of all high-order terms in the product
\begin{align*}
W_{L:1}(t+1) = \prod_i \left( W_i(t) - \eta \frac{\partial \ell}{\partial W_i}(t)  \right).
\end{align*}
Same as~\eqref{eqn:gradient-norm-bound}, we have $\norm{\frac{\partial \ell}{\partial W_i} (t)}_F \le \frac{2 \sqrt{\ell(t)}\norm{X}}{\sqrt{d_y}}$ ($\forall i\in[L]$).
Then we have
\begin{align*}
& \frac{1}{\sqrt{m^{L-1}d_y}} \norm{E(t) X}_F \\
\le\,& \frac{1}{\sqrt{m^{L-1}d_y}} \sum_{s=2}^L \binom{L}{s} \left(\eta \cdot \frac{2 \sqrt{\ell(t)}\norm{X}}{\sqrt{d_y}} \right)^s    m^{\frac{L-s}{2}} \norm{X} \\
\le\,& \sqrt{\frac{m}{d_y}} \norm{X} \sum_{s=2}^L L^s \left(\eta \cdot \frac{2 \sqrt{\ell(t)}\norm{X}}{\sqrt{d_y}} \right)^s    m^{-\frac{s}{2}}  \\
=\,& \sqrt{\frac{m}{d_y}} \norm{X} \sum_{s=2}^L  \left( \frac{2\eta L \sqrt{\ell(t)}\norm{X}}{\sqrt{m d_y}} \right)^s
\end{align*}
From $\eta \le \frac{d_y}{ 2L \norm{X}^2 }$, we have $\frac{2\eta L \sqrt{\ell(t)}\norm{X}}{\sqrt{m d_y}}  \le  \frac{  \sqrt{d_y \cdot\ell(t)}}{  \sqrt{m}\norm{X} } $.
Note that $m > C \cdot\frac{ d_y B}{\norm{X}^2} \ge C \cdot\frac{ d_y \ell(t)}{\norm{X}^2}$. Thus we have
\begin{align*}
& \frac{1}{\sqrt{m^{L-1}d_y}} \norm{E(t) X}_F
\le \sqrt{\frac{m}{d_y}} \norm{X}   \left( \frac{2\eta L \sqrt{\ell(t)}\norm{X}}{\sqrt{m d_y}} \right)^2 \sum_{s=2}^{L-2} 0.5^{s-2} \\
\le\,& 2 \sqrt{\frac{m}{d_y}} \norm{X}   \left( \frac{2\eta L \sqrt{\ell(t)}\norm{X}}{\sqrt{m d_y}} \right)^2
\le 2 \sqrt{\frac{m}{d_y}} \norm{X}  \cdot  \frac{2\eta L \sqrt{\ell(t)}\norm{X}}{\sqrt{m d_y}}  \cdot \frac{  \sqrt{d_y \cdot\ell(t)}}{  \sqrt{m}\norm{X} } \\
=\,& \frac{4\eta L \norm{X} \cdot \ell(t)}{\sqrt{md_y}}.
\end{align*}
It suffices to show that the above bound is at most $\frac16 \eta \lambda_{\min}(P_t) \norm{U(t)-Y}_F = \frac16 \eta \lambda_{\min}(P_t) \sqrt{2\ell(t)} $.
Since $\lambda_{\min}(P_t) \ge \frac{3}{5} L \sigma_{\min}^2(X) / d_y$, it suffices to have
\begin{align*}
\frac{4\eta L \norm{X} \cdot \ell(t)}{\sqrt{md_y}} \le \frac16 \eta \cdot \frac{3 L \sigma_{\min}^2(X) \sqrt{2\ell(t)} }{ 5 d_y},
\end{align*}
which is true since $m> C \cdot \frac{d_y B \norm{X}^2}{\sigma_{\min}^4(X)} \ge C \cdot \frac{d_y \ell(t)\norm{X}^2}{\sigma_{\min}^4(X)}$.

Finally, from~\eqref{eqn:U-dynamics} and $\eta \le \frac{d_y}{2L \norm{X}^2} \le \frac{1}{\lambda_{\max}(P_t)}$ we have
\begin{align*}
&\norm{U(t+1)-Y}_F= \norm{\vectorize{U(t+1) - Y} }\\
=\,& \norm{ (I-\eta P(t)) \cdot \vectorize{U(t)-Y} + \frac{1}{\sqrt{m^{L-1}d_y}} \vectorize{E(t)X}} \\
\le\,& (1 - \eta \lambda_{\min}(P(t)) ) \norm{\vectorize{U(t)-Y}} + \frac{1}{\sqrt{m^{L-1}d_y}} \norm{E(t) X}_F \\
\le\,& (1 - \eta \lambda_{\min}(P(t)) ) \norm{U(t)-Y}_F + \frac16 \eta \lambda_{\min}(P_t) \norm{U(t)-Y}_F \\
=\,& \left(1 - \frac56 \eta \lambda_{\min}(P(t)) \right) \norm{U(t)-Y}_F  \\
\le\,& \left(1 - \frac12 \eta L \sigma_{\min}^2(X)/d_y \right) \norm{U(t)-Y}_F .
\end{align*}
Therefore $\ell(t+1) \le \left(1 - \frac12 \eta L \sigma_{\min}^2(X)/d_y \right)^2 \ell(t) \le \left(1 - \frac12 \eta L \sigma_{\min}^2(X)/d_y \right)\ell(t)$.
Combined with ${\mathcal{A}}(t)$, this proves ${\mathcal{A}}(t+1)$.
\end{proof}

\section{Related Work} \label{sec:related}

\paragraph{Deep linear networks.}
Despite the simplicity of their input-output maps, deep linear networks define high-dimensional non-convex optimization landscapes whose properties closely reflect those of their non-linear counterparts. For this reason, deep linear networks have been the subject of extensive theoretical analysis. A line of work~\citep{kawaguchi2016deep,hardt2016identity,lu2017depth,yun2017global,zhou2018critical,laurent2018deep} studied the landscape properties of deep linear networks. Although it was established that all local minima are global under certain assumptions, these properties alone are still not sufficient to guarantee global convergence or to provide a concrete rate of convergence for gradient-based optimization algorithms.

Another line of work directly analyzed the trajectory taken by gradient descent and established conditions that guarantee convergence to global minimum~\citep{bartlett2018gradient, arora2018convergence, du2019width}.
Most relevant to our work is the result of \cite{du2019width}, which shows that if the width of hidden layers is larger than the depth, gradient descent with Gaussian initialization can efficiently converge to a global minimum. Our result establishes that for Gaussian initialization, this linear dependence between width and depth is necessary, while for orthogonal initialization, the width can be independent of depth.
Our negative result for Gaussian initialization also significantly generalizes the result of \cite{shamir2018exponential}, who proved a similar negative result for $1$-dimensional linear networks.

\paragraph{Orthogonal weight initializations.}
Orthogonal weight initializations have also found significant success in non-linear networks. In the context of feedforward models, the spectral properties of a network's input-output Jacobian have been empirically linked to convergence speed~\citep{saxe2014exact,pennington2017resurrecting,pennington2018emergence,xiao2018dynamical}. It was found that when this spectrum concentrates around $1$ at initialization, a property dubbed \emph{dynamical isometry}, convergence times improved by orders of magnitude. The conditions for attaining dynamical isometry in the infinite-width limit were established by~\citet{pennington2017resurrecting,pennington2018emergence} and basically require that input-output map to be approximately linear and for the weight matrices to be orthogonal. Therefore the training time benefits of dynamical isometry are likely rooted in the benefits of orthogonality for deep linear networks, which we establish in this work.

Orthogonal matrices are also frequently used in the context of recurrent neural networks, for which the stability of the state-to-state transition operator is determined by the spectrum of its Jacobian~\citep{haber2017stable,laurent2016recurrent}. Orthogonal matrices can improve the conditioning, leading to an ability to learn over long time horizons~\citep{le2015simple,henaff2016recurrent,chen2018dynamical,gilboa2019dynamical}. While the benefits of orthogonality can be quite large at initialization, little is known about whether or in what contexts these benefits persist during training, a scenario that has lead to the development of efficient methods of constraining the optimization to the orthogonal group~\citep{wisdom2016full,vorontsov2017orthogonality,mhammedi2017efficient}. Although we do not study the recurrent setting in this work, an extension of our analysis might help determine when orthogonality is beneficial in that setting.

Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC); Machine Learning (stat.ML)