Numerical Matrix Decomposition

Abstract: In 1954, Alston S. Householder published \textit{Principles of Numerical Analysis}, one of the first modern treatments on matrix decomposition that favored a (block) LU decomposition-the factorization of a matrix into the product of lower and upper triangular matrices. And now, matrix decomposition has become a core technology in machine learning, largely due to the development of the back propagation algorithm in fitting a neural network. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, Hilbert space, and things in the complex domain. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields.Keywords: Existence and computing of matrix decompositions, Floating point operations (flops), Low-rank approximation, Pivot, LU/PLU decomposition, CR/CUR/Skeleton decomposition, Coordinate transformation, ULV/URV decomposition, Rank decomposition, Rank revealing decomposition, Update/downdate, Tensor decomposition.

\part{Gaussian Elimination}
\section*{Introduction}
In linear algebra, the \textit{Gaussian elimination} is often referred to as the  \textit{row reduction}, which is an algorithm for solving systems of linear equations such that the original full linear equation is converted into the \textit{row echelon} form (or in some cases, an upper triangular one) whilst the computational complexity is reduced.

In the row reduction process, the Gaussian elimination performs elementary row operations that can be divided into two phases. The first phase is also known as the \textit{forward elimination} that reduces the linear system into an upper triangular one or the row echelon form in general, where the properties of the linear system is kept into the new one such as the rank and from which we can tell whether there are solutions, the uniqueness of the solution and so on. In the second phase, the process performs a \textit{back substitution} such that the linear system is converted into a \textit{reduced row echelon form} and the solution is found.

In this part, we will introduce the related LU, and Cholesky decompositions. And another highly relevant decomposition, CR decomposition, will be delayed into Part~\ref{part:data-interation} as it skeletons the matrix and compresses the matrix into a thin one whilst sparsity, nonnegativity of the matrices are kept.

\chapter{LU Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{LU Decomposition}
Perhaps the best known and the first matrix decomposition we should know about is the LU decomposition. We now illustrate the results in the following theorem and the proof of the existence of which will be delayed in the next sections.
\begin{theoremHigh}[LU Decomposition with Permutation]\label{theorem:lu-factorization-with-permutation}
Every nonsingular $n\times n$ square matrix $\bA$ can be factored as
\begin{equation}
	\bA = \bP\bL\bU, \nonumber
\end{equation}
where $\bP$ is a permutation matrix, $\bL$ is a unit lower triangular matrix (i.e., lower triangular matrix with all 1's on the diagonal), and $\bU$ is a nonsingular upper triangular matrix.
\end{theoremHigh}

The LU decomposition was the decomposition of Gauss's elimination algorithm, which he sketched in 1809 \citep{gauss1877theoria} and presented in full in 1810 \citep{gauss1828disquisitio}. See also the discussion by \citep{stewart2000decompositional}. Note that, in the remainder of this text, we will put the decomposition-related results in the blue box. And other claims will be in a gray box. This rule will be applied for the rest of the survey without special mention.

\begin{remark}[Decomposition Notation]
The above decomposition applies to any nonsingular matrix $\bA$. We will see that this decomposition arises from the elimination steps in which case row operations of subtraction and exchange of two rows are allowed where the subtractions are recorded in matrix $\bL$ and the row exchanges are recorded in matrix $\bP$. To make this row exchange explicit, the common form for the above decomposition is $\bQ\bA=\bL\bU$ where $\bQ=\bP^\top$ that records the exact row exchanges of the rows of $\bA$. Otherwise, the $\bP$ would record the row exchanges of $\bL\bU$. In our case, we will make the decomposition to be clear for matrix $\bA$ rather than for $\bQ\bA$. For this reason, we will put the permutation matrix on the right-hand side of the equation for the remainder of the text without special mention.
\end{remark}

Specifically, in some cases, we will not need the permutation matrix. This decomposition relies on the leading principal minors. We provide the definitions which are important for the illustration.
\begin{definition}[Principal Minors\index{Principal minors}]\label{definition:principle-minors}
	Let $\bA$ be an $n\times n$ square matrix. A $k \times k$ submatrix of $\bA$ obtained by deleting any $n-k$ columns and the same $n-k$ rows from $\bA$ is called a $k$-th order \textbf{principal submatrix} of $\bA$. The determinant of a $k \times k$ principal submatrix is called a $k$-th order \textbf{principal minor} of $\bA$.
\end{definition}

Under mild conditions on the selected indices for the submatrix, we may obtain a specific kind of principal minors.
\begin{definition}[Leading Principal Minors\index{Leading principal minors}]\label{definition:leading-principle-minors}
	Let $\bA$ be an $n\times n$ square matrix. A $k \times k$ submatrix of $\bA$ obtained by deleting the \textbf{last} $n-k$ columns and the \textbf{last} $n-k$ rows from $\bA$ is called a $k$-th order \textbf{leading principal submatrix} of $\bA$, that is, the $k\times k$ submatrix taken from the top left corner of $\bA$. The determinant of the $k \times k$ leading principal submatrix is called a $k$-th order \textbf{leading principal minor} of $\bA$.
\end{definition}

Given an $n\times n$ matrix $\bA$ with $(i,j)$-th entry being $a_{ij}$, let $\bA_{1:k,1:k}$ denote the $k\times k$ submatrix taken from the top left corner of $\bA$. That is,
$$
\bA_{1:k,1:k} =
\begin{bmatrix}
a_{11} & a_{12} & \ldots & a_{1k}\\
a_{21} & a_{22} & \ldots & a_{2k}\\
\vdots & \vdots & \ddots & \vdots \\
a_{k1} & a_{k2} & \ldots & a_{kk}\\
\end{bmatrix}.
$$
Then $\Delta_k=\det(\bA_{1:k,1:k} )$ is the $k$-th order leading principal minor of $\bA$.

Under specific conditions on the leading principal minors of matrix $\bA$, the LU decomposition will not involve the permutation matrix.
\begin{theoremHigh}[LU Decomposition without Permutation]\label{theorem:lu-factorization-without-permutation}
For any $n\times n$ square matrix $\bA$, if all the leading principal minors are nonzero, i.e., $\det(\bA_{1:k,1:k})\neq 0$, for all $k\in \{1,2,\ldots, n\}$, then $\bA$ can be factored as
\begin{equation}
	\bA = \bL\bU, \nonumber
\end{equation}
where $\bL$ is a unit lower triangular matrix (i.e., lower triangular matrix with all 1's on the diagonal), and $\bU$ is a \textbf{nonsingular} upper triangular matrix.

\item Specifically, this decomposition is \textbf{unique}. See Corollary~\ref{corollary:unique-lu-without-permutation}.
\end{theoremHigh}

\begin{remark}[Other Forms of the LU Decomposition without Permutation]
The leading principal minors are nonzero, in another word, means the leading principal submatrices are nonsingular.

\paragraph{Singular $\bA$} In the above theorem, we assume $\bA$ is nonsingular as well. The LU decomposition also exists for singular matrix $\bA$. However, the matrix $\bU$ will be singular as well in this case. This can be shown in the following section that, if matrix $\bA$ is singular, some pivots will be zero, and the corresponding diagonal values of $\bU$ will be zero.

\paragraph{Singular leading principal submatrices} Even if we assume matrix $\bA$ is nonsingular, the leading principal submatrices might be singular. Suppose further that some of the leading principal minors are zero, the LU decomposition also exists, but if so, it is again not unique.
\end{remark}

We will discuss where this decomposition comes from in the next section. There are also generalizations of LU decomposition to non-square or singular matrices, such as rank-revealing LU decomposition. Please refer to \citep{pan2000existence, miranian2003strong, dopico2006multiple} or we will have a short discussion in Section~\ref{section:rank-reveal-lu-short}.

\section{Relation to Gaussian Elimination}\label{section:gaussian-elimination}
Solving linear system equation $\bA\bx=\bb$ is the basic problem in linear algebra.
Gaussian elimination transforms a linear system into an upper triangular one by applying simple \textit{elementary row transformations} on the left of the linear system in $n-1$ stages if $\bA\in \real^{n\times n}$. As a result, it is much easier to solve by a backward substitution. The elementary transformation is defined rigorously as follows.
\begin{definition}[Elementary Transformation\index{Elementary transformation}]
For square matrix $\bA$, the following three transformations are referred as \textbf{elementary row/column transformations}:
\item 1. Interchanging two rows (or columns) of $\bA$;
\item 2. Multiplying all elements of a row (or a column) of $\bA$ by some nonzero number;
\item 3. Adding any row (or column) of $\bA$ multiplied by a nonzero number to any other row (or column);
\end{definition}
Specifically, the elementary row transformations of $\bA$ are unit lower triangular to multiply $\bA$ on the left, and the elementary column transformations of $\bA$ are unit upper triangular to multiply $\bA$ on the right.

The Gaussian elimination is described by the third type - elementary row transformation above. Suppose the upper triangular matrix obtained by Gaussian elimination is given by $\bU = \bE_{n-1}\bE_{n-2}\ldots\bE_1\bA$, and in the $k$-th stage, the $k$-th column of $\bE_{k-1}\bE_{k-2}\ldots\bE_1\bA$ is $\bx\in \real^n$. Gaussian elimination will introduce zeros below the diagonal of $\bx$ by
$$
\bE_k = \bI - \bz_k \be_k^\top,
$$
where $\be_k \in \real^n$ is the $k$-th unit basis vector, and $\bz_k\in \real^n$ is given by
$$
\bz_k = [0, \ldots, 0, z_{k+1}, \ldots, z_n]^\top, \qquad z_i= \frac{x_{i}}{x_{k}}, \gap\forall i \in \{k+1,\ldots, n\}.
$$
We realize that $\bE_k$ is a unit lower triangular matrix (with $1$'s on the diagonal) with only the $k$-th column of the lower submatrix being nonzero,
$$
\bE_k=
\begin{bmatrix}
1 & \ldots & 0&  0 & \ldots & 0\\
\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
0 & \ldots & 1  & 0 & \ldots & 0 \\
0 & \ldots & -z_{k+1}  & 1 & \ldots & 0\\
\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
0 & \ldots & -z_n & 0 & \ldots & 1
\end{bmatrix},
$$
and multiplying on the left by $\bE_k$ will introduce zeros below the diagonal:
$$
\bE_k \bx =
\begin{bmatrix}
	1 & \ldots & 0&  0 & \ldots & 0\\
	\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
	0 & \ldots & 1  & 0 & \ldots & 0 \\
	0 & \ldots & -z_{k+1}  & 1 & \ldots & 0\\
	\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
	0 & \ldots & -z_n & 0 & \ldots & 1
\end{bmatrix}
\begin{bmatrix}
	x_1 \\
	\vdots \\
	x_k\\
	x_{k+1} \\
	\vdots \\
	x_n
\end{bmatrix}=
\begin{bmatrix}
	x_1 \\
	\vdots \\
	x_k\\
0 \\
	\vdots \\
0
\end{bmatrix}.
$$

For example, we write out the Gaussian elimination steps for a $4\times 4$ matrix. For simplicity, we assume there are no row permutations. And in the following matrix, $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\begin{tcolorbox}[title={A Trivial Gaussian Elimination For a $4\times 4$ Matrix}]
\begin{equation}\label{equation:elmination-steps}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	 0 & \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	 0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
      0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bE_3}{\longrightarrow}
\begin{sbmatrix}{\bE_3\bE_2\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
    0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & 0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix},
\end{equation}
\end{tcolorbox}
\noindent where $\bE_1, \bE_2, \bE_3$ are lower triangular matrices. Specifically, as discussed above, Gaussian transformation matrices $\bE_i$'s are unit lower triangular matrices with $1$'s on the diagonal. This can be explained that for the $k$-th transformation $\bE_k$, working on the matrix $\bE_{k-1}\ldots\bE_1\bA$, the transformation subtracts multiples of the $k$-th row from rows $\{k+1, k+2, \ldots, n\}$ to get zeros below the diagonal in the $k$-th column of the matrix. And never use rows $\{1, 2, \ldots, k-1\}$.

For the transformation example above, at step $1$, we multiply left by $\bE_1$ so that multiples of the $1$-st row are subtracted from rows $2, 3, 4$ and the first entries of rows $2, 3, 4$ are set to zero. Similar situations for step 2 and step 3. By setting $\bL=\bE_1^{-1}\bE_2^{-1}\bE_3^{-1}$ and letting the matrix after elimination be $\bU$, \footnote{The inverses of unit lower triangular matrices are also unit lower triangular matrices. And the products of unit lower triangular matrices are also unit lower triangular matrices.} we get $\bA=\bL\bU$. Thus we obtain an LU decomposition for this $4\times 4$ matrix $\bA$.

\begin{definition}[Pivot\index{Pivot}]\label{definition:pivot}
First nonzero entry in the row after each elimination step is called a \textbf{pivot}. For example, the \textcolor{blue}{blue} crosses in Equation~\eqref{equation:elmination-steps} are pivots.
\end{definition}

But sometimes, it can happen that the value of $\bA_{11}$ is zero. No $\bE_1$ can make the next elimination step successful. So we need to interchange the first row and the second row via a permutation matrix $\bP_1$. This is known as the \textbf{pivoting}, or \textbf{permutation}:
\begin{tcolorbox}[title={Gaussian Elimination With a Permutation In the Beginning}]
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	0 & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\bP_1}{\longrightarrow}
\begin{sbmatrix}{\bP_1\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bP_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
&\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bE_1\bP_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bE_3}{\longrightarrow}
\begin{sbmatrix}{\bE_3\bE_2\bE_1\bP_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & 0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix}.
\end{aligned}
$$
\end{tcolorbox}
\noindent By setting $\bL=\bE_1^{-1}\bE_2^{-1}\bE_3^{-1}$ and $\bP=\bP_1^{-1}$, we get $\bA=\bP\bL\bU$. Therefore we obtain a full LU decomposition with permutation for this $4\times 4$ matrix $\bA$.

In some situations, other permutation matrices $\bP_2, \bP_3, \ldots$ will appear in between the lower triangular $\bE_i$'s. An example is shown as follows:
\begin{tcolorbox}[title={Gaussian Elimination With a Permutation In Between}]
$$
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bP_1}{\longrightarrow}
\begin{sbmatrix}{\bP_1\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bP_1\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & \bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix}.
$$
\end{tcolorbox}
\noindent In this case, we find $\bU=\bE_2\bP_1\bE_1\bA$. In Section~\ref{section:lu-perm}, Section~\ref{sec:compute-lu-pivoting} or Section~\ref{section:partial-pivot-lu}, we will show that the permutations in-between will result in the same form $\bA=\bP\bL\bU$ where $\bP$ takes account of all the permutations.

The above examples can be easily extended to any $n\times n$ matrix if we assume there are no row permutations in the process. And we will have $n-1$ such lower triangular transformations.
The $k$-th transformation $\bE_k$ introduces zeros below the diagonal in the $k$-th column of $\bA$ by subtracting multiples of the $k$-th row from rows $\{k+1, k+2, \ldots, n\}$.
Finally, by setting $\bL=\bE_1^{-1}\bE_2^{-1}\ldots \bE_{n-1}^{-1}$ we obtain the LU decomposition $\bA=\bL\bU$ (without permutation).

\section{Existence of the LU Decomposition without Permutation}\label{section:exist-lu-without-perm}
The Gaussian elimination or Gaussian transformation shows the origin of the LU decomposition. We then prove Theorem~\ref{theorem:lu-factorization-without-permutation} rigorously, i.e., the existence of the LU decomposition without permutation by induction.
\begin{proof}[\textbf{of Theorem~\ref{theorem:lu-factorization-without-permutation}: LU Decomposition without Permutation}]
We will prove by induction that every $n\times n$ square matrix $\bA$ with nonzero leading principal minors has a decomposition $\bA=\bL\bU$. The $1\times 1$ case is trivial by setting $L=1, U=A$, thus, $A=LU$.

Suppose for any $k\times k$ matrix $\bA_k$ with all the leading principal minors being nonzero has an LU decomposition without permutation. If we prove any $(k+1)\times(k+1)$ matrix $\bA_{k+1}$ can also be factored as this LU decomposition without permutation, then we complete the proof.

For any $(k+1)\times(k+1)$ matrix $\bA_{k+1}$, suppose the $k$-th order leading principal submatrix of $\bA_{k+1}$ is $\bA_k$ with size $k\times k$. Then $\bA_k$ can be factored as $\bA_k =  \bL_k\bU_k$ with $\bL_k$ being a unit lower triangular matrix and $\bU_k$ being a nonsingular upper triangular matrix from the assumption. Write out $\bA_{k+1}$ as
$
\bA_{k+1} = \begin{bmatrix}
	\bA_k & \bb \\
	\bc^\top & d
\end{bmatrix}.
$
Then it admits the factorization:
$$
\bA_{k+1} = \begin{bmatrix}
	\bA_k & \bb \\
	\bc^\top & d
\end{bmatrix}
=
\begin{bmatrix}
	\bL_k &\bzero \\
	\bx^\top  & 1
\end{bmatrix}
\begin{bmatrix}
	\bU_k & \by\\
	\bzero & z
\end{bmatrix} = \bL_{k+1}\bU_{k+1},
$$
where $\bb = \bL_k\by$, $\bc^\top = \bx^\top\bU_k$, $d = \bx^\top\by + z$, $\bL_{k+1}=\begin{bmatrix}
	\bL_k &\bzero \\
	\bx^\top  & 1
\end{bmatrix}$, and $\bU_{k+1}=\begin{bmatrix}
\bU_k & \by\\
\bzero & z
\end{bmatrix}$.
From the assumption, $\bL_k$ and $\bU_k$ are nonsingular. Therefore
$$
\by = \bL_k^{-1}\bb, \qquad \bx^\top=\bc^\top\bU_k^{-1}, \qquad  z=d - \bx^\top\by.
$$
If further, we could prove $z$ is nonzero such that $\bU_{k+1}$ is nonsingular, we complete the proof.

Since all the leading principal minors of $\bA_{k+1}$ are nonzero,
we have $\det(\bA_{k+1})=$
\footnote{By the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bA & \bB \\
		\bC & \bD
	\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$.}
$\det(\bA_k)\cdot$ $\det(d-\bc^\top\bA_k^{-1}\bb)
=\det(\bA_k)\cdot(d-\bc^\top\bA_k^{-1}\bb) \neq 0$, where $d-\bc^\top\bA_k^{-1}\bb$ is a scalar.
As $\det(\bA_k)\neq 0$ from the assumption, we obtain $d-\bc^\top\bA_k^{-1}\bb \neq 0$. Substitute $\bb = \bL_k\by$ and $\bc^\top = \bx^\top\bU_k$ into the formula, we have $d-\bx^\top\bU_k\bA_k^{-1}\bL_k\by =d-\bx^\top\bU_k(\bL_k\bU_k)^{-1}\bL_k\by =d-\bx^\top\by \neq 0$ which is exactly the form of $z\neq 0$. Thus we find $\bL_{k+1}$ with all the values on the diagonal being 1, and $\bU_{k+1}$ with all the values on the diagonal being nonzero which means $\bL_{k+1}$ and $\bU_{k+1}$ are nonsingular, \footnote{A triangular matrix (upper or lower) is nonsingular if and only if all the entries on its main diagonal are nonzero.}
from which the result follows.
\end{proof}

We further prove that if no permutation involves, the LU decomposition is unique.
\begin{corollary}[Uniqueness of the LU Decomposition without Permutation]\label{corollary:unique-lu-without-permutation}
Suppose the $n\times n$ square matrix $\bA$ has nonzero leaning principal minors. Then, the LU decomposition is unique.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:unique-lu-without-permutation}]
Suppose the LU decomposition is not unique, then we can find two decompositions such that $\bA=\bL_1\bU_1 = \bL_2\bU_2$ which implies $\bL_2^{-1}\bL_1=\bU_2\bU_1^{-1}$. The left of the equation is a unit lower triangular matrix and the right of the equation is an upper triangular matrix.
This implies both sides of the above equation are diagonal matrices. Since the inverse of a unit lower triangular matrix is also a unit lower triangular matrix, and the product of unit lower triangular matrices is also a unit lower triangular matrix, this results in that $\bL_2^{-1}\bL_1 = \bI$.
The equality implies that both sides are identity such that $\bL_1=\bL_2$ and $\bU_1=\bU_2$ and leads to a contradiction.
\end{proof}

In the proof of Theorem~\ref{theorem:lu-factorization-without-permutation}, we have shown that the diagonal values of the upper triangular matrix are all nonzero if the leading principal minors of $\bA$ are all nonzero. We then can formulate this decomposition in another form if we divide each row of $\bU$ by each diagonal value of $\bU$. This is called the \textit{LDU decomposition}.
\begin{corollaryHigh}[LDU Decomposition]\label{corollary:ldu-decom}
For any $n\times n$ square matrix $\bA$, if all the leading principal minors are nonzero, i.e., $\det(\bA_{1:k,1:k})\neq 0$, for all $k\in \{1,2,\ldots, n\}$, then $\bA$ can be \textbf{uniquely} factored as
\begin{equation}
	\bA = \bL\bD\bU, \nonumber
\end{equation}
where $\bL$ is a unit lower triangular matrix, $\bU$ is a \textbf{unit} upper triangular matrix, and $\bD$ is a diagonal matrix.
\end{corollaryHigh}
The proof is trivial that from the LU decomposition of $\bA=\bL\bR$, we can find a diagonal matrix $\bD=diag(\bR_{11}, \bR_{22}, \ldots, \bR_{nn})$ such that $\bD^{-1}\bR = \bU$ is a unit upper triangular matrix. And the uniqueness comes from the uniqueness of the LU decomposition.

\section{Existence of the LU Decomposition with Permutation}\label{section:lu-perm}
In Theorem~\ref{theorem:lu-factorization-without-permutation}, we require that matrix $\bA$ has nonzero leading principal minors. However, this is not necessarily. Even when the leading principal minors are zero, nonsingular matrices still have an LU decomposition, but with an additional permutation. The proof is still from induction.
\begin{proof}[\textbf{of Theorem~\ref{theorem:lu-factorization-with-permutation}: LU Decomposition with Permutation}]
We note that any $1\times 1$ nonsingular matrix has a full LU decomposition $A=PLU$ by simply setting $P=1$, $L=1$, $U=A$.
We will show that if every $(n-1)\times (n-1)$ nonsingular matrix has a full LU decomposition, then this is also true for every $n\times n$ nonsingular matrix. By induction, we prove that every nonsingular matrix has a full LU decomposition.

We will formulate the proof in the following order. \textcolor{black}{If $\bA$ is nonsingular, then its row permuted matrix $\bB$ is also nonsingular}. And Schur complement of $\bB_{11}$ in $\bB$ is also nonsingular. Finally, we formulate the decomposition of $\bA$ by $\bB$ from this property.

We notice that at least one element in the first column of $\bA$ must be nonzero otherwise $\bA$ will be singular. We can then apply a row permutation that makes the element in entry $(1,1)$ to be nonzero. That is, there exists a permutation $\bP_1$ such that $\bB = \bP_1 \bA$ in which case $\bB_{11} \neq 0$. Since $\bA$ and $\bP_1$ are both nonsingular and the product of nonsingular matrices is also nonsingular, then $\bB$ is also nonsingular.

\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={Schur complement of $\bB$ is also nonsingular:}]
Now consider the Schur complement of $\bB_{11}$ in $\bB$ with size $(n-1)\times (n-1)$
$$
\bar{\bB} = \bB_{2:n,2:n} -\frac{1}{\bB_{11}} \bB_{2:n,1} \bB_{1,2:n}.
$$
Suppose there is an $(n-1)$-vector $\bx$ satisfies
\begin{equation}\label{equ:lu-pivot1}
	\bar{\bB} \bx = 0.
\end{equation}
Then $\bx$ and $y=-\frac{1}{\bB_{11}}\bB_{1,2:n} \bx  $ satisfy
$$
\bB
\left[
\begin{matrix}
	\bx \\
	y
\end{matrix}
\right]
=
\left[
\begin{matrix}
	\bB_{11} & \bB_{1,2:n} \\
	\bB_{2:n,1} & \bB_{2:n,2:n}
\end{matrix}
\right]
\left[
\begin{matrix}
	\bx \\
	y
\end{matrix}
\right]
=
\left[
\begin{matrix}
	\bzero \\
	0
\end{matrix}
\right].
$$
Since $\bB$ is nonsingular, $\bx$ and $y$ must be zero. Hence, Equation~\eqref{equ:lu-pivot1} holds only if $\bx=\bzero$ which means that the null space of $\bar{\bB}$ is of dimension 0 and thus $\bar{\bB}$ is nonsingular with size $(n-1)\times(n-1)$.
\end{mdframed}

By the induction assumption that any $(n-1)\times(n-1)$ nonsingular matrix can be factorized as the full LU decomposition form
$$
\bar{\bB} = \bP_2\bL_2\bU_2.
$$
We then factor $\bA$ as
\begin{equation*}
	\begin{aligned}
		\bA &= \bP_1^\top
		\left[
		\begin{matrix}
			\bB_{11} & \bB_{1,2:n} \\
			\bB_{2:n,1} & \bB_{2:n,2:n}
		\end{matrix}
		\right]\\
		&= \bP_1^\top
		\left[
		\begin{matrix}
			1 & 0 \\
			0 & \bP_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			\bB_{11} & \bB_{1,2:n} \\
			\bP_2^\top \bB_{2:n,1} &\bP_2^\top \bB_{2:n,2:n}
		\end{matrix}
		\right]\\
		&= \bP_1^\top
		\left[
		\begin{matrix}
			1 & 0 \\
			0 & \bP_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			\bB_{11} & \bB_{1,2:n} \\
			\bP_2^\top \bB_{2:n,1} & \textcolor{blue}{\bL_2\bU_2}+\bP_2^\top \textcolor{blue}{\frac{1}{\bB_{11}} \bB_{2:n,1} \bB_{1,2:n}}
		\end{matrix}
		\right]\\
		&= \bP_1^\top
		\left[
		\begin{matrix}
			1 & 0 \\
			0 & \bP_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			1 & 0 \\
			\frac{1}{\bB_{11}}\bP_2^\top \bB_{2:n,1} & \bL_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			\bB_{11} & \bB_{1,2:n} \\
			\bzero & \bU_2
		\end{matrix}
		\right].\\
	\end{aligned}
\end{equation*}
Therefore, we find the full LU decomposition of $\bA=\bP\bL\bU$ by defining
$$
\bP = \bP_1^\top
\left[
\begin{matrix}
	1 & 0 \\
	0 & \bP_2
\end{matrix}
\right], \qquad
\bL=\left[
\begin{matrix}
	1 & 0 \\
	\frac{1}{\bB_{11}}\bP_2^\top \bB_{2:n,1} & \bL_2
\end{matrix}
\right], \qquad
\bU=
\left[
\begin{matrix}
	\bB_{11} & \bB_{1,2:n} \\
	\bzero & \bU_2
\end{matrix}
\right],
$$
from which the result follows. We will formulate this process into Algorithm~\ref{alg:lu-with-pivoting} to compute this decomposition.
\end{proof}

\section{Computing the LU without Pivoting Recursively: A=LU}\label{section:compute-lu-without-pivot}
As a start, we stay with the most frequent and easy case without involving row exchanges
\begin{equation}
	\bA = \bL\bU, \nonumber
\end{equation}
where $\bL$ is the unit lower triangular matrix and $\bU$ is the nonsingular upper triangular matrix. We refer to this decomposition as the LU decomposition without pivoting (or without permutation).

In Section~\ref{section:gaussian-elimination}, we mentioned the connection of the LU decomposition to the Gaussian elimination. We could find the LU decomposition of a matrix by first applying Gaussian elimination to $\bA$ to get $\bU$, and then examine the multipliers in the Gaussian elimination process to determine the entries below the main diagonal of $\bL$. We will now look at another method for finding the LU decomposition without going through the process of Gaussian elimination.

Again, we define $\bA_{i:j,m:n}$ to be the $(j-i+1)\times(n-m+1)$ submatrix of $\bA$ with rows $i, i+1, \ldots, j$ and columns $m, m+1, \ldots, n$ of $\bA$. And simply, $\bA_{ij}$ to be the ($i,j$)-th entry of $\bA$.

Assume $\bA$ has the form $	\bA = \bL\bU$ of the LU decomposition. We will see, this is equivalent to assuming $\bA_{11}$ is not zero, which implies leading principal minors are nonzero.

From the property of lower triangular matrices and upper triangular matrices, we suppose $\bA$ can be factored as
$$
\bA=\left[
\begin{matrix}
	\bA_{11} & \bA_{1,2:n} \\
	\bA_{2:n,1} & \bA_{2:n,2:n}
\end{matrix}
\right]
=\left[
\begin{matrix}
	1 & 0 \\
	\bL_{2:n,1} & \bL_{2:n,2:n}
\end{matrix}
\right]
\left[
\begin{matrix}
	\bU_{11} & \bU_{1,2:n} \\
	0 & \bU_{2:n,2:n}
\end{matrix}
\right] = \bL\bU.
$$
Writing out the product on the right-hand side of the above equation, we obtain
$$
\bA=\left[
\begin{matrix}
	\bA_{11} & \bA_{1,2:n} \\
	\bA_{2:n,1} & \bA_{2:n,2:n}
\end{matrix}
\right]
=\left[
\begin{matrix}
	\bU_{11} & \bU_{1,2:n} \\
	\bU_{11} \bL_{2:n,1} & \bL_{2:n,1}\bU_{1,2:n} +\bL_{2:n,2:n} \bU_{2:n,2:n}
\end{matrix}
\right] ,
$$
which helps us decide the values of $\bL$ and $\bU$ by
$$
\begin{aligned}
	\bU_{11} &= \bA_{11}   \\
	\bU_{1,2:n} &= \bA_{1,2:n}
\end{aligned}
\bigg\}
\qquad \mathrm{i.e.,} \qquad \bU_{1,1:n} = \bA_{1,1:n},
$$
$$
\bL_{2:n,1} = \frac{1}{\bA_{11}} \bA_{2:n,1},
$$
and
$$
\bL_{2:n,2:n}\bU_{2:n,2:n} = \bA_{2:n,2:n} - \bL_{2:n,1}\bU_{1,2:n} = \bA_{2:n,2:n} -\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n}.
$$
As $\bL_{2:n,2:n}\in \real^{(n-1)\times(n-1)}$ is also a unit lower triangular matrix and $\bU_{2:n,2:n}\in \real^{(n-1)\times(n-1)}$ is a nonsingular upper triangular matrix both of which are of size $(n-1)\times (n-1)$ from the context. Let $\bA_2 = \bA_{2:n,2:n} -\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n} $. So we can calculate $\bL_{2:n,2:n}$ and $\bU_{2:n,2:n}$ by factoring $\bA_2$ as
$$
\bA_2 = \bL_{2:n,2:n}\bU_{2:n,2:n},
$$
which is an LU decomposition of a matrix with size $(n-1)\times(n-1)$. This suggests a recursive algorithm: to factorize a matrix of size $n\times n $, we calculate the first column of $\bL$ (whose first row can be calculated implicitly as 1) and the first row of $\bU$ (whose first column can be calculated implicitly as $\bA_{11}$) leaving the other $n-1$ columns and $n-1$ rows of them to the next round. Continuing recursively, we arrive at a decomposition of a $1\times 1$ matrix.

\paragraph{A word on the leading principal minors} In this process, we only assume the elements in entry $(1,1)$ of $\bA, \bA_2, \bA_3, \ldots, \bA_n$ are nonzero. This is actually the same assumption as the leading principal minors of $\bA$ are all nonzero. The recursive process is formulated in Algorithm~\ref{alg:lu-without-pivoting}.

\begin{algorithm}[H]
\caption{LU Decomposition without Pivoting Recursively}
\label{alg:lu-without-pivoting}
\begin{algorithmic}[1]
\Require
Matrix $\bA$ is nonsingular and square with size $n\times n $;
\State Calculate the first row of $\bU$: $\bU_{1,1:n} = \bA_{1,1:n}$; \Comment{0 flops}
\State Calculate the first column of $\bL$: $\bL_{11}=1$ and $\bL_{2:n,1} = \frac{1}{\bA_{11}} \bA_{2:n,1}$; \Comment{$n-1$ flops}
\State Calculate the LU decomposition
$$\bA_2=\bA_{2:n,2:n} -\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n} = \bL_{2:n,2:n}\bU_{2:n,2:n};$$ \Comment{$2(n-1)^2$ flops}
\end{algorithmic}
\end{algorithm}

\textbf{Schur complement}: Assume $\bA_{11}$ is not zero, then $\bA_2 = \bA_{2:n,2:n} -\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n}$ is called the Schur complement of $\bA_{11}$ in $\bA$. More details about Schur complement can refer to Appendix~\ref{appendix:schur-complement}.

\textbf{Operation count}: The LU decomposition algorithm without pivoting is the first algorithm we have presented in this survey. It is important to assess its cost. To do so, we follow the classical route and count the number of floating-point operations (flops) that the algorithm requires. Each addition, subtraction, multiplication, division, and square root counts as one flop. Note that we have the convention that an assignment operation does not count as one flop.
\begin{theorem}[Algorithm Complexity: LU without Pivoting Recursively]\label{theorem:lu-complexity}
	Algorithm~\ref{alg:lu-without-pivoting} requires $\sim (2/3)n^3$ flops to compute the LU decomposition of an $n\times n$ matrix. Note that the theorem expresses only the leading term of the flop count. And the symbol ``$\sim$" has the usual asymptotic meaning
	\begin{equation*}
	\lim_{n \to +\infty} \frac{\mathrm{number\, of\, flops}}{(2/3)n^3} = 1.
	\end{equation*}
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:lu-complexity}]
The step 1 in Algorithm~\ref{alg:lu-without-pivoting} costs 0 flops and step 2 involves $(n-1)$ divisions.

In step 3, we can compute $\frac{1}{\bA_{11}} \bA_{2:n,1}$ firstly which costs $0$ flops as it has been calculated in step 2, and the outer product with $\bA_{1,2:n}$ costs $(n-1)^2$ multiplications/flops. So the computation of $\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n}$ involves $(n-1)^2$ multiplications.
If we calculate $\bA_{2:n,1}\bA_{1,2:n}$ firstly, then the costs of $\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n}$ is $2(n-1)^2$ totally.
So we choose the first way to do the computation. Furthermore, The subtraction of the two matrices requires $(n-1)^2$ flops. As a result, the cost of step 3 is $2(n-1)^2$ flops in total.

It can be shown that the costs of the final loop is $2(n-1)^2 + (n-1) = 2n^2-3n+1$ flops. Let $f(n)=2n^2-3n+1$, the final cost can then be calculated by
$$
\mathrm{cost}=f(n)+f(n-1)+\ldots+f(1).
$$
Simple calculation\footnote{By the fact that $1^2+2^2+\ldots+n^2 = \frac{2n^3+3n^2+n}{6}$ and $1+2+\ldots+n=\frac{n(n+1)}{2}$.} can show that the complexity is $(2/3)n^3-(1/2)n^2-(1/6)n$ flops, or $(2/3)n^3$ flops if we keep only the leading term.
\end{proof}

\subsection{Complexity of Matrix and Vector Operations}
The calculation of the complexity extensively relies on the complexity of the multiplication of two matrices so that we formulate the finding in the following lemma.
\begin{lemma}[Vector Inner Product Complexity]
Given two vectors $\bv,\bw\in \real^{n}$. The  inner product of the two vectors $\bv^\top\bw$ is given by $\bv^\top\bw=v_1w_1+v_2w_2+\ldots v_nw_n$ which involves $n$ scalar multiplications and $n-1$ scalar additions. Therefore the complexity for the inner product is $2n-1$ flops.
\end{lemma}

The matrix multiplication thus relies on the complexity of the inner product.
\begin{lemma}[Matrix Multiplication Complexity]\label{lemma:matrix-multi-complexity}
For matrix $\bA\in\real^{m\times n}$ and $\bB\in \real^{n\times k}$, the complexity of the multiplication $\bC=\bA\bB$ is $mk(2n-1)$ flops.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:matrix-multi-complexity}]
We notice that each entry of $\bC$ involves a vector inner product which requires $n$ multiplications and $n-1$ additions. And there are $mk$ such entries which leads to the conclusion.
\end{proof}

\section{Computing the LU without Pivoting Element-Wise: A=LU}\label{section:compute-lu-without-pivot-doolittle}
We notice that computing the LU decomposition is equivalent to solving the following equations
$$
\bA_{ij} = \sum_{s=1}^{\min(i,j)} \bL_{is} \bU_{sj}, \qquad \forall i,j\in \{1,2,\ldots, n\}.
$$
Furthermore, when $i\leq j$, the above equation can be decomposed into
$$
\bA_{ij} = \sum_{s=1}^{i-1} \bL_{is} \bU_{sj} + \bU_{ij}, \qquad \text{since $\bL_{ii}=1$}.
$$
Suppose we know the first $k-1$ columns of $\bL$ and the first $k-1$ rows of $\bU$, we have the following observations:
$$
\begin{aligned}
\bA_{kj} &= \sum_{s=1}^{k-1} \bL_{ks} \bU_{sj} + \bU_{kj}, \qquad &\text{for all $j\in \{k,k+1,\ldots, n\}$,} \qquad \text{since $k\leq j$}.\\
\bA_{ik} &= \sum_{s=1}^{k-1} \bL_{is} \bU_{sk}+\bL_{ik} \bU_{kk}, \qquad &\text{for all $i\in \{k+1,k+2,\ldots,n\}$},\qquad \text{since $i\geq k$}.
\end{aligned}
$$
Therefore, the $k$-th row of $\bU$ and $k$-th column of $\bL$ can be obtained by
$$
\begin{aligned}
	\bU_{kj} &= \bA_{kj} - \sum_{s=1}^{k-1} \bL_{is} \bU_{sj} , \qquad &\text{for all $j\in \{k,k+1,\ldots, n\}$,} \qquad \text{since $k\leq j$}.\\
	\bL_{ik}  &=(\bA_{ik} - \sum_{s=1}^{k-1} \bL_{is} \bU_{sk})/\bU_{kk}, \qquad &\text{for all $i\in \{k+1,k+2,\ldots,n\}$},\qquad \text{since $i\geq k$}.
\end{aligned}
$$
This is known as the \textit{Doolittle's method}, the values of $\bU$ and $\bL$ can be computed element-wise, and the process is formulated in Algorithm~\ref{alg:compute-lu-element-level}. We notice that, mathematically, the Doolittle's method is equivalent to the recursive algorithm, but from different perspectives.

\begin{algorithm}[H]
\caption{LU Decomposition without Pivoting Element-Wise}
\label{alg:compute-lu-element-level}
\begin{algorithmic}[1]
\Require
Matrix $\bA$ with size $n\times n$;
\State Calculate first of $\bR$ by $\bR_{11} = \sqrt{\bA_{11}}$;
\For{$k=1$ to $n$}
\State //i.e., the $k$-th column of $\bU$ and $k$-th row of $\bL$;
\For{$j=k$ to $n$}
\State $\bU_{kj} = \bA_{kj} - \sum_{s=1}^{k-1} \bL_{is} \bU_{sj}$;
\EndFor
\For{$i=k+1$ to $n$}
\State$\bL_{ik}  =(\bA_{ik} - \sum_{s=1}^{k-1} \bL_{is} \bU_{sk})/\bU_{kk}$;
\EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

	\begin{theorem}[Algorithm Complexity: LU without Pivoting Elementwise]\label{theorem:lu-complexity-doolittle}
	Algorithm~\ref{alg:compute-lu-element-level} requires $\sim (2/3)n^3$ flops to compute the LU decomposition of an $n\times n$ matrix.
	\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:lu-complexity-doolittle}]
The step 5 in Algorithm~\ref{alg:compute-lu-element-level} requires $(k-1)$ multiplications, $(k-2)$ additions, and 1 subtractions for each loop $k,j$. And there are $n-k+1$ such loop $j$'s, that is, $(2k-2)(n-k+1)=-2k^2+(2n+4)k-2(n+1)$ flops totally from step 5 for each loop $k$.

Similarly, the step 8 requires $(k-1)$ multiplications, $(k-2)$ additions, $1$ subtraction, and 1 division for each loop $k,j$. And there are $n-k$ such loop $j$'s, that is, $(2k-1)(n-k)=-2k^2 + (2n+1)k-n$ flops totally from step 8 for each loop $k$.

Thus, for each loop $k$, the total complexity is $(2k-2)(n-k+1)+(2k-1)(n-k) = -4k^2 + (4n+5)k -(3n+2)$ flops. Let $f(k)=-4k^2 + (4n+5)k -(3n+2)$, the final complexity can be computed by
$$
\mathrm{cost=} f(1)+f(2) +\ldots f(n).
$$
A simple calculation can show that the complexity is $(2/3)n^3$ flops if we keep only the leading term.
\end{proof}

\subsection{Extension to Thin Matrices}
We notice that the complexity of Algorithm~\ref{alg:compute-lu-element-level}  is the same as that of Algorithm~\ref{alg:lu-without-pivoting}. Doolittle's method is mathematically equivalent to the recursive algorithm. However, the Doolittle's method can be extended to compute the LU decomposition for $\bA\in\real^{m\times n}$ with $m\geq n$, i.e., a thin matrix. The LU decomposition is given by $\bA=\bL\bU$ where $\bL\in\real^{m\times n}$ and $\bU\in \real^{n\times n}$ are upper triangular. That is, $\bL$ is \textit{trapezoidal}: $\bL_{ij}=0$ for $i<j$. The procedure is similar and is formulated in Algorithm~\ref{alg:compute-lu-element-level-thin} where the difference is illustrated in blue text.

\begin{algorithm}[H]
	\caption{Thin LU Decomposition without Pivoting Element-Wise}
	\label{alg:compute-lu-element-level-thin}
	\begin{algorithmic}[1]
		\Require
		Matrix $\bA$ with size $\textcolor{blue}{m}\times n$;
		\State Calculate first of $\bR$ by $\bR_{11} = \sqrt{\bA_{11}}$;
		\For{$k=1$ to $n$}
		\State //i.e., the $k$-th column of $\bU$ and $k$-th row of $\bL$;
		\For{$j=k$ to $n$}
		\State $\bU_{kj} = \bA_{kj} - \sum_{s=1}^{k-1} \bL_{is} \bU_{sj}$;
		\EndFor
		\For{$i=k+1$ to $\textcolor{blue}{m}$}
		\State$\bL_{ik}  =(\bA_{ik} - \sum_{s=1}^{k-1} \bL_{is} \bU_{sk})/\bU_{kk}$;
		\EndFor
		\EndFor
	\end{algorithmic}
\end{algorithm}

	\begin{theorem}[Algorithm Complexity: LU Thin Matrix ]\label{theorem:lu-complexity-doolittle-thin}
		Algorithm~\ref{alg:compute-lu-element-level-thin} requires $\sim n^2(m-n/3)$ flops to compute the LU decomposition of an $n\times n$ matrix.
		\item When $m=n$, the complexity of Algorithm~\ref{alg:compute-lu-element-level-thin} is $(2/3)n^3$ flops, which is the same as that of Algorithm~\ref{alg:compute-lu-element-level}.
	\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:lu-complexity-doolittle-thin}]
The complexity of step 5 is the same as that in Algorithm~\ref{alg:compute-lu-element-level}, which is $(2k-2)(n-k+1)=-2k^2+(2n+4)k-2(n+1)$ flops totally from step 5 for each loop $k$.

The complexity of step 8 is slightly different where we replace $n$ by $m$, and it requires $(2k-1)(m-k)=-2k^2 + (2m+1)k-m$ flops totally from step 8 for each loop $k$.

Thus, for each loop $k$, the total complexity is $(2k-2)(n-k+1)+(2k-1)(m-k) = -4k^2 + (2m+2n+5)k -(2n+m+2)$ flops. Let $f(k)=-4k^2 + (2m+2n+5)k -(2n+m+2)$, the final complexity can be calculated by
$$
\mathrm{cost=} f(1)+f(2) +\ldots+ f(n).
$$
A simple calculation can show that the complexity is $n^2(m-n/3)$ flops if we keep only the leading term.
\end{proof}

\section{Computing the LU with Pivoting: A=PLU}\label{sec:compute-lu-pivoting}
Further, we extend Algorithm~\ref{alg:lu-without-pivoting} to the full LU decomposition with $\bA=\bP\bL\bU$. Note that we assume $\bA_{11}$ is nonzero in Algorithm~\ref{alg:lu-without-pivoting}. This is not necessarily true. We will avoid this assumption by permutation matrix. The following algorithm is just formulated from the proof of Theorem~\ref{theorem:lu-factorization-with-permutation}.

\begin{algorithm}[H]
\caption{LU Decomposition with Pivoting}
\label{alg:lu-with-pivoting}
\begin{algorithmic}[1]
	\Require
	matrix $\bA$ is nonsingular and square with size $n\times n$;
	\State Choose permutation matrix $\bP_1$ such that $\bB = \bP_1 \bA$ and $\bB_{11}\neq 0$; \Comment{0 flops}
	\State Calculate the $\bar{\bB}$ for next round: $\bar{\bB}=\bB_{2:n,2:n} -\frac{1}{\bB_{11}} \bB_{2:n,1} \bB_{1,2:n} = \bP_2\bL_2 \bU_2$; \Comment{$2(n-1)^2+(n-1)$ flops}
	\State Calculate the full LU decomposition of $\bA=\bP\bL\bU$ with
	$$
	\bP = \bP_1^\top
	\left[
	\begin{matrix}
		1 & 0 \\
		0 & \bP_2
	\end{matrix}
	\right], \qquad
	\bL=\left[
	\begin{matrix}
		1 & 0 \\
		\frac{1}{\bB_{11}}\bP_2^\top \bB_{2:n,1} & \bL_2
	\end{matrix}
	\right], \qquad
	\bU=
	\left[
	\begin{matrix}
		\bB_{11} & \bB_{1,2:n} \\
		\bzero & \bU_2
	\end{matrix}
	\right]
	$$
	\gap \gap \Comment{$n-1$ flops}
\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: LU with Pivoting]\label{theorem:lu-complexity-with-pivoting}
	Algorithm~\ref{alg:lu-with-pivoting} requires $\sim (2/3)n^3$ flops to compute a full LU decomposition of an $n\times n$ nonsingular matrix.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:lu-complexity-with-pivoting}]
The step 1 costs 0 flops as it only involves assignment operations and step 2 involves $(n-1)^2$ multiplications and $(n-1)^2+(n-1)$ divisions which costs $2(n-1)^2+(n-1)$ flops to compute $\bar{\bB}=\bB_{2:n,2:n} -\frac{1}{\bB_{11}} \bB_{2:n,1} \bB_{1,2:n}$ as shown in the proof of Theorem~\ref{theorem:lu-complexity}.

The computation of step 3 results from $\frac{1}{\bB_{11}}\bP_2^\top \bB_{2:n,1}$ which costs $n-1$ flops as the permutation does not count.

So it costs $(2n^2-2n)$ flops in the final loop. Let $f(n) = 2n^2-2n$, the final complexity can be calculated by
$$
\mathrm{cost=} f(n)+f(n-1)+\ldots+f(1).
$$
Simple calculations can show that the complexity is $(2/3)n^3-(2/3)n$ flops, or $(2/3)n^3$ flops if we keep only the leading term.
\end{proof}

\section{Bandwidth Preserving in the LU Decomposition without Permutation}

For any matrix, the bandwidth of it can be defined as follows.
\begin{definition}[Matrix Bandwidth\index{Matrix bandwidth}]\label{defin:matrix-bandwidth}
For any matrix $\bA\in \real^{n\times n}$ with entry $(i,j)$ element denoted by $\bA_{ij}$. Then $\bA$ has \textbf{upper bandwidth $q$} if $\bA_{ij} =0$ for $j>i+q$, and \textbf{lower bandwidth $p$} if $\bA_{ij}=0$ for $i>j+p$.

An example of a $6\times 6$ matrix with upper bandwidth $2$ and lower bandwidth $3$ is shown as follows:
$$
\begin{bmatrix}
\boxtimes & \boxtimes & \boxtimes & 0& 0 & 0\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes& 0 & 0\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes & 0\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes & \boxtimes\\
0 & 0 & \boxtimes & \boxtimes& \boxtimes & \boxtimes\\
\end{bmatrix}.
$$
\end{definition}

Then, we prove that the bandwidth after the LU decomposition without permutation is preserved.
\begin{lemma}[Bandwidth Preserving]\label{lemma:lu-bandwidth-presev}
For any matrix $\bA\in \real^{n\times n}$ with upper bandwidth $q$ and lower bandwidth $p$. If $\bA$ has an LU decomposition $\bA=\bL\bU$, then $\bU$ has upper bandwidth $q$ and $\bL$ has lower bandwidth $p$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:lu-bandwidth-presev}]
Following from the computation of the LU decomposition without permutation in Section~\ref{section:compute-lu-without-pivot}, and
from the property of lower triangular matrices and upper triangular matrices, we have the decomposition for $\bA$ as follows
$$
\bA=\left[
\begin{matrix}
	\bA_{11} & \bA_{1,2:n} \\
	\bA_{2:n,1} & \bA_{2:n,2:n}
\end{matrix}
\right]
=\left[
\begin{matrix}
	1 & 0 \\
\frac{1}{\bA_{11}} \bA_{2:n,1}  & \bI_{n-1}
\end{matrix}
\right]
\left[
\begin{matrix}
	\bA_{11} & \bA_{1,2:n}\\
	0 & \bS
\end{matrix}
\right] = \bL_1 \bU_1,
$$
where $\bS =\bA_{2:n,2:n} - \frac{1}{\bA_{11}}\bA_{2:n,1}\bA_{1,2:n}$ is the Schur complement of $\bA_{11}$ in $\bA$.
We can name this decomposition of $\bA$ as the $s$-decomposition of $\bA$.
The first column of $\bL_1$ and the first row of $\bU_1$ have the required structure (bandwidth $p$ and $q$ respectively), and the Schur complement $\bS$ of $\bA_{11}$ has upper bandwidth $q-1$ and lower bandwidth $p-1$ respectively. The result follows by induction on the $s$-decomposition of $\bS$.
\end{proof}

\section{Block LU Decomposition}
Another form of the LU decomposition is to factor the matrix into  block triangular matrices.

\begin{theoremHigh}[Block LU Decomposition without Permutation]\label{theorem:block-lu-factorization-without-permutation}
For any $n\times n$ square matrix $\bA$, if the first $m$ leading principal block submatrices are nonsingular, then $\bA$ can be factored as
\begin{equation}
\bA = \bL\bU
=
\begin{bmatrix}
\bI  & & & \\
\bL_{21} & \bI & & \\
\vdots & & \ddots & \\
\bL_{m1} & \ldots & \bL_{m,m-1} & \bI
\end{bmatrix}
\begin{bmatrix}
\bU_{11}  &\bU_{12} & \ldots & \bU_{1m}\\
  & \bU_{22} & &  \vdots \\
  & & \ddots & \bU_{m-1,m}\\
  & & & \bU_{mm}\\
\end{bmatrix}
, \nonumber
\end{equation}
where $\bL_{i,j}$'s and $\bU_{ij}$'s are some block matrices.

Specifically, this decomposition is unique.
\end{theoremHigh}
Note that the $\bU$ in the above theorem is not necessarily upper triangular. An example can be shown as follows:
$$
\bA =
\left[\begin{array}{cc;{2pt/2pt}cc}
0& 1 &   1 & 1\\
-1& 2 &   -1 & 2\\\hdashline[2pt/2pt]
2& 1 &   4 & 2\\
1& 2 &   3 & 3\\
\end{array}\right]
=
\left[\begin{array}{cc;{2pt/2pt}cc}
	1& 0 &   0& 0\\
	0& 1 &   0 & 0\\\hdashline[2pt/2pt]
	5& -2 &   1 & 0\\
	4& -1 &   0 & 1\\
\end{array}\right]
\left[\begin{array}{cc;{2pt/2pt}cc}
	0& 1 &   1 & 1\\
	-1& 2 &   -1 & 2\\\hdashline[2pt/2pt]
	0& 0 &   -3 & 1\\
	0& 0 &   -2 & 1\\
\end{array}\right].
$$
The trivial non-block LU decomposition fails on $\bA$ since the entry $(1,1)$ is zero. However, the block LU decomposition exists.

\section{Application: Linear System via the LU Decomposition}\label{section:lu-linear-sistem}
 Consider the well-determined linear system $\bA\bx = \bb$ with $\bA$ of size $n\times n $ and nonsingular. Avoid solving the system by computing the inverse of $\bA$, we solving linear equation by the LU decomposition. Suppose $\bA$ admits the LU decomposition $\bA = \bP\bL\bU$. The solution is given by the following algorithm.

\begin{algorithm}[H]
	\caption{Solving Linear Equations by LU Decomposition}
	\label{alg:linear-equation-by-LU}
	\begin{algorithmic}[1]
		\Require
		matrix $\bA$ is nonsingular and square with size $n\times n $, solve $\bA\bx=\bb$;
		\State LU Decomposition: factor $\bA$ as $\bA=\bP\bL\bU$; \Comment{(2/3)$n^3$ flops}
		\State Permutation: $\bw = \bP^\top\bb$; \Comment{0 flops }
		\State Forward substitution: solve $\bL\bv = \bw$; \Comment{$1+3+... + (2n-1)=n^2$ flops}
		\State Backward substitution: solve $\bU\bx= \bv$; \Comment{$1+3+... + (2n-1)=n^2$ flops}
	\end{algorithmic}
\end{algorithm}

The complexity of the decomposition step is $(2/3)n^3$ flops, the backward and forward substitution steps both cost $1+3+... + (2n-1)=n^2$ flops. Therefore, the total cost for computing the linear system via the LU factorization is $(2/3)n^3 + 2n^2$ flops. If we keep only the leading term, the Algorithm~\ref{alg:linear-equation-by-LU} costs $(2/3)n^3$ flops where the most cost comes from the LU decomposition.

\paragraph{Linear system via the block LU decomposition} For a block LU decomposition of $\bA=\bL\bU$, we need to solve $\bL\bv = \bw$ and $\bU\bx = \bv$. But the latter system is not triangular and requires some extra computations.

\section{Application: Computing the Inverse of Nonsingular Matrices}\label{section:inverse-by-lu}
By Theorem~\ref{theorem:lu-factorization-with-permutation}, for any nonsingular matrix $\bA\in \real^{n\times n}$, we have a full LU factorization $\bA=\bP\bL\bU$. Then the inverse can be obtained by solving the matrix equation
$$
\bA\bX = \bI,
$$
which contains $n$ linear systems computation: $\bA\bx_i = \be_i$ for all $i \in \{1, 2, \ldots, n\}$ where $\bx_i$ is the $i$-the column of $\bX$ and $\be_i$ is the $i$-th column of $\bI$ (i.e., the $i$-th unit vector).

\begin{theorem}[Inverse of Nonsingular Matrix by Linear System]
Computing the inverse of a nonsingular matrix $\bA \in \real^{n\times n}$ by $n$ linear systems needs $\sim (2/3)n^3 + n(2n^2)=(8/3)n^3$ flops where $(2/3)n^3$ comes from the computation of the LU decomposition of $\bA$.
\end{theorem}
The proof is trivial by using Algorithm~\ref{alg:linear-equation-by-LU}.

However, the complexity can be reduced by taking the advantage of the structures of $\bU, \bL$.
We find that the inverse of the nonsingular matrix is $\bA^{-1} = \bU^{-1}\bL^{-1}\bP^{-1}=\bU^{-1}\bL^{-1}\bP^{T}$.
\begin{theorem}[Inverse of Nonsingular Matrix by LU Factorization]\label{theorem:inverse-by-lu2}
	Computing the inverse of a nonsingular matrix $\bA \in \real^{n\times n}$ by $\bA^{-1} = \bU^{-1}\bL^{-1}\bP^{T}$ needs $\sim (2/3)n^3 + (4/3)n^3=2n^3$ flops where $(2/3)n^3$ comes from the computation of the LU decomposition of $\bA$.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:inverse-by-lu2}]
We notice the computation of $\bU^{-1}\bL^{-1}\bP^{T}$ comes from $\bY=\bU^{-1}\bL^{-1}$. And $\bU^{-1}$ is an upper triangular, $\bL^{-1}$ is a unit lower triangular matrix.
Suppose
$$\bU^{-1} = \bZ=\begin{bmatrix}
	-\bz_1^\top-\\
	-\bz_2^\top-\\
	\vdots \\
	-\bz_n^\top-
\end{bmatrix}\qquad
\text{ and }\qquad
\bU = [\bu_1, \bu_2, \ldots, \bu_n],
$$
are the row partitions and column partitions of $\bU^{-1}$ and $\bU$ respectively.
Since both $\bZ$ and $\bU$ are upper triangular matrices. Thus we have
$$
\bI=
\begin{bmatrix}
	\bz_1^\top\bu_1=1 & \bz_1^\top\bu_2=0 & \bz_1^\top\bu_3=0 & \bz_1^\top\bu_4=0 & \ldots  & \bz_1^\top\bu_n=0 \\
	\bz_2^\top\bu_1=0 & \bz_2^\top\bu_2=1 & \bz_2^\top\bu_3=0            & \bz_2^\top\bu_4=0 & \ldots  & \bz_2^\top\bu_n=0 \\
	\bz_3^\top\bu_1=0 & \bz_3^\top\bu_2=0 & \bz_3^\top\bu_3=1 &\bz_3^\top\bu_4=0  & \ldots & \bz_3^\top\bu_n=0 \\
	\bz_4^\top\bu_1=0 & \bz_2^\top\bu_2=0 & \bz_4^\top\bu_3=0 &\bz_4^\top\bu_4=1  & \ldots & \bz_4^\top\bu_n=0 \\
	\vdots  &\vdots  & \vdots &\vdots  & \ldots & \vdots \\
	\bz_n^\top\bu_1=0 & \bz_n^\top\bu_2=0 & \bz_n^\top\bu_3=0&\bz_n^\top\bu_4=0  & \ldots & \bz_n^\top\bu_n=1
\end{bmatrix}.
$$
By $\bz_1^\top\bu_1=1$, we can compute the first component of $\bz_1$ with 1 flop; By $\bz_1^\top\bu_2=0$, we can compute the second component of $\bz_1$ with 3 flops as the first component is already calculated; $\ldots$.
Then we list the complexity of each inner product in an $n\times n$ matrix, where each entry ($i,j$) denotes the cost to calculate the $(i,j)$ element of $\bU^{-1}$:
$$
\mathrm{cost}=
\begin{bmatrix}
\bz_1^\top\bu_1=1 & \bz_1^\top\bu_2=3           & \bz_1^\top\bu_3=5 & \bz_1^\top\bu_4=7 & \ldots  & \bz_1^\top\bu_n=2n-1 \\
0 & \bz_2^\top\bu_2=1 & \bz_2^\top\bu_3=3            & \bz_2^\top\bu_4=5 & \ldots  & \bz_2^\top\bu_n=2n-3 \\
0 & 0 & \bz_3^\top\bu_3=1 &\bz_3^\top\bu_4=3  & \ldots & \bz_3^\top\bu_n=2n-5 \\
0 & 0 & 0 &\bz_4^\top\bu_4=1  & \ldots & \bz_4^\top\bu_n=2n-7 \\
\vdots  &\vdots  & \vdots &\vdots  & \ldots & \vdots \\
0 & 0 & 0& 0 & \ldots & \bz_n^\top\bu_n=1
\end{bmatrix}=
\begin{bmatrix}
	n^2 \\
	(n-1)^2 \\
	(n-2)^2 \\
	(n-3)^2 \\
	\vdots \\
	1
\end{bmatrix},
$$
which is a sum of $n$ sets of arithmetic sequences and the sum of each sequence is shown in the last equality above. Thus the total cost to compute $\bU^{-1}$ is \fbox{$\frac{2n^3+3n^2+n}{6}$} flops. Similarly, to calculate $\bL^{-1}$ also costs \fbox{$\frac{2n^3+3n^2+n}{6}$} flops.

A moment of reflexion on the multiplication of $\bY=\bU^{-1}\bL^{-1}$ would reveal that:

$\bullet$ The entry $(1,1)$ of $\bY_{11}$ involves the computation of an inner product with $n$-dimension which takes $2n-1$ flops (i.e., $n$ multiplications and $n-1$ additions).

$\bullet$ The entry $(1,2)$ of $\bY_{12}$ involves the computation of an inner product with $(n-1)$-dimension which takes $2n-3$ flops.

$\bullet$ The process can go on, we write the flops in an $n\times n$ matrix with each entry $(i,j)$ meaning the number of flops to calculate the $(i,j)$ element of $\bY$:
$$
\text{costs of $\bY=\bU^{-1}\bL^{-1}$}=
\begin{bmatrix}
	\{ \textcolor{red}{2n-1}             & \textcolor{red}{2n-3} & \textcolor{red}{2n-5} & \textcolor{red}{2n-7} & \textcolor{red}{\ldots} & \textcolor{red}{1}\} \\
    \overbrace{\textcolor{cyan}{2n-3}}	 & \{ \textcolor{green}{2n-3} & \textcolor{green}{2n-5} & \textcolor{green}{2n-7} & \textcolor{green}{\ldots} & \textcolor{green}{1}\} \\
	\textcolor{cyan}{2n-5}        & \overbrace{\textcolor{magenta}{2n-5}} & \{ \textcolor{blue}{2n-5} & \textcolor{blue}{2n-7} & \textcolor{blue}{\ldots} & \textcolor{blue}{1}\} \\
	\textcolor{cyan}{2n-7}        & \textcolor{magenta}{2n-7}  &  \overbrace{\textcolor{olive}{2n-7}} & \{\textcolor{orange}{2n-7} &\textcolor{orange}{\ldots}  & \textcolor{orange}{1}\} \\
	\textcolor{cyan}{\vdots}      & \textcolor{magenta}{\vdots} & \textcolor{olive}{\vdots} & \textcolor{purple}{\vdots} & \ldots & \vdots  \\
	\underbrace{\textcolor{cyan}{1}}  & \underbrace{\textcolor{magenta}{1}}          & \underbrace{\textcolor{olive}{1}}      & \underbrace{\textcolor{purple}{1}}  & \vdots & \textcolor{teal}{1} \\
\end{bmatrix},
$$
which is a symmetric matrix and the total complexity can be calculated by the sum of several arithmetic sequences where each sequence is denoted in a different color. To make it clearer, each arithmetic sequence is also denoted by a brace in the above matrix. Simple calculations will show the complexity is \fbox{$(2/3)n^3 + (1/3)n$}.

As a result, the total cost is then $(2/3)n^3 + (1/3)n^3 + (1/3)n^3 + (2/3)n^3=2n^3$ flops if we keep only the leading term, where the first $(2/3)n^3$ comes from the computation of the LU decomposition of $\bA$.
\end{proof}
\section{Application: Computing the Determinant via the LU Decomposition}
We can find the determinant easily by using the LU decomposition.
If $\bA=\bL\bU$, then $\det(\bA) = \det(\bL\bU) = \det(\bL)\det(\bU) = \bU_{11}\bU_{22}\ldots\bU_{nn}$ where $\bU_{ii}$ is the $i$-th diagonal of $\bU$ for $i\in \{1,2,\ldots,n\}$. \footnote{The determinant of a lower triangular matrix (or an upper triangular matrix) is the product of the diagonal entries.}

Further, for the LU decomposition with permutation $\bA=\bP\bL\bU$, $\det(\bA) = \det(\bP\bL\bU) = \det(\bP)\bU_{11}\bU_{22}\ldots\bU_{nn}$.
The determinant of a permutation matrix is either 1 or –1 because after
changing rows around (which changes the sign of the determinant \footnote{The determinant changes sign when two rows are exchanged (sign reversal).}) a permutation matrix
becomes identity matrix $\bI$, whose determinant is one.

\section{Pivoting}
\subsection{Partial Pivoting}\label{section:partial-pivot-lu}
In practice, it is desirable to pivot even when it is not necessary. When dealing with a linear system via the LU decomposition as shown in Algorithm~\ref{alg:linear-equation-by-LU}, if the diagonal entries of $\bU$ are small, it can lead to inaccurate solutions for the linear solution. Thus, it is common to pick the largest entry to be the pivot. This is known as the \textit{partial pivoting}. For example,
\begin{tcolorbox}[title={Partial Pivoting For a $4\times 4$ Matrix}]
	\begin{equation}\label{equation:elmination-steps2}
		\begin{sbmatrix}{\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes
		\end{sbmatrix}
		\stackrel{\bE_1}{\longrightarrow}
		\begin{sbmatrix}{\bE_1\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \bm{2} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & \bm{5} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & \bm{7} & \bm{\boxtimes} & \bm{\boxtimes}
		\end{sbmatrix}
		\stackrel{\bP_1}{\longrightarrow}
		\begin{sbmatrix}{\bP_1\bE_1\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \textcolor{blue}{7} & \textcolor{blue}{\boxtimes} & \textcolor{blue}{\boxtimes} \\
			0  & 5 & \boxtimes & \boxtimes \\
			0 & \textcolor{blue}{2} & \textcolor{blue}{\boxtimes} & \textcolor{blue}{\boxtimes}
		\end{sbmatrix}
		\stackrel{\bE_2}{\longrightarrow}
		\begin{sbmatrix}{\bE_2\bP_1\bE_1\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 &  7 & \boxtimes & \boxtimes \\
			0 & \bm{0}  & \textcolor{red}{\bm{\boxtimes}} & \bm{\boxtimes} \\
			0 & \bm{0} & \bm{0} & \textcolor{red}{\bm{\boxtimes}}
		\end{sbmatrix},
	\end{equation}
\end{tcolorbox}
\noindent in which case, we pick 7 as the pivot after transformation by $\bE_1$ even when it is not necessary. This interchange permutation can guarantee that no multiplier is greater than 1 in absolute value during the Gaussian elimination.
A specific example is provided as follows.
\begin{example}[Partial Pivoting]
Suppose
$$
\bA=
\begin{bmatrix}
2 & 10 & 5\\
1 & 4  & -2 \\
6 & 8  & 4
\end{bmatrix}
$$
To get the smallest possible multipliers in the first Gaussian transformation, we need to interchange the largest value in the first column to entry $(1,1)$. The permutation matrix $\bP_1$ is doing so
$$
\bP_1 =
\begin{bmatrix}
&& 1\\
&1& \\
1& &
\end{bmatrix}
, \qquad \text{such that} \qquad
\bU=\bP_1\bA =
\begin{bmatrix}
6 & 8  & 4\\
1 & 4  & -2 \\
2 & 10 & 5
\end{bmatrix}.
$$
Then, it follows that $\bE_1$ will introduce zero below the entry $(1,1)$,
$$
\bE_1 =
\begin{bmatrix}
1 & &\\
-1/6 & 1 \\
-1/3 & 0 & 1
\end{bmatrix},
\qquad \text{and} \qquad
\bU=\bE_1\bP_1\bA=
\begin{bmatrix}
	6 & 8  & 4\\
    0  & 8/3  & -8/3 \\
    0  &  22/3  & 11/3
\end{bmatrix}
$$
Now, we pivot 22/3 before 8/3, and the permutation is given by
$$
\bP_2 =
\begin{bmatrix}
1	&& \\
	& &1 \\
	&1 &
\end{bmatrix}
, \qquad \text{such that} \qquad
\bU=\bP_2\bE_1\bP_1\bA =
\begin{bmatrix}
	6 & 8  & 4\\
	0  &  22/3  & 11/3\\
		0  & 8/3  & -8/3 \\
\end{bmatrix}.
$$
Finally, the Gaussian transformation to introduce zero below entry ($2,2$) is given by
$$
\bE_2 =
\begin{bmatrix}
1 & & \\
& 1 & \\
& -4/11 & 1
\end{bmatrix}
\qquad \text{and} \qquad
\bU=\bE_2 \bP_2\bE_1\bP_1\bA=
\begin{bmatrix}
	6 & 8  & 4\\
	0  &  22/3  & 11/3\\
	0  & 0  & -4\\
\end{bmatrix}.
$$
And output the final $\bU$.
\exampbar
\end{example}
As discussed above, the Gaussian transformation $\bE_k$ in $k$-th step of the partial pivoting is given by
$$
\bE_k = \bI - \bz_k \be_k^\top,
$$
where $\be_k \in \real^n$ is the $k$-th unit vector, and $\bz_k\in \real^n$ is given by
$$
\bz_k = [0, \ldots, 0, z_{k+1}, \ldots, z_n]^\top, \qquad z_i= \frac{\bU_{ik}}{\bU_{kk}}, \forall i \in \{k+1,\ldots, n\}.
$$
We realize that $\bE_k$ is a unit lower triangular matrix with only the $k$-th column of the lower submatrix being nonzero,
$$
\begin{blockarray}{ccccccc}
	\begin{block}{c[ccccc]c}
		&	\bI & \bzero & 0 & \bzero & \bzero &    \\
		& \bzero & \ddots & 0 & \bzero	 & \bzero  &   \\
		\bE_k=	& \bzero&\bzero & 1 &\bzero	 &  \bzero  &  \\
		& \bzero & \bzero & \boxtimes & \bI	 & \bzero  &  \\
		& \bzero & \bzero & \boxtimes & \bzero  & \bI &  \\
	\end{block}
	&  & &  k&  &  &  \\
\end{blockarray}.
$$
More generally, the procedure for computing the LU decomposition with partial pivoting of $\bA\in\real^{n\times n}$ is given in Algorithm~\ref{alg:lu-partial-pivot}.

\begin{algorithm}[H]
	\caption{LU Decomposition with Partial Pivoting}
	\label{alg:lu-partial-pivot}
	\begin{algorithmic}[1]
		\Require
		Matrix $\bA$ with size $n\times n$;
		\State Let $\bU = \bA$;
		\For{$k=1$ to $n-1$} \Comment{i.e., get the $k$-th column of $\bU$}
		\State Find a row permutation matrix $\bP_k$ that swaps $\bU_{kk}$ with the largest element in $|\bU_{k:n,k}|$;
		\State $\bU =\bP_k\bU$;
		\State Determine the Gaussian transformation $\bE_k $ to introduce zeros below the diagonal of the $k$-th column of $\bU$;
		\State $\bU = \bE_k\bU$;
		\EndFor
		\State Output $\bU$;
	\end{algorithmic}
\end{algorithm}
The algorithm requires $~2/3(n^3)$ flops and $(n-1)+(n-2)+\ldots + 1 \sim O(n^2)$ comparisons resulting from the pivoting procedure.
Upon completion, the upper triangular matrix $\bU$ is given by
$$
\bU = \bE_{n-1}\bP_{n-1} \ldots \bE_2\bP_2\bE_1\bP_1\bA.
$$

\paragraph{Computing the final $\bL$} And we here show that Algorithm~\ref{alg:lu-partial-pivot} computes the LU decomposition in the following form
$$
\bA = \bP\bL\bU,
$$
where $\bP=\bP_1 \bP_2\ldots\bP_{n-1}$ takes account of all the interchanges, $\bU$ is the upper triangular matrix results directly from the algorithm, $\bL$ is unit lower triangular with $|\bL_{ij}|\leq 1$ for all $1 \leq i,j\leq n$. $\bL_{k+1:n,k}$ is a permuted version of $\bE_k$'s multipliers. To see this, we notice that the permutation matrices used in the algorithm fall into a special kind of permutation matrix since we only interchange two rows of the matrix. \textit{This implies the $\bP_k$'s are symmetric and $\bP_k^2 = \bI$ for $k\in \{1,2,\ldots,n-1\}$}.
Suppose
$$
\bM_k = (\bP_{n-1} \ldots \bP_{k+1}) \bE_k (\bP_{k+1} \ldots \bP_{n-1}).
$$
Then, $\bU$ can be written as
$$
\bU = \bM_{n-1}\ldots \bM_2\bM_1 \bP^\top \bA.
$$
To see what $\bM_k$ is, we realize that $\bP_{k+1}$ is a permutation with the upper left $k\times k$ block being an identity matrix. And thus we have
$$
\begin{aligned}
\bM_k &= (\bP_{n-1} \ldots \bP_{k+1}) (\bI_n-\bz_k\be_k^\top) (\bP_{k+1} \ldots \bP_{n-1})\\
&=\bI_n - (\bP_{n-1} \ldots \bP_{k+1})(\bz_k\be_k^\top)(\bP_{k+1} \ldots \bP_{n-1}) \\
&=\bI_n - (\bP_{n-1} \ldots \bP_{k+1}\bz_k) (\be_k^\top\bP_{k+1} \ldots \bP_{n-1}) \\
&=\bI_n - (\bP_{n-1} \ldots \bP_{k+1}\bz_k)\be_k^\top.  \qquad &\text{(since $\be_k^\top\bP_{k+1} \ldots \bP_{n-1} = \be_k^\top$)}
\end{aligned}
$$
This implies that $\bM_k$ is unit lower triangular with the $k$-th column being the permuted version of $\bE_k$. And the final lower triangular $\bL$ is thus given by
$$
\bL = \bM_1^{-1}\bM_2^{-1} \ldots \bM_{n-1}^{-1}.
$$

\subsection{Complete Pivoting}\label{section:complete-pivoting}
In partial pivoting, when introducing zeros below the diagonal of the $k$-th column of $\bU$, the $k$-th pivot is determined by scanning the current subcolumn $\bU_{k:n,k}$. In complete pivoting, the largest absolute entry in the current submatrix $\bU_{k:n,k:n}$ is interchanged into the entry $(k,k)$ of $\bU$. Therefore, an additional \textit{column permutation} $\bQ_k$ is needed in each step. The final upper triangular matrix $\bU$ is obtained by
$$
\bU = \bE_{n-1}\bP_{n-1}\ldots (\bE_2\bP_2(\bE_1\bP_1\bA\bQ_1)\bQ_2) \ldots \bQ_{n-1}.
$$
Similarly, the complete pivoting algorithm is formulated in Algorithm~\ref{alg:lu-complete-pivot}.
\begin{algorithm}[H]
	\caption{LU Decomposition with Complete Pivoting}
	\label{alg:lu-complete-pivot}
	\begin{algorithmic}[1]
		\Require
		Matrix $\bA$ with size $n\times n$;
		\State Let $\bU = \bA$;
		\For{$k=1$ to $n-1$} \Comment{the value $k$ is to get the $k$-th column of $\bU$}
		\State Find a row permutation matrix $\bP_k$, and a column permutation $\bQ_k$ that swaps $\bU_{kk}$ with the largest element in $|\bU_{k:n,k:n}|$, say $\bU_{u,v} = \max{|\bU_{k:n,k:n}|}$;
		\State $\bU =\bP_k\bU\bQ_k$;
		\State Determine the Gaussian transformation $\bE_k $ to introduce zeros below the diagonal of the $k$-th column of $\bU$;
		\State $\bU = \bE_k\bU$;
		\EndFor
		\State Output $\bU$;
	\end{algorithmic}
\end{algorithm}
The algorithm requires $~2/3(n^3)$ flops and $(n^2+(n-1)^2+\ldots +1^2)\sim O(n^3)$ comparisons resulting from the pivoting procedure. Again, let $\bP=\bP_1 \bP_2\ldots\bP_{n-1}$, $\bQ = \bQ_1 \bQ_2\ldots\bQ_{n-1}$,
$$
\bM_k = (\bP_{n-1} \ldots \bP_{k+1}) \bE_k (\bP_{k+1} \ldots \bP_{n-1}), \qquad \text{for all $k\in \{1,2,\ldots,n-1\}$}
$$
and
$$
\bL = \bM_1^{-1}\bM_2^{-1} \ldots \bM_{n-1}^{-1}.
$$
We have $\bA = \bP\bL\bU \bQ^\top$ or $\bP^\top\bA\bQ = \bL\bU$ as the final decomposition.

\subsection{Rook Pivoting}
The \textit{rook pivoting} provides an alternative to the partial and complete pivoting. Instead of choosing the larges value in $|\bU_{k:n,k:n}|$ in the $k$-th step, it searches for an element of $\bU_{k:n,k:n}$ that is maximal in both its row and column. Apparently, the rook pivoting is not unique such that we could find many entries that satisfy the criteria. For example, for a submatrix $\bU_{k:n,k:n}$ as follows
$$
\bU_{k:n,k:n} =
\begin{bmatrix}
1 & 2 & 3 & 4 \\
2 & 3 & 7 & 3 \\
5 & 2 & 1 & 2 \\
2 & 1 & 2 & 1 \\
\end{bmatrix},
$$
where the $7$ will be chosen by complete pivoting. And one of $5, 4, 7$ will be identified as a rook pivot.

\section{Rank-Revealing LU Decomposition}\label{section:rank-reveal-lu-short}
In many applications, a factorization produced by Gaussian elimination with pivoting when $\bA$ has rank $r$ will reveal rank in the following form\index{Rank-revealing}\index{Rank-revealing LU}
$$
\bP\bA\bQ =
\begin{bmatrix}
\bL_{11} & \bzero \\
\bL_{21}^\top & \bI
\end{bmatrix}
\begin{bmatrix}
\bU_{11} & \bU_{12} \\
\bzero & \bzero
\end{bmatrix},
$$
where $\bL_{11}\in \real^{r\times r}$ and $\bU_{11}\in \real^{r\times r}$ are nonsingular, $\bL_{21}, \bU_{21}\in \real^{r\times (n-r)}$, and $\bP,\bQ$ are permutations. Gaussian elimination with rook pivoting or complete pivoting can result in such decomposition \citep{hwang1992rank, higham2002accuracy}.

\section{Rate of Change of L and U*}
If $\bA$ has a unique LU decomposition, and a small perturbation $\Delta\bA$ such that the LU decomposition of $\bA+\Delta\bA$ also exists and is given by $\bA+\Delta\bA = (\bL+\Delta\bL)(\bU+\Delta\bU)$. Then $\Delta\bA$ can be obtained by
$$
\Delta\bA = \Delta\bL \cdot \bU + \bL \cdot  \Delta\bU.
$$
And we have
$$
\bL^{-1}\cdot \Delta\bA \cdot \bU^{-1} = \bL^{-1}\cdot  \Delta\bL + \Delta\bU \cdot \bU^{-1}.
$$
Since both $\bL$ and $(\bL+\Delta\bL)$ are unit lower triangular matrices, $\Delta \bL$ is strictly lower triangular, i.e., lower triangular with 0's on the diagonal. And both $\bU$ and $\bU+\Delta\bU$ are upper triangular matrices, $\Delta\bU$ is therefore upper triangular. Thus
$$
\Delta\bL = \bL \cdot slt(\bL^{-1}\cdot \Delta\bA \cdot \bU^{-1}),
\qquad
\Delta\bU = ut(\bL^{-1}\cdot \Delta\bA \cdot \bU^{-1})  \bU,
$$
where $slt(\bB)$ is the strictly lower triangular part of $\bB$, and $ut(\bB)$ is the upper triangular part of $\bB$. Clearly, the sensitivity, aka, the rate of change of $\bL$ and $\bU$ depends on the inverse of $\bL$ and $\bU$. More generally, we have the following result.

\begin{theorem}[Rate of Change of $\bL$ and $\bU$]\label{theorem:rate-lu-decomposition}
Suppose matrix $\bA \in \real^{n\times n}$ has nonzero leading principal minors, i.e., $\det(\bA_{1:k,1:k})\neq 0$, for all $k\in \{1,2,\ldots, n\}$, with LU decomposition $\bA = \bL\bU$. Let $\bG$ be a real $n\times n$ matrix and let $\Delta\bA = \epsilon \bG$, for some $\epsilon \geq 0$. If $\epsilon$ is sufficiently small such that all the leading principal minors of $\bA+t\bG$ are nonzero for all $|t|\leq \epsilon$. Then $\bA+\Delta\bA$ has the LU decomposition
$$
\bA+\Delta\bA = (\bL+\Delta\bL)(\bU+\Delta\bU),
$$
with $\Delta\bL$ and $\Delta\bU$ satisfying
$$
\begin{aligned}
\Delta \bL &= \epsilon \dotL(0) + O(\epsilon^2), \\
\Delta\bU &= \epsilon \dotU(0) + O(\epsilon^2),
\end{aligned}
$$
where $\dotL(0) = \dotL(t)|_{t=0}$ and $\dotU(0) = \dotU(t)|_{t=0}$, and $\dotL(t)$ and $\dotU(t)$ are defined by the unique LU decomposition
\begin{equation}\label{equation:partial-eq}
\bA + t\bG = \bL(t)\bU(t), \qquad |t| \leq \epsilon.
\end{equation}
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:rate-lu-decomposition}]
We notice that
\begin{equation}\label{equation:rage-lu-eq1}
\begin{aligned}
	\bL(t) &= \bL +\Delta\bL, \\
	\dotL(t)&=\frac{\partial \bL(t)}{\partial t}
\end{aligned}
\end{equation}
By Taylor expansion, we have
$$
\dotL(\epsilon) =\frac{\partial \bL(\epsilon)}{\partial \epsilon} \approx \dotL(0) + \ddotL(\epsilon) \epsilon + O(\epsilon^2).
$$
Since $\epsilon$ is small enough, such that Equation~\eqref{equation:rage-lu-eq1} can be written as
$$
\dotL(\epsilon)=\frac{\partial \bL(\epsilon)}{\partial \epsilon} = \frac{\Delta\bL}{\epsilon},
$$
which results in
$$
\Delta\bL = \epsilon \dotL(0) +  O(\epsilon^2).
$$
Similarly, we can also obtain
$$
\Delta\bU = \epsilon \dotU(0) + O(\epsilon^2).
$$
This completes the proof.
\end{proof}
Following from the Theorem~\ref{theorem:rate-lu-decomposition} above, we can also claim that

\begin{align}
\bL\dotU(0) &+ \dotL(0)\bU = \bG, \label{equation:partial-lu-rate-1}\\
\dotL(0) &= \bL \cdot slt(\bL^{-1}\bG\bU^{-1}), \label{equation:partial-lu-rate-2}\\
\dotU(0) &= ut(\bL^{-1}\bG\bU^{-1}) \bU. \label{equation:partial-lu-rate-3}
\end{align}
Note that $\bL(0) = \bL$, $\bL(\epsilon) =\bL+\Delta\bL$, $\bU(0) = \bU$, $\bU(\epsilon)=\bU +\Delta\bU$. If we differentiate Equation~\eqref{equation:partial-eq}, and set $t=0$, we can obtain result in Equation~\eqref{equation:partial-lu-rate-1}. Further, multiply Equation~\eqref{equation:partial-lu-rate-1} left by $\bL^{-1}$ and right by $\bU^{-1}$, we have
$$
\dotU(0)\bU^{-1} + \bL^{-1}\dotL(0) =\bL^{-1}\bG\bU^{-1},
$$
where $\dotL(0)$ is strictly lower triangular and $\dotU(0)$ is upper triangular. Thus the results in Equation~\eqref{equation:partial-lu-rate-2} and \eqref{equation:partial-lu-rate-3} can be proved.

In the above theorem, we proved that the sensitivity of $\Delta\bL$ and $\Delta\bU$ is bounded by $\dotL(0)$ and $\dotU(0)$ to some $\epsilon$ value. More results of the perturbation on the LU decomposition can be found in \citep{baarland1991perturbation,sun1992rounding, sun1992componentwise, stewart1993perturbation, stewart1997perturbation, chang1997pertubation, chang1998sensitivity, higham2002accuracy, bueno2004stability}, similar results for the QR decomposition \citep{stewart1993perturbation, chang1997perturbationqr, chang1997pertubation}, and the Cholesky decomposition \citep{sun1992rounding, stewart1997perturbation, chang1997pertubation}.

\chapter{Nonnegative Matrix Factorization (NMF)}\index{NMF}\label{section:nmf}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Nonnegative Matrix Factorization}
Following from the matrix factorization via the ALS, we now consider algorithms for solving the nonnegative matrix factorization (NMF) problem:
\begin{itemize}
\item Given a nonnegative matrix $\bA\in \real^{M\times N}$, find nonnegative matrix factors $\bW\in \real^{M\times K}$ and $\bZ\in \real^{K\times N}$ such that:
$$
\bA\approx\bW\bZ.
$$
\end{itemize}
To measure the approximation, the loss to evaluate is still from the Frobenius norm of the difference between the two matrices:
$$
L(\bW,\bZ) = ||\bW\bZ-\bA||^2.
$$

\section{NMF via Multiplicative Update}
Following from Section~\ref{section:als-netflix}, given $\bW\in \real^{M\times K}$, we want to update $\bZ\in \real^{K\times N}$, the gradient with respect to $\bZ$ is given by Equation~\eqref{equation:givenw-update-z-allgd}:
$$
\begin{aligned}
	\frac{\partial L(\bZ|\bW)}{\partial \bZ} =2  \bW^\top(\bW\bZ-\bA) \in \real^{K\times N}.
\end{aligned}
$$
Applying the gradient descent idea in Section~\ref{section:als-gradie-descent}, the trivial update on $\bZ$ can be done by
$$
(\text{GD on $\bZ$})\gap \bZ \leftarrow \bZ - \eta \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)=\bZ - \eta \left(2  \bW^\top\bW\bZ-2\bW^\top\bA\right),
$$
where $\eta$ is a small positive step size. Now if we suppose a different step size for each entry of $\bZ$ and incorporate the constant 2 into the step size, the update can be obtained by
$$
(\text{GD$^\prime$ on $\bZ$})\gap
\begin{aligned}
	\bZ_{kn} &\leftarrow \bZ_{kn} - \frac{\eta_{kn}}{2} \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)_{kn}\\
	&=\bZ_{kn} - \eta_{kn}(\bW^\top\bW\bZ-\bW^\top\bA)_{kn}, \gap k\in [1,K], n\in[1,N],
\end{aligned}
$$
where $\bZ_{kn}$ is the $(k,n)$-th entry of $\bZ$. Now if we rescale the step size:
$$
\eta_{kn} = \frac{\bZ_{kn}}{(\bW^\top\bW\bZ)_{kn}},
$$
then we obtain the update rule:
$$
(\text{Multiplicative update on $\bZ$})\gap
\bZ_{kn} \leftarrow \bZ_{kn}\frac{(\bW^\top\bA)_{kn}}{(\bW^\top\bW\bZ)_{kn}}, \gap k\in [1,K], n\in[1,N],
$$
which is known as the \textit{multiplicative update} and is first developed in \citep{lee2001algorithms} and further discussed in \citep{pauca2006nonnegative}. Analogously, the multiplicative update on $\bW$ can be obtained by
\begin{equation}\label{equation:multi-update-w}
(\text{Multiplicative update on $\bW$})\gap
\bW_{mk} \leftarrow \bW_{mk} \frac{(\bA\bZ^\top)_{mk}}{(\bW\bZ\bZ^\top)_{mk}}, \gap m\in[1,M], k\in[1,K].
\end{equation}

\begin{theorem}[Convergence of Multiplicative Update]
The loss $L(\bW,\bZ)=||\bW\bZ-\bA||^2$ is non-increasing under the multiplicative update rules:
$$
\left\{
\begin{aligned}
	\bZ_{kn} &\leftarrow \bZ_{kn}\frac{(\bW^\top\bA)_{kn}}{(\bW^\top\bW\bZ)_{kn}}, \gap k\in [1,K], n\in[1,N];\\
	\bW_{mk} &\leftarrow \bW_{mk} \frac{(\bA\bZ^\top)_{mk}}{(\bW\bZ\bZ^\top)_{mk}}, \gap m\in[1,M], k\in[1,K].
\end{aligned}
\right.
$$
\end{theorem}
We refer the proof of the above theorem to \citep{lee2001algorithms}.  Clearly the approximations $\bW$ and $\bZ$ remain nonnegative during the updates.
It
is generally best to update $\bW$ and $\bZ$ ``simultaneously”, instead of updating each
matrix fully before the other. In this case, after updating a row of $\bZ$, we update the corresponding column of $\bW$.
In
the implementation, a small positive quantity, say the square root of the machine
precision, should be added to the denominators in the approximations of $\bW$ and $\bZ$
at each iteration step. And a trivial $\epsilon=10^{-9}$ can do the job. The full procedure is shown in Algorithm~\ref{alg:nmf-multiplicative}.

\begin{algorithm}[h]
\caption{NMF via Multiplicative Updates}
\label{alg:nmf-multiplicative}
\begin{algorithmic}[1]
\Require $\bA\in \real^{M\times N}$;
\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ randomly with nonnegative entries.
\State choose a stop criterion on the approximation error $\delta$;
\State choose maximal number of iterations $C$;
\State $iter=0$;
\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
\State $iter=iter+1$;
\For{$k=1$ to $K$}
\For{$n=1$ to $N$} \Comment{udate $k$-th row of $\bZ$}
\State $\bZ_{kn} \leftarrow \bZ_{kn}\frac{(\bW^\top\bA)_{kn}}{(\bW^\top\bW\bZ)_{kn}+\textcolor{blue}{\epsilon}}$;
\EndFor
\For{$m=1$ to $M$} \Comment{udate $k$-th column of $\bW$}
\State $\bW_{mk} \leftarrow \bW_{mk} \frac{(\bA\bZ^\top)_{mk}}{(\bW\bZ\bZ^\top)_{mk}+\textcolor{blue}{\epsilon}}$;
\EndFor
\EndFor
\EndWhile
\State Output $\bW,\bZ$;
\end{algorithmic}
\end{algorithm}
\section{Regularization}
\begin{algorithm}[h]
	\caption{NMF via Regularized Multiplicative Updates}
	\label{alg:nmf-multiplicative-regularization}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$;
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ randomly with nonnegative entries.
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose maximal number of iterations $C$;
		\State choose regularization parameter $\lambda_z, \lambda_w$;
		\State $iter=0$;
		\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\For{$k=1$ to $K$}
		\For{$n=1$ to $N$} \Comment{udate $k$-th row of $\bZ$}
		\State $\bZ_{kn} \leftarrow \bZ_{kn}\frac{(\bW^\top\bA)_{kn}-
			\textcolor{blue}{\lambda_z\bZ_{kn}}}{(\bW^\top\bW\bZ)_{kn}+\textcolor{blue}{\epsilon}}$;
		\EndFor
		\For{$m=1$ to $M$} \Comment{udate $k$-th column of $\bW$}
		\State $\bW_{mk} \leftarrow \bW_{mk} \frac{(\bA\bZ^\top)_{mk}	-\textcolor{blue}{\lambda_w\bW_{mk}}}{(\bW\bZ\bZ^\top)_{mk}+\textcolor{blue}{\epsilon}}$;
		\EndFor
		\EndFor
		\EndWhile
		\State Output $\bW,\bZ$;
	\end{algorithmic}
\end{algorithm}

Similar to the ALS with regularization in Section~\ref{section:regularization-extention-general}, recall the regularization helps employ the ALS into general matrices. We can also add a regularization in the context of NMF:
$$
	L(\bW,\bZ)  =||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2, \qquad \lambda_w>0, \lambda_z>0,
$$
where the induced matrix norm is still the Frobenius norm. The gradient with respect to $\bZ$ given $\bW$ is the same as that in Equation~\eqref{equation:als-regulari-gradien}:
$$
\begin{aligned}
	\frac{\partial L(\bZ|\bW)}{\partial \bZ} =2\bW^\top(\bW\bZ-\bA) + \textcolor{blue}{2\lambda_z\bZ}  \in \real^{K\times N}.
\end{aligned}
$$
The trivial gradient descent update can be obtained by
$$
(\text{GD on }\bZ) \gap \bZ \leftarrow \bZ - \eta \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)=\bZ - \eta \left(2  \bW^\top\bW\bZ-2\bW^\top\bA+\textcolor{blue}{2\lambda_z\bZ}\right),
$$
Analogously, if we suppose a different step size for each entry of $\bZ$ and incorporate the constant 2 into the step size, the update can be obtained by
$$
(\text{GD$^\prime$ on $\bZ$})\gap
\begin{aligned}
	\bZ_{kn} &\leftarrow \bZ_{kn} - \frac{\eta_{kn}}{2} \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)_{kn}\\
	&=\bZ_{kn} - \eta_{kn}(\bW^\top\bW\bZ-\bW^\top\bA+\textcolor{blue}{\lambda_z\bZ})_{kn}, \gap k\in [1,K], n\in[1,N],
\end{aligned}
$$
Now if we rescale the step size:
$$
\eta_{kn} = \frac{\bZ_{kn}}{(\bW^\top\bW\bZ)_{kn}},
$$
then we obtain the update rule:
$$
(\text{Multiplicative update on $\bZ$})\gap
\bZ_{kn} \leftarrow \bZ_{kn}\frac{(\bW^\top\bA)_{kn}-
\textcolor{blue}{\lambda_z\bZ_{kn}}
}{(\bW^\top\bW\bZ)_{kn}}, \gap k\in [1,K], n\in[1,N].
$$
Similarly, the multiplicative update on $\bW$ can be obtained by
$$
(\text{Multiplicative update on $\bW$})\gap
\bW_{mk} \leftarrow \bW_{mk} \frac{(\bA\bZ^\top)_{mk}
	-\textcolor{blue}{\lambda_w\bW_{mk}}
}{(\bW\bZ\bZ^\top)_{mk}}, \gap m\in[1,M], k\in[1,K].
$$
The procedure is then formulated in Algorithm~\ref{alg:nmf-multiplicative-regularization}.
\section{Initialization}
In the above discussion, we initialize $\bW$ and $\bZ$ randomly. Whereas, there are also alternative strategies designed to obtain better initial estimates in the hope of converging more rapidly to a good solution \citep{boutsidis2008svd, gillis2014and}. We sketch the methods as follows for a reference:
\begin{itemize}
\item \textit{Clustering techniques.} Use some clustering methods on the columns of $\bA$, and make the cluster means of the top $K$ clusters as the columns of $\bW$, and initialize $\bZ$ as a proper scaling of
the cluster indicator matrix (that is, $\bZ_{kn}\neq 0$ indicates $\ba_n$ belongs to the $k$-th cluster);
\item \textit{Subset selection.} Pick $K$ columns of $\bA$ and set those as the initial columns for $\bW$, and analogously, $K$ rows of $\bA$ are selected to form the rows of $\bZ$;
\item \textit{SVD-based.} Suppose the SVD of $\bA=\sum_{i=1}^{r}\sigma_i\bu_i\bv_i^\top$ where each factor $\sigma_i\bu_i\bv_i^\top$ is a rank-one matrix with possible negative values in $\bu_i, \bv_i$, and nonnegative $\sigma_i$. Denote $[x]_+=\max(x, 0)$, we notice
$$
\bu_i\bv_i^\top = [\bu_i]_+[\bv_i]_+^\top+[-\bu_i]_+[-\bv_i]_+^\top-[-\bu_i]_+[\bv_i]_+^\top-[\bu_i]_+[-\bv_i]_+^\top.
$$
Either $[\bu_i]_+[\bv_i]_+^\top$ or $[-\bu_i]_+[-\bv_i]_+^\top$ can be selected as a column and a row in $\bW,\bZ$.
\end{itemize}

\section{Movie Recommender Context}
Both the NMF and the ALS approximate the matrix and reconstruct the entries in the matrix with a set of basis vectors. The basis in the NMF is composed of vectors with nonnegative elements while the basis in the ALS can have positive or negative values.
The difference then is that the NMF reconstructs each vector as a positive summation of the basis vectors with a ``relative" small component in the direction of each basis vector.
Whereas, In the ALS the data is modeled as a linear combination of the basis you can add or subtract vectors as needed and the components in the direction of each basis vector can be large positive values or negative values. Therefore, depending on the application one or the other factorization can be utilized to describe the data with different meanings.

In the context of a movie recommender system then the rows of $\bW$ represent the feature of the movies and columns of $\bZ$ represent the features of a user.
In the NMF you can say that a movie is 0.5 comedy, 0.002 action, and 0.09 romantic. However, In the ALS you can get combinations such as 4 comedy, -0.05 comedy, and -3 drama, i.e., a positive or negative component on that feature.

The ALS and NMF are similar in the sense that the importance of each basis vector is not in a hierarchical manner. Whereas, The key difference between the ALS and the SVD is in that, in the SVD, the importance of each vector in the basis is relative to the value of the singular value associated with that vector. For the SVD of $\bA=\sum_{i=1}^{r}\sigma_i\bu_i\bv_i^\top$,
this usually means that the reconstruction $\sigma_1\bu_1\bv_i^\top$ via the first set of basis vectors dominates and is the most used set to reconstruct data, then the second set and so on, so the basis in the SVD has an implicit hierarchy and that doesn't happen in the ALS or the NMF.
Recall the low-rank approximation on the flag image in Section~\ref{section:als-low-flag} where we find the second component $\bw_2\bz_2^\top$ via the ALS in Figure~\ref{fig:als52} plays an important role in the reconstruction of the original figure, whereas the second component $\sigma_2\bu_2\bv_2^\top$ via the SVD in Figure~\ref{fig:svd22} plays a small role in the reconstruction.

\chapter{Alternating Least Squares}\label{section:als}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Preliminary: Least Squares Approximations}
The linear model is the main technique in regression problems and the primary
tool for it is the least squares approximation which minimizes a sum of squared errors. This is a natural choice when we’re interested in finding the regression function which minimizes the
corresponding expected squared error. Over the recent decades, linear models have been used in a wide range of applications, e.g., decision making \citep{dawes1974linear}, time series \citep{christensen1991linear, lu2017machine}, and in many fields of study, production science, social science and soil science \citep{fox1997applied, lane2002generalized, schaeffer2004application, mrode2014linear}.

Let's consider the overdetermined system $\bb = \bA\bx $, where $\bA\in \real^{m\times n}$ is the input data matrix, $\bb\in \real^m$ is the observation matrix (target matrix), and the sample number $m$ is larger than the dimension number $n$. $\bx$ is a vector of weights of the linear model. Normally $\bA$ will have full column rank since the data from real world has large chance to be unrelated. In practice, a bias term is added to the first column of $\bA$ such that the least square is to find the solution of
\begin{equation}\label{equation:ls-bias}
	\widetildebA \widetildebx =
[\bm{1} ,\bA ]
\begin{bmatrix}
	x_0\\
	\bx
\end{bmatrix}
 = \bb .
\end{equation}

It often happens that $\bb = \bA\bx$ has no solution. The usual reason is: too many equations, i.e., the matrix has more rows than columns.
Define the column space of $\bA$ by $\{\bA\bgamma: \,\, \forall \bgamma \in \real^n\}$  and denoted by $\cspace(\bA)$.
Thus the meaning of $\bb = \bA\bx$ has no solution is that $\bb$ is outside the column space of $\bA$. In another word, the error $\be = \bb -\bA\bx$ cannot get down to zero. When the error $\be$ is as small as possible in the sense of mean squared error, $\bx_{LS}$ is a least squares solution, i.e., $||\bb-\bA\bx_{LS}||^2$ is minimum.

\paragraph{Least squares by calculus}
When $||\bb-\bA\bx||^2$ is differentiable, and parameter space of $\bx$ is an open set (the least achievable value is obtained inside the parameter space), the least squares estimator must be the root of $||\bb-\bA\bx||^2$. We thus come into the following lemma.

	\begin{lemma}[Least Squares by Calculus]\label{lemma:ols}
		Assume $\bA \in \real^{m\times n}$ is fixed and has full rank (i.e., the columns of $\bA$ are linearly independent) with $m>n$. Consider the overdetermined system $\bb = \bA\bx$, the least squares solution by calculus via setting the derivative in every direction of $||\bb-\bA\bx||^2$ to be zero is $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$. The value $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ is known as the ordinary least squares (OLS) estimator or simply least squares (LS) estimator of $\bx$.
	\end{lemma}

To prove the lemma above, we must show $\bA^\top\bA$ is invertible. Since we assume $\bA$ has full rank and $m>n$. $\bA^\top\bA \in \real^{n\times n}$ is invertible if it has rank $n$ which is the same as the rank of $\bA$. This has been proved in Lemma~\ref{lemma:rank-of-ata} (p.~\pageref{lemma:rank-of-ata}).
Apply the observation to $\bA^\top$, we can also prove that $\bA\bA^\top$ and $\bA$ have same rank. This result brings about the ordinary least squares estimator as follows.
\begin{proof}[of Lemma \ref{lemma:ols}]
	Recall from calculus that a minimum of a function $f(\bx)$ occurs at a value $\bx_{LS}$ such that the derivative $\frac{\partial}{\partial \bx} f(\bx)=\bzero$. The differential of $||\bb-\bA\bx||^2$ is $2\bA^\top\bA\bx -2\bA^\top\bb$. $\bA^\top\bA$ is invertible since we assume $\boldsymbol{A}$ is fixed and has full rank with $m>n$ (Lemma~\ref{lemma:rank-of-ata}, p.~\pageref{lemma:rank-of-ata}). So the OLS solution of $\bx$ is $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ which completes the proof.
\end{proof}

\begin{definition}[Normal Equation]\label{definition:normal-equation-als}
	We can write the zero derivative of $||\bb-\bA\bx||^2$ as $\bA^\top\bA \bx_{LS} = \bA^\top\bb$. The equation is also known as the \textit{normal equation}. In the assumption, $\bA$ has full rank with $m>n$. So $\bA^\top\bA$ is invertible which implies $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$.
\end{definition}

\begin{figure}[h!]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[A convex funtion]{\label{fig:convex-1}
		\includegraphics[width=0.31\linewidth]{./imgs/convex.pdf}}
	\subfigure[A concave function]{\label{fig:convex-2}
		\includegraphics[width=0.31\linewidth]{./imgs/concave.pdf}}
	\subfigure[A random function]{\label{fig:convex-3}
		\includegraphics[width=0.31\linewidth]{./imgs/convex-none.pdf}}
	\caption{Three functions.}
	\label{fig:convex-concave-none}
\end{figure}
However, we do not actually know the least squares estimator obtained in Lemma~\ref{lemma:ols} is the least or largest achievable estimator (or neither). An example is shown in Figure~\ref{fig:convex-concave-none}. All we can get is that there only exists one root for the function $||\bb-\bA\bx||^2$. The following remark can address this concern.
\begin{remark}[Verification of Least Squares Solution]
	Why does the zero derivative imply least mean squared error? The usual reason is from the convex analysis as we shall see shortly. But here we verify directly that the OLS solution finds the least squares. For any $\bbeta \neq \hat{\bbeta}$, we have
	\begin{equation}
		\begin{aligned}
			||\bb - \bA\bx ||^2 &= ||\bb - \bA\bx_{LS} + \bA\bx_{LS} - \bA\bx||^2 \\
			&= ||\bb-\bA\bx_{LS} + \bA (\bx_{LS} - \bx)||^2 \\
			&=||\bb-\bA\bx_{LS}||^2 + ||\bA(\bx_{LS} - \bx)||^2 + 2(\bA(\bx_{LS} - \bx))^\top(\bb-\bA\bx_{LS}) \\
			&=||\bb-\bA\bx_{LS}||^2 + ||\bA(\bx_{LS} - \bx)||^2 + 2(\bx_{LS} - \bx)^\top(\bA^\top\bb - \bA^\top\bA\bx_{LS}), \nonumber
		\end{aligned}
	\end{equation}
	where the third term is zero from the normal equation and $||\bA(\bx_{LS} - \bx)||^2 \geq 0$. Therefore,
	\begin{equation}
		||\bb - \bA\bx ||^2 \geq ||\bb-\bA\bx_{LS}||^2. \nonumber
	\end{equation}
	Thus we show that the OLS estimator indeed gives the minimum, not the maximum or a
	saddle point via the calculus approach.
\end{remark}

Further question would be posed: why does this normal equation magically produce solution for $\bx$? A simple example would give the answer. $x^2=-1$ has no real solution. But $x\cdot x^2 = x\cdot (-1)$ has a real solution $\hat{x} = 0$ in which case $\hat{x}$ makes $x^2$ and $-1$ as close as possible.

\begin{example}[Multiply From Left Can Change The Solution Set ]
	Consider the matrix
	$$
	\bA=\left[
	\begin{matrix}
		-3 & -4 \\
		4 & 6  \\
		1 & 1
	\end{matrix}
	\right] \mathrm{\,\,\,and\,\,\,}
	\bb=\left[
	\begin{matrix}
		1  \\
		-1   \\
		0
	\end{matrix}
	\right].
	$$
	It can be easily verified that $\bA\bx = \bb$ has no solution for $\bx$. However, if we multiply on the left by
	$$
	\bB=\left[
	\begin{matrix}
		0 & -1 & 6\\
		0 & 1  & -4
	\end{matrix}
	\right].
	$$
	Then we have $\bx_{LS} = [1/2, -1/2]^\top$ as the solution of $\bB\bA\bx= \bB\bb$. This specific example shows why the normal equation can give rise to the least square solution. Multiply on the left of a linear system will change the solution set.\exampbar
\end{example}

\begin{mdframed}[hidealllines=true,backgroundcolor=gray!10,frametitle={Rank Deficiency}]
	Note here, we assume $\bA\in \real^{m\times n}$ has full rank with $m>n$ to make $\bA^\top\bA$ invertible. But when two or more columns of $\bA$ are perfectly correlated, the matrix $\bA$ will be deficient and $\bA^\top\bA$ is singular. Choose the $\bx$ that minimizes $\bx_{LS}^\top\bx_{LS}$ which meets the normal equation can help to solve the problem. I.e., choose the shortest magnitude least squares solution. But this is not the main interest of the text. We will leave this topic to the readers. In Section~\ref{section:ls-utv}, ~\ref{section:application-ls-svd}, we have shortly discussed how to use UTV decomposition and SVD to tackle the rank deficient least squares problem.
\end{mdframed}

\section{Netflix Recommender and Matrix Factorization}\label{section:als-netflix}
In the Netflix prize \citep{bennett2007netflix}, the goal was to predict ratings of users for different movies, given the existing ratings of those users for other movies.
We index $M$ movies with $m= 1, 2,\ldots,M$ and $N$ users
by $n = 1, 2,\ldots,N$. We denote the rating of the $n$-th user for the $m$-th movie by $a_{mn}$. Define $\bA$ to be an $M \times N$ rating matrix with columns $\ba_n \in \real^M$ containing ratings of the $n$-th user. Note that many ratings $a_{mn}$ are missing and our goal is to predict those missing ratings accurately.

We formally consider algorithms for solving the following problem: The matrix $\bA$ is approximately factorized into an $M\times K$ matrix $\bW$ and a $K \times  N$ matrix $\bZ$. Usually $K$ is chosen to be smaller than $M$ or $N$, so that $\bW$ and $\bZ$ are smaller than the original
matrix $\bA$. This results in a compressed version of the original data matrix.
An
appropriate decision on the value of $K$ is critical in practice, but the choice of $K$ is very
often problem dependent.
The factorization is significant in the sense, suppose $\bA=[\ba_1, \ba_2, \ldots, \ba_N]$ and $\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$ are the column partitions of $\bA,\bZ$ respectively, then $\ba_n = \bW\bz_n$, i.e., each column $\ba_n$ is approximated by a linear combination of the columns of $\bW$ weighted by the components in $\bz_n$. Therefore, columns of $\bW$ can be thought of containing column basis of $\bA$. This is similar to the factorization in the data interpretation part (Part~\ref{part:data-interation}, p.~\pageref{part:data-interation}). What's different is that we are not restricting $\bW$ to be exact columns from $\bA$.

To find the approximation $\bA\approx\bW\bZ$, we need to define a loss function such that the distance between $\bA$ and $\bW\bZ$ can be measured. The loss function is selected to be the Frobenius norm between two matrices which vanishes to zero if $\bA=\bW\bZ$ where the advantage will be seen shortly.

To simplify the problem, let us
assume that there are no missing
ratings firstly. Project data vectors $\ba_n$ to a smaller
dimension $\bz_n \in \real^K$  with $K<M$,
such that the \textit{reconstruction error} measured by Frobenius norm is minimized (assume $K$ is known):
\begin{equation}\label{equation:als-per-example-loss}
\mathop{\min}_{\bW,\bZ}  \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2,
\end{equation}
where $\bW=[\bw_1^\top; \bw_2^\top; \ldots; \bw_M^\top]\in \real^{M\times K}$ and $\bZ=[\bz_1, \bz_2, \ldots, \bz_N] \in \real^{K\times N}$ containing $\bw_m$'s and $\bz_n$'s as rows and columns respectively. The loss form in Equation~\eqref{equation:als-per-example-loss} is known as the \textit{per-example loss}. It can be equivalently written as
$$
L(\bW,\bZ) = \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2 = ||\bW\bZ-\bA||^2.
$$
Moreover, the loss $L(\bW,\bZ)=\sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)$ is convex with respect to $\bZ$ given $\bW$ and vice versa. Therefore, we can first minimize with respect to $\bZ$
given $\bW$ and then minimize with respect to $\bW$
given $\bZ$:
$$
\left\{
\begin{aligned}
	\bZ &\leftarrow \mathop{\arg \min}_{\bZ} L(\bW,\bZ);    \qquad \text{(ALS1)} \\
	\bW &\leftarrow \mathop{\arg \min}_{\bW} L(\bW,\bZ). \qquad \text{(ALS2)}
\end{aligned}
\right.
$$
This is known as the \textit{coordinate descent algorithm} in which case we employ the least squares, it is also called the \textit{alternating least squares (ALS)} \citep{comon2009tensor, takacs2012alternating, giampouras2018alternating}. The convergence is guaranteed if the loss function $L(\bW,\bZ)$ decreases at each iteration and we shall discuss more on this in the sequel.\index{ALS}

\begin{remark}[Convexity and Global Minimum]
Although the loss function defined by Frobenius norm $|| \bW\bZ-\bA||^2$ is convex in $\bW$ given $\bZ$ or vice versa, it is not convex in both variables together. Therefore we are not able to find the global minimum. However, the convergence is assured to find local minima.
\end{remark}

\paragraph{Given $\bW$, Optimizing $\bZ$}

Now, let's see what is in the problem of $\bZ \leftarrow \mathop{\arg \min}_{\bZ} L(\bW,\bZ)$. When there exists a unique minimum of the loss function $L(\bW,\bZ)$ with respect to $\bZ$, we speak of the \textit{least squares} minimizer of $\mathop{\arg \min}_{\bZ} L(\bW,\bZ)$.
Given $\bW$, $L(\bW,\bZ)$ can be written as $L(\bZ|\bW)$ to emphasize on the variable of $\bZ$:
$$
\begin{aligned}
	L(\bZ|\bW) &= ||\bW\bZ-\bA||^2= \left\Vert\bW[\bz_1,\bz_2,\ldots, \bz_N]-[\ba_1,\ba_2,\ldots,\ba_N]\right\Vert^2=\left\Vert
	\begin{bmatrix}
		\bW\bz_1 - \ba_1 \\
		\bW\bz_2 - \ba_2\\
		\vdots \\
		\bW\bz_N - \ba_N
	\end{bmatrix}
	\right\Vert^2.
\end{aligned}\footnote{The matrix norm used here is the Frobenius norm (Definition~\ref{definition:frobenius}, p.~\pageref{definition:frobenius}) such that $|| \bA ||= \sqrt{\sum_{i=1,j=1}^{m,n} (\bA_{ij})^2}$ if $\bA\in \real^{m\times n}$. And the vector norm used here is the $l_2$ norm (Section~\ref{section:vector-norm}, p.~\pageref{section:vector-norm}) such that $||\bx||_2  = \sqrt{\sum_{i=1}^{n}x_i^2}$ if $\bx\in \real^n$.}
$$
Now, if we define
$$
\widetildebW =
\begin{bmatrix}
	\bW & \bzero & \ldots & \bzero\\
	\bzero & \bW & \ldots & \bzero\\
	\vdots & \vdots & \ddots & \vdots \\
	\bzero & \bzero & \ldots & \bW
\end{bmatrix}
\in \real^{MN\times KN},
\gap
\widetildebz=
\begin{bmatrix}
	\bz_1 \\ \bz_2 \\ \vdots \\ \bz_N
\end{bmatrix}
\in \real^{KN},
\gap
\widetildeba=
\begin{bmatrix}
	\ba_1 \\ \ba_2 \\ \vdots \\ \ba_N
\end{bmatrix}
\in \real^{MN},
$$
then the (ALS1) problem can be reduced to the normal least squares for minimizing  $||\widetildebW \widetildebz - \widetildeba||^2$ with respect to $\widetildebz$. And the solution is given by
$$
\widetildebz = (\widetildebW^\top\widetildebW)^{-1} \widetildebW^\top\widetildeba.
$$
The construction may seem reasonable at first glance. But since $rank(\widetildebW) =\min\{M,K\}$, $(\widetildebW^\top\widetildebW)$ is not invertible.
A direct way to solve (ALS1) is to find the differential of $L(\bZ|\bW)$ with respect to $\bZ$:
\begin{equation}\label{equation:givenw-update-z-allgd}
\begin{aligned}
	\frac{\partial L(\bZ|\bW)}{\partial \bZ} &=
	\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial \bZ}\\
	&=\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial (\bW\bZ-\bA)}
	\frac{\partial (\bW\bZ-\bA)}{\partial \bZ}\\
	&\stackrel{\star}{=}2  \bW^\top(\bW\bZ-\bA) \in \real^{K\times N},
\end{aligned}
\end{equation}
where the first equality is from the definition of Frobenius (Definition~\ref{definition:frobenius}, p.~\pageref{definition:frobenius}) such that $|| \bA || = \sqrt{\sum_{i=1,j=1}^{m,n} (\bA_{ij})^2}=\sqrt{tr(\bA\bA^\top)}$, and equality ($\star$) comes from the fact that $\frac{\partial tr(\bA\bA^\top)}{\partial \bA} = 2\bA$. When the loss function is a differentiable function of $\bZ$, we may determine the least squares solution by differential calculus, and a minimum of the function
$L(\bZ|\bW)$ must be a root of the equation:
$$
\frac{\partial L(\bZ|\bW)}{\partial \bZ}  = \bzero.
$$
By finding the root of the above equation, we have the ``candidate" update on $\bZ$ that find the minimizer of $L(\bZ|\bW)$
\begin{equation}\label{equation:als-z-update}
	\bZ = (\bW^\top\bW)^{-1} \bW^\top \bA  \leftarrow \mathop{\arg \min}_{\bZ} L(\bZ|\bW).
\end{equation}
Before we declare a root of the above equation is actually a minimizer rather than a maximizer (that's why we call the update a ``candidate" update above), we need to verify the function is convex such that if the function is twice differentiable, this can be equivalently done by verifying
$$
\frac{\partial^2 L(\bZ|\bW)}{\partial \bZ^2} > 0,
$$
i.e., the Hessian matrix is positive definite (recall the definition of positive definiteness, Definition~\ref{definition:psd-pd-defini}, p.~\pageref{definition:psd-pd-defini}). To see this, we write out the twice differential
$$
\frac{\partial^2 L(\bZ|\bW)}{\partial \bZ^2}= 2\bW^\top\bW \in \real^{K\times K},
$$
which has full rank if $\bW\in \real^{M\times K}$ has full rank (Lemma~\ref{lemma:rank-of-ata}, p.~\pageref{lemma:rank-of-ata}) and $K<M$. We here claim that if $\bW$ has full rank, then $\frac{\partial^2 L(\bZ|\bW)}{\partial \bZ^2}$ is positive definite. This can be done by checking that when $\bW$ has full rank, $\bW\bx=\bzero$ only when $\bx=\bzero$ since the null space of $\bW$ is of dimension 0. Therefore,
$$
\bx^\top (2\bW^\top\bW)\bx >0, \qquad \text{for any nonzero vector $\bx\in \real^K$}.
$$
Now, the thing is that we need to check \textcolor{blue}{if $\bW$ has full rank so that the Hessian of $L(\bZ|\bW)$ is positive definiteness}, otherwise, we cannot claim the update of $\bZ$ in Equation~\eqref{equation:als-z-update} decreases the loss so that the matrix decomposition is going into the right way to better approximate the original matrix $\bA$ by $\bW\bZ$.
We will shortly come back to the positive definiteness of the Hessian matrix in the sequel which relies on the following lemma
\begin{lemma}[Rank of $\bZ$ after Updating]\label{lemma:als-update-z-rank}
	Suppose $\bA\in \real^{M\times N}$ has full rank with $M\leq N$ and $\bW\in \real^{M\times K}$ has full rank with $K<M$, then the update of $\bZ=(\bW^\top\bW)^{-1} \bW^\top \bA \in \real^{K\times N}$ in Equation~\eqref{equation:als-z-update} has full rank.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:als-update-z-rank}]
	Since $\bW^\top\bW\in \real^{K\times K}$ has full rank if $\bW$ has full rank (Lemma~\ref{lemma:rank-of-ata}, p.~\pageref{lemma:rank-of-ata}) such that $(\bW^\top\bW)^{-1} $ has full rank.

	Suppose $\bW^\top\bx=\bzero$, this implies $(\bW^\top\bW)^{-1} \bW^\top\bx=\bzero$. Thus
	$$
	\nspace(\bW^\top) \subseteq \nspace\left((\bW^\top\bW)^{-1} \bW^\top\right).
	$$
	Moreover, suppose $(\bW^\top\bW)^{-1} \bW^\top\bx=\bzero$, and since $(\bW^\top\bW)^{-1} $ is invertible. This implies $ \bW^\top\bx=(\bW^\top\bW)\bzero=\bzero$, and
	$$
	\nspace\left((\bW^\top\bW)^{-1} \bW^\top\right)\subseteq \nspace(\bW^\top).
	$$
	As a result, by ``sandwiching", it follows that
	\begin{equation}\label{equation:als-z-sandiwch1}
		\nspace(\bW^\top) = \nspace\left((\bW^\top\bW)^{-1} \bW^\top\right).
	\end{equation}
	Therefore, $(\bW^\top\bW)^{-1} \bW^\top$ has full rank $K$. Let $\bT=(\bW^\top\bW)^{-1} \bW^\top\in \real^{K\times M}$, and suppose $\bT^\top\bx=\bzero$. This implies $\bA^\top\bT^\top\bx=\bzero$, and
	$$
	\nspace(\bT^\top) \subseteq \nspace(\bA^\top\bT^\top).
	$$
	Similarly, suppose $\bA^\top(\bT^\top\bx)=\bzero$. Since $\bA$ has full rank with the dimension of the null space being 0: $dim\left(\nspace(\bA^\top)\right)=0$, $(\bT^\top\bx)$ must be zero. The claim follows from that since $\bA$ has full rank $M$ with the row space of $\bA^\top$ being equal to the column space of $\bA$ where $dim\left(\cspace(\bA)\right)=M$ and the $dim\left(\nspace(\bA^\top)\right) = M-dim\left(\cspace(\bA)\right)=0$. Therefore, $\bx$ is in the null space of $\bT^\top$ if $\bx$ is in the null space of $\bA^\top\bT^\top$:
	$$
	\nspace(\bA^\top\bT^\top)\subseteq \nspace(\bT^\top).
	$$
	By ``sandwiching" again,
	\begin{equation}\label{equation:als-z-sandiwch2}
		\nspace(\bT^\top) = \nspace(\bA^\top\bT^\top).
	\end{equation}
	Since $\bT^\top$ has full rank $K<M<N$, $dim\left(\nspace(\bT^\top) \right) = dim\left(\nspace(\bA^\top\bT^\top)\right)=0$.
	Therefore,
	$\bZ^\top=\bA^\top\bT^\top$ has full rank $K$.
	We complete the proof.
\end{proof}

\paragraph{Given $\bZ$, Optimizing $\bW$}

Given $\bZ$, $L(\bW,\bZ)$ can be written as $L(\bW|\bZ)$ to emphasize on the variable of $\bW$:
$$
\begin{aligned}
	L(\bW|\bZ) &= ||\bW\bZ-\bA||^2.
\end{aligned}
$$
A direct way to solve (ALS2) is to find the differential of $L(\bW|\bZ)$ with respect to $\bW$:
$$
\begin{aligned}
	\frac{\partial L(\bW|\bZ)}{\partial \bW} &=
	\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial \bW}\\
	&=\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial (\bW\bZ-\bA)}
	\frac{\partial (\bW\bZ-\bA)}{\partial \bW}\\
	&=  2(\bW\bZ-\bA)\bZ^\top \in \real^{M\times K}.
\end{aligned}
$$
The ``candidate" update on $\bW$ is similarly to find the root of the differential $\frac{\partial L(\bW|\bZ)}{\partial \bW}$:
\begin{equation}\label{equation:als-w-update}
	\bW^\top = (\bZ\bZ^\top)^{-1}\bZ\bA^\top  \leftarrow \mathop{\arg\min}_{\bW} L(\bW|\bZ).
\end{equation}
Again, we emphasize that the update is only a ``candidate" update. We need to further check whether the Hessian is positive definite or not.
The Hessian matrix is given by
$$
\begin{aligned}
	\frac{\partial^2 L(\bW|\bZ)}{\partial \bW^2} =2\bZ\bZ^\top \in \real^{K\times K}.
\end{aligned}
$$
Therefore, by analogous analysis, if $\bZ$ has full rank with $K<N$, the Hessian matrix is positive definite.

\begin{lemma}[Rank of $\bW$ after Updating]\label{lemma:als-update-w-rank}
	Suppose $\bA\in \real^{M\times N}$ has full rank with $M\leq N$ and $\bZ\in \real^{K\times N}$ has full rank with $K<N$, then the update of $\bW^\top = (\bZ\bZ^\top)^{-1}\bZ\bA^\top$ in Equation~\eqref{equation:als-w-update} has full rank.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:als-update-w-rank}]
	The proof is slightly different to that of Lemma~\ref{lemma:als-update-z-rank}. Since $\bZ \in \real^{K\times N}$ and $\bA^\top \in \real^{N\times M}$ have full rank, i.e., $\det(\bZ)>0$ and $\det(\bA^\top)>0$. The determinant of their product $\det(\bZ\bA^\top)=\det(\bZ)\det(\bA^\top)>0$ such that $\bZ\bA^\top$ has full rank (rank $K$). Similarly argument can find $\bW^\top$ also has full rank.
\end{proof}

Combine the observations in Lemma~\ref{lemma:als-update-z-rank} and Lemma~\ref{lemma:als-update-w-rank}, as long as we \textcolor{blue}{initialize $\bZ, \bW$ to have full rank}, the updates in Equation~\eqref{equation:als-z-update} and Equation~\eqref{equation:als-w-update} are reasonable. \textcolor{blue}{The requirement on the $\leq N$ is reasonable in that there are always more users than the number of movies}. We conclude the process in Algorithm~\ref{alg:als}.

\begin{algorithm}[H]
	\caption{Alternating Least Squares}
	\label{alg:als}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$ \textcolor{blue}{with $M\leq N$};
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{with full rank and $K<M\leq N$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$||\bA-\bW\bZ||>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\State $\bZ = (\bW^\top\bW)^{-1} \bW^\top \bA  \leftarrow \mathop{\arg \min}_{\bZ} L(\bZ|\bW)$;
		\State $\bW^\top = (\bZ\bZ^\top)^{-1}\bZ\bA^\top  \leftarrow \mathop{\arg\min}_{\bW} L(\bW|\bZ)$;
		\EndWhile
		\State Output $\bW,\bZ$;
	\end{algorithmic}
\end{algorithm}

\section{Regularization: Extension to General Matrices}\label{section:regularization-extention-general}
We can add a regularization to minimize the following loss:
\begin{equation}\label{equation:als-regularion-full-matrix}
L(\bW,\bZ)  =||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2, \qquad \lambda_w>0, \lambda_z>0,
\end{equation}
where the differential with respect to $\bZ, \bW$ are given respectively by
\begin{equation}\label{equation:als-regulari-gradien}
\left\{
\begin{aligned}
	\frac{\partial L(\bW,\bZ) }{\partial \bZ} &= 2\bW^\top(\bW\bZ-\bA) + 2\lambda_z\bZ \in \real^{K\times N};\\
	\frac{\partial L(\bW,\bZ) }{\partial \bW} &= 2(\bW\bZ-\bA)\bZ^\top + 2\lambda_w\bW \in \real^{M\times K}.
\end{aligned}
\right.
\end{equation}
The Hessian matrices are given respectively by
$$
\left\{
\begin{aligned}
	\frac{\partial^2 L(\bW,\bZ) }{\partial \bZ^2} &= 2\bW^\top\bW+ 2\lambda_z\bI \in \real^{K\times K};\\
	\frac{\partial^2 L(\bW,\bZ) }{\partial \bW^2} &= 2\bZ\bZ^\top + 2\lambda_w\bI \in \real^{K\times K}, \\
\end{aligned}
\right.
$$
which are positive definite due to the perturbation by the regularization. To see this,
$$
\left\{
\begin{aligned}
	\bx^\top (2\bW^\top\bW +2\lambda_z\bI)\bx &= \underbrace{2\bx^\top\bW^\top\bW\bx}_{\geq 0} + 2\lambda_z ||\bx||^2>0, \gap \text{for nonzero $\bx$};\\
	\bx^\top (2\bZ\bZ^\top +2\lambda_w\bI)\bx &= \underbrace{2\bx^\top\bZ\bZ^\top\bx}_{\geq 0} + 2\lambda_w ||\bx||^2>0,\gap \text{for nonzero $\bx$}.
\end{aligned}
\right.
$$
\textcolor{blue}{The regularization makes the Hessian matrices positive definite even if $\bW, \bZ$ are rank deficient}. And now the matrix decomposition can be extended to any matrix even when $M>N$. In rare cases, $K$ can be chosen as $K>\max\{M, N\}$ such that a high-rank approximation of $\bA$ is obtained. However, in most scenarios, we want to find the low-rank approximation of $\bA$ such that $K<\min\{M, N\}$. For example, the ALS can be utilized to find the low-rank neural networks to reduce the memory of the neural networks whilst increase the performance (Section~\ref{section:low-rank-neural}, p.~\pageref{section:low-rank-neural}).

Therefore, the minimizers are given by finding the roots of the differential:
\begin{equation}\label{equation:als-regular-final-all}
	\left\{
	\begin{aligned}
		\bZ &= (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \bA  ;\\
		\bW^\top &= (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bA^\top .
	\end{aligned}
	\right.
\end{equation}
The regularization parameters $\lambda_z, \lambda_2\in \real$ are used to balance the trade-off
between the accuracy of the approximation and the smoothness of the computed solution. The choice on the selection of the parameters is typically problem dependent and can be obtained by \textit{cross-validation}. Again, we conclude the process in Algorithm~\ref{alg:als-regularizer}.
\begin{algorithm}[H]
	\caption{Alternating Least Squares with Regularization}
	\label{alg:als-regularizer}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$;
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose regularization parameters $\lambda_w, \lambda_z$;
		\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$||\bA-\bW\bZ||>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\State $\bZ = (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \bA  \leftarrow \mathop{\arg \min}_{\bZ} L(\bZ|\bW)$;
		\State $\bW^\top = (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bA^\top  \leftarrow \mathop{\arg\min}_{\bW} L(\bW|\bZ)$;
		\EndWhile
		\State Output $\bW,\bZ$;
	\end{algorithmic}
\end{algorithm}

\section{Missing Entries}\label{section:alt-columb-by-column}
Since the matrix decomposition via the ALS is extensively used in the Netflix recommender data, where many entries are missing since many users have not watched some movies or they will not rate the movies for some reasons. We can make an additional mask matrix $\bM\in \real^{M\times N}$ where $\bM_{mn}\in \{0,1\}$ means if the user $n$ has rated the movie $m$ or not. Therefore, the loss function can be defined as
$$
L(\bW,\bZ) = ||\bM\circledast  \bA- \bM\circledast (\bW\bZ)||^2,
$$
where $\circledast$ is the \textit{Hadamard product} between matrices. For example, the Hadamard product for a $3 \times 3$ matrix $\bA$ with a $3\times 3$ matrix $\bB$ is
$$
\bA\circledast \bB =
\begin{bmatrix}
	a_{11} & a_{12} & a_{13} \\
	a_{21} & a_{22} & a_{23} \\
	a_{31} & a_{32} & a_{33}
\end{bmatrix}
\circledast
\begin{bmatrix}
	b_{11} & b_{12} & b_{13} \\
	b_{21} & b_{22} & b_{23} \\
	b_{31} & b_{32} & b_{33}
\end{bmatrix}
=
\begin{bmatrix}
	a_{11}b_{11} &  a_{12}b_{12} & a_{13}b_{13} \\
	a_{21}b_{21} & a_{22}b_{22} & a_{23}b_{23} \\
	a_{31}b_{31} & a_{32}b_{32} & a_{33}b_{33}
\end{bmatrix}.
$$
To find the solution of the problem, let's decompose the updates in Equation~\eqref{equation:als-regular-final-all} into:
\begin{equation}\label{equation:als-ori-all-wz}
	\left\{
	\begin{aligned}
		\bz_n &= (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \ba_n, &\gap& \text{for $n\in \{1,2,\ldots, N\}$}  ;\\
		\bw_m &= (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bb_m,  &\gap& \text{for $m\in \{1,2,\ldots, M\}$} ,
	\end{aligned}
	\right.
\end{equation}
where $\bZ=[\bz_1, \bz_2, \ldots, \bz_N], \bA=[\ba_1,\ba_2, \ldots, \ba_N]$ are the column partitions of $\bZ, \bA$ respectively. And $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M], \bA^\top=[\bb_1,\bb_2, \ldots, \bb_M]$ are the column partitions of $\bW^\top, \bA^\top$ respectively. The factorization of the updates indicates the update can be done via a column by column fashion.

\paragraph{Given $\bW$}
Let $\bo_n\in \real^M$ denote the movies rated by user $n$ where $o_{nm}=1$ if user $n$ has rated movie $m$, and $o_{nm}=1$ otherwise. Then the $n$-th column of $\bA$ without missing entries can be denoted as the matlab style notation $\ba_n[\bo_n]$. And we want to approximate the existing $n$-th column by $\ba_n[\bo_n] \approx \bW[\bo_n, :]\bz_n$ which is actually a rank-one least squares problem:
\begin{equation}\label{equation:als-ori-all-wz-modif-z}
	\begin{aligned}
		\bz_n &= \left(\bW[\bo_n, :]^\top\bW[\bo_n, :]+ \lambda_z\bI\right)^{-1} \bW[\bo_n, :]^\top \ba_n[\bo_n], &\gap& \text{for $n\in \{1,2,\ldots, N\}$} .
	\end{aligned}
\end{equation}
Moreover, the loss function with respect to $\bz_n$:
$$
L(\bz_n|\bW) =\sum_{m\in \bo_n} \left(a_{mn} - \bw_m^\top\bz_n\right)^2
$$
and if we are concerned about the loss for all users:
$$
L(\bZ|\bW) =\sum_{n=1}^N\ \sum_{m\in \bo_n} \left(a_{mn} - \bw_m^\top\bz_n\right)^2
$$

\paragraph{Given $\bZ$}
Similarly, if $\bp_m \in \real^{N}$ denotes the users that have rated the movie $m$ with $\bp_{dn}=1$ if the movie $m$ has been rated by user $n$. Then the $m$-th row of $\bA$ without missing entries can be denoted as the matlab style notation $\bb_m[\bp_m]$. And we want to approximate the existing $m$-th row by $\bb_m[\bp_m] \approx \bZ[:, \bp_m]^\top\bw_m$,
\footnote{Note that $\bZ[:, \bp_m]^\top$ is the transpose of $\bZ[:, \bp_m]$, which is equal to $\bZ^\top[\bp_m,:]$, i.e., transposing first and then selecting.}
which again is a rank-one least squares problem:
\begin{equation}\label{equation:als-ori-all-wz-modif-w}
	\begin{aligned}
		\bw_m &= (\bZ[:, \bp_m]\bZ[:, \bp_m]^\top+\lambda_w\bI)^{-1}\bZ[:, \bp_m]\bb_m[\bp_m],  &\gap& \text{for $m\in \{1,2,\ldots, M\}$} .
	\end{aligned}
\end{equation}
Moreover, the loss function with respect to $\bw_n$:
$$
L(\bw_n|\bZ) =\sum_{n\in \bp_m} \left(a_{mn} - \bw_m^\top\bz_n\right)^2
$$
and if we are concerned about the loss for all users:
$$
L(\bW|\bZ) =\sum_{d=1}^M  \sum_{n\in \bp_m} \left(a_{mn} - \bw_m^\top\bz_n\right)^2
$$
The procedure is again formulated in Algorithm~\ref{alg:als-regularizer-missing-entries}.
\begin{algorithm}[H]
	\caption{Alternating Least Squares with Missing Entries and Regularization}
	\label{alg:als-regularizer-missing-entries}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$;
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose regularization parameters $\lambda_w, \lambda_z$;
		\State compute the mask matrix $\bM$ from $\bA$;
			\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{\textcolor{blue}{$||\bM\circledast  \bA- \bM\circledast (\bW\bZ)||^2>\delta $} and $iter<C$}
		\State $iter=iter+1$;
		\For{$n=1,2,\ldots, N$}
		\State $\bz_n = \left(\bW[\bo_n, :]^\top\bW[\bo_n, :]+ \lambda_z\bI\right)^{-1} \bW[\bo_n, :]^\top \ba_n[\bo_n]$; \Comment{$n$-th column of $\bZ$}
		\EndFor

		\For{$m=1,2,\ldots, M$}
		\State $\bw_m = (\bZ[:, \bp_m]\bZ[:, \bp_m]^\top+\lambda_w\bI)^{-1}\bZ[:, \bp_m]\bb_m[\bp_m]$;\Comment{$m$-th column of $\bW^\top$}
		\EndFor
		\EndWhile
		\State Output $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$;
	\end{algorithmic}
\end{algorithm}

\section{Vector Inner Product}\label{section:als-vector-product}
We have seen the ALS is to find matrices $\bW, \bZ$ such that $\bW\bZ$ can approximate $\bA\approx \bW\bZ$ in terms of minimum least squared loss:
$$
\mathop{\min}_{\bW,\bZ}  \sum_{n=1}^N \sum_{d=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2,
$$
that is, each entry $a_{mn}$ in $\bA$ can be approximated by the inner product between the two vectors $\bw_m^\top\bz_n$. The geometric definition of vector inner product is given by
$$
\bw_m^\top\bz_n = ||\bw||\cdot ||\bz|| \cos \theta,
$$
where $\theta$ is the angle between $\bw$ and $\bz$. So if the vector norms of $\bw, \bz$ are determined, the smaller the angle, the larger the inner product.

Come back to the Netflix data, where the rating are ranging from 0 to 5 and the larger the better. If $\bw_m$ and $\bz_n$ fall ``close" enough, then $\bw^\top\bz$ will have a larger value. This reveals the meaning behind the ALS where $\bw_m$ represents the features of movie $m$, whilst $\bz_n$ contains the features of user $n$. And each element in $\bw_m$ and $\bz_n$ represents a same feature. For example, it could be that the second feature $\bw_{m2}$ \footnote{$\bw_{m2}$ is the second element of vector $\bw_{m2}$} represents if the movie is an action movie or not, and $\bz_{n2}$ denotes if the user $n$ likes action movies or not. If it happens the case, then $\bw_m^\top\bz_n$ will be large and approximates $a_{mn}$ well.

Note that, in the decomposition $\bA\approx \bW\bZ$, we know the rows of $\bW$ contain the hidden features of the movies and the columns of $\bZ$ contain the hidden features of the users. However, we cannot identify what are the meanings of the rows of $\bW$ or the columns of $\bZ$. We know they could be something like categories or genres of the movies, that provide some underlying connections between the users and the movies, but we cannot be sure what exactly they are. This is where the terminology ``hidden" comes from.

\section{Gradient Descent}\label{section:als-gradie-descent}
In Equation~\eqref{equation:als-ori-all-wz}, we obtain the column-by-column update directly from the full matrix way in Equation~\eqref{equation:als-regular-final-all} (with regularization considered). Now let's see what's behind the idea. Following from Equation~\eqref{equation:als-regularion-full-matrix}, the loss under the regularization:
\begin{equation}
	L(\bW,\bZ)  =||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2, \qquad \lambda_w>0, \lambda_z>0,
\end{equation}
Since we are now considering the minimization of above loss with respect to $\bz_n$, we can decompose the loss into
\begin{equation}\label{als:gradient-regularization-zn}
\begin{aligned}
L(\bz_n)  &=||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2\\\
&= ||\bW\bz_n-\ba_n||^2 + \lambda_z ||\bz_n||^2 +
\underbrace{\sum_{i\neq n} ||\bW\bz_i-\ba_i||^2  + \lambda_z \sum_{i\neq n}||\bz_i||^2   +    \lambda_w ||\bW||^2 }_{C_{z_n}},
\end{aligned}
\end{equation}
where $C_{z_n}$ is a constant with respect to $\bz_n$, and $\bZ=[\bz_1, \bz_2, \ldots, \bz_N], \bA=[\ba_1,\ba_2, \ldots, \ba_N]$ are the column partitions of $\bZ, \bA$ respectively. Taking the differential
$$
\frac{\partial L(\bz_n)}{\partial \bz_n} = 2\bW^\top\bW\bz_n - 2\bW^\top\ba_n + 2\lambda_z\bz_n,
$$
under which the root is exactly the first update of the column fashion in Equation~\eqref{equation:als-ori-all-wz}:
$$
\bz_n = (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \ba_n, \gap \text{for $n\in \{1,2,\ldots, N\}$}.
$$
Similarly, we can decompose the loss with respect to $\bw_m$,
\begin{equation}\label{als:gradient-regularization-wd}
\begin{aligned}
L(\bw_m )  &=
||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2\\
&=||\bZ^\top\bW-\bA^\top||^2 +\lambda_w ||\bW^\top||^2 + \lambda_z ||\bZ||^2\\\
&= ||\bZ^\top\bw_m-\bb_n||^2 + \lambda_w ||\bw_m||^2 +
\underbrace{\sum_{i\neq m} ||\bZ^\top\bw_i-\bb_i||^2  + \lambda_w \sum_{i\neq m}||\bw_i||^2   +    \lambda_z ||\bZ||^2 }_{C_{w_m}},
\end{aligned}
\end{equation}
where $C_{w_m}$ is a constant with respect to $\bw_m$, and $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M], \bA^\top=[\bb_1,\bb_2, \ldots,$ $\bb_M]$ are the column partitions of $\bW^\top, \bA^\top$ respectively.
Analogously, taking the differential with respect to $\bw_m$, it follows that
$$
\frac{\partial L(\bw_m)}{\partial \bw_m} = 2\bZ\bZ^\top\bw_m - 2\bZ\bb_n + 2\lambda_w\bw_m,
$$
under which the root is exactly the second update of the column fashion in Equation~\eqref{equation:als-ori-all-wz}:
$$
\bw_m = (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bb_m,  \gap \text{for $m\in \{1,2,\ldots, M\}$} .
$$

Now suppose we write out the iteration as the superscript and we want to find the updates $\{\bz^{(k+1)}_n, \bw^{(k+1)}_m\}$ base on $\{\bZ^{(k)}, \bW^{(k)}\}$:
$$
\left\{
\begin{aligned}
	\bz^{(k+1)}_n    &\leftarrow \mathop{\arg \min}_{\bz_n^{(k)}} L(\bz_n^{(k)});\\
	\bw_m^{(k+1)}    &\leftarrow \mathop{\arg\min}_{\bw_m^{(k)}} L(\bw_m^{(k)}).
\end{aligned}
\right.
$$
For simplicity, we will be looking at $\bz^{(k+1)}_n    \leftarrow \mathop{\arg \min}_{\bz_n^{(k)}} L(\bz_n^{(k)}|-)$, and the derivation for the update on $\bw_m^{(k+1)}$ will be the same. Suppose we want to approximate $\bz^{(k+1)}_n$ by a linear update on $\bz^{(k)}_n$:
$$
\bz^{(k+1)}_n = \bz^{(k)}_n + \eta \bv.
$$
The problem now turns to the solution of $\bv$ such that
$$
\bv=\mathop{\arg \min}_{\bv} L(\bz^{(k)}_n + \eta \bv) .
$$
\begin{algorithm}[h]
	\caption{Alternating Least Squares with Full Entries and Gradient Descent}
	\label{alg:als-regularizer-missing-stochas-gradient}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$;
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose regularization parameters $\lambda_w, \lambda_z$, and step size $\eta_w, \eta_z$;
		\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\For{$n=1,2,\ldots, N$}
		\State $\bz^{(k+1)}_n =\bz^{(k)}_n - \eta_z \frac{\nabla L(\bz^{(k)}_n )}{||\nabla L(\bz^{(k)}_n )||}$; \Comment{$n$-th column of $\bZ$}
		\EndFor

		\For{$m=1,2,\ldots, M$}
		\State $\bw^{(k+1)}_m  = \bw^{(k)}_m - \eta_w \frac{\nabla L(\bw^{(k)}_m )}{||\nabla L(\bw^{(k)}_m )||}$;\Comment{$m$-th column of $\bW^\top$}
		\EndFor
		\EndWhile
		\State Output $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$;
	\end{algorithmic}
\end{algorithm}
By Taylor's formula (Appendix~\ref{appendix:taylor-expansion}, p.~\pageref{appendix:taylor-expansion}), $L(\bz^{(k)}_n + \eta \bv)$ can be approximated by
$$
L(\bz^{(k)}_n + \eta \bv) \approx L(\bz^{(k)}_n ) + \eta \bv^\top \nabla  L(\bz^{(k)}_n ),
$$
when $\eta$ is small enough. Then an search under the condition $||\bv||=1$ given positive $\eta$ is as follows:
$$
\bv=\mathop{\arg \min}_{||\bv||=1} L(\bz^{(k)}_n + \eta \bv) \approx\mathop{\arg \min}_{||\bv||=1}
 \left\{L(\bz^{(k)}_n ) + \eta \bv^\top \nabla  L(\bz^{(k)}_n )\right\}.
$$
This is known as the \textit{greedy search}. The optimal $\bv$ can be obtained by
$$
\bv = -\frac{\nabla L(\bz^{(k)}_n )}{||\nabla L(\bz^{(k)}_n )||},
$$
i.e., $\bv$ is in the opposite direction of $\nabla L(\bz^{(k)}_n )$. Therefore, the update of $\bz_n^{(k+1)}$ is reasonable to be taken as
$$
\bz^{(k+1)}_n =\bz^{(k)}_n + \eta \bv = \bz^{(k)}_n - \eta \frac{\nabla L(\bz^{(k)}_n )}{||\nabla L(\bz^{(k)}_n )||},
$$
which usually called the \textit{gradient descent}. Similarly, the gradient descent of $\bw_m^{(k+1)}$ is given by
$$
\bw^{(k+1)}_m =\bw^{(k)}_m + \eta \bv = \bw^{(k)}_m - \eta \frac{\nabla L(\bw^{(k)}_m )}{||\nabla L(\bw^{(k)}_m )||}.
$$
The updated procedure on Algorithm~\ref{alg:als-regularizer} by a gradient descent way is then formulated in Algorithm~\ref{alg:als-regularizer-missing-stochas-gradient}.

\paragraph{Geometrical Interpretation of Gradient Descent}
\begin{lemma}[Direction of Gradients]\label{lemm:direction-gradients}
An important fact is that gradients are orthogonal to level curves (a.k.a., level surface).
\end{lemma}
\begin{proof}[of Lemma~\ref{lemm:direction-gradients}]
This is equivalently to prove that the gradients is orthogonal to the tangent of the level curve. For simplicity, let's first look at the 2-dimensional case. Suppose the level curve has the form $f(x,y)=c$. This implicitly gives a relation between $x$ and $y$ such that $y=y(x)$ where $y$ can be thought of as a function of $x$. Therefore, the level curve can be written as
$$
f(x, y(x)) = c.
$$
The chain rule indicates
$$
\frac{\partial f}{\partial x} \underbrace{\frac{dx}{dx}}_{=1} + \frac{\partial f}{\partial y} \frac{dy}{dx}=0.
$$
Therefore, the gradients is perpendicular to the tangent:
$$
\left\langle \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right\rangle
\cdot
\left\langle \frac{dx}{dx}, \frac{dy}{dx}\right\rangle=0.
$$
Let us now treat the problem in full generality, suppose the level curve of a vector $\bx\in \real^n$: $f(\bx) = f(x_1, x_2, \ldots, x_n)=c$. Each variable $x_i$ can be regarded as a function of a variable $t$ on the level curve $f(\bx)=c$: $f(x_1(t), x_2(t), \ldots, x_n(t))=c$. Differentiate the equation with respect to $t$ by chain rule:
$$
\frac{\partial f}{\partial x_1} \frac{dx_1}{dt} + \frac{\partial f}{\partial x_2} \frac{dx_2}{dt}
+\ldots + \frac{\partial f}{\partial x_n} \frac{dx_n}{dt}
=0.
$$
Therefore, the gradients is perpendicular to the tangent in $n$-dimensional case:
$$
\left\langle \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n}\right\rangle
\cdot
\left\langle \frac{dx_1}{dt}, \frac{dx_2}{dt}, \ldots \frac{dx_n}{dt}\right\rangle=0.
$$
This completes the proof.
\end{proof}
The lemma above reveals the geometrical interpretation of gradient descent. For finding a solution to minimize a convex function $L(\bz)$, gradient descent goes to the negative gradient direction that can decrease the loss. Figure~\ref{fig:alsgd-geometrical} depicts a $2$-dimensional case, where $-\nabla L(\bz)$ pushes the loss to decrease for the convex function $L(\bz)$.
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[A 2-dimensional convex function $L(\bz)$]{\label{fig:alsgd1}
		\includegraphics[width=0.47\linewidth]{./imgs/alsgd1.pdf}}
	\subfigure[$L(\bz)=c$ is a constant]{\label{fig:alsgd2}
		\includegraphics[width=0.44\linewidth]{./imgs/alsgd2.pdf}}
	\caption{Figure~\ref{fig:alsgd1} shows a function ``density" and a contour plot (\textcolor{bluepigment}{blue}=low, \textcolor{canaryyellow}{yellow}=high) where the upper graph is the ``density", and the lower one is the projection of it (i.e., contour). Figure~\ref{fig:alsgd2}: $-\nabla L(\bz)$ pushes the loss to decrease for the convex function $L(\bz)$.}
	\label{fig:alsgd-geometrical}
\end{figure}
\section{Regularization: A Geometrical Interpretation}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.95\textwidth]{./imgs/alsgd3.pdf}
	\caption{Constrained gradient descent with $\bz^\top\bz\leq C$. The \textcolor{green}{green} vector $\bw$ is the projection of $\bv_1$ into $\bz^\top\bz\leq C$ where $\bv_1$ is the component of $-\nabla l(\bz)$ perpendicular to $\bz_1$. The right picture is the next step after the update in the left picture. $\bz^\star$ denotes the optimal solution of \{$\min l(\bz)$\}.}
	\label{fig:alsgd3}
\end{figure}
We have seen in Section~\ref{section:regularization-extention-general} that the regularization can extend the ALS to general matrices. The gradient descent can reveal the geometric meaning of the regularization. To avoid confusion, we denote the loss function without regularization by $l(\bz)$ and the loss with regularization by $L(\bz) = \l(\bz)+\lambda_z ||\bz||^2$ where $l(\bz): \real^n \rightarrow \real$. When minimizing $l(\bz)$, descent method will search in $\real^n$ for a solution. However, in machine learning, searching in the whole space can cause overfitting. A partial solution is to search in a subset of the vector space, e.g., searching in $\bz^\top\bz < C$ for some constant $C$. That is
$$
\mathop{\arg\min}_{\bz} \gap l(\bz), \qquad s.t., \gap \bz^\top\bz\leq C.
$$
As shown above, a trivial gradient descent method will go further in the direction of $-\nabla l(\bz)$, i.e., update $\bz$ by $\bz\leftarrow \bz-\eta \nabla l(\bz)$ for small step size $\eta$. When the level curve is $l(\bz)=c_1$ and the current position of $\bz=\bz_1$ where $\bz_1$ is the intersection of $\bz^\top\bz=C$ and $l(\bz)=c_1$, the descent direction $-\nabla l(\bz_1)$ will be perpendicular to the level curve of $l(\bz_1)=c_1$ as shown in the left picture of Figure~\ref{fig:alsgd3}. However, if we further restrict the optimal value can only be in $\bz^\top\bz\leq C$, the trivial descent direction $-\nabla l(\bz_1)$ will lead $\bz_2=\bz_1-\eta \nabla l(\bz_1)$ outside of $\bz^\top\bz\leq C$. A solution is to decompose the step $-\nabla l(\bz_1)$ into
$$
-\nabla l(\bz_1) = a\bz_1 + \bv_1,
$$
where $a\bz_1$ is the component perpendicular to the curve of $\bz^\top\bz=C$, and $\bv_1$ is the component parallel to the curve of $\bz^\top\bz=C$. Keep only the step $\bv_1$, then the update
$$
\bz_2 = \text{project}(\bz_1+\eta \bv_1) = \text{project}\left(\bz_1 + \eta
\underbrace{(-\nabla l(\bz_1) -a\bz_1)}_{\bv_1}\right)\footnote{where the project($\bx$)
will project the vector $\bx$ to the closest point inside $\bz^\top\bz\leq C$. Notice here the direct update $\bz_2 = \bz_1+\eta \bv_1$ can still make $\bz_2$ outside the curve of $\bz^\top\bz\leq C$.}
$$
will lead to a smaller loss from $l(\bz_1)$ to $l(\bz_2)$ and still match $\bz^\top\bz\leq C$. This is known as the \textit{projection gradient descent}. It is not hard to see that the update $\bz_2 = \text{project}(\bz_1+\eta \bv_1)$ is equivalent to finding a vector $\bw$ (shown by the \textcolor{green}{green} vector in the left picture of Figure~\ref{fig:alsgd3}) such that $\bz_2=\bz_1+\bw$ is inside the curve of $\bz^\top\bz\leq C$. Mathematically, the $\bw$ can be obtained by $-\nabla l(\bz_1) -2\lambda \bz_1$ for some $\lambda$ as shown in the middle picture of Figure~\ref{fig:alsgd3}. This is exactly the negative gradient of $L(\bz)=l(\bz)+\lambda||\bz||^2$ such that
$$
\nabla L(\bz) = \nabla l(\bz) + 2\lambda \bz,
$$
and
$$
\begin{aligned}
\bw &= -\nabla L(\bz) \leadto \bz_2 &= \bz_1+ \bw =\bz_1 -  \nabla L(\bz).
\end{aligned}
$$
And in practice, a small step size $\eta$ can avoid going outside the curve of $\bz^\top\bz\leq C$:
$$
\bz_2  =\bz_1 -  \eta\nabla L(\bz),
$$
which is exactly what we have discussed in Section~\ref{section:regularization-extention-general}, the regularization term.

\paragraph{Sparsity}
In rare cases, we want to find sparse solution $\bz$ such that $l(\bz)$ is minimized. Constrained in $||\bz||_1 \leq C$ exists to this purpose where $||\cdot||_1$ is the $l_1$ norm of a vector or a matrix. The illustration of the $l_1$ in 2-dimensional and 3-dimensional space is shown in Figure~\ref{fig:p-norm-2d} and \ref{fig:p-norm-comparison-3d}. Similar to the previous case, the $l_1$ constrained optimization pushes the gradient descent towards the border of the level of $||\bz||_1=C$. The situation in the 2-dimensional case is shown in Figure~\ref{fig:alsgd4}. In a high-dimensional case, many elements in $\bz$ will be pushed into the breakpoint of $||\bz||_1=C$ as shown in the right picture of Figure~\ref{fig:alsgd4}.
\begin{figure}[h]
	\centering
	\includegraphics[width=0.95\textwidth]{./imgs/alsgd4.pdf}
	\caption{Constrained gradient descent with $||\bz||_1\leq C$, where the \textcolor{red}{red} dot denotes the breakpoint in $l_1$ norm. The right picture is the next step after the update in the left picture. $\bz^\star$ denotes the optimal solution of \{$\min l(\bz)$\}.}
	\label{fig:alsgd4}
\end{figure}

\section{Stochastic Gradient Descent}
Now suppose we come back to the per-example loss:
$$
L(\bW,\bZ)=  \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2 + \lambda_w||\bw_m||^2 +\lambda_z||\bz_n||^2.
$$
And when we iteratively decrease the per-example loss term $l(\bw_m, \bz_n)=\left(a_{mn} - \bw_m^\top\bz_n\right)^2$ for all $m\in [0,M], n\in[1,N]$, the full loss $L(\bW,\bZ)$ can also be decreased. This is known as the \textit{stochastic coordinate descent}. The differentials with respect to $\bw_m, \bz_n$, and their roots are given by
$$
\left\{
\begin{aligned}
\nabla l(\bz_n)=\frac{\partial l(\bw_m,\bz_n)}{\partial \bz_n} &= 2\bw_m\bw_m^\top \bz_n + 2\lambda_w\bw_m -2a_{mn} \bw_m \\
&\qquad \leadtosmall \bz_n= a_{mn}(\bw_m\bw_m^\top+\lambda_z\bI)^{-1}\bw_m;\\
\nabla l(\bw_m)=\frac{\partial l(\bw_m, \bz_n)}{\partial \bw_m} &= 2\bz_n\bz_n^\top\bw_m +2\lambda_z\bz_n - 2a_{mn}\bz_n\\
&\qquad \leadtosmall \bw_m= a_{mn}(\bz_n\bz_n^\top+\lambda_w\bI)^{-1}\bw_n.
\end{aligned}
\right.
$$
or analogously, the update can be done by gradient descent, and since we update by per-example loss, it is also known as the \textit{stochastic gradient descent}
$$
\left\{
\begin{aligned}
\bz_n&= \bz_n - \eta_z \frac{\nabla l(\bz_n)}{||\nabla l(\bz_n)||}; \\
\bw_m&= \bw_m - \eta_w \frac{\nabla l(\bw_m)}{||\nabla l(\bw_m)||}.
\end{aligned}
\right.
$$
The stochastic gradient descent update for ALS is formulated in Algorithm~\ref{alg:als-regularizer-missing-stochas-gradient-realstoch}. And in practice, the $m,n$ in the algorithm can be randomly produced, that's where the name \textit{stochastic} comes from.
\begin{algorithm}[h]
\caption{Alternating Least Squares with Full Entries and Stochastic Gradient Descent}
\label{alg:als-regularizer-missing-stochas-gradient-realstoch}
\begin{algorithmic}[1]
\Require $\bA\in \real^{M\times N}$;
\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
\State choose a stop criterion on the approximation error $\delta$;
\State choose regularization parameters $\lambda_w, \lambda_z$, and step size $\eta_w, \eta_z$;
\State choose maximal number of iterations $C$;
\State $iter=0$;
\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
\State $iter=iter+1$;
\For{$n=1,2,\ldots, N$}
\For{$m=1,2,\ldots, M$} \Comment{in practice, $m,n$ can be randomly produced}
\State $\bz_n= \bz_n - \eta_z \frac{\nabla l(\bz_n)}{||\nabla l(\bz_n)||}$;\Comment{$n$-th column of $\bZ$}
\State $\bw_m= \bw_m - \eta_w \frac{\nabla l(\bw_m)}{||\nabla l(\bw_m)||}$;\Comment{$m$-th column of $\bW^\top$}
\EndFor
\EndFor

\EndWhile
\State Output $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$;
\end{algorithmic}
\end{algorithm}
\section{Bias Term}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.95\textwidth]{./imgs/als-bias.pdf}
	\caption{Bias terms in alternating least squares where the \textcolor{canaryyellow}{yellow} entries denote ones (which are fixed) and \textcolor{cyan}{cyan} entries denote the added features to fit the bias terms. The dotted boxes give an example on how the bias terms work.}
	\label{fig:als-bias}
\end{figure}
In ordinary least squares, a bias term is added to the raw matrix as shown in Equation~\eqref{equation:ls-bias}. A similar idea can be applied to the ALS problem. We can add a fixed column with all 1's to the last column of $\bW$, thus an extra row should be added to last row of $\bZ$ to fit the features introduced by the bias term in $\bW$. Analogously, a fixed row with all 1's can be added to the first row of $\bZ$, and an extra column in the first column of $\bW$ to fit the features. The situation is shown in Figure~\ref{fig:als-bias}.

Following from the loss with respect to the columns of $\bZ$ in Equation~\eqref{als:gradient-regularization-zn}, suppose $\widetildebz_n
=
\begin{bmatrix}
	1\\
	\bz_n
\end{bmatrix}
$ is the $n$-th column of $\widetildebZ$, we have
\begin{equation}
	\begin{aligned}
		L(\bz_n)  &=||\widetildebW\widetildebZ-\bA||^2 +\lambda_w ||\widetildebW||^2 + \lambda_z ||\widetildebZ||^2\\\
		&= \left\Vert
		\widetildebW
		\begin{bmatrix}
			1 \\
			\bz_n
		\end{bmatrix}-\ba_n
		\right\Vert^2
		+
		\underbrace{\lambda_z ||\widetildebz_n||^2}_{=\lambda_z ||\bz_n||^2+\lambda_z}
		+
		\sum_{i\neq n} ||\widetildebW\widetildebz_i-\ba_i||^2  + \lambda_z \sum_{i\neq n}||\widetildebz_i||^2   +    \lambda_w ||\widetildebW||^2 \\
		&= \left\Vert
		\begin{bmatrix}
			\widebarbw_0 & \widebarbW
		\end{bmatrix}
		\begin{bmatrix}
			1 \\
			\bz_n
		\end{bmatrix}-\ba_n
		\right\Vert^2
		+ \lambda_z ||\bz_n||^2 + C_{z_n}
		= \left\Vert
		\widebarbW \bz_n -
		\underbrace{(\ba_n-\widebarbw_0)}_{\widebarba_n}
		\right\Vert^2
		+ \lambda_z ||\bz_n||^2 + C_{z_n},
	\end{aligned}
\end{equation}
where $\widebarbw_0$ is the first column of $\widetildebW$ and $C_{z_n}$ is a constant with respect to $\bz_n$. Let $\widebarba_n = \ba_n-\widebarbw_0$, the update of $\bz_n$ is just like the one in Equation~\eqref{als:gradient-regularization-zn} where the differential is given by:
$$
\frac{\partial L(\bz_n)}{\partial \bz_n} = 2\widebarbW^\top\widebarbW\bz_n - 2\widebarbW^\top\widebarba_n + 2\lambda_z\bz_n.
$$
Therefore the update on $\bz_n$ is given by the root of the above differential:
$$
\text{update on $\widetildebz_n$}=
\left\{
\begin{aligned}
	\bz_n &= (\widebarbW^\top\widebarbW+ \lambda_z\bI)^{-1} \widebarbW^\top \widebarba_n, \gap \text{for $n\in \{1,2,\ldots, N\}$};\\
	\widetildebz_n &= \begin{bmatrix}
		1\\\bz_n
	\end{bmatrix}.
\end{aligned}
\right.
$$
Similarly, follow from the loss with respect to each row of $\bW$ in Equation~\eqref{als:gradient-regularization-wd}, suppose $\widetildebw_m =
\begin{bmatrix}
	\bw_m \\
	1
\end{bmatrix}$ is the $m$-th row of $\widetildebW$ (or $m$-th column of $\widetildebW^\top$), we have
\begin{equation}
	\begin{aligned}
		L(\bw_m )
		&=||\widetildebZ^\top\widetildebW-\bA^\top||^2 +\lambda_w ||\widetildebW^\top||^2 + \lambda_z ||\widetildebZ||^2\\\
		&=
		||\widetildebZ^\top\widetildebw_m-\bb_m||^2 +
		\underbrace{\lambda_w ||\widetildebw_m||^2}_{=\lambda_w ||\bw_m||^2+\lambda_w}
		+
		\sum_{i\neq m} ||\widetildebZ^\top\widetildebw_i-\bb_i||^2  + \lambda_w \sum_{i\neq m}||\widetildebw_i||^2   +    \lambda_z ||\widetildebZ||^2 \\
		&=
		\left\Vert
		\begin{bmatrix}
			\widebarbZ^\top&
			\widebarbz_0
		\end{bmatrix}
		\begin{bmatrix}
			\bw_m \\
			1
		\end{bmatrix}
		-\bb_m\right\Vert^2 +
		\lambda_w ||\bw_m||^2
		+
		C_{w_m}\\
		&=
		\left\Vert
		\widebarbZ^\top\bw_m
		-(\bb_m-\widebarbz_0) \right\Vert^2+
		\lambda_w ||\bw_m||^2
		+
		C_{w_m},
	\end{aligned}
\end{equation}
where $\widebarbz_0$ is the last column of $\widetildebZ^\top$ and $\widebarbZ^\top$ is the left columns of it: $\widetildebZ^\top=\begin{bmatrix}
	\bw_m \\
	1
\end{bmatrix}$,
$C_{w_m}$ is a constant with respect to $\bw_m$, and $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M], \bA^\top=[\bb_1,\bb_2, \ldots, \bb_M]$ are the column partitions of $\bW^\top, \bA^\top$ respectively. Let $\widebarbb_m = \bb_m-\widebarbz_0$, the update of $\bw_m$ is again just like the on in Equation~\eqref{als:gradient-regularization-wd} where the differential is given by:
$$
\frac{\partial L(\bw_md)}{\partial \bw_m} = 2\widebarbZ\cdot \widebarbZ^\top\bw_m - 2\widebarbZ\cdot \widebarbb_m + 2\lambda_w\bw_m.
$$
Therefore the update on $\bw_m$ is given by the root of the above differential
$$
\text{update on $\widetildebw_m$}=
\left\{
\begin{aligned}
	\bw_m&=(\widebarbZ\cdot \widebarbZ^\top+\lambda_w\bI)^{-1}\widebarbZ\cdot \widebarbb_m,  \gap \text{for $m\in \{1,2,\ldots, M\}$} ;\\
	\widetildebw_m &= \begin{bmatrix}
		\bw_m \\ 1
	\end{bmatrix}.
\end{aligned}
\right.
$$
Similar updates by gradient descent under the bias terms or treatment on missing entries can be deduced and we shall not repeat the details (see Section~\ref{section:als-gradie-descent} and \ref{section:alt-columb-by-column} for a reference).
\section{Applications}

\subsection{Low-Rank Approximation}\label{section:als-low-flag}
We have discussed and compared the effects of SVD and pseudoskeleton on low-rank approximation in Section~\ref{section:svd-low-rank-approxi} (p.~\pageref{section:svd-low-rank-approxi}). The image to be compressed is shown in Figure~\ref{fig:eng300} with size of $600\times 1200$ and rank 402. Figure~\ref{fig:svdd-by-parts} shows the image reconstructed by the first singular value already approximates the original image very well. Figure~\ref{fig:svdd-pseudoskeleton} shows the difference of each compression with rank 90, 60, 30, 10. We find SVD does well with rank 90, 60, 30. Pseudoskeleton compresses well in the black horizontal and vertical lines in the image. But it performs poor in the details of the flag.

\noindent
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[$\sigma_1\bu_1\bv_1^\top$\protect\newline$F_1=60217$]{\label{fig:svd12}
		\includegraphics[width=0.15\linewidth]{./imgs/svd_pic1.png}}
	\subfigure[$\sigma_2\bu_2\bv_2^\top$\protect\newline$F_2=120150$]{\label{fig:svd22}
		\includegraphics[width=0.15\linewidth]{./imgs/svd_pic2.png}}
	\subfigure[$\sigma_3\bu_3\bv_3^\top$\protect\newline$F_3=124141$]{\label{fig:svd32}
		\includegraphics[width=0.15\linewidth]{./imgs/svd_pic3.png}}
	\subfigure[$\sigma_4\bu_4\bv_4^\top$\protect\newline$F_4=125937$]{\label{fig:svd42}
		\includegraphics[width=0.15\linewidth]{./imgs/svd_pic4.png}}
	\subfigure[$\sigma_5\bu_5\bv_5^\top$\protect\newline$F_5=126127$]{\label{fig:svd52}
		\includegraphics[width=0.15\linewidth]{./imgs/svd_pic5.png}}
	\subfigure[All 5 singular values: $\sum_{i=1}^{5}\sigma_i\bu_i\bv_i^\top$,\protect\newline$F=$\textbf{44379}]{\label{fig:svd62}
		\includegraphics[width=0.15\linewidth]{./imgs/svd_pic6_all.png}}
	\quad
\subfigure[$\bc_1\br_1^\top$\protect\newline$G_1=60464$]{\label{fig:skeleton51}
	\includegraphics[width=0.15\linewidth]{./imgs/skeleton_5_1.png}}
\subfigure[$\bc_2\br_2^\top$\protect\newline$G_2=122142$]{\label{fig:skeleton52}
	\includegraphics[width=0.15\linewidth]{./imgs/skeleton_5_2.png}}
\subfigure[$\bc_3\br_3^\top$\protect\newline$G_3=123450$]{\label{fig:skeleton53}
	\includegraphics[width=0.15\linewidth]{./imgs/skeleton_5_3.png}}
\subfigure[$\bc_4\br_4^\top$\protect\newline$G_4=125975$]{\label{fig:skeleton54}
	\includegraphics[width=0.15\linewidth]{./imgs/skeleton_5_5.png}}
\subfigure[$\bc_5\br_5^\top$\protect\newline$G_5=124794$]{\label{fig:skeleton55}
	\includegraphics[width=0.15\linewidth]{./imgs/skeleton_5_4.png}}
\subfigure[Pseudoskeleton Rank 5 $\sum_{i=1}^{5}\bc_i\br_i^\top$,\protect\newline$G=45905$.]{\label{fig:skeleton5_all}
	\includegraphics[width=0.15\linewidth]{./imgs/skeleton_5_all.png}}
	\quad
	\subfigure[$\bw_1\bz_1^\top$\protect\newline$S_1=82727$]{\label{fig:als51}
		\includegraphics[width=0.15\linewidth]{./imgs/als_rank5_3.png}}
	\subfigure[$\bw_2\bz_2^\top$\protect\newline$S_2=107355$]{\label{fig:als52}
		\includegraphics[width=0.15\linewidth]{./imgs/als_rank5_2.png}}
	\subfigure[$\bw_3\bz_3^\top$\protect\newline$S_3=119138$]{\label{fig:als53}
		\includegraphics[width=0.15\linewidth]{./imgs/als_rank5_5.png}}
	\subfigure[$\bw_4\bz_4^\top$\protect\newline$S_4=120022$]{\label{fig:als54}
		\includegraphics[width=0.15\linewidth]{./imgs/als_rank5_1.png}}
	\subfigure[$\bw_5\bz_5^\top$\protect\newline$S_5=120280$]{\label{fig:als55}
		\includegraphics[width=0.15\linewidth]{./imgs/als_rank5_4.png}}
	\subfigure[ALS Rank 5 $\sum_{i=1}^{5}\bw_i\bz_i^\top$,\protect\newline$S=52157$.]{\label{fig:als5_all}
		\includegraphics[width=0.15\linewidth]{./imgs/als_rank3.png}}
	\caption{Image compression for gray flag image into a rank-5 matrix via the SVD, and decompose into 5 parts where $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_{5}$, i.e., $F_1\leq F_2\leq \ldots \leq F_5$ with $F_i = ||\sigma_i\bu_i\bv^\top - \bA||_F$ for $i\in \{1,2,\ldots, 5\}$. And reconstruct images by single singular value and its corresponding left and right singular vectors, $\bc_i\br_i^\top$, $\bw_i\bz_i^\top$ respectively.}
	\label{fig:svdd-by-parts-als}
\end{figure}

Similar results can be observed for the low-rank approximation via the ALS decomposition. The ALS approximation is given by $\bA\approx\bW\bZ$ where $\bW\in \real^{m\times \gamma}, \bZ\in \real^{\gamma\times n}$ if $\bA\in \real^{m\times n}$ such that $\bW$ and $\bZ$ are rank-$\gamma$ matrices. Suppose $\gamma=5$, and
$$
\bW=[\bw_1, \bw_2, \ldots, \bw_5],
\qquad
\text{and}
\qquad
\bZ =
\begin{bmatrix}
	\bz_1^\top \\
	\bz_2^\top \\
	\vdots \\
	\bz_5^\top
\end{bmatrix},
$$
are the column and row partitions of $\bW, \bZ$ respectively \footnote{For simplicity, note that this definition is different to what we have defined in Section~\ref{section:als-netflix} where we define $\bw_i$ as the rows of $\bW$. }. Then $\bA$ can be approximated by $\sum_{i=1}^{5}\bw_i\bz_i^\top$. The partitions are ordered such that
$$
\underbrace{||\bw_1\bz_1^\top-\bA||_F}_{S_1} \leq
\underbrace{||\bw_2\bz_2^\top-\bA||_F}_{S_2}
\leq \ldots \leq
\underbrace{||\bw_5\bz_5^\top-\bA||_F}_{S_5}.
$$
We observe that $\bw_1\bz_1^\top$ works slightly \textbf{different} to that of $\sigma_1\bu_1\bv^\top$ where the reconstruction error measured by Frobenius norm are not close as well (82,727 in the pseudoskeleton case compared to that of 60,217 in the SVD case). As we mentioned previously, $\bc_1\br_1^\top$ works similarly to that of $\sigma_1\bu_1\bv^\top$ since the pseudoskeleton relies on the SVD. However, in ALS, the reconstruction is all from least squares optimization.
The key difference between ALS and SVD is in that, in SVD, the importance of each vector in the basis is relative to the value of the singular value associated with that vector. This usually means that the first vector of the basis dominates and is the most used vector to reconstruct data, then the second vector and so on, so the basis in SVD has an implicit hierarchy and that doesn't happen in ALS where we find the second component $\bw_2\bz_2^\top$ via ALS in Figure~\ref{fig:als52} plays an important role in the reconstruction of the original figure, whereas the second component $\sigma_2\bu_2\bv_2^\top$ via SVD in Figure~\ref{fig:svd22} plays a small role in the reconstruction.
\begin{SCfigure}
	\caption{Comparison of reconstruction errors measured by Frobenius norm among the SVD, pseudoskeleton, and ALS where the approximated rank ranges from $3$ to 100. ALS with well-selected parameters works similar to SVD.}
	\includegraphics[width=0.5\textwidth]{./imgs/svd_skeleton_als_fnorm.pdf}
	\label{fig:svd_skeleton_als_fnorm}
\end{SCfigure}

We finally compare low-rank approximation among the SVD, pseudoskeleton, and ALS with different ranks. Figure~\ref{fig:svdd-pseudoskeleton-als} shows the difference of each compression with rank 90, 60, 30, 10. We observe that the SVD does well with rank 90, 60, 30. The pseudoskeleton-approximation compresses well in the black horizontal and vertical lines in the image. But it performs poor in the details of the flag. ALS works similarly to the SVD in terms of visual expression and reconstruction errors measured by Frobenius norm.
Figure~\ref{fig:svd_skeleton_als_fnorm} shows the comparison of the reconstruction errors among the SVD, the pseudoskeleton, and the ALS approximations measured by Frobenius norm ranging from rank $3$ to $100$ where we find in all cases, the truncated SVD does best in terms of Frobenius norm. Again, similar results can be observed when applied to spectral norm. The ALS works better than the pseudoskeleton decomposition when $\lambda_w=\lambda_z=0.15$. An interesting cutoff happens when $\lambda_w=\lambda_z=\{0.03, 0.08, 0.15\}$. That is, when the rank increases, the ALS will be very close to the SVD in the sense of low-rank approximation.

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[SVD with rank 90\protect\newline Frobenius norm=\textbf{6,498}]{\label{fig:svd902}
		\includegraphics[width=0.3\linewidth]{./imgs/svd90.png}}
	\quad
	\subfigure[Pseudoskeleton with rank 90\protect\newline Frobenius norm=13,751]{\label{fig:skeleton902}
		\includegraphics[width=0.3\linewidth]{./imgs/skeleton90.png}}
	\subfigure[ALS with rank 90\protect\newline Frobenius norm=6,622]{\label{fig:als_90}
		\includegraphics[width=0.3\linewidth]{./imgs/als_rank90.png}}\\
	\subfigure[SVD with rank 60\protect\newline Frobenius norm=\textbf{8,956}]{\label{fig:svd602}
		\includegraphics[width=0.3\linewidth]{./imgs/svd60.png}}
	\quad
	\subfigure[Pseudoskeleton with rank 60\protect\newline Frobenius norm=14,217]{\label{fig:skeleton602}
		\includegraphics[width=0.3\linewidth]{./imgs/skeleton60.png}}
	\subfigure[ALS with rank 60\protect\newline Frobenius norm=9,028]{\label{fig:als_60}
		\includegraphics[width=0.3\linewidth]{./imgs/als_rank60.png}}\\
	\subfigure[SVD with rank 30\protect\newline Frobenius norm=\textbf{14,586}]{\label{fig:svd302}
		\includegraphics[width=0.3\linewidth]{./imgs/svd30.png}}
	\quad
	\subfigure[Pseudoskeleton with rank 30\protect\newline Frobenius norm=17,853]{\label{fig:skeleton302}
		\includegraphics[width=0.3\linewidth]{./imgs/skeleton30.png}}
	\subfigure[ALS with rank 30\protect\newline Frobenius norm=18,624]{\label{fig:als_30}
		\includegraphics[width=0.3\linewidth]{./imgs/als_rank30.png}}
	\subfigure[SVD with rank 10\protect\newline Frobenius norm=\textbf{31,402}]{\label{fig:svd102}
		\includegraphics[width=0.3\linewidth]{./imgs/svd10.png}}
	\quad
	\subfigure[Pseudoskeleton with rank 10\protect\newline Frobenius norm=33,797]{\label{fig:skeleton102}
		\includegraphics[width=0.3\linewidth]{./imgs/skeleton10.png}}
	\subfigure[ALS with rank 10\protect\newline Frobenius norm=33,449]{\label{fig:als_10}
		\includegraphics[width=0.3\linewidth]{./imgs/als_rank10.png}}
	\caption{Image compression for gray flag image with different rank.}
	\label{fig:svdd-pseudoskeleton-als}
\end{figure}

\subsection{Movie Recommender}
The ALS is extensively developed for the movie recommender system. To see this, we obtain
the ``movielens100k" dataset from MovieLens \citep{harper2015movielens}\footnote{http://grouplens.org}. It consists of 100,000 ratings from 943 users on 1,682 movies. The rating values go from 0 to 5. The data was collected through the MovieLens website
during the seven-month period from September 19th,
1997 through April 22nd, 1998. This data has been cleaned up - users
who had less than 20 ratings or did not have complete demographic
information were removed from this data set such that simple demographic info for the users (age, gender, occupation, zip) can be obtained. However, we will only work on the trivial rating matrix.

The dataset is split into training and validation data, around 95,015 and 4,985 ratings respectively.
The error is measured by root mean squared error (RMSE). The RMSE is frequently used as a measure of the differences between values. For a set of values $\{x_1, x_2, \ldots, x_n\}$ and its predictions $\{\hat{x}_1, \hat{x}_2, \ldots, \hat{x}_n\}$, the RMSE can be described as
$$
\text{RMSE}(\bx, \hat{\bx}) = \sqrt{\frac{1}{n} \sum_{i=1}^{n}(x_i-\hat{x}_i)}.
$$
The minimal RMSE for validation is obtained when $K=185$ and $\lambda_w=\lambda_z=0.15$, and it is equal to $0.806$ as shown in Figure~\ref{fig:movie100k}. Therefore, when the rating ranges from 0 to 5, the ALS at least can predict whether the user likes to watch the movie (ranges 4 to 5) or not much (ranges 0 to 2).
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Training]{\label{fig:movie100k1}
		\includegraphics[width=0.44\linewidth]{./imgs/movielen100k.pdf}}
	\quad
	\subfigure[Validation]{\label{fig:movie100k2}
		\includegraphics[width=0.44\linewidth]{./imgs/movielen100k_val.pdf}}
	\caption{Comparison of training and validation error for ``movielens100k" data set with different reduction dimension and regularization parameters.}
	\label{fig:movie100k}
\end{figure}

A recommender system can work simply by pushing the movie $m$ when $a_{mn}\geq4$ if user $n$ have not rated the movie $m$. Or in rare cases, it happens that the user $n$ have rated all the movies he likes (say rates $\geq4$). Then, a partial solution is to find out similar movies to high-rating movies to push. Suppose user $n$ likes movie $m$ very much and he has rated with $5$: $a_{mn}=5$. Under the ALS approximation $\bA=\bW\bZ$ where each row of $\bW$ represents the hidden features of each movie (see Section~\ref{section:als-vector-product} on the vector product). The solution is given by finding the most similar movies to movie $m$ that user $n$ has not rated (or watched). In mathematical language,
$$
\mathop{\arg \max}_{\bw_i} \gap \text{similarity}(\bw_i, \bw_m), \qquad \text{for all} \gap i \neq \bo_n,
$$
where $\bw_i$'s are the rows of $\bW$ representing the hidden feature of movie $i$ and $\bo_n$ is a mask vector indicating the movies that user $n$ has rated.

The method above relies on the similarity function between two vectors. The \textit{cosine similarity} is the most commonly used measure. It is defined to equal the cosine of the angle between the two vectors:
$$
\cos(\bx, \by) = \frac{\bx^\top\by}{||\bx||\cdot ||\by||},
$$
where the value ranges from $-1$ to 1 with -1 is perfectly dissimilar and 1 is perfectly similar. From the above definition, it follows that the cosine similarity depends only on the angle between the two non-zero vectors, but not on their magnitudes since it can be regarded as the inner product between the normalized vectors. A second measure for similarity is known as the \textit{Pearson similarity}:
$$
\text{Pearson}(\bx,\by) =\frac{\Cov(\bx,\by)}{\sigma_x \cdot \sigma_y}
= \frac{\sum_{i=1}^{n} (x_i - \bar{x} ) (y_i -\bar{y})}{ \sqrt{\sum_{i=1}^{n} (x_i-\bar{x})^2}\sqrt{ \sum_{i=1}^{n} (y_i-\bar{y})^2 }},
$$
whose range varies between -1 and 1, where -1 is perfectly dissimilar and 1 is perfectly similar. The Pearson similarity is usually used to measure the linear correlation between two sets of data. It is the ratio between the covariance of two variables and the product of their standard deviations.
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Cosine Bin Plot]{\label{fig:als-cosine}%
		\includegraphics[width=0.33\linewidth]{./imgs/als-bin-cosine.pdf}}%
	\subfigure[Pearson Bin Plot]{\label{fig:als-pearson}%
		\includegraphics[width=0.33\linewidth]{./imgs/als-bin-pearson.pdf}}%
	\subfigure[PR Curve]{\label{fig:als-prcurve}%
		\includegraphics[width=0.33\linewidth]{./imgs/als-prcurve.pdf}}%
	\caption{Distribution of the insample and outsample under cosine and Pearson similarity and the Precision-Recall curve form them.}
	\label{fig:als-prcurive-bin}
\end{figure}

Following from the example above on the movielens100k dataset, we choose $\lambda_w=\lambda_z=0.15$ for the regularization and the rank $62$ to minimize the RMSE. We want to look at the similarity between different movie hidden vectors.
Define further the ``insample" as the similarity between the movies having rates $5$ for each user, and ``outsample" as the similarity between the movies having rate $5$ and $1$ for each user. Figure~\ref{fig:als-pearson} and \ref{fig:als-cosine} depict the bin plot of the distribution of insample and outsample under cosine and Pearson similarity. Figure~\ref{fig:als-prcurve} shows the precision-recall (PR) curve of them where we find the cosine similarity works better such that it can find out more than $73\%$ of the potential high-rating movies with a $90\%$ precision. However, Pearson similarity can only separate out about $64\%$ of the high-rating movies to have a $90\%$ precision. In practice, other measure can also be explored, such as negative Euclidean distance in which case the Euclidean distance can measure the ``dissimilarity" between two vectors, and a negative one thus represents the similarity of them.

\chapter{Acknowledgments}
We thank Gilbert Strang for raising the question formulated in Corollary~\ref{corollary:invertible-intersection}, checking the writing of the survey, for a stream of ideas and references about the three factorizations from the steps of elimination, and for the generous sharing of the manuscript of \citep{strang2021three}.

\newpage
\chapter{Appendix}

\chapter{Modern Applications}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Low-Rank Neural Networks}\label{section:low-rank-neural}
We start with the basic LeNet5 neural network to illustrate this low-rank neural networks idea. But we modify the fully connected layers as $LenetModified$ as shown below in the bluebox. Also, we notice that a layer with input feature 120, output feature 100 is just a matrix of size $120\times 100$. If we put a "regularize" on the matrix, such as the rank of the matrix is 50. All in all, a matrix with $120\times 100$ or a matrix via matrix multiplication of two matrices $120\times 50, 50\times 100$ are similar in some sense. The matrix multiplication result also has a size of $120\times 100$. Thus, we will have a low-rank version of the fully connected layer. We call this $LeNetDecom$ structured. The fully connected layers in $LenetModified$ and $LenetDecom$ are shown as follows:\index{Low-rank neural networks}
\begin{svgraybox}
\begin{lstlisting}
LenetModified{
	Convolutional Layers Omitted;
	(Fully Connected Layers):
	(0): Linear(in_features=120, out_features=100)
	(1): Tanh()
	(2): Linear(in_features=100, out_features=80)
	(3): Tanh()
	(4): Linear(in_features=80, out_features=60)
	(5): Tanh()
	(6): Linear(in_features=60, out_features=40)
	(7): Tanh()
	(8): Linear(in_features=40, out_features=10)
	(9): LogSoftmax()
}
\end{lstlisting}
\end{svgraybox}

\begin{svgraybox}
\begin{lstlisting}
LenetDecom{
	Convolutional Layers Omitted;
	(Fully Connected Layers):
	(0): Linear(in_features=120, out_features=50)
	(1): Linear(in_features=50, out_features=100)
	(2): Tanh()
	(3): Linear(in_features=100, out_features=40)
	(4): Linear(in_features=40, out_features=80)
	(5): Tanh()
	(6): Linear(in_features=80, out_features=30)
	(7): Linear(in_features=30, out_features=60)
	(8): Tanh()
	(9): Linear(in_features=60, out_features=20)
	(10): Linear(in_features=20, out_features=40)
	(11): Tanh()
	(12): Linear(in_features=40, out_features=10)
	(13): LogSoftmax()
}
\end{lstlisting}
\end{svgraybox}

We realize that reducing a fully connected layer with size $m\times n$ to a low-rank layer $m\times r, r\times n$ can reduce the space to save the model from $mn$ values to $r(m+n)$ values. In this specific example above, we reduce $120\times 100=12000$ to $50(120+100)=11000$ values. This reduction also involves the matrix multiplication operations.

After training 100 epochs, we find the minimal training loss in $LenetModified$ is smaller than that in $LenetDecom$ as shown in Figure~\ref{fig:lenetLoss_train}. But the validation loss of $LenetModified$ (larger than 0.04) is larger than that in $LenetDecom$ (smaller than 0.04) as shown in Figure~\ref{fig:lenetLoss_val}. We see this "regularization" property in low-rank neural networks from this simple example.

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Training loss]{\label{fig:lenetLoss_train}
		\includegraphics[width=0.47\linewidth]{./imgs/lenetLoss_train.pdf}}
	\quad
	\subfigure[Validation loss]{\label{fig:lenetLoss_val}
		\includegraphics[width=0.47\linewidth]{./imgs/lenetLoss_val.pdf}}
	\caption{Comparison of full neural networks and low-rank neural networks.}
	\label{fig:lenetLoss}
\end{figure}

Further, if we have already trained $LenetModified$, and we want to use $LenetDecom$ as it saves space and matrix operations. To avoid training from scratch, we could load the weights from $LenetModified$ to $LenetDecom$. Then, one method could be used is applying SVD to decompose the weight matrix and keep only the first $r$ singular values, i,e., set the singular values $\sigma_{r+1}, \sigma_{r+2}, \ldots=0$. That is, a weight matrix $\bW = \bU_r\bSigma_r\bV_r = \bW_1\bW_2$, where we can set $\bW_1=\bU_r\bSigma_r, \bW_2=\bV_r$. Then we load the convolutional layers of $LenetDecom$ from the convolutional layers of $LenetModified$, and load the fully connected layers of $LenetDecom$ via the decomposition of $LenetModified$. The fine-tuning of the network will make the result better. We denote this method as $LenetDecomSVD$. The training and validation losses of $LenetDecomSVD$ are shown in Figure~\ref{fig:lenetLoss}. We find the training loss of $LenetDecomSVD$ approaches 0 even in epoch 1 and the validation loss of $LenetDecomSVD$ is also smaller than 0.04 which is better than the original $LenetModified$.

This is just one simple example of how the low-rank neural network works. We do not claim that the low-rank neural networks work better in all the scenarios. And also we can apply this matrix decomposition into convolutional layers via the equivalence of convolutional layer and fully connected layers in \citep{ma2017equivalence}. Also, a more detailed exploration of going deep in neural networks is explained in \citep{chen2015net2net, wei2016network, lu2018compnet}

\section{One More Step: Adding a Nonlinear Function Layer}
In the above section, we approximate a fully connected layer by a multiplication of two low-rank matrices. The further goal can be putting a nonlinear function in between the two low-rank matrices, i.e., the final layer would be $f(\bW_1) \bW_2$, where $f()$ is a nonlinear function. We denote this method for the above example as $LenetDecomNonlinear$.\index{Nonlinear function layer}

Moreover, if we already trained $LenetModified$, we can also modify the structure of previously trained models to a new structure to avoid training from scratch every time. We then again factor the fully networks by matrix decomposition.

\begin{SCfigure
	\centering
	\includegraphics[width=0.5\textwidth]{imgs/tanh.pdf}
	\caption{Demonstration of y=Tanh(x) vs y=x. We notice that in the input field between $(-0.25, 0.25)$, $Tanh$ is almost close to a linear function.}
	\label{fig:tanh}
\end{SCfigure}

Specifically, when we use $Tanh$ function in neural networks as the nonlinear function, where $Tanh(x) = \frac{e^x-e^{-x}}{e^x + e^{-x}}$. We notice that in the input field between $(-0.25, 0.25)$, $Tanh$ is almost close to a linear function as shown in Figure~\ref{fig:tanh} where the orange line is the difference between $y=Tanh(x)$ and $y=x$ and the difference is almost 0 when $x\in (-0.25, 0.25)$.

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Training loss]{\label{fig:nonlinear_lenetLoss_train}
		\includegraphics[width=0.47\linewidth]{./imgs/lenetLoss_train_Tanh.pdf}}
	\quad
	\subfigure[Validation loss]{\label{fig:nonlinear_lenetLoss_val}
		\includegraphics[width=0.47\linewidth]{./imgs/lenetLoss_val_Tanh.pdf}}
	\caption{Comparison of full neural networks and $Tanh$ in low-rank neural networks.}
	\label{fig:nonlinear_lenetLoss}
\end{figure}

After the matrix multiplication of $\bW=\bW_1\bW_2$, if we can make all the values $\bA_{in}\bW_1$ in the range of $(-0.25, 0.25)$ where $\bA_1$ is the output from the previous layer, then we can add the $Tanh$ without any suffer as it is in the linear space of $Tanh$. Note that matrix decomposition has this equivalence: $\bW = \bW_1*\bW_2 = (\sigma \bW_1)*(\frac{1}{\sigma}\bW_2)$ where $\sigma$ is any nonzero scalar. Then we can set the value of $\sigma$ to make the maximal absolute value of $\bA_{in}(\sigma \bW_1)$ to be 0.25. Following the specific example in the previous section, we denote this method as $LenetDecomNonlinear\_SVD$ where we put an extra $Tanh$ function in between the factored matrices as follows:
\begin{svgraybox}
\begin{lstlisting}
LenetDecomNonlinear{
	Convolutional Layers Omitted;
	(Fully Connected Layers):
	(0): Linear(in_features=120, out_features=50)
	(1): Tanh()  [Differece]
	(2): Linear(in_features=50, out_features=100)
	(3): Tanh()
	(4): Linear(in_features=100, out_features=40)
	(5): Tanh() [Differece]
	(6): Linear(in_features=40, out_features=80)
	(7): Tanh()
	(8): Linear(in_features=80, out_features=30)
	(9): Tanh() [Differece]
	(10): Linear(in_features=30, out_features=60)
	(11): Tanh()
	(12): Linear(in_features=60, out_features=20)
	(13): Tanh() [Differece]
	(14): Linear(in_features=20, out_features=40)
	(15): Tanh()
	(16): Linear(in_features=40, out_features=10)
	(17): LogSoftmax()
}
\end{lstlisting}
\end{svgraybox}

After training 100 epochs, we find the minimal training loss in $LenetModified$,\\ $LenetDecomNonlinear$, and $LenetDecomNonlinear\_SVD$ are similar as shown in Figure~\ref{fig:nonlinear_lenetLoss_train}. And the validation loss in $LenetModified$ and $LenetDecomNonlinear$ are also similar which are larger than 0.04. However, the minimal validation loss in \\ $LenetDecomNonlinear\_SVD$ is smaller than 0.04 which gives us a promising result for this matrix decomposition method used in neural networks  as shown in Figure~\ref{fig:nonlinear_lenetLoss_val}. Again, we refer the readers to \citep{chen2015net2net, wei2016network, lu2018compnet} for more details about these neural architecture search methods.

\chapter{Biconjugate Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup
\section{Existence of the Biconjugate Decomposition}
The biconjugate decomposition was proposed in \citep{chu1995rank} and discussed in \citep{yang2000matrix}.
The existence of the biconjugate decomposition relies on the rank-one reduction theorem shown below. And a variety of matrix decomposition methods can be unified via this biconjugate decomposition.
\begin{theorem}[Rank-One Reduction\index{Rank-one reduction}]\label{theorem:rank-1-reduction}
Any $m\times n$ matrix $\bA\in \real^{m\times n}$ with rank $r$, a pair of vectors $\bx\in \real^n$ and $\by\in \real^m$ such that $w=\by^\top\bA\bx \neq 0$, then the matrix $\bB=\bA-w^{-1}\bx\by^\top\bA$ has rank $r-1$ which has exactly one less than the rank of $\bA$.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:rank-1-reduction}]
If we can show that the dimension of $\nspace(\bB)$ is one larger than that of $\bA$. Then this implicitly shows $\bB$ has rank exactly one less than the rank of $\bA$.

For any vector $\bn \in \nspace(\bA)$, i.e., $\bA\bn=\bzero$, we then have $\bB\bn =\bA\bn-w^{-1}\bx\by^\top\bA\bn=\bzero $ which means $\nspace(\bA)\subseteq \nspace(\bB)$.

Now for any vector $\bmm \in \nspace(\bB)$, then $\bB\bmm = \bA\bmm-w^{-1}\bx\by^\top\bA\bmm =\bzero$.

Let $k=w^{-1}\by^\top\bA\bmm$, which is a scalar, thus $\bA(\bmm - k\bx)=\bzero$, i.e., for any vector $\bn\in \nspace(\bA)$, we could find a vector $\bmm\in \nspace(\bB)$ such that $\bn=(\bmm - k\bx)\in \nspace(\bA)$. Note that $\bA\bx\neq \bzero$ from the definition of $w$. Thus, the null space of $\bB$ is therefore obtained from the null space of $\bA$ by adding $\bx$ to its basis which will increase the order of the space by 1. Thus the dimension of $\nspace(\bA)$ is smaller than the dimension of $\nspace(\bB)$ by 1 which completes the proof.
\end{proof}

Suppose matrix $\bA\in \real^{m\times n}$ has rank $r$, we can define a rank reducing process to generate a sequence of Wedderburn matrices $\{\bA_k\}$:
$$
\bA_1 = \bA, \qquad \text{and}\qquad \bA_{k+1} = \bA_k-w_k^{-1}\bA_k\bx_k\by_k^\top\bA_k,
$$
where $\bx_k \in \real^n$ and $\by_k\in \real^m$ are any vectors satisfying $w_k = \by_k^\top\bA\bx_k \neq 0$. The sequence will terminate in $r$ steps since the rank of $\bA_k$ decreases by exactly one at each step. Write out the sequence:
$$
\begin{aligned}
	\bA_1 &= \bA, \\
\bA_1-\bA_{2} &= w_1^{-1}\bA_1\bx_1\by_1^\top\bA_1,\\
\bA_2-\bA_3 &=w_2^{-1}\bA_2\bx_2\by_2^\top\bA_2, \\
\bA_3-\bA_4 &=w_3^{-1}\bA_3\bx_3\by_3^\top\bA_3, \\
\vdots &=\vdots\\
\bA_{r-1}-\bA_{r} &=w_{r-1}^{-1}\bA_{r-1}\bx_{r-1}\by_{r-1}^\top\bA_{r-1}, \\
\bA_r-\bzero &=w_{r}^{-1}\bA_{r}\bx_{r}\by_{r}^\top\bA_{r}.
\end{aligned}
$$
By adding the sequence we will get
$$
(\bA_1-\bA_2)+(\bA_2-\bA_3)+\ldots+(\bA_{r-1}-\bA_{r})+(\bA_r-\bzero ) =\bA= \sum_{i=1}^{r}w_i^{-1}\bA_i\bx_i\by_i^\top\bA_i.
$$
\begin{theoremHigh}[Biconjugate Decomposition: Form 1]\label{theorem:biconjugate-form1}
This equality from rank-reducing process implies the following matrix decomposition
$$
\bA = \bPhi \bOmega^{-1} \bPsi^\top,
$$
where $\bOmega=diag(w_1, w_2, \ldots, w_r)$, $\bPhi=[\bphi_1,\bphi_2, \ldots, \bphi_r]\in \real^{m\times r}$ and $\bPsi=[\bpsi_1, \bpsi_2, \ldots, \bpsi_r]$ with
$$
\bphi_k = \bA_k\bx_k, \qquad \text{and}\qquad \bpsi_k=\bA_k^\top \by_k.
$$
\end{theoremHigh}
Obviously, different choices of $\bx_k$'s and $\by_k$'s will result in different factorizations. So this factorization is rather general and we will show its connection to some well-known decomposition methods.\index{Wedderburn sequence}

\begin{remark}
For the vectors $\bx_k, \by_k$ in the Wedderburn sequence, we have the following property
$$
\begin{aligned}
\bx_k &\in \nspace(\bA_{k+1}) \bot \cspace(\bA_{k+1}^\top), \\
\by_k &\in \nspace(\bA_{k+1}^\top) \bot \cspace(\bA_{k+1}).
\end{aligned}
$$
\end{remark}

\begin{lemma}[General Term Formula of Wedderburn Sequence: V1]\label{lemma:wedderburn-sequence-general}
For each matrix with $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$, then $\bA_{k+1} $ can be written as
$$
\bA_{k+1} = \bA - \sum_{i=1}^{k}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA,
$$
where
$$
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{w_i}\bu_i,\qquad \text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\by_k^\top \bA\bu_i}{w_i}\bv_i.
$$
\end{lemma}

The proof of this lemma is provided in Appendix~\ref{appendix:wedderburn-general-term}. We notice that $w_i =\by_k^\top\bA_k\bx_k$ in the general term formula is related to $\bA_k$. So it's not the true general term formula. We will write $w_i$ to be related to $\bA$ rather than $\bA_k$ later.
From the general term formula of Wedderburn sequence, we have
$$
\begin{aligned}
\bA_{k+1} &= \bA - \sum_{i=1}^{k}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA \\
\bA_{k} &= \bA - \sum_{i=1}^{k-1}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA.
\end{aligned}
$$
Thus, $\bA_{k+1} - \bA_{k} = -w_{k}^{-1} \bA\bu_k \bv_k^\top \bA$. Since we define the sequence by $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$. We then find $w_{k}^{-1} \bA\bu_k \bv_k^\top \bA = w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$. It is trivial to see
\begin{equation}\label{equation:wedderburn-au-akxk}
\begin{aligned}
\bA\bu_k &=\bA_k\bx_k,\\
\bv_k^\top \bA&=\by_k^\top \bA_k.
\end{aligned}
\end{equation}

Let $z_{k,i} = \frac{\bv_i^\top \bA\bx_k}{w_i}$ which is a scalar. From the definition of $\bu_k$ and $\bv_k$ in the above lemma, then

$\bullet$ $\bu_1=\bx_1$;

$\bullet$ $\bu_2 = \bx_2 - z_{2,1}\bu_1$;

$\bullet$ $\bu_3 = \bx_3 - z_{3,1}\bu_1-z_{3,2}\bu_2$;

$\bullet$ $\ldots$.

This process is just similar to the Gram-Schmidt process. But now, we do not project $\bx_2$ onto $\bx_1$ with the smallest distance. The vector of $\bx_2$ along $\bx_1$ is now defined by $z_{2,1}$. This process is shown in Figure~\ref{fig:projection-wedd}. In Figure~\ref{fig:project-line-wedd}, $\bu_2$ is not perpendicular to $\bu_1$. But $\bu_2$ does not lie on the same line of $\bu_1$ so that $\bu_1, \bu_2$ still could span a $\real^2$ subspace. Similarly, in Figure~\ref{fig:project-space-wedd}, $\bu_3= \bx_3 - z_{3,1}\bu_1-z_{3,2}\bu_2$ does not lie in the space spanned by $\bu_1, \bu_2$ so that $\bu_1, \bu_2, \bu_3$ could still span a $\real^3$ subspace.

A moment of reflexion would reveal that the span of $\bx_2, \bx_1$ is the same as the span of $\bu_2, \bu_1$. Similarly for $\bv_i$'s. We have the following property:
\begin{equation}\label{equation:wedderburn-span-same}
	\left\{
\begin{aligned}
	\textrm{span}\{\bx_1, \bx_2, \ldots, \bx_j\} &= \textrm{span}\{\bu_1, \bu_2, \ldots, \bu_j\};\\
	\textrm{span}\{\by_1, \by_2, \ldots, \by_j\} &= \textrm{span}\{\bv_1, \bv_2, \ldots, \bv_j\}.\\
\end{aligned}
\right.
\end{equation}

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[``Project" onto a line]{\label{fig:project-line-wedd}
		\includegraphics[width=0.47\linewidth]{./imgs/projectline_wedderburn.pdf}}
	\quad
	\subfigure[``Project" onto a space]{\label{fig:project-space-wedd}
		\includegraphics[width=0.47\linewidth]{./imgs/projectspace_wedderburn.pdf}}
	\caption{``Project" a vector onto a line and onto a space.}
	\label{fig:projection-wedd}
\end{figure}

Further, from the rank-reducing property in the Wedderburn sequence, we have
$$
\left\{
\begin{aligned}
	\cspace(\bA_1) &\supset \cspace(\bA_2) \supset \cspace(\bA_3) \supset \ldots;\\
	\nspace(\bA_1^\top) &\subset \nspace(\bA_2^\top) \subset \nspace(\bA_3^\top) \subset \ldots.
\end{aligned}
\right.
$$
Since $\by_k \in \nspace(\bA_{k+1}^\top)$, it then follows that $\by_j \in \nspace(\bA_{k+1}^\top)$ for all $j<k+1$, i.e., $\bA_{k+1}^\top \by_j=\bzero$ for all $j<k+1$. Which also holds true for $\bx_{k+1}^\top \bA_{k+1}^\top \by_j=0$ for all $j<k+1$. From Equation~\eqref{equation:wedderburn-au-akxk}, we also have $\bu_{k+1}^\top \bA^\top \by_j=0$ for all $j<k+1$. Following from Equation~\eqref{equation:wedderburn-span-same}, we obtain
$$
\bv_j^\top\bA\bu_{k+1}=0 \text{ for all } j<k+1.
$$
Similarly, we can prove
$$
\bv_{k+1}^\top\bA\bu_{j}=0 \text{ for all } j<k+1.
$$
Moreover, we defined $w_k = \by_k^\top \bA_k\bx_k$. By Equation~\eqref{equation:wedderburn-au-akxk}, we can write the $w_k$ as:
$$
\begin{aligned}
	w_k &= \by_k^\top \bA_k\bx_k\\
	&=\bv_k^\top \bA\bx_k \\
	&=\bv_k^\top\bA (\bu_k +\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{w_i}\bu_i) \qquad &(\text{by the definition of }\bu_k \text{ in Lemma~\ref{lemma:wedderburn-sequence-general}})\\
	&=\bv_k^\top\bA \bu_k,\qquad &(\text{by } \bv_{k}^\top\bA\bu_{j}=0 \text{ for all } j<k)
\end{aligned}
$$
which can be used to substitute the $w_k$ in Lemma~\ref{lemma:wedderburn-sequence-general} and we then have the full version of the general term formula of the Wedderburn sequence such that the formula does not depend on $\bA_k$'s (in the form of $w_k$'s) with
\begin{equation}\label{equation:uk-vk-to-mimic-gram-process}
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{\textcolor{blue}{\bv_i^\top\bA\bu_i}}\bu_i,\qquad \text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\by_k^\top \bA\bu_i}{\textcolor{blue}{\bv_i^\top\bA\bu_i}}\bv_i.
\end{equation}

\textbf{Gram-Schmidt Process from Wedderburn Sequence}: If $\bX=[\bx_1,
\bx_2, \ldots, \bx_r]\in \real^{n\times r}$, $\bY=[\by_1, \by_2, \ldots, \by_r]\in \real^{m\times r}$ effects a rank-reducing process for $\bA$. Let $\bA$ be the identity matrix and $(\bX, \bY)$ are identical and contain the vectors for which an orthogonal basis is desired, then $(\bU = \bV)$ give the resultant orthogonal basis.

This form of $\bu_k$ and $\bv_k$ in Equation~\eqref{equation:uk-vk-to-mimic-gram-process} is very close to the projection to the perpendicular space of the Gram-Schmidt process in Equation~\eqref{equation:gram-schdt-eq2}. We then define \fbox{$<\bx, \by>:=\by^\top\bA\bx$} to explicitly mimic the form of projection in Equation~\eqref{equation:gram-schdt-eq2}.
We formulate the results so far in the following lemma which can help us have a clear vision about what we have been working on and we will use these results extensively in the sequel:
\begin{lemma}[Properties of Wedderburn Sequence]\label{lemma:wedderburn-sequence-general-v2}
For each matrix with $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$, then $\bA_{k+1} $ can be written as
$$
\bA_{k+1} = \bA - \sum_{i=1}^{k}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA,
$$
where
\begin{equation}\label{equation:properties-of-wedderburn-ukvk}
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\textcolor{blue}{<\bx_k, \bv_i>}}{\textcolor{blue}{<\bu_i,\bv_i>}}\bu_i,\qquad
\text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\textcolor{blue}{<\bu_i,\by_k>}}{\textcolor{blue}{<\bu_i,\bv_i>}}\bv_i.
\end{equation}
Further, we have the following properties:
\begin{equation}\label{equation:wedderburn-au-akxk-2}
	\begin{aligned}
		\bA\bu_k &=\bA_k\bx_k,\\
		\bv_k^\top \bA&=\by_k^\top \bA_k.
	\end{aligned}
\end{equation}
\begin{equation}
<\bu_k, \bv_j>=<\bu_j, \bv_k>=0 \text{ for all } j<k.
\end{equation}
\begin{equation}\label{equation:wk-by-ukvk}
	w_k = \by_k^\top\bA_k\bx_k = <\bu_k, \bv_k>
\end{equation}

\end{lemma}

By substituting Equation~\eqref{equation:wedderburn-au-akxk-2} into Form 1 of biconjugate decomposition, and using Equation~\eqref{equation:wk-by-ukvk} which implies $w_k = \bv_k^\top\bA\bu_k$, we have the Form 2 and Form 3 of this decomposition:
\begin{theoremHigh}[Biconjugate Decomposition: Form 2 and Form 3]\label{theorem:biconjugate-form2}
The equality from rank-reducing process implies the following matrix decomposition
$$
\bA = \bA\bU_r \bOmega_r^{-1} \bV_r^\top\bA,
$$
where $\bOmega_r=diag(w_1, w_2, \ldots, w_r)$, $\bU_r=[\bu_1,\bu_2, \ldots, \bu_r]\in \real^{m\times r}$ and  $\bV_r=[\bv_1, \bv_2,$ $\ldots,$ $\bv_r] \in \real^{n\times r}$ with
\begin{equation}\label{equation:properties-of-wedderburn-ukvk2-inform2}
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{<\bx_k, \bv_i>}{<\bu_i,\bv_i>}\bu_i,\qquad
\text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{<\bu_i,\by_k>}{<\bu_i,\bv_i>}\bv_i.
\end{equation}
And also the following decomposition
\begin{equation}\label{equation:wedderburn-vgamma-ugamma}
\bV_\gamma^\top \bA \bU_\gamma = \bOmega_\gamma,
\end{equation}
where $\bOmega_\gamma=diag(w_1, w_2, \ldots, w_\gamma)$, $\bU_\gamma=[\bu_1,\bu_2, \ldots, \bu_\gamma]\in \real^{m\times \gamma}$ and $\bV_\gamma=[\bv_1, \bv_2,$ $\ldots,$ $\bv_\gamma]\in \real^{n\times \gamma}$. Note the difference between the subscripts $r$ and $\gamma$ we used here with $\gamma \leq r$.
\end{theoremHigh}
We notice that, in these two forms of biconjugate decomposition, they are independent of the Wedderburn matrices $\{\bA_k\}$.

\textbf{A word on the notation}: we will use the subscript to indicate the dimension of the matrix avoiding confusion in the sequel, e.g., the $r, \gamma$ in the above theorem.

\section{Properties of the Biconjugate Decomposition}
\begin{corollary}[Connection of $\bU_\gamma$ and $\bX_\gamma$]\label{corollary:biconjugate-connection-u-x}
If $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$ effects a rank-reducing process for $\bA$, then there are unique unit upper triangular matrices $\bR_\gamma^{(x)}\in\real^{\gamma\times \gamma}$ and $\bR_\gamma^{(y)}\in\real^{\gamma\times \gamma}$ such that
$$
\bX_\gamma = \bU_\gamma \bR_\gamma^{(x)}, \qquad \text{and} \qquad \bY_\gamma=\bV_\gamma\bR_\gamma^{(y)},
$$
where $\bU_\gamma$ and $\bV_\gamma$ are matrices with columns resulting from the Wedderburn sequence as in Equation~\eqref{equation:wedderburn-vgamma-ugamma}.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:biconjugate-connection-u-x}]
The proof is trivial from the definition of $\bu_k$ and $\bv_k$ in Equation~\eqref{equation:properties-of-wedderburn-ukvk} or Equation~\eqref{equation:properties-of-wedderburn-ukvk2-inform2} by setting the $j$-th column of $\bR_\gamma^{(x)}$ and $\bR_\gamma^{(y)}$ as
$$
\left[\frac{<\bx_j,\bv_1>}{<\bu_1,\bv_1>}, \frac{<\bx_j,\bv_2>}{<\bu_2,\bv_2>}, \ldots, \frac{<\bx_j,\bv_{j-1}>}{<\bu_{j-1},\bv_{j-1}>}, 1, 0, 0, \ldots, 0  \right]^\top,
$$
and
$$
\left[\frac{<\bu_1, \by_j>}{<\bu_1,\bv_1>},\frac{<\bu_2, \by_j>}{<\bu_2,\bv_2>}, \ldots, \frac{<\bu_{j-1}, \by_j>}{<\bu_{j-1},\bv_{j-1}>}, 1, 0, 0, \ldots, 0  \right]^\top.
$$
This completes the proof.
\end{proof}

The $(\bU_\gamma, \bV_\gamma) \in \real^{m\times \gamma}\times \real^{n\times \gamma}$ in Theorem~\ref{theorem:biconjugate-form2} is called a \textbf{biconjugate pair} with respect to $\bA$ if $\Omega_\gamma$ is nonsingular and diagonal. And let $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$ effect a rank-reducing process for $\bA$, then $(\bX_\gamma, \bY_\gamma)$ is said to be \textbf{biconjugatable} and \textbf{biconjugated into a biconjugate pair} of matrices $(\bU_\gamma, \bV_\gamma)$, if there exist unit upper triangular matrices $\bR_\gamma^{(x)},\bR_\gamma^{(y)}$ such that $\bX_\gamma = \bU_\gamma \bR_\gamma^{(x)}$ and $\bY_\gamma=\bV_\gamma\bR_\gamma^{(y)}$.

\section{Connection to Well-Known Decomposition Methods}

\subsection{LDU Decomposition}
\begin{theorem}[LDU, \cite{chu1995rank} Theorem 2.4]\label{theorem:biconjugate-ldu}
If $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$
and $\bA\in \real^{m\times n}$ with $\gamma$ in $\{1, 2, \ldots, r\}$. Then $(\bX_\gamma, \bY_\gamma)$ can be biconjugated if and only if $\bY_\gamma^\top\bA\bX_\gamma$ has an LDU decomposition.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:biconjugate-ldu}]
Suppose $\bX_\gamma$ and $\bY_\gamma$ are biconjugatable, then, there exists a unit upper triangular matrices $\brx$ and $\bry$ such that $\bxgamma = \bugamma\brx$, $\bygamma = \bvgamma\bry$ and $\bvgamma^\top\bA\bugamma = \bomegagamma$ is a nonsingular diagonal matrix. Then, it follows that
$$
\bygamma^\top\bA\bxgamma = \bryt \bvgamma^\top \bA \bugamma\brx = \bryt \bomegagamma \brx
$$
is the unique unit triangular LDU decomposition of $\bygamma^\top\bA\bxgamma$. This form above can be seen as the \textbf{fourth form of biconjugate decomposition}, thus we put the proof into a graybox.

Conversely, suppose $\bygamma^\top\bA\bxgamma = \bR_2^\top \bD\bR_1$ is an LDU decomposition with both $\bR_1$ and $\bR_2$ being unit upper triangular matrices. Then since $\bR_1^{-1}$ and $\bR_2^{-1}$ are also unit upper triangular matrices, and $(\bX_\gamma, \bY_\gamma)$ biconjugates into $(\bX_\gamma\bR_1^{-1}, \bY_\gamma\bR_2^{-1})$.
\end{proof}

\begin{corollary}[Determinant]\label{corollary:lu-determinant}
Suppose $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$ are biconjugatable. Then
$$
\det(\bygamma^\top \bA\bxgamma) = \prod_{i=1}^{\gamma} w_i.
$$
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:lu-determinant}]
By Theorem~\ref{theorem:biconjugate-ldu}, since $(\bX_\gamma, \bY_\gamma)$ are biconjugatable, then there are unit upper triangular matrices $\brx$ and $\bry$ such that $\bygamma^\top\bA\bxgamma = \bryt \bomegagamma \brx$. The determinant is just product of the trace.
\end{proof}

\begin{lemma}[Biconjugatable in Principal Minors]\label{lemma:Biconjugatable-in-Principal-Minors}
Let $r=rank(\bA) \geq \gamma$ with $\bA\in \real^{m\times n}$. In the Wedderburn sequence, take $\bx_i$ as the $i$-th basis in $\real^n$ for $i \in \{1, 2, \ldots, \gamma\}$ (i.e., $\bx_i = \be_i \in \real^n$) and $\by_i$ as the $i$-th basis in $\real^m$ for $i \in \{1, 2, \ldots, \gamma\}$ (i.e., $\by_i=\be_i \in
\real^m$). That is $\bygamma^\top \bA\bxgamma$ is the leading principal submatrix of $\bA$, i.e., $\bygamma^\top \bA\bxgamma = \bA_{1:\gamma, 1:\gamma}$. Then, $(\bX_\gamma, \bY_\gamma)$ is biconjugatable if and only if the $\gamma$-th leading principal minor of $\bA$ is nonzero. In this case, the $\gamma$-th leading principal minor of $\bA$ is given by $\prod_{i=1}^{\gamma} w_i$.\index{Leading principal minors}
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:Biconjugatable-in-Principal-Minors}]
The proof is trivial that the $\gamma$-th leading principal minor of $\bA$ is nonzero will imply that $w_i \neq 0$ for all $i\leq \gamma$. Thus the Wedderburn sequence can be successfully obtained. The converse holds since Corollary~\ref{corollary:lu-determinant} implies $\det(\bygamma^\top \bA\bxgamma)$ is nonzero.
\end{proof}

We thus finally come to the LDU decomposition for square matrices.
\begin{theorem}[LDU: Biconjugate Decomposition for Square Matrices]\label{theorem:biconjugate-square-ldu}
For any matrix $\bA\in \real^{n\times n}$, $(\bI_n, \bI_n)$ is biconjugatable if and only if all the leading principal minors of $\bA$ are nonzero. In this case, $\bA$ can be factored as
$$
\bA = \bV_n^{-\top} \bOmega_n \bU_n^{-1} = \bL\bD\bU,
$$
where $\bOmega_n = \bD$ is a diagonal matrix with nonzero values on the diagonal, $\bV_n^{-\top} = \bL$ is a unit lower triangular matrix and $\bU_n^{-1} = \bU$ is a unit upper triangular matrix.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:biconjugate-square-ldu}]
From Lemma~\ref{lemma:Biconjugatable-in-Principal-Minors}, it is trivial that $(\bI_n, \bI_n)$ is biconjugatable. From Corollary~\ref{corollary:biconjugate-connection-u-x}, we have $\bU_n \bR_n^{(x)} = \bI_n$ and $\bI_n=\bV_n\bR_n^{(y)}$, thus $\bR_n^{(x)} = \bU_n^{-1}$ and $\bR_n^{(y)} = \bV_n^{-1}$ are well defined and we complete the proof.
\end{proof}

\subsection{Cholesky Decomposition}
For symmetric and positive definite, the leading principal minors are positive for sure. The proof is provided in Section~\ref{appendix:leading-minors-pd}.
\begin{theorem}[Cholesky: Biconjugate Decomposition for PD Matrices]\label{theorem:biconjugate-square-cholesky}
For any symmetric and positive definite matrix $\bA\in \real^{n\times n}$, the Cholesky decomposition of $\bA$ can be obtained from the Wedderburn sequence applied to $(\bI_n, \bI_n)$ as $(\bX_n, \bY_n)$.
In this case, $\bA$ can be factored as
$$
\bA = \bU_n^{-\top} \bOmega_n \bU_n^{-1} = (\bU_n^{-\top} \bOmega_n^{1/2})( \bOmega_n^{1/2} \bU_n^{-1}) =\bR^\top\bR,
$$
where $\bOmega_n$ is a diagonal matrix with positive values on the diagonal, and $\bU^{-1}$ is a unit upper triangular matrix.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:biconjugate-square-cholesky}]
Since the leading principal minors of positive definite matrices are positive, $w_i>0$ for all $i\in \{1, 2,\ldots, n\}$. It can be easily verified via the LDU from biconjugation decomposition and the symmetric property of $\bA$ that $\bA = \bU_n^{-\top} \bOmega_n \bU_n^{-1}$. And since $w_i$'s are positive, thus $\bOmega_n$ is positive definite and can be factored as $\bOmega_n = \bOmega_n^{1/2}\bOmega_n^{1/2}$ which implies $\bOmega_n^{1/2} \bU_n^{-1}$ is the Cholesky factor.
\end{proof}

\subsection{QR Decomposition}

Without loss of generality, we shall assume that $\bA\in \real^{n\times n}$ has full column rank so that the columns of $\bA$ can be factored as $\bA=\bQ\bR$ with $\bQ, \bR \in \real^{n\times n}$

\begin{theorem}[QR: Biconjugate Decomposition for Nonsingular Matrices]\label{theorem:biconjugate-square-qr}
For any nonsingular matrix $\bA\in \real^{n\times n}$, the QR decomposition of $\bA$ can be obtained from the Wedderburn sequence applied to $(\bI_n, \bA)$ as $(\bX_n, \bY_n)$.
In this case, $\bA$ can be factored as
$$
\bA = \bQ\bR,
$$
where $\bQ=\bV_n \bOmega_n^{-1/2}$ is an orthogonal matrix and $\bR = \bOmega_n^{1/2}\bR_n^{(x)}$ is an upper triangular matrix with \textcolor{blue}{Form 4} in Theorem~\ref{theorem:biconjugate-ldu} and let $\gamma=n$
$$
\bY_n^\top\bA\bX_n = \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)}=\bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}.
$$
where we set $\gamma=n$ since $\gamma$ is any value that $\gamma\leq r$ and the rank $r=n$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:biconjugate-square-qr}]
Since $(\bX_n, \bY_n) = (\bI_n, \bA)$. Then
By Theorem~\ref{theorem:biconjugate-ldu}, we have the decomposition
$$
\bY_n^\top\bA\bX_n = \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)}=\bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}.
$$
Substitute $(\bI_n, \bA)$ into above decomposition, we have
\begin{equation}\label{equation:biconjugate-qr-ata1}
\begin{aligned}
	\bY_n^\top\bA\bX_n &= \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)} = \bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}\\
	\bA^\top \bA &= \bR_n^{(y)\top} \bOmega_n \bR_n^{(x)} \\
	\bA^\top \bA &= \bR_1^\top \bOmega_n \bR_1 \qquad (\text{$\bA^\top\bA$ is symmetric and let $\bR_1=\bR_n^{(x)}=\bR_n^{(y)}$})\\
	\bA^\top \bA &= (\bR_1^\top \bOmega_n^{1/2\top}) (\bOmega_n^{1/2}\bR_1) \\
	\bA^\top \bA &= \bR^\top\bR. \qquad (\text{Let $\bR = \bOmega_n^{1/2}\bR_1$})
\end{aligned}
\end{equation}
To see why $\bOmega_n$ can be factored as $\bOmega_n = \bOmega_n^{1/2\top}\bOmega_n^{1/2}$.
Suppose $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$. We obtain $w_i = \ba_i^\top \ba_i>0$ since $\bA$ is nonsingular. Thus $\bOmega_n=diag(w_1, w_2, \ldots, w_n)$ is positive definite and can be factored as
\begin{equation}\label{equation:omega-half-qr}
\bOmega_n = \bOmega_n^{1/2}\bOmega_n^{1/2}= \bOmega_n^{1/2\top}\bOmega_n^{1/2}.
\end{equation}

By $\bxgamma = \bugamma\brx$ in Theorem~\ref{theorem:biconjugate-ldu} for all $\gamma\in \{1, 2, \ldots, n\}$, we have
$$
\begin{aligned}
\bX_n &= \bU_n\bR_1 \\
\bI_n &= \bU_n\bR_1, \qquad (\text{Since $\bX_n = \bI_n$}) \\
\bU_n &= \bR_1^{-1}
\end{aligned}
$$
By $\bygamma = \bvgamma\bry$ in Theorem~\ref{theorem:biconjugate-ldu}  for all $\gamma\in \{1, 2, \ldots, n\}$, we have
\begin{equation}\label{equation:biconjugate-qr-ata2}
\begin{aligned}
	\bY_n &= \bV_n\bR_1\\
	\bA &= \bV_n\bR_1, \qquad &(\text{$\bA=\bY_n$}) \\
	\bA^\top \bA &= \bR_1^\top\bV_n^\top \bV_n\bR_1 \\
	\bR_1^\top \bOmega_n \bR_1&=\bR_1^\top\bV_n^\top \bV_n\bR_1, \qquad &(\text{From Equation~\eqref{equation:biconjugate-qr-ata1}})\\
	(\bR_1^\top \bOmega_n^{1/2\top}) (\bOmega_n^{1/2}\bR_1) &= (\bR_1^\top \bOmega_n^{1/2\top} \bOmega_n^{-1/2\top})\bV_n^\top \bV_n (\bOmega_n^{-1/2}\bOmega_n^{1/2}\bR_1), \qquad &\text{(From Equation~\eqref{equation:omega-half-qr})} \\
	\bR^\top\bR &= \bR^\top (\bOmega_n^{-1/2\top} \bV_n^\top) (\bV_n \bOmega_n^{-1/2})  \bR
\end{aligned}
\end{equation}
Thus, $\bQ=\bV_n \bOmega_n^{-1/2}$ is an orthogonal matrix.
\end{proof}

\subsection{SVD}
To differentiate the notation, let $\bA=\bU^\mathrm{svd} \bSigma^\mathrm{svd} \bV^{\mathrm{svd}\top}$ be the SVD of $\bA$ where $\bU^\mathrm{svd} = [\bu_1^\mathrm{svd}, \bu_2^\mathrm{svd}, \ldots, \bu_n^\mathrm{svd}]$, $\bV^\mathrm{svd} = [\bv_1^\mathrm{svd}, \bv_2^\mathrm{svd}, \ldots, \bv_n^\mathrm{svd}]$ and $\bSigma^\mathrm{svd} = diag(\sigma_1, \sigma_2, \ldots, \sigma_n)$. Without loss of generality, we assume $\bA\in \real^{n\times n}$ and $rank(\bA)=n$. Readers can prove the equivalence for $\bA\in \real^{m\times n}$.

If $\bX_n=\bV^\mathrm{svd}$, $\bY_n=\bU^\mathrm{svd}$ effects a rank-reducing process for $\bA$.
From the definition of $\bu_k$ and $\bv_k$ in Equation~\eqref{equation:properties-of-wedderburn-ukvk} or Equation~\eqref{equation:properties-of-wedderburn-ukvk2-inform2}, we have
$$
\bu_k = \bv_k^\mathrm{svd} \qquad \text{and} \qquad \bv_k = \bu_k^\mathrm{svd} \qquad \text{and} \qquad w_k = \by_k^\top \bA \bx_k=\sigma_k.
$$
That is $\bV_n = \bU^\mathrm{svd}$, $\bU_n = \bV^\mathrm{svd}$, and $\bOmega_n = \bSigma^\mathrm{svd}$, where we set $\gamma=n$ since $\gamma$ is any value that $\gamma\leq r$ and the rank $r=n$.

By $\bX_n= \bU_n\bR_n^{(x)}$ in Theorem~\ref{theorem:biconjugate-ldu}, we have
$$
\bX_n = \bU_n\bR_n^{(x)} \leadto
\bV^\mathrm{svd} = \bV^\mathrm{svd}\bR_n^{(x)}	 \leadto
\bI_n = \bR_n^{(x)}
$$
By $\bY_n = \bV_n\bR_n^{(y)}$ in Theorem~\ref{theorem:biconjugate-ldu}, we have
$$
\bY_n = \bV_n\bR_n^{(y)}\leadto
\bU^\mathrm{svd} = \bU^\mathrm{svd}\bR_n^{(y)} \leadto
\bI_n=\bR_n^{(y)}
$$
Again, from Theorem~\ref{theorem:biconjugate-ldu} and let $\gamma=n$, we have
$$
\bY_n^\top\bA\bX_n = \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)}=\bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}.
$$
That is
$$
\bU^{\mathrm{svd}\top}\bA \bV^\mathrm{svd} = \bSigma^\mathrm{svd},
$$
which is exactly the form of SVD and we prove the equivalence of SVD and biconjugate decomposition when the Wedderburn sequence is applied to $(\bV^\mathrm{svd}, \bU^\mathrm{svd})$ as $(\bX_n, \bY_n)$.

\chapter{Cholesky Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Cholesky Decomposition}
Positive definiteness or positive semidefiniteness is one of the highest accolades to which a matrix can aspire.
In this section, we will introduce decompositional approaches for the two special kinds of matrices and we first illustrate the most famous Cholesky decomposition as follows.
\begin{theoremHigh}[Cholesky Decomposition]\label{theorem:cholesky-factor-exist}
	Every positive definite matrix $\bA\in \real^{n\times n}$ can be factored as
	$$
	\bA = \bR^\top\bR,
	$$
	where $\bR \in \real^{n\times n}$ is an upper triangular matrix \textbf{with positive diagonal elements}. This decomposition is known as the \textbf{Cholesky decomposition}  of $\bA$. $\bR$ is known as the \textbf{Cholesky factor} or \textbf{Cholesky triangle} of $\bA$.

	\item Alternatively, $\bA$ can be factored as $\bA=\bL\bL^\top$ where $\bL=\bR^\top$ is a lower triangular matrix \textbf{with positive diagonals}.

	\item Specifically, the Cholesky decomposition is unique (Corollary~\ref{corollary:unique-cholesky-main}, p.~\pageref{corollary:unique-cholesky-main}).
\end{theoremHigh}

The Cholesky decomposition is named after a French military officer and mathematician, Andr\'{e}-Louis Cholesky (1875-1918), who developed the Cholesky decomposition in his surveying work. Similar to the LU decomposition for solving linear systems, the Cholesky decomposition is further used primarily to solve positive definite linear systems. The development on the solution is similar to that of the LU decomposition in Section~\ref{section:lu-linear-sistem} (p.~\pageref{section:lu-linear-sistem}), and we shall not repeat the details.

\section{Existence of the Cholesky Decomposition via Recursive Calculation}

In this section, we will prove the existence of the Cholesky decomposition via recursive calculation. In Section~\ref{section:cholesky-by-qr-spectral} (p.~\pageref{section:cholesky-by-qr-spectral}), we will also prove the existence of the Cholesky decomposition via the QR decomposition and spectral decomposition.

Before showing the existence of Cholesky decomposition, we need the following definitions and lemmas.
\begin{definition}[Positive Definite and Positive Semidefinite\index{Positive definite}\index{Positive semidefinite}]\label{definition:psd-pd-defini}
	A matrix $\bA\in \real^{n\times n}$ is positive definite (PD) if $\bx^\top\bA\bx>0$ for all nonzero $\bx\in \real^n$.
	And a matrix $\bA\in \real^{n\times n}$ is positive semidefinite (PSD) if $\bx^\top\bA\bx \geq 0$ for all $\bx\in \real^n$.
\end{definition}

One of the prerequisites for the Cholesky decomposition is the definition of the above positive definiteness of a matrix. We sketch several properties of this PD matrix as follows:
\begin{tcolorbox}[title={Positive Definite Matrix Property 1 of 6}]
	We will show the equivalent definition on the positive definiteness of a matrix $\bA$ is that $\bA$ only has positive eigenvalues, or on the positive semidefiniteness of a matrix $\bA$ is that $\bA$ only has nonnegative eigenvalues. The proof is provided in Section~\ref{section:equivalent-pd-psd} (p.~\pageref{section:equivalent-pd-psd}) as a consequence of the spectral theorem.
\end{tcolorbox}

\begin{tcolorbox}[title={Positive Definite Matrix Property 2 of 6}]

	\begin{lemma}[Positive Diagonals of Positive Definite Matrices]\label{lemma:positive-in-pd}
		The diagonal elements of a positive definite matrix $\bA$ are all \textbf{positive}. And similarly, the diagonal elements of a positive semidefinite matrix $\bB$ are all \textbf{nonnegative}.
	\end{lemma}

\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:positive-in-pd}]
	From the definition of positive definite matrices, we have $\bx^\top\bA \bx >0$ for all nonzero $\bx$. In particular, let $\bx=\be_i$ where $\be_i$ is the $i$-th unit vector with the $i$-th entry being equal to 1 and other entries being equal to 0. Then,
	$$
	\be_i^\top\bA \be_i = a_{ii}>0, \qquad \forall i \in \{1, 2, \ldots, n\},
	$$
	where $a_{ii}$ is the $i$-th diagonal component. The proof for the second part follows similarly. This completes the proof.
\end{proof}

The complete pivoting in the Cholesky decomposition is simpler than that in the LU decomposition in the sense that the maximal value in a positive definite matrix is on the diagonal
\begin{tcolorbox}[title={Positive Definite Matrix Property 3 of 6}]
	\begin{lemma}[Maximal Value of Positive Definite Matrices]\label{lemma:positive-maximal-diagonal}
		The maximum element in a positive definite matrix lies on the diagonal. And this argument works similarly to positive semidefinite matrices.
	\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:positive-maximal-diagonal}]
	Suppose $\be_i, \be_j$ are the $i$-th and $j$-th unit vector, and $a_{ij}$ being the entry $(i,j)$ of the positive definite matrix $\bA$. Then, it follows that
	$$
	(\be_i-\be_j)^\top\bA(\be_i-\be_j) = a_{ii}+a_{jj}-2a_{ij}>0.
	$$
	Therefore, either $a_{ii}$ or $a_{jj}$ is larger than $a_{ij}$. If we loop around all the entries, the result follows. When it comes to the positive semidefinite matrices, it follows that the largest element appear on the diagonal, with possibility that some non-diagonal elements are equal to the largest element.
\end{proof}

\begin{tcolorbox}[title={Positive Definite Matrix Property 4 of 6}]

	\begin{lemma}[Schur Complement of Positive Definite Matrices\index{Schur complement}]\label{lemma:pd-of-schur}
		For any positive definite matrix $\bA\in \real^{n\times n}$, its Schur complement of $\bA_{11}$ is given by $\bS_{n-1}=\bA_{2:n,2:n}-\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top$ which is also positive definite.

		\paragraph{A word on the notation} Note that the subscript $n-1$ of $\bS_{n-1}$ means it is of size $(n-1)\times (n-1)$ and it is a Schur complement of an $n\times n$ positive definite matrix. We will use this notation in the following sections.
	\end{lemma}

\end{tcolorbox}

\begin{proof}[of Lemma~\ref{lemma:pd-of-schur}]
	For any nonzero vector $\bv\in \real^{n-1}$, we can construct a vector $\bx\in \real^n$ by the following equation:
	$$
	\bx =
	\begin{bmatrix}
		-\frac{1}{\bA_{11}} \bA_{2:n,1}^\top  \bv \\
		\bv
	\end{bmatrix},
	$$
	which is nonzero. Then
	$$
	\begin{aligned}
		\bx^\top\bA\bx
		&= \left[-\frac{1}{\bA_{11}} \bv^\top \bA_{2:n,1}\qquad \bv^\top\right]
		\begin{bmatrix}
			\bA_{11} & \bA_{2:n,1}^\top \\
			\bA_{2:n,1} & \bA_{2:n,2:n}
		\end{bmatrix}
		\begin{bmatrix}
			-\frac{1}{\bA_{11}} \bA_{2:n,1}^\top  \bv \\
			\bv
		\end{bmatrix} \\
		&= \left[-\frac{1}{\bA_{11}} \bv^\top \bA_{2:n,1}\qquad \bv^\top\right]
		\begin{bmatrix}
			0 \\
			\bS_{n-1}\bv
		\end{bmatrix} \\
		&= \bv^\top\bS_{n-1}\bv.
	\end{aligned}
	$$
	Since $\bA$ is positive definite, we have $\bx^\top\bA\bx = \bv^\top\bS_{n-1}\bv >0$ for all nonzero $\bv$. Thus, the Schur complement $\bS_{n-1}$ is positive definite as well.
\end{proof}

The above argument can be extended to PSD matrices as well. If $\bA$ is PSD, then the Schur complement $\bS_{n-1}$ is also PSD.

\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
	\paragraph{A word on the Schur complement} In the proof of Theorem~\ref{theorem:lu-factorization-with-permutation}, we have shown this Schur complement $\bS_{n-1}=\bA_{2:n,2:n}-\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top$ is also nonsingular if $\bA$ is nonsngular and $\bA_{11}\neq 0$. Similarly, the Schur complement of $\bA_{nn}$ in $\bA$ is $\bar{\bS}_{n-1} =\bA_{1:n-1,1:n-1} - \frac{1}{\bA_{nn}}\bA_{1:n-1,n} \bA_{1:n-1,n}^\top$ which is also positive definite if $\bA$ is positive definite. This property can help prove the fact that the leading principal minors of positive definite matrices are all positive. See Section~\ref{appendix:leading-minors-pd} for more details.
\end{mdframed}

We then prove the existence of Cholesky decomposition using these lemmas.
\begin{proof}[\textbf{of Theorem~\ref{theorem:cholesky-factor-exist}: Existence of Cholesky Decomposition Recursively}]
	For any positive definite matrix $\bA$, we can write out (since $\bA_{11}$ is positive by Lemma~\ref{lemma:positive-in-pd})
	$$
	\begin{aligned}
		\bA &=
		\begin{bmatrix}
			\bA_{11} & \bA_{2:n,1}^\top \\
			\bA_{2:n,1} & \bA_{2:n,2:n}
		\end{bmatrix} \\
		&=\begin{bmatrix}
			\sqrt{\bA_{11}} &\bzero\\
			\frac{1}{\sqrt{\bA_{11}}} \bA_{2:n,1} &\bI
		\end{bmatrix}
		\begin{bmatrix}
			\sqrt{\bA_{11}} & \frac{1}{\sqrt{\bA_{11}}}\bA_{2:n,1}^\top \\
			\bzero & \bA_{2:n,2:n}-\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top
		\end{bmatrix}\\
		&=\begin{bmatrix}
			\sqrt{\bA_{11}} &\bzero\\
			\frac{1}{\sqrt{\bA_{11}}} \bA_{2:n,1} &\bI
		\end{bmatrix}
		\begin{bmatrix}
			1 & \bzero \\
			\bzero & \bA_{2:n,2:n}-\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top
		\end{bmatrix}
		\begin{bmatrix}
			\sqrt{\bA_{11}} & \frac{1}{\sqrt{\bA_{11}}}\bA_{2:n,1}^\top \\
			\bzero & \bI
		\end{bmatrix}\\
		&=\bR_1^\top
		\begin{bmatrix}
			1 & \bzero \\
			\bzero & \bS_{n-1}
		\end{bmatrix}
		\bR_1.
	\end{aligned}
	$$
	where
	$$\bR_1 =
	\begin{bmatrix}
		\sqrt{\bA_{11}} & \frac{1}{\sqrt{\bA_{11}}}\bA_{2:n,1}^\top \\
		\bzero & \bI
	\end{bmatrix}.
	$$
	Since we proved the Schur complement $\bS_{n-1}$ is positive definite in Lemma~\ref{lemma:pd-of-schur}, then we can factor it in the same way as
	$$
	\bS_{n-1}=
	\hat{\bR}_2^\top
	\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bS_{n-2}
	\end{bmatrix}
	\hat{\bR}_2.
	$$
	Therefore, we have
	$$
	\begin{aligned}
		\bA &= \bR_1^\top
		\begin{bmatrix}
			1 & \bzero \\
			\bzero & \hat{\bR}_2^\top
			\begin{bmatrix}
				1 & \bzero \\
				\bzero & \bS_{n-2}
			\end{bmatrix}
			\hat{\bR}_2.
		\end{bmatrix}
		\bR_1\\
		&=
		\bR_1^\top
		\begin{bmatrix}
			1 &\bzero \\
			\bzero &\hat{\bR}_2^\top
		\end{bmatrix}
		\begin{bmatrix}
			1 &\bzero \\
			\bzero &\begin{bmatrix}
				1 & \bzero \\
				\bzero & \bS_{n-2}
			\end{bmatrix}
		\end{bmatrix}
		\begin{bmatrix}
			1 &\bzero \\
			\bzero &\hat{\bR}_2
		\end{bmatrix}
		\bR_1\\
		&=
		\bR_1^\top \bR_2^\top
		\begin{bmatrix}
			1 &\bzero \\
			\bzero &\begin{bmatrix}
				1 & \bzero \\
				\bzero & \bS_{n-2}
			\end{bmatrix}
		\end{bmatrix}
		\bR_2 \bR_1.
	\end{aligned}
	$$
	The same formula can be recursively applied. This process gradually continues down to the bottom-right corner giving us the decomposition
	$$
	\begin{aligned}
		\bA &= \bR_1^\top\bR_2^\top\ldots \bR_n^\top \bR_n\ldots \bR_2\bR_1\\
		&= \bR^\top \bR,
	\end{aligned}
	$$
	where $\bR_1, \bR_2, \ldots, \bR_n$ are upper triangular matrices with positive diagonal elements and $\bR=\bR_1\bR_2\ldots\bR_n$ is also an upper triangular matrix with positive diagonal elements from which the result follows.
\end{proof}
The process in the proof can also be used to calculate the Cholesky decomposition and compute the complexity of the algorithm. In Section~\ref{section:compute-cholesky}, we will do the computation in a similar way but from a different point of view.
\begin{lemma}[$\bR^\top\bR$ is PD]\label{lemma:r-to-pd}
	For any upper triangular matrix $\bR$ with positive diagonal elements, then
	$
	\bA = \bR^\top\bR
	$
	is positive definite.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:r-to-pd}]
	If an upper triangular matrix $\bR$ has positive diagonals, it has full column rank, and the null space of $\bR$ is of dimension 0 by the fundamental theorem of linear algebra (discussed in Appendix~\ref{appendix:fundamental-rank-nullity}, p.~\pageref{appendix:fundamental-rank-nullity}). As a result, $\bR\bx \neq \bzero$ for any nonzero vector $\bx$. Thus $\bx^\top\bA\bx = ||\bR\bx||^2 >0$ for any nonzero vector $\bx$.
\end{proof}
This corollary above works not only for the upper triangular matrices $\bR$, but can be extended to any $\bR$ with linearly independent columns.

\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
	\paragraph{A word on the two claims} Combine Theorem~\ref{theorem:cholesky-factor-exist} and Lemma~\ref{lemma:r-to-pd}, we can claim that matrix $\bA$ is positive definite if and only if $\bA$ can be factored as $\bA=\bR^\top\bR$ where $\bR$ is an upper triangular matrix with positive diagonals.
\end{mdframed}

\section{Sylvester's Criterion: Leading Principal Minors of PD Matrices}\label{appendix:leading-minors-pd}

In Lemma~\ref{lemma:pd-of-schur}, we proved for any positive definite matrix $\bA\in \real^{n\times n}$, its Schur complement of $\bA_{11}$ is $\bS_{n-1}=\bA_{2:n,2:n}-\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top$ and it is also positive definite.
This is also true for its Schur complement of $\bA_{nn}$, i.e., $\bS_{n-1}^\prime = \bA_{1:n-1,1:n-1} -\frac{1}{\bA_{nn}} \bA_{1:n-1,n}\bA_{1:n-1,n}^\top$ is also positive definite.

We then claim that all the leading principal minors (Definition~\ref{definition:leading-principle-minors}, p.~\pageref{definition:leading-principle-minors}) of a positive definite matrix $\bA \in \real^{n\times n}$ are positive. This is also known as the Sylvester's criterion \citep{swamy1973sylvester, gilbert1991positive}. Recall that these positive leading principal minors imply the existence of the LU decomposition for positive definite matrix $\bA$ by Theorem~\ref{theorem:lu-factorization-without-permutation} (p.~\pageref{theorem:lu-factorization-without-permutation}).

To show the Sylvester's criterion, we need the following lemma.
\begin{tcolorbox}[title={Positive Definite Matrix Property 5 of 6}]

	\begin{lemma}[Quadratic PD]\label{lemma:quadratic-pd}
		Let $\bE$ be any invertible matrix. Then $\bA$ is positive definite if and only if $\bE^\top\bA\bE$ is also positive definite.
	\end{lemma}

\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:quadratic-pd}]
	We will prove by forward implication and reverse implication separately as follows.
	\paragraph{Forward implication}
	Suppose $\bA$ is positive definite, then for any nonzero vector $\bx$, we have $\bx^\top \bE^\top\bA\bE \bx = \by^\top\bA\by > 0$, since $\bE$ is invertible such that $\bE\bx$ is nonzero. \footnote{Since the null space of $\bE$ is of dimension 0 and the only solution for $\bE\bx=\bzero$ is the trivial solution $\bx=0$.} This implies $\bE^\top\bA\bE$ is PD.

	\paragraph{Reverse implication}
	Conversely, suppose $\bE^\top\bA\bE$ is positive definite, for any nonzero $\bx$, we have $\bx^\top \bE^\top\bA\bE \bx>0$. For any nonzero $\by$, there exists a nonzero $\bx$ such that $\by =\bE\bx$ since $\bE$ is invertible. This implies $\bA$ is PD as well.
\end{proof}

We then provide the rigorous proof for Sylvester's criterion.
\begin{tcolorbox}[title={Positive Definite Matrix Property 6 of 6}]

	\begin{theorem}[Sylvester's Criterion\index{Sylvester's criterion}]\label{lemma:sylvester-criterion}
		The real symmetric matrix $\bA\in \real^{n\times n}$ is positive definite if and only if all the leading principal minors of $\bA$ are positive.

	\end{theorem}

\end{tcolorbox}

\begin{proof}[of Theorem~\ref{lemma:sylvester-criterion}]
	We will prove by forward implication and reverse implication separately as follows.
	\paragraph{Forward implication: } We will prove by induction for the forward implication. Suppose $\bA$ is positive definite. Since all the components on the diagonal of positive definite matrices are all positive (Lemma~\ref{lemma:positive-in-pd}, p.~\pageref{lemma:positive-in-pd}). The case for $n=1$ is trivial that $\det(\bA)> 0$ if $\bA$ is a scalar.

	Suppose all the leading principal minors for $k\times k$ matrices are all positive. If we could prove this is also true for $(k+1)\times (k+1)$ PD  matrices, then we complete the proof.

	For a $(k+1)\times (k+1)$ matrix with the block form $\bM=\begin{bmatrix}
		\bA & \bb\\
		\bb^\top & d
	\end{bmatrix}$, where $\bA$ is a $k\times k$ submatrix. Then its Schur complement of $d$, $\bS_{k} = \bA - \frac{1}{d} \bb\bb^\top$ is also positive definite and its determinant is positive from the assumption. Therefore, $\det(\bM) = \det(d)\det( \bA - \frac{1}{d} \bb\bb^\top) $=
	\footnote{By the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
			\bA & \bB \\
			\bC & \bD
		\end{bmatrix}$, then $\det(\bM) = \det(\bD)\det(\bA-\bB\bD^{-1}\bC)$.}
	$d\cdot \det( \bA - \frac{1}{d} \bb\bb^\top)>0$, which completes the proof.

	\paragraph{Reverse implication:} Conversely, suppose all the leading principal minors of $\bA\in \real^{n\times n}$ are positive, i.e., leading principal submatrices are nonsingular. Suppose further the $(i,j)$-th entry of $\bA$ is denoted by $a_{ij}$, we realize that $a_{11}>0$ by the assumption.

	Subtract multiples of the first row of $\bA$ to zero out the entries in the first column of $\bA$ below the first diagonal $a_{11}$. That is,
	$$
	\bA =
	\begin{bmatrix}
		a_{11} & a_{12} & \ldots &a_{1n}\\
		a_{21} & a_{22} & \ldots &a_{2n}\\
		\vdots & \vdots & \ddots &\vdots\\
		a_{n1} & a_{n2} & \ldots &a_{nn}\\
	\end{bmatrix}
	\stackrel{\bE_1 \bA}{\longrightarrow}
	\begin{bmatrix}
		a_{11} & a_{12} & \ldots &a_{1n}\\
		0 & a_{22} & \ldots &a_{2n}\\
		\vdots & \vdots & \ddots &\vdots\\
		0 & a_{n2} & \ldots &a_{nn}\\
	\end{bmatrix}.
	$$
	This operation preserves the values of the principal minors of $\bA$. The $\bE_1$ might be mysterious to the readers. Actually, the $\bE_1$ contains two steps $\bE_1 = \bZ_{12}\bZ_{11}$. The first step $\bZ_{11}$ is to subtract the 2-nd row to the $n$-th row by a multiple of the first row, that is
	$$
	\begin{aligned}
		\bA =
		\begin{bmatrix}
			a_{11} & a_{12} & \ldots &a_{1n}\\
			a_{21} & a_{22} & \ldots &a_{2n}\\
			\vdots & \vdots & \ddots &\vdots\\
			a_{n1} & a_{n2} & \ldots &a_{nn}\\
		\end{bmatrix}
		&\stackrel{\bZ_{11} \bA}{\longrightarrow}
		\begin{bmatrix}
			1 & 0 & \ldots &0\\
			-\frac{a_{21}}{a_{11}} & 1 & \ldots &0\\
			\vdots & \vdots & \ddots &\vdots\\
			-\frac{a_{n1}}{a_{11}} & 0 & \ldots &1\\
		\end{bmatrix}
		\begin{bmatrix}
			a_{11} & a_{12} & \ldots &a_{1n}\\
			a_{21} & a_{22} & \ldots &a_{2n}\\
			\vdots & \vdots & \ddots &\vdots\\
			a_{n1} & a_{n2} & \ldots &a_{nn}\\
		\end{bmatrix}\\
		&=
		\begin{bmatrix}
			a_{11} & a_{12} & \ldots &a_{1n}\\
			0 & (a_{22} -\frac{a_{21}}{a_{11}} a_{12})  & \ldots &(a_{2n}-\frac{a_{21}}{a_{11}} a_{1n})\\
			\vdots & \vdots & \ddots &\vdots\\
			0 & (a_{n2}-\frac{a_{n1}}{a_{11}}a_{12}) & \ldots &(a_{nn}-\frac{a_{n1}}{a_{11}}a_{1n})\\
		\end{bmatrix},
	\end{aligned}
	$$
	where we subtract the bottom-right $(n-1)\times (n-1)$ by some terms additionally. $\bZ_{12}$ is to add back these terms.
	$$
	\begin{aligned}
		\bZ_{11}\bA &=
		\begin{bmatrix}
			a_{11} & a_{12} & \ldots &a_{1n}\\
			0 & (a_{22} -\frac{a_{21}}{a_{11}} a_{12})  & \ldots &(a_{2n}-\frac{a_{21}}{a_{11}} a_{1n})\\
			\vdots & \vdots & \ddots &\vdots\\
			0 & (a_{n2}-\frac{a_{n1}}{a_{11}}a_{12}) & \ldots &(a_{nn}-\frac{a_{n1}}{a_{11}}a_{1n})\\
		\end{bmatrix}\\
		&\stackrel{\bZ_{12}(\bZ_{11} \bA)}{\longrightarrow}
		\begin{bmatrix}
			1 & 0 & \ldots &0\\
			\frac{a_{21}}{a_{11}} & 1 & \ldots &0\\
			\vdots & \vdots & \ddots &\vdots\\
			\frac{a_{n1}}{a_{11}} & 0 & \ldots &1\\
		\end{bmatrix}
		\begin{bmatrix}
			a_{11} & a_{12} & \ldots &a_{1n}\\
			0 & (a_{22} -\frac{a_{21}}{a_{11}} a_{12})  & \ldots &(a_{2n}-\frac{a_{21}}{a_{11}} a_{1n})\\
			\vdots & \vdots & \ddots &\vdots\\
			0 & (a_{n2}-\frac{a_{n1}}{a_{11}}a_{12}) & \ldots &(a_{nn}-\frac{a_{n1}}{a_{11}}a_{1n})\\
		\end{bmatrix}\\
		&=\begin{bmatrix}
			a_{11} & a_{12} & \ldots &a_{1n}\\
			0 & a_{22} & \ldots &a_{2n}\\
			\vdots & \vdots & \ddots &\vdots\\
			0 & a_{n2} & \ldots &a_{nn}\\
		\end{bmatrix}=\bE_1\bA.
	\end{aligned}
	$$

	Now subtract multiples of the first column of $\bE_1\bA$, from the other columns of $\bE_1\bA$ to zero out the entries in the first row of $\bE_1\bA$ to the right of the first column. Since $\bA$ is symmetric, we can multiply on the right by $\bE_1^\top$ to get what we want. We then have
	$$
	\bA =
	\begin{bmatrix}
		a_{11} & a_{12} & \ldots &a_{1n}\\
		a_{21} & a_{22} & \ldots &a_{2n}\\
		\vdots & \vdots & \ddots &\vdots\\
		a_{n1} & a_{n2} & \ldots &a_{nn}\\
	\end{bmatrix}
	\stackrel{\bE_1 \bA}{\longrightarrow}
	\begin{bmatrix}
		a_{11} & a_{12} & \ldots &a_{1n}\\
		0 & a_{22} & \ldots &a_{2n}\\
		\vdots & \vdots & \ddots &\vdots\\
		0 & a_{n2} & \ldots &a_{nn}\\
	\end{bmatrix}
	\stackrel{\bE_1 \bA\bE_1^\top}{\longrightarrow}
	\begin{bmatrix}
		a_{11} & 0 & \ldots &0\\
		0 & a_{22} & \ldots &a_{2n}\\
		\vdots & \vdots & \ddots &\vdots\\
		0 & a_{n2} & \ldots &a_{nn}\\
	\end{bmatrix}.
	$$
	This operation also preserves the values of the principal minors of $\bA$. The leading principal minors of $\bE_1 \bA\bE_1^\top$ are exactly the same as those of $\bA$.

	Continue this process, we will transform $\bA$ into a diagonal matrix $\bE_n \ldots \bE_1 \bA\bE_1^\top\ldots\bE_n^\top$ whose diagonal values are exactly the same as the diagonals of $\bA$ and are positive. Let $\bE = \bE_n \ldots \bE_1$, which is an invertible matrix. Apparently, $\bE \bA \bE^\top$ is PD, which implies $\bA$ is PD as well from Lemma~\ref{lemma:quadratic-pd}.
\end{proof}

\section{Existence of the Cholesky Decomposition via the LU Decomposition without Permutation}
By Theorem~\ref{lemma:sylvester-criterion} on Sylvester's criterion and the existence of LU decomposition without permutation in Theorem~\ref{theorem:lu-factorization-without-permutation} (p.~\pageref{theorem:lu-factorization-without-permutation}), there is a unique LU decomposition for positive definite matrix $\bA=\bL\bU_0$ where $\bL$ is a unit lower triangular matrix and $\bU_0$ is an upper triangular matrix. Since \textit{the signs of the pivots of a symmetric matrix are the same as the signs of the eigenvalues} \citep{strang1993introduction}:
$$
\text{number of positive pivots = number of positive eigenvalues. \index{Pivot} }
$$
And $\bA = \bL\bU_0$ has the following form
$$
\begin{aligned}
	\bA = \bL\bU_0 &=
	\begin{bmatrix}
		1 & 0 & \ldots & 0 \\
		l_{21} & 1 & \ldots & 0\\
		\vdots & \vdots & \ddots & \vdots\\
		l_{n1} & l_{n2} & \ldots & 1
	\end{bmatrix}
	\begin{bmatrix}
		u_{11} & u_{12} & \ldots & u_{1n} \\
		0 & u_{22} & \ldots & u_{2n}\\
		\vdots & \vdots & \ddots & \vdots\\
		0 & 0 & \ldots & u_{nn}
	\end{bmatrix}.\\

\end{aligned}
$$
This implies that the diagonals of $\bU_0$ contain the pivots of $\bA$. And all the eigenvalues of PD matrices are positive (see Lemma~\ref{lemma:eigens-of-PD-psd}, p.~\pageref{lemma:eigens-of-PD-psd}, which is a consequence of spectral decomposition). Thus the diagonals of $\bU_0$ are positive.

Taking the diagonal of $\bU_0$ out into a diagonal matrix $\bD$, we can rewrite $\bU_0=\bD\bU$ as shown in the following equation
$$
\begin{aligned}
	\bA = \bL\bU_0 =
	\begin{bmatrix}
		1 & 0 & \ldots & 0 \\
		l_{21} & 1 & \ldots & 0\\
		\vdots & \vdots & \ddots & \vdots\\
		l_{n1} & l_{n2} & \ldots & 1
	\end{bmatrix}
	\begin{bmatrix}
		u_{11} & 0 & \ldots & 0 \\
		0 & u_{22} & \ldots & 0\\
		\vdots & \vdots & \ddots & \vdots\\
		0 & 0 & \ldots & u_{nn}
	\end{bmatrix}
	\begin{bmatrix}
		1 & u_{12}/u_{11} & \ldots & u_{1n}/u_{11} \\
		0 & 1 & \ldots & u_{2n}/u_{22}\\
		\vdots & \vdots & \ddots & \vdots\\
		0 & 0 & \ldots & 1
	\end{bmatrix}=\bL\bD\bU,
\end{aligned}
$$
where $\bU$ is a \textit{unit} upper triangular matrix.
By the uniqueness of the LU decomposition without permutation in Corollary~\ref{corollary:unique-lu-without-permutation} (p.~\pageref{corollary:unique-lu-without-permutation}) and the symmetry of $\bA$, it follows that $\bU = \bL^\top$, and $\bA = \bL\bD\bL^\top$. Since the diagonals of $\bD$ are positive, we can set $\bR = \bD^{1/2}\bL^\top$ where $\bD^{1/2}=\diag(\sqrt{u_{11}}, \sqrt{u_{22}}, \ldots, \sqrt{u_{nn}})$ such that $\bA = \bR^\top\bR$ is the Cholesky decomposition of $\bA$, and $\bR$ is upper triangular with positive diagonals.

\subsection{Diagonal Values of the Upper Triangular Matrix}\label{section:cholesky-diagonals}

Suppose $\bA$ is a PD matrix, take $\bA$ as a block matrix $\bA = \begin{bmatrix}
	\bA_{11} & \bA_{12} \\
	\bA_{21} & \bA_{22}
\end{bmatrix}$ where $\bA_{11}\in \real^{k\times k}$, and its block LU decomposition is given by
$$
\begin{aligned}
	\bA &= \begin{bmatrix}
		\bA_{11} & \bA_{12} \\
		\bA_{21} & \bA_{22}
	\end{bmatrix}
	=\bL\bU_0=
	\begin{bmatrix}
		\bL_{11} & \bzero \\
		\bL_{21} & \bL_{22}
	\end{bmatrix}
	\begin{bmatrix}
		\bU_{11} & \bU_{12} \\
		\bzero & \bU_{22}
	\end{bmatrix} \\
	&=\begin{bmatrix}
		\bL_{11}\bU_{11} & \bL_{11}\bU_{12} \\
		\bL_{21}\bU_{11}  & \bL_{21}\bU_{12}+\bL_{22}\bU_{22}
	\end{bmatrix}.
\end{aligned}
$$
Then the leading principal minor (Definition~\ref{definition:leading-principle-minors}, p.~\pageref{definition:leading-principle-minors}), $\Delta_k=\det(\bA_{1:k,1:k} ) = \det(\bA_{11})$  is given by
$$
\Delta_k = \det(\bA_{11}) = \det(\bL_{11}\bU_{11} ) = \det(\bL_{11} )\det(\bU_{11}).
$$
We notice that $\bL_{11}$ is a unit lower triangular matrix and $\bU_{11}$ is an upper triangular matrix. By the fact that the determinant of a lower triangular matrix (or an upper triangular matrix) is the product of the diagonal entries, we obtain
$$
\Delta_k = \det(\bU_{11})= u_{11} u_{22}\ldots u_{kk},
$$
i.e., the $k$-th leading principal minor of $\bA$ is the determinant of the $k\times k$ submatrix of $\bU_0$. That is also the product of the first $k$ diagonals of $\bD$ ($\bD$ is the matrix from $\bA = \bL\bD\bL^\top$). Let $\bD = \diag(d_1, d_2, \ldots, d_n)$, therefore, we have
$$
\Delta_k = d_1 d_2\ldots d_k = \Delta_{k-1}d_k.
$$
This gives us an alternative form of $\bD$, i.e., the \textbf{squared} diagonal values of $\bR$ ($\bR$ is the Cholesky from $\bA=\bR^\top\bR$), and it is given by
$$
\bD = \diag\left(\Delta_1, \frac{\Delta_2}{\Delta_1}, \ldots, \frac{\Delta_n}{\Delta_{n-1}}\right),
$$
where $\Delta_k$ is the $k$-th leading principal minor of $\bA$, for all $k\in \{1,2,\ldots, n\}$. That is, the diagonal values of $\bR$ are given by
$$
\diag\left(\sqrt{\Delta_1}, \sqrt{\frac{\Delta_2}{\Delta_1}}, \ldots, \sqrt{\frac{\Delta_n}{\Delta_{n-1}}}\right).
$$

\subsection{Block Cholesky Decomposition}
Following from the last section, suppose $\bA$ is a PD matrix, take $\bA$ as a block matrix $\bA = \begin{bmatrix}
	\bA_k & \bA_{12} \\
	\bA_{21} & \bA_{22}
\end{bmatrix}$ where $\bA_k\in \real^{k\times k}$, and its block LU decomposition is given by
$$
\begin{aligned}
	\bA &= \begin{bmatrix}
		\bA_k & \bA_{12} \\
		\bA_{21} & \bA_{22}
	\end{bmatrix}
	=\bL\bU_0=
	\begin{bmatrix}
		\bL_k & \bzero \\
		\bL_{21} & \bL_{22}
	\end{bmatrix}
	\begin{bmatrix}
		\bU_k & \bU_{12} \\
		\bzero & \bU_{22}
	\end{bmatrix} \\
	&=\begin{bmatrix}
		\bL_k\bU_k & \bL_{11}\bU_{12} \\
		\bL_{21}\bU_{11}  & \bL_{21}\bU_{12}+\bL_{22}\bU_{22}
	\end{bmatrix}.\\

\end{aligned}
$$
where the $k$-th leading principal submatrix $\bA_k$ of $\bA$ also has its LU decomposition $\bA_k=\bL_k\bU_k$. Then, it is trivial that the Cholesky decomposition of an $n\times n$ matrix contains $n-1$ other Cholesky decompositions within it: $\bA_k = \bR_k^\top\bR_k$, for all $k\in \{1,2,\ldots, n-1\}$. This is particularly true that any leading principal submatrix $\bA_k$ of the positive definite matrix $\bA$ is also positive definite. This can be shown that for positive definite matrix $\bA_{k+1} \in \real^{(k+1)\times(k+1)}$, and any nonzero vector $\bx_k\in\real^k$ appended by a zero element $\bx_{k+1} = \begin{bmatrix}
	\bx_k\\
	0
\end{bmatrix}$. It follows that
$$
\bx_k^\top\bA_k\bx_k = \bx_{k+1}^\top\bA_{k+1}\bx_{k+1} >0,
$$
and $\bA_k$ is positive definite. If we start from $\bA\in \real^{n\times n}$, we will recursively get that $\bA_{n-1}$ is PD, $\bA_{n-2}$ is PD, $\ldots$. And all of them admit a Cholesky decomposition.

\section{Existence of the Cholesky Decomposition via Induction}
In the last section, we proved the existence of the Cholesky decomposition via the LU decomposition without permutation. Following from the proof of the LU decomposition in Section~\ref{section:exist-lu-without-perm}, we realize that the existence of Cholesky decomposition can be a direct consequence of induction as well.
\begin{proof}[\textbf{of Theorem~\ref{theorem:cholesky-factor-exist}: Existence of Cholesky Decomposition by Induction\index{Induction}}]
	We will prove by induction that every $n\times n$ positive definite matrix $\bA$ has a decomposition $\bA=\bR^\top\bR$. The $1\times 1$ case is trivial by setting $R=\sqrt{A}$, thus, $A=R^2$.

	Suppose for any $k\times k$ PD matrix $\bA_k$ has a Cholesky decomposition. If we prove any $(k+1)\times(k+1)$ PD matrix $\bA_{k+1}$ can also be factored as this Cholesky decomposition, then we complete the proof.

	For any $(k+1)\times(k+1)$ PD matrix $\bA_{k+1}$, Write out $\bA_{k+1}$ as
	$$
	\bA_{k+1} = \begin{bmatrix}
		\bA_k & \bb \\
		\bb^\top & d
	\end{bmatrix}.
	$$
	We note that $\bA_k$ is PD.
	By the inductive hypothesis, it admits a Cholesky decomposition $\bA_k$ is given by $\bA_k = \bR_k^\top\bR_k$. We can construct the upper triangular matrix
	$$
	\bR=\begin{bmatrix}
		\bR_k & \br\\
		0 & s
	\end{bmatrix},
	$$
	such that it follows that
	$$
	\bR_{k+1}^\top\bR_{k+1} =
	\begin{bmatrix}
		\bR_k^\top\bR_k & \bR_k^\top \br\\
		\br^\top \bR_k & \br^\top\br+s^2
	\end{bmatrix}.
	$$
	Therefore, if we can prove $\bR_{k+1}^\top \bR_{k+1} = \bA_{k+1}$ is the Cholesky decomposition of $\bA_{k+1}$ (which requires the value $s$ to be positive), then we complete the proof. That is, we need to prove
	$$
	\begin{aligned}
		\bb &= \bR_k^\top \br, \\
		d &= \br^\top\br+s^2.
	\end{aligned}
	$$
	Since $\bR_k$ is nonsingular, we have a unique solution for $\br$ and $s$ that
	$$
	\begin{aligned}
		\br &= \bR_k^{-\top}\bb, \\
		s &= \sqrt{d - \br^\top\br} = \sqrt{d - \bb^\top\bA_k^{-1}\bb},
	\end{aligned}
	$$
	since we assume $s$ is nonnegative. However, we need to further prove that $s$ is not only nonnegative, but also positive. Since $\bA_k$ is PD, from Sylvester's criterion, and the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bA & \bB \\
		\bC & \bD
	\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$. We have
	$$
	\det(\bA_{k+1}) = \det(\bA_k)\det(d- \bb^\top\bA_k^{-1}\bb) =  \det(\bA_k)(d- \bb^\top\bA_k^{-1}\bb)>0.
	$$
	Since $ \det(\bA_k)>0$, we then obtain that $(d- \bb^\top\bA_k^{-1}\bb)>0$ and this implies $s>0$.
	We complete the proof.

		\end{proof}

		%

		\section{Uniqueness of the Cholesky Decomposition}

		\begin{corollary}[Uniqueness of Cholesky Decomposition\index{Uniqueness}]\label{corollary:unique-cholesky-main}
			The Cholesky decomposition $\bA=\bR^\top\bR$ for any positive definite matrix $\bA\in \real^{n\times n}$ is unique.
		\end{corollary}

		The uniqueness of the Cholesky decomposition can be an immediate consequence of the uniqueness of the LU decomposition without permutation. Or, an alternative rigorous proof is provided as follows.
		\begin{proof}[of Corollary~\ref{corollary:unique-cholesky-main}]
			Suppose the Cholesky decomposition is not unique, then we can find two decompositions such that $\bA=\bR_1^\top\bR_1 = \bR_2^\top\bR_2$ which implies
			$$
			\bR_1\bR_2^{-1}= \bR_1^{-\top} \bR_2^\top.
			$$
			From the fact that the inverse of an upper triangular matrix is also an upper triangular matrix, and the product of two upper triangular matrices is also an upper triangular matrix, \footnote{Same for lower triangular matrices: the inverse of a lower triangular matrix is also a lower triangular matrix, and the product of two lower triangular matrices is also a lower triangular matrix.} we realize that the left-side of the above equation is an upper triangular matrix and the right-side of it is a lower triangular matrix. This implies $\bR_1\bR_2^{-1}= \bR_1^{-\top} \bR_2^\top$ is a diagonal matrix, and $\bR_1^{-\top} \bR_2^\top= (\bR_1^{-\top} \bR_2^\top)^\top = \bR_2\bR_1^{-1}$.
			Let $\bLambda = \bR_1\bR_2^{-1}= \bR_2\bR_1^{-1}$ be the diagonal matrix. We notice that the diagonal value of $\bLambda$ is the product of the corresponding diagonal values of $\bR_1$ and $\bR_2^{-1}$ (or $\bR_2$ and $\bR_1^{-1}$). That is, for
			$$
			\bR_1=\begin{bmatrix}
				r_{11} & r_{12} & \ldots & r_{1n} \\
				0 & r_{22} & \ldots & r_{2n}\\
				\vdots & \vdots & \ddots & \vdots\\
				0 & 0 & \ldots & r_{nn}
			\end{bmatrix},
			\qquad
			\bR_2=
			\begin{bmatrix}
				s_{11} & s_{12} & \ldots & s_{1n} \\
				0 & s_{22} & \ldots & s_{2n}\\
				\vdots & \vdots & \ddots & \vdots\\
				0 & 0 & \ldots & s_{nn}
			\end{bmatrix},
			$$
			we have,
			$$
			\begin{aligned}
				\bR_1\bR_2^{-1}=
				\begin{bmatrix}
					\frac{r_{11}}{s_{11}} & 0 & \ldots & 0 \\
					0 & \frac{r_{22}}{s_{22}} & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & \frac{r_{nn}}{s_{nn}}
				\end{bmatrix}
				=
				\begin{bmatrix}
					\frac{s_{11}}{r_{11}} & 0 & \ldots & 0 \\
					0 & \frac{s_{22}}{r_{22}} & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & \frac{s_{nn}}{r_{nn}}
				\end{bmatrix}
				=\bR_2\bR_1^{-1}.
			\end{aligned}
			$$
			Since both $\bR_1$ and $\bR_2$ have positive diagonals, this implies $r_{11}=s_{11}, r_{22}=s_{22}, \ldots, r_{nn}=s_{nn}$. And $\bLambda = \bR_1\bR_2^{-1}= \bR_2\bR_1^{-1}  =\bI$.
			That is, $\bR_1=\bR_2$ and this leads to a contradiction. The Cholesky decomposition is thus unique.
		\end{proof}

		\section{Computing the Cholesky Decomposition Recursively}\label{section:compute-cholesky}
		Similar to computing the LU decomposition,
		to compute the Cholesky decomposition, we write out the equality $\bA=\bR^\top\bR$:
		$$
		\begin{aligned}
			\bA=\left[
			\begin{matrix}
				\bA_{11} & \bA_{1,2:n} \\
				\bA_{2:n,1} & \bA_{2:n,2:n}
			\end{matrix}
			\right]
			&=\left[
			\begin{matrix}
				\bR_{11} & 0 \\
				\bR_{1,2:n}^\top & \bR_{2:n,2:n}^\top
			\end{matrix}
			\right]
			\left[
			\begin{matrix}
				\bR_{11} & \bR_{1,2:n} \\
				0 & \bR_{2:n,2:n}
			\end{matrix}
			\right]\\
			&=
			\left[
			\begin{matrix}
				\bR_{11}^2 & \bR_{11}\bR_{1,2:n} \\
				\bR_{11}\bR_{1,2:n}^\top & \bR_{1,2:n}^\top\bR_{1,2:n} + \bR_{2:n,2:n}^\top\bR_{2:n,2:n}
			\end{matrix}
			\right],
		\end{aligned}
		$$
		which allows to determine the first row of $\bR$ by
		$$
		\bR_{11} = \sqrt{\bA_{11}}, \qquad \bR_{1,2:n} = \frac{1}{\bR_{11}}\bA_{1,2:n}.
		$$
		Let $\bA_2=\bR_{2:n,2:n}^\top\bR_{2:n,2:n}$. The equality $\bA_{2:n,2:n} = \bR_{1,2:n}^\top\bR_{1,2:n} + \bR_{2:n,2:n}^\top\bR_{2:n,2:n}$ indicates
		$$
		\begin{aligned}
			\bA_2=\bR_{2:n,2:n}^\top\bR_{2:n,2:n} &= \bA_{2:n,2:n} - \bR_{1,2:n}^\top\bR_{1,2:n} \\
			&= \bA_{2:n,2:n} - \frac{1}{\bA_{11}} \bA_{1,2:n}^\top\bA_{1,2:n} \\
			&= \bA_{2:n,2:n} - \frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n}, \qquad &(\bA \mbox{ is symmetric})
		\end{aligned}
		$$
		where $\bA_2$ is the Schur complement of $\bA_{11}$ in $\bA$ of size $(n-1)\times (n-1)$. And to get $\bR_{2:n,2:n}$ we must compute the Cholesky decomposition of matrix $\bA_2$ of shape $(n-1)\times (n-1)$. Again, this is a recursive algorithm and is formulated in Algorithm~\ref{alg:compute-choklesky}.

		\begin{algorithm}[H]
			\caption{Cholesky Decomposition via Recursive Algorithm}
			\label{alg:compute-choklesky}
			\begin{algorithmic}[1]
				\Require
				Positive definite matrix $\bA$ with size $n\times n$;
				\State Calculate first row of $\bR$ by $\bR_{11} = \sqrt{\bA_{11}}, \bR_{1,2:n} = \frac{1}{\bR_{11}}\bA_{1,2:n}$; \Comment{$n$ flops}
				\State Compute the Cholesky decomposition of the $(n-1)\times (n-1)$ matrix
				$$
				\bA_2=\bR_{2:n,2:n}^\top\bR_{2:n,2:n}=\bA_{2:n,2:n} - \frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n};
				$$
				\Comment{$n^2-n$ flops}

			\end{algorithmic}
		\end{algorithm}

		\begin{theorem}[Algorithm Complexity: Cholesky Recursively]\label{theorem:cholesky-complexity}
			Algorithm~\ref{alg:compute-choklesky} requires $\sim(1/3)n^3$ flops to compute the Cholesky decomposition of an $n\times n$ positive definite matrix.
		\end{theorem}

		\begin{proof}[of Theorem~\ref{theorem:cholesky-complexity}]
			Step 1 takes 1 square root and $(n-1)$ divisions which take $n$ flops totally.

			For step 2, note that $\frac{1}{\bA_{11}} \bA_{2:n,1}\bA_{1,2:n} = (\frac{1}{\sqrt{\bA_{11}}} \bA_{2:n,1})(\frac{1}{\sqrt{\bA_{11}}}\bA_{1,2:n}) = \bR_{1,2:n}^\top \bR_{1,2:n}$. If we calculate the complexity directly from the equation in step 2, we will get the same complexity as the LU decomposition. But the symmetry of $\bR_{1,2:n}^\top \bR_{1,2:n}$ can be adopted, the complexity of $\bR_{1,2:n}^\top \bR_{1,2:n}$ reduces from $(n-1)\times(n-1)$ multiplications to $1+2+\ldots+(n-1)=\frac{n^2-n}{2}$ multiplications, which is almost half of the original complexity. The cost of matrix division reduces from $(n-1)\times(n-1)$ to $1+2+\ldots+(n-1)=\frac{n^2-n}{2}$ as well. So the total cost for step 2 is $n^2-n$ flops.

			Let $f(n) = n^2-n + n = n^2$, the total complexity is
			$$
			\mathrm{cost} = f(n)+f(n-1)+\ldots +f(1).
			$$
			Simple calculations will show the total complexity for all the recursive steps is $\frac{2n^3+3n^2+n}{6}$ flops which is $(1/3)n^3$ flops if we keep only the leading term.
		\end{proof}

		An important use of the Cholesky decomposition computation above is for testing whether a symmetric matrix is positive definite. The test is simply to run the algorithm above and declare the matrix positive definite if the algorithm completes without encountering any negative or zero pivots (in step 1 above) and not positive definite otherwise.

		To end up this section, we provide the full pseudo code for Algorithm~\ref{alg:compute-choklesky} as shown in Algorithm~\ref{alg:compute-choklesky11} (compare the two algorithms).
		\begin{algorithm}[H]
			\caption{Cholesky Decomposition via Recursive Algorithm: Full Pseudo Code}
			\label{alg:compute-choklesky11}
			\begin{algorithmic}[1]
				\Require
				Positive definite matrix $\bA$ with size $n\times n$;
				\For{$k=1$ to $n$} \Comment{compute the $k$th row of $\bR$}
				\State $\bR_{kk} = \sqrt{\bA_{kk}}$; \Comment{first element of $k$-th row, 1 flop}
				\State $\bR_{k,k+1:n} = \frac{1}{\bR_{kk}} \bA_{k,k+1:n}$; \Comment{the rest elements of $k$-th row, $n-k$ flops}
				\State $\bA_{k+1:n,k+1:n} = \bA_{k+1:n,k+1:n} - \bR_{k,k+1:n}^\top\bR_{k,k+1:n}$; \Comment{$2(1+2+\ldots+(n-k))$ flops}
				\EndFor

			\end{algorithmic}
		\end{algorithm}
		\section{Computing the Cholesky Decomposition Element-Wise}

		It is also common to compute the Cholesky decomposition via element-level equations which come from directly solving the matrix equation $\bA=\bR^\top\bR$. We notice that the entry $(i,j)$ of $\bA$ is $\bA_{ij} = \bR_{:,i}^\top \bR_{:,j} = \sum_{k=1}^{i} \bR_{ki}\bR_{kj}$ if $i<j$. This further implies,
		$$
		\begin{aligned}
			\bA_{ij} &= \bR_{:,i}^\top \bR_{:,j} = \sum_{k=1}^{i} \bR_{ki}\bR_{kj} \\
			&= \sum_{k=1}^{i-1} \bR_{ki}\bR_{kj} + \bR_{ii}\bR_{ij},
		\end{aligned}
		$$
		and
		$$
		\bR_{ij} = (\bA_{ij} - \sum_{k=1}^{i-1} \bR_{ki}\bR_{kj})/\bR_{ii},
		$$
		if $i<j$.
		If we equate elements of $\bR$ by taking a column at a time and start with $\bR_{11} = \sqrt{\bA_{11}}$, the element-level algorithm is formulated in Algorithm~\ref{alg:compute-choklesky-element-level}.

		\begin{algorithm}[H]
			\caption{Cholesky Decomposition Element-Wise}
			\label{alg:compute-choklesky-element-level}
			\begin{algorithmic}[1]
				\Require
				Positive definite matrix $\bA$ with size $n\times n$;
				\State Calculate first of $\bR$ by $\bR_{11} = \sqrt{\bA_{11}}$;
				\For{$j=1$ to $n$}
				\For{$i=1$ to $j-1$}
				\State $\bR_{ij} = (\bA_{ij} - \sum_{k=1}^{i-1} \bR_{ki}\bR_{kj})/\bR_{ii}$, since $i<j$;
				\EndFor
				\State $\bR_{jj} = \sqrt{\bA_{jj}- \sum_{k=1}^{j-1}\bR_{kj}^2}$;
				\EndFor
			\end{algorithmic}
		\end{algorithm}

		\begin{theorem}[Algorithm Complexity: Cholesky Element-wise]\label{theorem:cholesky-complexity-element}
			Algorithm~\ref{alg:compute-choklesky-element-level} requires $\sim(1/3)n^3$ flops to compute the Cholesky decomposition of an $n\times n$ positive definite matrix.
		\end{theorem}

		\begin{proof}[of Theorem~\ref{theorem:cholesky-complexity-element}]
			For step 4 in Algorithm~\ref{alg:compute-choklesky-element-level}, that is, for each $j, i$, step 4 involves $(i-1)$ multiplications, $(i-2)$ additions, $1$ subtraction, and 1 division, which is $2i-1$ flops. Let $f(k)=2k-1$, the total flops required from step 4 for each loop $j$ is given by
			$$
			f(1)+f(2)+\ldots +f(j-1) = j^2-2j+1 \,\,\,\,\mathrm{flops}.
			$$
			Further, for each loop $j$, step 6 requires $j-1$ multiplications, $j-2$ additions, $1$ subtraction, $1$ square root. That is, step 6 involves $2j-1$ flops for each loop $j$. Combine the flops needed in step 4, it needs $j^2$ flops for each loop $j$ totally. Let $g(k)=k^2$, the total complexity is thus given by
			$$
			g(1)+g(2)+\ldots +g(n) = \frac{2n^3+3n^2+n}{6} \,\,\,\,\mathrm{flops},
			$$
			which is $(1/3)n^3$ flops if we keep only the leading term.
		\end{proof}
		The complexity of Algorithm~\ref{alg:compute-choklesky-element-level} is the same as that of 	Algorithm~\ref{alg:compute-choklesky}. And indeed, the ideas behind them are similar as well.

		\section{More Properties of Positive Definite Matrices}

		\begin{lemma}[PD Properties\index{Positive definite}]\label{lemma:pd-more-properties}
			For positive definite matrix $\bA\in \real^{n\times n}$, we have the following properties
			\item 1). Any principal minors are positive (see Definition~\ref{definition:principle-minors}, p.~\pageref{definition:principle-minors}, not necessarily to be the leading principal minors);
			\item 2). Suppose the diagonal values of $\bA$ are $a_{ii}$, for all $i\in \{1,2,\ldots, n\}$, then $\det(\bA) \leq \prod_{i=1}^{n}a_{ii}$. The equality can be obtained when $\bA$ is a diagonal matrix.
		\end{lemma}

		\begin{proof}[of Lemma~\ref{lemma:pd-more-properties}]
			For 1). Similar to the permutation matrix (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}), we can define a selection matrix. For a row selection matrix $\bP$, if we select a row of the matrix $\bA$ by $\bP\bA$, the corresponding diagonal of $\bP$ will be 1, and 0 otherwise. In this sense, the $\bP^\top=\bP$ is a column selection matrix such that $\bA\bP^\top$ is to select the corresponding columns. Then, any $k\times k$ submatrix of $\bA$ can be obtained by $\bP\bA\bP^\top$. For any vector $\bx\in \real^n$ where not all the corresponding $k$ entries are zero (such that $\bx^\top\bP$ is nonzero), we have
			$$
			\bx^\top \bP\bA\bP^\top \bx >0.
			$$
			Since $(n-k)$ rows of $\bP$ are zero and the corresponding $(n-k)$ columns of $\bP$ are zero as well, these rows and columns will not count any value for the above equation, we can just remove them away.
			This implies the $k\times k$ submatrix $\bA_k \in \real^{k\times k}$ of $\bA$, and its corresponding $k\times k$ sub-selection matrix $\bP_k \in \real^{k\times k}$ (which is an identity matrix here), such that
			$$
			\bx_k\bP_k\bA_k\bP_k^\top \bx_k >0,
			$$
			where $\bx_k \in \real^{k}$. Since the vector $\bx$ is any vector in $\real^n$, the $\bx_k$ is thus also any vector in $\real^k$.
			This results in that $\bA_k$ is PD.

			For 2). For the LU decomposition of $\bA$,
			$\bA = \bL\bU_0$ has the following form
			$$
			\begin{aligned}
				\bA = \bL\bU_0 &=
				\begin{bmatrix}
					1 & 0 & \ldots & 0 \\
					l_{21} & 1 & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					l_{n1} & l_{n2} & \ldots & 1
				\end{bmatrix}
				\begin{bmatrix}
					u_{11} & u_{12} & \ldots & u_{1n} \\
					0 & u_{22} & \ldots & u_{2n}\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & u_{nn}
				\end{bmatrix}\\
				&=
				\begin{bmatrix}
					1 & 0 & \ldots & 0 \\
					l_{21} & 1 & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					l_{n1} & l_{n2} & \ldots & 1
				\end{bmatrix}
				\begin{bmatrix}
					u_{11} & 0 & \ldots & 0 \\
					0 & u_{22} & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & u_{nn}
				\end{bmatrix}
				\begin{bmatrix}
					1 & u_{12}/u_{11} & \ldots & u_{1n}/u_{11} \\
					0 & 1 & \ldots & u_{2n}/u_{22}\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & 1
				\end{bmatrix}\\
				&=
				\begin{bmatrix}
					1 & 0 & \ldots & 0 \\
					l_{21} & 1 & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					l_{n1} & l_{n2} & \ldots & 1
				\end{bmatrix}
				\begin{bmatrix}
					u_{11} & 0 & \ldots & 0 \\
					0 & u_{22} & \ldots & 0\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & u_{nn}
				\end{bmatrix}
				\begin{bmatrix}
					1 & l_{21} & \ldots & l_{n1} \\
					0 & 1 & \ldots & l_{n2}\\
					\vdots & \vdots & \ddots & \vdots\\
					0 & 0 & \ldots & 1
				\end{bmatrix}=\bL\bD\bL^\top.
			\end{aligned}
			$$
			We have discussed in Section~\ref{section:cholesky-diagonals} that $\det(\bA) = \det(\bL)\det(\bU_0) = \prod_{i=1}^{n}u_{ii}\leq \prod_{i=1}^{n}a_{ii}$, where the last inequality comes from the fact that $\bA$ is symmetric such that
			$$
			a_{ii} = (\sum_{j=1}^{i} l_{ij}u_{ji}) +u_{ii} = (\sum_{j=1}^{i} l_{ij}^2\cdot u_{ii}) +u_{ii} \geq u_{ii}.
			$$
			And the equality can be obtained when $\bA$ is a diagonal matrix.
		\end{proof}

		\section{Last Words on Positive Definite Matrices}
		In Section~\ref{section:equivalent-pd-psd} (p.~\pageref{section:equivalent-pd-psd}), we will prove that a matrix $\bA$ is PD if and only if $\bA$ can be factored as $\bA=\bP^\top\bP$ where $\bP$ is nonsingular. And in Section~\ref{section:unique-posere-pd} (p.~\pageref{section:unique-posere-pd}), we will prove that PD matrix $\bA$ can be uniquely factored as $\bA =\bB^2$ where $\bB$ is also PD. The two results are both consequences of the spectral decomposition of PD matrices.

		To conclude, for PD matrix $\bA$, we can factor it into $\bA=\bR^\top\bR$ where $\bR$ is an upper triangular matrix with positive diagonals as shown in Theorem~\ref{theorem:cholesky-factor-exist} by Cholesky decomposition, $\bA = \bP^\top\bP$ where $\bP$ is nonsingular in Theorem~\ref{lemma:nonsingular-factor-of-PD} (p.~\pageref{lemma:nonsingular-factor-of-PD}), and $\bA = \bB^2$ where $\bB$ is PD in Theorem~\ref{theorem:unique-factor-pd} (p.~\pageref{theorem:unique-factor-pd}). For clarity, the different factorizations of positive definite matrix $\bA$ are summarized in Figure~\ref{fig:pd-summary}.

		\begin{figure}[htbp]
			\centering

			\centering
			\resizebox{0.9\textwidth}{!}{%
				\begin{tikzpicture}[>=latex]

					\tikzstyle{state} = [draw, very thick, fill=white, rectangle, minimum height=3em, minimum width=6em, node distance=8em, font={\sffamily\bfseries}]
					\tikzstyle{stateEdgePortion} = [black,thick];
					\tikzstyle{stateEdge} = [stateEdgePortion,->];
					\tikzstyle{stateEdge2} = [stateEdgePortion,<->];
					\tikzstyle{edgeLabel} = [pos=0.5, text centered, font={\sffamily\small}];

					\node[ellipse, name=pdmatrix, draw,font={\sffamily\bfseries},  node distance=7em, xshift=-9em, yshift=-1em,fill={colorals}]  {PD Matrix $\bA$};
					\node[state, name=bsqure, below of=pdmatrix, xshift=0em, yshift=1em, fill={colorlu}] {$\bB^2$};
					\node[state, name=ptp, right of=bsqure, xshift=3em, fill={colorlu}] {$\bP^\top\bP$};
					\node[state, name=rsqure, left of=bsqure, xshift=-3em, fill={colorlu}] {$\bR^\top\bR$};
					\node[ellipse, name=utv, below of=pdmatrix,draw,  node distance=7em, xshift=0em, yshift=-4em,font={\tiny},fill={coloruppermiddle}]  {PD $\bB$};
					\node[ellipse, name=upperr, left of=utv, draw, node distance=8em, xshift=-3em,font={\tiny},fill={coloruppermiddle}]  {\parbox{6em}{Upper \\Triangular $\bR$}};
					\node[ellipse, name=nonp, right of=utv,draw,  node distance=8em, xshift=3em, font={\tiny},fill={coloruppermiddle}]  {Nonsingular $\bP$};

					\coordinate (lq2inter3) at ($(pdmatrix.east -| ptp.north) + (-0em,0em)$);
					\draw (pdmatrix.east) edge[stateEdgePortion] (lq2inter3);
					\draw (lq2inter3) edge[stateEdge]
					node[edgeLabel, text width=7.25em, yshift=0.8em]{\parbox{5em}{Spectral\\Decomposition}} (ptp.north);

					\coordinate (rqr2inter1) at ($(pdmatrix.west) + (0,0em)$);
					\coordinate (rqr2inter3) at ($(rqr2inter1-| rsqure.north) + (-0em,0em)$);
					\draw (rqr2inter1) edge[stateEdgePortion] (rqr2inter3);
					\draw (rqr2inter3) edge[stateEdge]
					node[edgeLabel, text width=8em, yshift=0.8em]{\parbox{2em}{LU/\\Spectral/\\Recursive}} (rsqure.north);

					\draw (pdmatrix.south)
					edge[stateEdge] node[edgeLabel,yshift=0.5em]{\parbox{5em}{Spectral\\Decomposition} }
					(bsqure.north);

					\draw (upperr.north)
					edge[stateEdge] node[edgeLabel,yshift=0.5em]{}
					(rsqure.south);

					\draw (utv.north)
					edge[stateEdge] node[edgeLabel,yshift=0.5em]{}
					(bsqure.south);

					\draw (nonp.north)
					edge[stateEdge] node[edgeLabel,yshift=0.5em]{}
					(ptp.south);

					\begin{pgfonlayer}{background}
						\draw [join=round,cyan,dotted,fill={colormiddle}] ($(upperr.south west) + (-1.6em, -1em)$) rectangle ($( pdmatrix.east-|ptp.north east) + (1.6em, +1.8em)$);
					\end{pgfonlayer}

				\end{tikzpicture}
			}

			\caption{Demonstration of different factorizations on positive definite matrix $\bA$.}
			\label{fig:pd-summary}
		\end{figure}

		\section{Pivoted Cholesky Decomposition}\label{section:volum-picot-cholesjy}
		If $\bP$ is a permutation matrix and $\bA$ is positive definite, then $\bP^\top\bA\bP$ is said to be a diagonal permutation of $\bA$ (among other things, it permutes the diagonals of A). Any diagonal permutation of A
		is positive definite and has a Cholesky factor. Such a factorization is called a pivoted Cholesky factorization. There are many ways to pivot a
		Cholesky decomposition, but the most common one is the complete pivoting (see Section~\ref{section:complete-pivoting}, p.~\pageref{section:complete-pivoting}) such that
		$$
		\bP\bA\bP^\top = \bR^\top\bR
		$$
		is the column-pivoted Cholesky decomposition of $\bA$, where $\bP$ is a permutation matrix, and $\bR$ is upper triangular.

		Following the Cholesky decomposition via recursive calculation in Algorithm~\ref{alg:compute-choklesky11}, we notice from Lemma~\ref{lemma:positive-maximal-diagonal} that the maximal element of a PD matrix lies on the diagonal. Therefore, the complete pivoting algorithm for Cholesky decomposition can only search in the diagonals. The procedure is shown in Algorithm~\ref{alg:cholesky-complete-pivot22}.
		\begin{algorithm}[h]
			\caption{Cholesky Decomposition via Recursive Algorithm: Complete Pivoting}
			\label{alg:cholesky-complete-pivot22}
			\begin{algorithmic}[1]
				\Require
				Positive definite matrix $\bA$ with size $n\times n$;
				\State $\bR \in \real^{n\times n}$ is initialized with all zeros;
				\For{$k=1$ to $n$} \Comment{compute the $k$th row of $\bR$}
				\State Search in the diagonals of $\bA_{k:n,k:n}$ such that $\bA_{vv} = \max \bA_{k:n,k:n}$;
				\State Swap the $k$-th and $v$-th column of $\bA$: $\bA_{:,k}\leftrightarrow\bA_{:,v}$ by column permutation;
				\State Swap the $k$-th and $v$-th column of $\bR$: $\bR_{:,k}\leftrightarrow\bR_{:,v}$ by column permutation;
				\State Swap the $k$-th and $v$-th row of $\bA$: $\bA_{k,:}\leftrightarrow\bA_{v,:}$ by row permutation;
				\State $\bR_{kk} = \sqrt{\bA_{kk}}$; \Comment{first element of $k$-th row, 1 flop}
				\State $\bR_{k,k+1:n} = \frac{1}{\bR_{kk}} \bA_{k,k+1:n}$; \Comment{the rest elements of $k$-th row, $n-k$ flops}
				\State $\bA_{k+1:n,k+1:n} = \bA_{k+1:n,k+1:n} - \bR_{k,k+1:n}^\top\bR_{k,k+1:n}$; \Comment{$2(1+2+\ldots+(n-k))$ flops}
				\EndFor
			\end{algorithmic}
		\end{algorithm}

		\section{Decomposition for Semidefinite Matrices}
		For positive semidefinite matrices, the Cholesky decomposition also exists with slight modification.

		\begin{theoremHigh}[Semidefinite Decomposition\index{Positive semidefinite}]\label{theorem:semidefinite-factor-exist}
			Every positive semidefinite matrix $\bA\in \real^{n\times n}$ can be factored as
			$$
			\bA = \bR^\top\bR,
			$$
			where $\bR \in \real^{n\times n}$ is an upper triangular matrix with possible \textbf{zero} diagonal elements and the factorization is \textbf{not unique} in general.
		\end{theoremHigh}

		For such decomposition, the diagonal of $\bR$ may not display the rank of $\bA$ \citep{higham2009cholesky}.
		\begin{example}[\citep{higham2009cholesky}]
			Suppose
			$$
			\bA =
			\begin{bmatrix}
				1 & -1 & 1 \\
				-1 & 1 & -1 \\
				1 & -1 & 2
			\end{bmatrix}.
			$$
			The semidefinite decomposition is given by
			$$
			\bA =
			\begin{bmatrix}
				1 & 0 & 0 \\
				-1 & 0 & 0 \\
				1 & 1 & 0
			\end{bmatrix}
			\begin{bmatrix}
				1 & -1 & 1 \\
				0 & 0 & 1 \\
				0 & 0 & 0
			\end{bmatrix}
			=\bR^\top\bR.
			$$
			$\bA$ has rank 2, but $\bR$ has only one nonzero diagonal element.
			\exampbar
		\end{example}

		We notice that all PD matrices have full rank, and this fact permeates many of our proofs discussed above. This can be proved by the Sylvester's criterion (Theorem~\ref{lemma:sylvester-criterion}, p.~\pageref{lemma:sylvester-criterion}) that all the leading principal minors of PD matrices are positive. Or, we can simply prove that if a PD matrix $\bA$ is rank deficient, this implies the null space of $\bA$ has positive dimension and there exists a vector $\bx$ in the null space of $\bA$ such that $\bA\bx=\bzero$, i.e., $\bx^\top\bA\bx=0$ and it leads to a contradiction to the definition of the PD matrices.

		However, this is not necessarily true for PSD matrices where the dimension of the null space can be larger than 0.
		Therefore, and more generally, a rank-revealing decomposition for semidefinite decomposition is provided as follows.\index{Rank-revealing}\index{Semidefinite rank-revealing}

		\begin{theoremHigh}[Semidefinite Rank-Revealing Decomposition\index{Rank-revealing}]\label{theorem:semidefinite-factor-rank-reveal}
			Every positive semidefinite matrix $\bA\in \real^{n\times n}$ with rank $r$ can be factored as
			$$
			\bP^\top \bA\bP  = \bR^\top\bR, \qquad \mathrm{with} \qquad
			\bR = \begin{bmatrix}
				\bR_{11} & \bR_{12}\\
				\bzero &\bzero
			\end{bmatrix} \in \real^{n\times n},
			$$
			where $\bR_{11} \in \real^{r\times r}$ is an upper triangular matrix with positive diagonal elements, and $\bR_{12}\in \real^{r\times (n-r)}$.
		\end{theoremHigh}

		The proof for the existence of the above rank-revealing decomposition for semidefinite matrices is delayed in Section~\ref{section:semi-rank-reveal-proof} (p.~\pageref{section:semi-rank-reveal-proof}) as a consequence of the spectral decomposition (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}) and the column-pivoted QR decomposition (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}). Whereas, the rigorous proof for the trivial Semidefinite Decomposition Theorem~\ref{theorem:semidefinite-factor-exist} can be a direct result of the spectral decomposition and trivial QR decomposition (Theorem~\ref{theorem:qr-decomposition}, p.~\pageref{theorem:qr-decomposition})
		.
		\section{Application: Rank-One Update/Downdate}\label{section:cholesky-rank-one-update}
		Updating linear systems after low-rank modifications of the system matrix is widespread in machine learning, statistics, and many other fields. However, it is well known that this update can lead to serious instabilities in the presence
		of roundoff error \citep{seeger2004low}. If the system matrix is positive definite, it is almost always
		possible to use a representation based on the Cholesky decomposition which is
		much more numerically stable. We will shortly provide the proof for this rank one update/downdate via Cholesky decomposition in this section.
		\subsection{Rank-One Update}\index{Rank-one update}
		A rank-one update $\bA^\prime$ of matrix $\bA$ by vector $\bx$ is of the form \citep{gill1974methods, bojanczyk1987note,  chang1997pertubation, davis1999modifying, seeger2004low, chen2008algorithm, davis2008user, higham2009cholesky}:
		\begin{equation*}
			\begin{aligned}
				\bA^\prime &= \bA + \bv \bv^\top\\
				\bR^{\prime\top}\bR^\prime &= \bR^\top\bR + \bv \bv^\top.
			\end{aligned}
		\end{equation*}
		If we have already calculated the Cholesky factor $\bR$ of $\bA \in \real^{n\times n}$, then the Cholesky factor $\bR^\prime$ of $\bA^\prime$ can be calculated efficiently.  Note that $\bA^\prime$ differs from $\bA$ only via the symmetric rank-one matrix. Hence we can compute $\bR^\prime$ from $\bR$ using the rank-one Cholesky update, which takes $O(n^2)$ operations each saving from $O(n^3)$ if we do know $\bR$, the Cholesky decomposition of $\bA$ up front, i.e., we want to compute the Cholesky decomposition of $\bA^\prime$ via that of $\bA$.
		To see this,
		suppose there is a set of orthogonal matrices $\bQ_n \bQ_{n-1}\ldots \bQ_1$ such that that
		$$
		\bQ_n \bQ_{n-1}\ldots \bQ_1
		\begin{bmatrix}
			\bv^\top \\
			\bR
		\end{bmatrix}
		=
		\begin{bmatrix}
			\bzero \\
			\bR^\prime
		\end{bmatrix}.
		$$
		Then we find out the expression for the Cholesky factor of $\bA^\prime$ by $\bR^\prime$.
		Specifically, multiply the left-hand side (l.h.s.,) of above equation by its transpose,
		$$
		\begin{bmatrix}
			\bv & \bR^\top
		\end{bmatrix}
		\bQ_1 \ldots \bQ_{n-1}\bQ_n
		\bQ_n \bQ_{n-1}\ldots \bQ_1
		\begin{bmatrix}
			\bv^\top \\
			\bR
		\end{bmatrix}
		= \bR^\top\bR + \bv \bv^\top.
		$$
		And multiply the right-hand side (r.h.s.,) by its transpose,
		$$
		\begin{bmatrix}
			\bzero & \bR^{\prime\top}
		\end{bmatrix}
		\begin{bmatrix}
			\bzero \\
			\bR^\prime
		\end{bmatrix}=\bR^{\prime\top}\bR^\prime,
		$$
		which agrees with the l.h.s., equation. Givens rotations are such orthogonal matrices that can transfer $\bR,\bv$ into $\bR^\prime$. We will discuss the intrinsic meaning of Givens rotation shortly to prove the existence of QR decomposition in Section~\ref{section:qr-givens} (p.~\pageref{section:qr-givens}). Here, we only introduce the definition of it and write out the results directly. Feel free to skip this section for a first reading.

		\begin{definition}[$n$-th Order Givens Rotation]
			A Givens rotation is represented by a matrix of the following form

			$$
			\bG_{kl}=
			\begin{bmatrix}
				1 &          &   &  &   &   & &  & &\\
				& \ddots  &  &  &  & && & &\\
				&      & 1 &  & & &  && &\\
				&      &  & c &  &  &  & s & &\\
				&& &   & 1 & & && &\\
				&& &   &   &\ddots &  && &\\
				&& &  &   &  & 1&& &\\
				&& & -s &  &  & &c& &\\
				&& & &  &  & & &1 & \\
				&& & &  &  & & & &\ddots
			\end{bmatrix}_{n\times n},
			$$
			where the $(k,k), (k,l), (l,k), (l,l)$ entries are $c, s, -s, c$ respectively, and $s = \cos \theta$ and $c=\cos \theta$ for some $\theta$.

			Let $\bdelta_k \in \real^n$ be the zero vector except that the entry $k$ is 1. Then mathematically, the Givens rotation defined above can be denoted by
			$$
			\bG_{kl}= \bI + (c-1)(\bdelta_k\bdelta_k^\top + \bdelta_l\bdelta_l^\top) + s(\bdelta_k\bdelta_l^\top -\bdelta_l\bdelta_k^\top ).
			$$
		\end{definition}

		It can be easily verified that the $n$-th order Givens rotation is an orthogonal matrix and its determinant is 1. For any vector $\bx =[x_1, x_2, \ldots, x_n]^\top \in \real^n$, we have $\by = \bG_{kl}\bx$, where
		$$
		\left\{
		\begin{aligned}
			&y_k = c \cdot x_k + s\cdot x_l,   \\
			&y_l = -s\cdot x_k +c\cdot x_l,  \\
			&y_j = x_j , &  (j\neq k,l)
		\end{aligned}
		\right.
		$$
		That is, a Givens rotation applied to $\bx$ rotates two components of $\bx$ by some angle $\theta$ and leaves all other components the same.

		Now, suppose we have an $(n+1)$-th order Givens rotation indexed from $0$ to $n$, and it is given by
		$$
		\bG_k = \bI + (c_k-1)(\bdelta_0\bdelta_0^\top + \bdelta_k\bdelta_k^\top) + s_k(\bdelta_0\bdelta_k^\top -\bdelta_k\bdelta_0^\top ),
		$$
		where $c_k = \cos \theta_k, s_k=\sin\theta_k$ for some $\theta_k$, $\bG_k \in \real^{(n+1)\times (n+1)}$, $\delta_k\in \real^{n+1}$ is a zero vector except that the $(k+1)$-th entry is 1.

		\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
			Taking out the $k$-th column of the following equation
			$$
			\begin{bmatrix}
				\bv^\top \\
				\bR
			\end{bmatrix}
			=
			\begin{bmatrix}
				\bzero \\
				\bR^\prime
			\end{bmatrix},
			$$
			where we let the $k$-th element of $\bv$ be $v_k$, and the $k$-th diagonal of $\bR$ be $r_{kk}$.
			We realize that $\sqrt{v_k^2 + r_{kk}^2} \neq 0$,
			let $c_k = \frac{r_{kk}}{\sqrt{v_k^2 + r_{kk}^2}}$, $s_k=-\frac{v_k}{\sqrt{v_k^2 + r_{kk}^2}}$. Then,
			$$
			\left\{
			\begin{aligned}
				&v_k \rightarrow c_kv_k+s_kr_{kk}=0;   \\
				&r_{kk}\rightarrow -s_k v_k +c_kr_{kk}= \sqrt{v_k^2 + r_{kk}^2} = r^\prime_{kk} .  \\

			\end{aligned}
			\right.
			$$
			That is, $G_k$ will introduce zero value to the $k$-th element to $\bv$ and nonzero value to $r_{kk}$.
		\end{mdframed}
		This finding above is essential for the rank-one update. And we obtain
		$$
		\bG_n \bG_{n-1}\ldots \bG_1
		\begin{bmatrix}
			\bv^\top \\
			\bR
		\end{bmatrix}
		=
		\begin{bmatrix}
			\bzero \\
			\bR^\prime
		\end{bmatrix}.
		$$
		For each Givens rotation, it takes $6n$ flops. And there are $n$ such rotations, which requires $6n^2$ flops if keeping only the leading term. The complexity to calculate the Cholesky factor of $\bA^\prime$ is thus reduced from $\frac{1}{3} n^3$ to $6n^2$ flops if we already know the Cholesky factor of $\bA$ by the rank-one update. The above algorithm is essential to reduce the complexity of the posterior calculation in the Bayesian inference for Gaussian mixture model \citep{lu2021bayes}.

		\subsection{Rank-One Downdate}
		Now suppose we have calculated the Cholesky factor of $\bA$, and the $\bA^\prime$ is the downdate of $\bA$ as follows:
		\begin{equation*}
			\begin{aligned}
				\bA^\prime &= \bA - \bv \bv^\top\\
				\bR^{\prime\top}\bR^\prime &= \bR^\top\bR - \bv \bv^\top.
			\end{aligned}
		\end{equation*}
		The algorithm is similar by proceeding as follows:
		\begin{equation}\label{equation:rank-one-downdate}
			\bG_1 \bG_{2}\ldots \bG_n
			\begin{bmatrix}
				\bzero \\
				\bR
			\end{bmatrix}
			=
			\begin{bmatrix}
				\bv^\top \\
				\bR^\prime
			\end{bmatrix}.
		\end{equation}
		Again, $
		\bG_k = \bI + (c_k-1)(\bdelta_0\bdelta_0^\top + \bdelta_k\bdelta_k^\top) + s_k(\bdelta_0\bdelta_k^\top -\bdelta_k\bdelta_0^\top ),
		$ can be constructed as follows:

		\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
			Taking out the $k$-th column of the following equation
			$$
			\begin{bmatrix}
				\bzero \\
				\bR
			\end{bmatrix}
			=
			\begin{bmatrix}
				\bv^\top \\
				\bR^\prime
			\end{bmatrix}.
			$$
			We realize that $r_{kk} \neq 0$,
			let $c_k=\frac{\sqrt{r_{kk}^2 - v_k^2}}{r_{kk}}$, $s_k = \frac{v_k}{r_{kk}}$. Then,
			$$
			\left\{
			\begin{aligned}
				& 0 \rightarrow s_kr_{kk}=v_k;   \\
				&r_{kk}\rightarrow c_k r_{kk}= \sqrt{r_{kk}^2-v_k^2  }=r^\prime_{kk} .  \\

			\end{aligned}
			\right.
			$$
			This requires $r^2_{kk} > v_k^2$ to make $\bA^\prime$ to be positive definite. Otherwise, $c_k$ above will not exist.
		\end{mdframed}
		Again, one can check that, multiply the l.h.s., of Equation~\eqref{equation:rank-one-downdate} by its transpose, we have
		$$
		\begin{bmatrix}
			\bzero & \bR^\top
		\end{bmatrix}
		\bG_n \ldots \bG_{2}\bG_1
		\bG_1 \bG_{2}\ldots \bG_n
		\begin{bmatrix}
			\bzero \\
			\bR
		\end{bmatrix} =\bR^\top\bR.
		$$
		And multiply the r.h.s., by its transpose, we have
		$$
		\begin{bmatrix}
			\bv & \bR^{\prime\top}
		\end{bmatrix}
		\begin{bmatrix}
			\bv^\top \\
			\bR^\prime
		\end{bmatrix}=\bv\bv^\top + \bR^{\prime\top}\bR^\prime.
		$$
		This results in $\bR^{\prime\top}\bR^\prime = \bR^\top\bR - \bv \bv^\top$.

		\section{Application: Indefinite Rank Two Update}\index{Rank-two update}

		Let $\bA = \bR^\top\bR$ be the Cholesky decomposition of $\bA$, \citep{goldfarb1976factorized, seeger2004low} give a stable method for the indefinite rank-two update of the form
		$$
		\bA^\prime = (\bI+\bv\bu^\top)\bA(\bI+\bu\bv^\top).
		$$
		Let
		$$
		\bigg\{
		\begin{aligned}
			\bz &= \bR^{-\top}\bv,   \\
			\bw &= \bR\bu,
		\end{aligned}
		\qquad
		\rightarrow
		\qquad
		\bigg\{
		\begin{aligned}
			\bv &= \bR^{\top}\bz,   \\
			\bu &= \bR^{-1}\bw.
		\end{aligned}
		$$
		And suppose the LQ decomposition \footnote{We will shortly introduce in Theorem~\ref{theorem:lq-decomposition} (p.~\pageref{theorem:lq-decomposition}).} of $\bI+\bz\bw^\top$ is given by $\bI+\bz\bw^\top =\bL\bQ$, where $\bL$ is lower triangular and $\bQ$ is orthogonal. Thus, we have
		$$
		\begin{aligned}
			\bA^\prime &= (\bI+\bv\bu^\top)\bA(\bI+\bu\bv^\top)\\
			&= (\bI+\bR^{\top}\bz   \bw^\top \bR^{-\top })\bA(\bI+\bR^{-1}\bw \bz^\top\bR)\\
			&= \bR^\top (\bI+\bz\bw^\top)(\bI+\bw\bz^\top)\bR\\
			&= \bR^\top\bL \bQ\bQ^\top \bL^\top\bR \\
			&= \bR^\top\bL \bL^\top\bR.
		\end{aligned}
		$$
		Let $\bR^\prime = \bR^\top\bL$ which is lower triangular, we find the Cholesky decomposition of $\bA^\prime$.

\part{Special Topics}

\newpage
\chapter{Coordinate Transformation in Matrix Decomposition}\label{section:coordinate-transformation}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

Suppose a vector $\bv\in \real^3$ and it has elements $\bv = [3;7; 2]$. But what do these values 3, 7, and 2 mean? In the Cartesian coordinate system, it means it has a component of 3 on the $x$-axis, a component of 7 on the $y$-axis, and a component of 2 on the $z$-axis.\index{Coordinate transformation}

\section{An Overview of Matrix Multiplication}
\paragraph{Coordinate defined by a nonsingular matrix} Suppose further a $3\times 3$ nonsingular matrix $\bB$ which means $\bB$ is invertible and columns of $\bB$ are linearly independent. Thus the 3 columns of $\bB$ form a basis for the space $\real^{3}$. One step forward, we can take the 3 columns of $\bB$ as a basis for a \textcolor{blue}{new coordinate system}, which we call the \textcolor{blue}{$B$ coordinate system}. Going back to the Cartesian coordinate system, we also have three vectors as a basis, $\be_1, \be_2, \be_3$. If we put the three vectors into columns of a matrix, the matrix will be an identity matrix. So $\bI\bv = \bv$ means \textcolor{blue}{transfer $\bv$ from the Cartesian coordinate system into the Cartesian coordinate system}, the same coordinate. Similarly, \textcolor{blue}{$\bB\bv=\bu$ is to transfer $\bv$ from the Cartesian coordinate system into the $B$ system}. Specifically, for $\bv = [3;7; 2]$ and $\bB=[\bb_1, \bb_2, \bb_3]$, we have $\bu=\bB\bv = 3\bb_1+7\bb_2+2\bb_3$, i.e., $\bu$ contains 3 of the first basis $\bb_1$ of $\bB$, 7 of the second basis $\bb_2$ of $\bB$, and 2 of the third basis $\bb_3$ of $\bB$.
If again, we want to transfer the vector $\bu$ from $B$ coordinate system back to the Cartesian coordinate system, we just need to multiply by $\bB^{-1}\bu = \bv$.

\paragraph{Coordinate defined by an orthogonal matrix} A $3\times 3$ orthogonal matrix $\bQ$ defines a ``better" coordinate system since the three columns (i.e., basis) are orthonormal to each other. $\bQ\bv$ is to transfer $\bv$ from the Cartesian to the coordinate system defined by the orthogonal matrix. Since the basis vectors from the orthogonal matrix are orthonormal, just like the three vectors $\be_1, \be_2, \be_3$ in the Cartesian coordinate system, the transformation defined by the orthogonal matrix just rotates or reflects the Cartesian system.
$\bQ^\top$ can help transfer back to the Cartesian coordinate system.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/eigenRotate.pdf}
	\caption{Eigenvalue Decomposition: $\bX^{-1}$ transforms to a different coordinate system. $\bLambda$ stretches and $\bX$ transforms back. $\bX^{-1}$ and $\bX$ are nonsingular, which will change the basis of the system, and the angle between the vectors $\bv_1$ and $\bv_2$ will \textbf{not} be preserved, that is, the angle between $\bv_1$ and $\bv_2$ is \textbf{different} from the angle between $\bv_1^\prime$ and $\bv_2^\prime$. The length of $\bv_1$ and $\bv_2$ are also \textbf{not} preserved, that is, $||\bv_1|| \neq ||\bv_1^\prime||$ and $||\bv_2|| \neq ||\bv_2^\prime||$.}
	\label{fig:eigen-rotate}
\end{figure}

\section{Eigenvalue Decomposition}
A square matrix $\bA$ with linearly independent eigenvectors can be factored as $\bA = \bX\bLambda\bX^{-1}$ where $\bX$ and $\bX^{-1}$ are nonsingular so that they define a system transformation intrinsically. $\bA\bu = \bX\bLambda\bX^{-1}\bu$ firstly transfers $\bu$ into the system defined by $\bX^{-1}$. Let's call this system the \textbf{eigen coordinate system}. $\bLambda$ is to stretch each component of the vector in the eigen system by the length of the eigenvalue. And then $\bX$ helps to transfer the resulting vector back to the Cartesian coordinate system. A demonstration of how the eigenvalue decomposition transforms between coordinate systems is shown in Figure~\ref{fig:eigen-rotate} where $\bv_1, \bv_2$ are two linearly independent eigenvectors of $\bA$ such that they form a basis for $\real^2$.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/spectralrotate.pdf}
	\caption{Spectral Decomposition $\bQ\bLambda \bQ^\top$: $\bQ^\top$ rotates or reflects, $\bLambda$ stretches cycle to ellipse, and $\bQ$ rotates or reflects back. Orthogonal matrices $\bQ^\top$ and $\bQ$ only change the basis of the system. However, they preserve the angle between the vectors $\bq_1$ and $\bq_2$, and the lengths of them.}
	\label{fig:spectral-rotate}
\end{figure}

\section{Spectral Decomposition}
A symmetric matrix $\bA$ can be factored as $\bA = \bQ\bLambda\bQ^\top$ where $\bQ$ and $\bQ^\top$ are orthogonal so that they define a system transformation intrinsically. $\bA\bu = \bQ\bLambda\bQ^\top\bu$ firstly rotates or reflects $\bu$ into the system defined by $\bQ^\top$. Let's call this system the \textbf{spectral coordinate system}. $\bLambda$ is to stretch each component of the vector in the spectral system by the length of eigenvalue. And then $\bQ$ helps to rotate or reflect the resulting vector back to the original coordinate system. A demonstration of how the spectral decomposition transforms between coordinate systems is shown in Figure~\ref{fig:spectral-rotate} where $\bq_1, \bq_2$ are two linearly independent eigenvectors of $\bA$ such that they form a basis for $\real^2$. The coordinate transformation in the spectral decomposition is similar to that of the eigenvalue decomposition. Except that in the spectral decomposition, the orthogonal vectors transferred by $\bQ^\top$ are still orthogonal. This is also a property of orthogonal matrices. That is, orthogonal matrices can be viewed as matrices which change the basis of other matrices. Hence they preserve the angle (inner product) between the vectors
$$
\bu^\top \bv = (\bQ\bu)^\top(\bQ\bv).
$$
The above invariance of the inner products of angles between the vectors are preserved, which also relies on the invariance of their lengths:
$$
 ||\bQ\bu|| = ||\bu||.
$$

\section{SVD}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/svdrotate.pdf}
	\caption{SVD: $\bV^\top$ and $\bU$ rotate or reflect, $\bSigma$ stretches the circle to an ellipse. Orthogonal matrices $\bV^\top$ and $\bU$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$, and the lengths of them.}
	\label{fig:svd-rotate}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/polarrotate.pdf}
	\caption{$\bV\bSigma \bV^\top$ from SVD or Polar decomposition: $\bV^\top$ rotates or reflects, $\bSigma$ stretches cycle to ellipse, and $\bV$ rotates or reflects back. Orthogonal matrices $\bV^\top$ and $\bV$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$, and the lengths of them.}
	\label{fig:polar-rotate}
\end{figure}
Any $m\times n$ matrix can be factored as $\bA=\bU\bSigma\bV^\top$. $\bA\bu=\bU\bSigma\bV^\top\bu$ then firstly rotates or reflects $\bu$ into the system defined by $\bV^\top$, which we call the \textbf{$V$ coordinate system}. $\bSigma$ stretches the first $r$ components of the resulted vector in the $V$ system by the lengths of the singular value. If $n\geq m$, then $\bSigma$ only keeps additional $m-r$ components which are stretched to zero while removing the final $n-m$ components. If $m>n$, the $\bSigma$ stretches $n-r$ components to zero, and also adds additional $m-n$ zero components. Finally, $\bU$ rotates or reflects the resulting vector into the \textbf{$U$ coordinate system} defined by $\bU$. A demonstration of how the SVD transforms in a $2\times 2$ example is shown in Figure~\ref{fig:svd-rotate}. Further, Figure~\ref{fig:polar-rotate} demonstrates the transformation of $\bV\bSigma \bV^\top$ in a $2\times 2$ example.
Similar to the spectral decomposition, orthogonal matrices $\bV^\top$ and $\bU$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/polarrotate2.pdf}
	\caption{Polar decomposition: $\bV^\top$ rotates or reflects, $\bSigma$ stretches cycle to ellipse, and $\bV$ rotates or reflects back. Orthogonal matrices $\bV^\top$, $\bV$, $\bQ_l$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$, and the lengths of them.}
	\label{fig:polar-rotate2}
\end{figure}

\section{Polar Decomposition}
Any $n\times n$ square matrix $\bA$ can be factored as left polar decomposition $\bA = (\bU\bV^\top)( \bV\bSigma \bV^\top) = \bQ_l\bS$. Similarly, $\bA\bv = \bQ_l( \bV\bSigma \bV^\top)\bu$ is to transfer $\bu$ into the system defined by $\bV^\top$ and stretch each component by the lengths of the singular values. Then the resulted vector is transferred back into the Cartesian coordinate system by $\bV$. Finally, $\bQ_l$ will rotate or reflect the resulting vector from the Cartesian coordinate system into the $Q$ system defined by $\bQ_l$. The meaning of right polar decomposition shares a similar description. Similar to the spectral decomposition, orthogonal matrices $\bV^\top$ and $\bV$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$.

\part{Data Interpretation and Information Distillation}\label{part:data-interation}
\section*{Introduction}
For matrix $\bA\in \real^{m\times n}$ with rank $r$, there exist $r$ linearly independent columns and rows respectively by the fundamental theorem of linear algebra (Theorem~\ref{theorem:fundamental-linear-algebra}, p.~\pageref{theorem:fundamental-linear-algebra}). Then $\bA$ admits
$$
\text{(DI1)} \qquad \underset{m\times n}{\bA} = \underset{m\times r}{\bC} \gap \underset{r\times n}{\bF},
$$
where $\bC$ contains $r$ linearly independent columns of $\bA$ and $\bF$ is to reconstruct all the columns of $\bA$ since all the columns of $\bC\bF$ are combinations of the columns of $\bC$. To see this, suppose $\bF=[\bff_1, \bff_2, \ldots, \bff_n]$ is the column partition of $\bC$, we have
$$
\bA = \bC\bF=[\bC\bff_1, \bC\bff_2, \ldots, \bC\bff_n],
$$
where each column $\bC\bff_i$ is a combination of the columns of $\bC$. And the columns of $\bC$ are known as the \textit{spanning columns} of $\bA$.
Or $\bA$ admits
$$
\text{(DI2)} \qquad \underset{m\times n}{\bA} = \underset{m\times r}{\bD} \gap \underset{r\times n}{\bR},
$$
where $\bR$ contains $r$ linearly independent rows of $\bA$ and $\bD$ is to reconstruct all the rows of $\bA$ since all the rows of $\bD\bR$ are combinations of the rows of $\bR$. To see this, suppose $\bD=[\bd_1^\top; \bd_2^\top; \ldots; \bd_m^\top]$ is the row partition of $\bD$, we have
$$
\bA = \bD\bR=
\begin{bmatrix}
	\bd_1^\top\bR \\
	\bd_2^\top\bR \\
	\vdots \\
	\bd_m^\top\bR
\end{bmatrix},
$$
where each row $\bd_i^\top\bR$ is a combination of the rows of $\bR$. And the rows of $\bR$ are known as the \textit{spanning rows} of $\bA$.

This factorization in (DI1) is similar to the QR decomposition where the first factor spans the column space of $\bA$ (orthogonal in the latter), whilst (DI2) is similar to the LQ decomposition.  However (DI1) has several advantages, as compared to, e.g., the QR:
\begin{itemize}
\item It is sometimes advantageous to work with a basis that consists of a subset of the
columns of $\bA$ itself. In this case, one typically has to give up on the requirement that the basis
vectors are orthogonormal;
\item If $\bA$ is sparse and nonnegative, then $\bC$ shares these properties;
\item The (DI1) requires less memory to store and is more efficient to calculate than the QR in general. One can see from the above representation that only $r(m+n)$ entries have to be
stored instead of $mn$ entries of the original matrix $\bA$ or $mn+\frac{(n+1)n}{2}$ entries in the \textit{reduced} QR decomposition (Figure~\ref{fig:qr-comparison}, p.~\pageref{fig:qr-comparison});
\item Finding the indices associated with the spanning columns is often helpful for the purpose of data interpretation and analysis, it can be very useful to identify a subset of the columns that distills the information in the matrix;
\item Finding the least squares solution $\bA\bx=\bb$ can be done by calculating the least squares solution of $\bC\widetildebx=\bb$ where the former one is to project $\bb$ into the column space of $\bA$ and the latter one is to project it into the column space of $\bC$. This factorization (DI1) can be regarded as the first phase of the variable selection in least squares or linear models in general. A short review of the variable selection procedure is given in Section~\ref{section:append-column-qr} (p.~\pageref{section:append-column-qr}) as an application of QR decomposition, and more details can refer to \citep{lu2021rigorous};
\item The (DI1) often preserves ``the physics" of a problem in a way that the QR does not.
\end{itemize}

For the rest of this part, we will discuss several variations of the factorization above with different focuses, e.g., in terms of description, easy to interpret or not, or well-condition, and how they select the basis vectors for the column space, for the row space, or for both. The big picture can be shown in Figure~\ref{fig:data-interpretation-world-picture}.

\begin{figure}[htbp]
\centering
\begin{widepage}
	\centering
\resizebox{0.65\textwidth}{!}{%
\begin{tikzpicture}[>=latex]
\tikzstyle{state} = [draw, very thick, fill=white, rectangle, minimum height=3em, minimum width=6em, node distance=8em, font={\sffamily\bfseries}]
\tikzstyle{stateEdgePortion} = [black,thick];
\tikzstyle{stateEdge} = [stateEdgePortion,->];
\tikzstyle{stateEdge2} = [stateEdgePortion,<->];
\tikzstyle{edgeLabel} = [pos=0.5, text centered, font={\sffamily\small}];

\node[state, name=cr, fill={colorcr}] {CR};
\node[state, name=rank, above of=cr,yshift=-3.2em,  fill={colorcr}] {Rank};
\node[state, name=interpolative, left of=cr, yshift=-2em,  fill={colorcr}] {Interpolative};
\node[state, name=skeleton, right of=cr, yshift=-2em, fill={colorcr}] {\parbox{3.5em}{ Skeleton (CUR)}};

\draw (rank.south)
edge[stateEdge] node[edgeLabel, yshift=0em, xshift=-0em]{Special Case}
(cr.north) ;

\draw ($(cr.east) + (0,1em)$)
edge[stateEdge, bend left=12.5] node[edgeLabel, xshift=0.5em, yshift=0.8em]{}
(skeleton.north);
\draw[decoration={text along path,
	text={|\sffamily|Same C},text align={center}},decorate] ($(cr.east) + (-1.8em,1.2em)$) to [bend left=20.5]  ($(skeleton.north) + (2.5em,-1.5em)$);

\draw (rank.west)
edge[stateEdge, bend left=-30] node[edgeLabel, yshift=0em, xshift=-0em]{}
($(interpolative.north) + (-2.5em,0em)$) ;
\draw[decoration={text along path,
text={|\sffamily|Special Case},text align={center}},decorate] ($(interpolative.north) + (-3em,0.3em)$) to [bend left=30]  ($(rank.west) + (-0em,0.em)$) ;

\draw (rank.east)
edge[stateEdge, bend left=30] node[edgeLabel, yshift=0em, xshift=-0em]{}
($(skeleton.north) + (2.5em,0em)$) ;
\draw[decoration={text along path,
text={|\sffamily|``Special" Case},text align={center}},decorate] ($(rank.east) + (0.7em,0.2em)$) to [bend left=30]  ($(skeleton.north) + (2.5em,0.5em)$);

\draw ($(cr.west) + (0,1em)$)
edge[stateEdge, bend left=-12.5] node[edgeLabel, xshift=-0.5em, yshift=0.8em]{}
(interpolative.north);
\draw[decoration={text along path,
text={|\sffamily|Independent},text align={center}},decorate] ($(interpolative.north) + (-0.5em,0.1em)$) to [bend left=12.5]  ($(cr.west) + (.5em,1.2em)$);
\draw[decoration={text along path,
text={|\sffamily|Columns},text align={center}},decorate] ($(interpolative.north) + (-0.8em,-1.1em)$) to [bend left=12.5]  ($(cr.west) + (2.5em,0.3em)$);

\begin{pgfonlayer}{background}
\draw [join=round,cyan,dotted,fill={coloruppermiddle}] ($(rank.north west  -| interpolative.north west) + (-0.8em, +0.8em)$) rectangle ($(skeleton.south east) + (0.8em, -0.8em)$);
\end{pgfonlayer}

\end{tikzpicture}
}
\end{widepage}
\caption{Data interpretation relationship. See also where it's lying in the matrix decomposition world map Figure~\ref{fig:matrix-decom-world-picture}.}
\label{fig:data-interpretation-world-picture}
\end{figure}

\subsection*{\textbf{A Brief Introduction for SVD}}
For the analysis of the	 decomposition in this part, especially the interpolative decomposition, we need the SVD and Eckart-Young-Misky Theorem that we shall briefly introduce here. More details are delayed in Section~\ref{section:SVD} (p.~\pageref{section:SVD}). For any matrix $\bA\in \real^{m\times n}$ with rank $r$, it admits
$$
 \bA = \bU \bSigma \bV^\top = \sum_{i=1}^r \sigma_i \bu_i \bv_i^\top,
 $$
 where $\bu_i$ and $\bv_i$ are left and right singular vectors respectively.  The matrix $\bSigma\in \real^{m\times n}$ is a rectangular diagonal matrix whose entries are singular values in descending
order, $\sigma_1\geq \sigma_2\geq \ldots \geq \sigma_r \textcolor{blue}{>} \sigma_{r+1}=\ldots=0$, along the main diagonal with only $r$ nonzero singular values.

Given further $1\leq k\leq r$, and let $\bA_k$ be the \textit{truncated SVD (TSVD)} of $\bA$ with the largest $k$ terms, i.e., $\bA_k = \sum_{i=1}^{k} \sigma_i\bu_i\bv_i^\top$ from SVD of $\bA=\sum_{i=1}^{r} \sigma_i\bu_i\bv_i^\top$ by zeroing out the $r-k$ trailing singular values of $\bA$. Then $\bA_k$ is the best rank-$k$ approximation to $\bA$ in terms of the spectral norm. That is, for any rank $r$ matrix $\bB$, we have $||\bA-\bA_k||_2 \leq ||\bA-\bB||_2$ or $||\bA-\bA_k||_F \leq ||\bA-\bB||_F$ which are measured by Frobenius norm and spectral norm respectively. And the distance between $\bA,\bA_k$ is given by
$$
||\bA-\bA_k||_F = \sqrt{\sigma_{k+1}^2+\ldots +\sigma_{r}^2},
$$
or
$$
||\bA-\bA_k||_2 = \sigma_{r}.
$$
\subsection*{\textbf{Subset Selection Problem}}

Subset selection is a method for selecting a subset of columns from a real matrix so that the subset represents the entire matrix well and is far from being rank deficient. (DI1) above is just an example of column selection with full rank $r$ columns. However, given integer $k<r$, we always attempt to find $k$ linearly independent columns that best represent the information in the matrix. The mathematical formulation of the subset selection problems is: Determine a permutation matrix $\bP$ such that
$$
\bA\bP = [\bA_1, \bA_2],
$$
where $\bA_1\in \real^{m\times k}$ contains $k$ linearly independent columns of $\bA$ such that
\begin{enumerate}
\item The smallest singular value is as large as possible (similar to SVD!). That is, there exists a value $\eta$ such that the $k$-th singular value of $\bA_1$ is bounded by that of $\bA$:
$$
\frac{\sigma_k(\bA)}{\eta} \leq \sigma_k(\bA_1) \leq \sigma_k(\bA).
$$
\item The rest $n-k$ redundant columns of $\bA_2$ are well represented by $k$ columns of $\bA_1$. That
is
$$
\mathop{\min}_{\bW \in \real^{k\times (n-k)}} ||\bA_1\bW-\bA_2||_2
$$
is small, i.e., there exists a value $\eta$ such that the distance is bounded by $(k+1)$-th singular value of $\bA$:
$$
\sigma_{k+1}(\bA) \leq \mathop{\min}_{\bW \in \real^{k\times (n-k)}} ||\bA_1\bW-\bA_2||_2 \leq \eta \sigma_{k+1}(\bA).
$$
\end{enumerate}

\newpage
\chapter{CR Decomposition}\label{section:cr-decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{CR Decomposition}
The CR decomposition is proposed in \citep{strang2021every, stranglu}. As usual, we firstly give the result and we will discuss the existence and the origin of this decomposition in the following sections.
\begin{theoremHigh}[CR Decomposition]\label{theorem:cr-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA} = \underset{m\times r}{\bC} \gap \underset{r\times n}{\bR}
$$
where $\bC$ is the first $r$ linearly independent columns of $\bA$, and $\bR$ is an $r\times n$ matrix to reconstruct the columns of $\bA$ from columns of $\bC$. In particular, $\bR$ is the row reduced echelon form (RREF) of $\bA$ without the zero rows.

\item The storage for the decomposition is then reduced or potentially increased from $mn$ to $r(m+n)$.
\end{theoremHigh}

\section{Existence of the CR Decomposition}

Since matrix $\bA$ is of rank $r$, there are some $r$ linearly independent columns in $\bA$. We then choose linearly independent columns from $\bA$ and put them into $\bC$:

\begin{tcolorbox}[title={Find $r$ linearly Independent Columns From $\bA$}]
1. If column 1 of $\bA$ is not zero, put it into the column of $\bC$;

2. If column 2 of $\bA$ is not a multiple of column 1, put it into the column of $\bC$;

3. If column 3 of $\bA$ is not a combination of columns 1 and 2, put it into the column of $\bC$;

4. Continue this process until we find $r$ linearly independent columns (or all the linearly independent columns if we do not know the rank $r$ beforehand).
\end{tcolorbox}

When we have the $r$ linearly independent columns from $\bA$, we can prove the existence of CR decomposition by the column space view of matrix multiplication.

\paragraph{Column space view of matrix multiplication} A multiplication of two matrices $\bD\in \real^{m\times k}, \bE\in \real^{k\times n}$ is $\bA=\bD\bE=\bD[\be_1, \be_2, \ldots, \be_n] = [\bD\be_1, \bD\be_2, \ldots, \bD\be_n]$, i.e., each column of $\bA$ is a combination of columns from $\bD$.

\begin{proof}[of Theorem~\ref{theorem:cr-decomposition}]
As the rank of matrix $\bA$ is $r$ and $\bC$ contains $r$ linearly independent columns from $\bA$, the column space of $\bC$ is equivalent to the column space of $\bA$.
If we take any other column $\ba_i$ of $\bA$, $\ba_i$ can be represented as a linear combination of the columns of $\bC$, i.e., there exists a vector $\br_i$ such that $\ba_i = \bC \br_i$, $\forall i\in \{1, 2, \ldots, n\}$. Put these $\br_i$'s into the columns of matrix $\bR$, we obtain
$$
\bA = [\ba_1, \ba_2, \ldots, \ba_n] = [\bC \br_1, \bC \br_2, \ldots, \bC \br_n]= \bC \bR,
$$
from which the result follows.
\end{proof}

\section{Reduced Row Echelon Form (RREF)}\label{section:rref-cr}
In Gaussian elimination Section~\ref{section:gaussian-elimination}, we introduced the elimination matrix (a lower triangular matrix) and permutation matrix to transform $\bA$ into an upper triangular form. We rewrite the Gaussian elimination for a $4\times 4$ square matrix, where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed:
\begin{tcolorbox}[title={Gaussian Elimination for a Square Matrix}]
$$
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bP_1}{\longrightarrow}
\begin{sbmatrix}{\bP_1\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bP_1\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & \bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix}.
$$
\end{tcolorbox}

Furthermore, the Gaussian elimination can also be applied on a rectangular matrix, we give an example for a $4\times 5$ matrix as follows:
\begin{tcolorbox}[title={Gaussian Elimination for a Rectangular Matrix}]
	$$
	\begin{sbmatrix}{\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\end{sbmatrix}
	\stackrel{\bE_1}{\longrightarrow}
	\begin{sbmatrix}{\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		\bm{0} & \bm{0} & \textcolor{blue}{\bm{5}} & \bm{6} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \bm{2} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\end{sbmatrix}
	\stackrel{\bE_2}{\longrightarrow}
	\begin{sbmatrix}{\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		0 & 0 & \textcolor{blue}{5} & 6 & \boxtimes\\
		0 & 0 & \bm{0} & \textcolor{blue}{\bm{3}} & \bm{\boxtimes}\\
		0 & 0 & \bm{0} & \bm{0} & \bm{0}\\
	\end{sbmatrix},
	$$
\end{tcolorbox}
\noindent where the \textcolor{blue}{blue}-colored numbers are \textbf{pivots} as we defined previously and we call the last matrix above \textbf{row echelon form}. Note that we get the 4-th row as a zero row in this specific example. Going further, if we subtract each row by a multiple of the next row to make the entries above the pivots to be zero:
\begin{tcolorbox}[title={Reduced Row Echelon Form: Get Zero Above Pivots}]
	$$
	\begin{sbmatrix}{\bE_2\bE_1\bA}
	\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
	0 & 0 & \textcolor{blue}{5} & 6 & \boxtimes\\
	0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
\end{sbmatrix}
\stackrel{\bE_3}{\longrightarrow}
	\begin{sbmatrix}{\bE_3\bE_2\bE_1\bA}
	\textcolor{blue}{2} & \boxtimes & \bm{0}  & \bm{-3}  & \bm{\boxtimes} \\
	0 & 0 & \textcolor{blue}{5} & 6 & \boxtimes\\
	0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
\end{sbmatrix}
\stackrel{\bE_4}{\longrightarrow}
\begin{sbmatrix}{\bE_4\bE_3\bE_2\bE_1\bA}
	\textcolor{blue}{2} & \boxtimes & 0 & \bm{0} & \bm{\boxtimes}\\
	0 & 0 & \textcolor{blue}{5} & \bm{0} & \bm{\boxtimes}\\
	0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
\end{sbmatrix},
	$$
\end{tcolorbox}
\noindent where $\bE_3$ subtracts 2 times the $2$-nd row from the $1$-st row, and $\bE_4$ adds the $3$-rd row to the $1$-st row and subtracts 2 times the $3$-rd row from the $2$-nd row. Finally, we get the full row reduced echelon form by making the pivots to be 1:
\begin{tcolorbox}[title={Reduced Row Echelon Form: Make The Pivots To Be 1}]
	$$
	\begin{sbmatrix}{\bE_4\bE_3\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 0 & 0 & \boxtimes\\
		0 & 0 & \textcolor{blue}{5} & 0 & \boxtimes\\
		0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
		0 & 0 & 0 & 0 & 0\\
	\end{sbmatrix}
\stackrel{\bE_5}{\longrightarrow}
\begin{sbmatrix}{\bE_5\bE_4\bE_3\bE_2\bE_1\bA}
	\textcolor{blue}{\bm{1}} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{\boxtimes}\\
	\bm{0} & \bm{0} & \textcolor{blue}{\bm{1}} & \bm{0} & \bm{\boxtimes}\\
	\bm{0} & \bm{0} & \bm{0} & \textcolor{blue}{\bm{1}} & \bm{\boxtimes}\\
	0 & 0 & 0 & 0 & 0\\
\end{sbmatrix},
	$$
\end{tcolorbox}
\noindent where $\bE_5$ makes the pivots to be 1. Note here, the transformation matrix $\bE_1, \bE_2, \ldots, \bE_5$ are not necessarily to be lower triangular matrices as they are in LU decomposition. They can also be permutation matrices or other matrices. We call this final matrix the \textbf{reduced row echelon form} of $\bA$ where it has 1's as pivots and zeros above the pivots.

\begin{lemma}[Rank and Pivots]\label{lemma:rank-is-pivots}
	The rank of $\bA$ is equal to the number of pivots.
\end{lemma}

\begin{lemma}[RREF in CR\index{Reduced row echelon form}]\label{lemma:r-in-cr-decomposition}
	The reduced row echelon form of the matrix $\bA$ without zero rows is the matrix $\bR$ in the CR decomposition.
\end{lemma}

\begin{proof}[Informal Proof of Lemma~\ref{lemma:r-in-cr-decomposition} and Lemma~\ref{lemma:rank-is-pivots}]
Following the steps in Gaussian elimination, the number of pivot elements indicates the number of linearly independent columns in matrix $\bA$, which is on the other hand, exactly the rank of the matrix.

Following the example above, we have
$$
\bE_5\bE_4\bE_3\bE_2\bE_1\bA = \bR \quad \longrightarrow \quad  \bA = (\bE_5\bE_4\bE_3\bE_2\bE_1)^{-1}\bR.
$$
We notice that the columns 1, 3, 4 of $\bR$ has only one element 1 which means we could construct a matrix $\bC$ (exactly the same column matrix in CR decomposition) whose first 3 columns to be equal to columns 1, 3, 4 of matrix $\bA$, i.e., $\bC=[\ba_1, \ba_3, \ba_4]$. Furthermore, since the last row of $\bR$ is all zero, the column 4 of $\bC$ does not account for any computation we can just ignore column 4 of $\bC$ and row 4 of $\bR$. And this $\bC$ is the only matrix that can reconstruct the columns 1, 3, 4 of $\bA$ as the pivots of $\bR$ are all 1. We obtain
$$
\bA=\bC\bR.
$$
This completes the proof.
\end{proof}
In short, we first compute the reduced row echelon form of matrix $\bA$ by $rref(\bA)$, Then $\bC$ is obtained by removing from $\bA$ all the non-pivot columns (which can be determined by looking for columns in $rref(\bA)$ which do not contain a pivot). And $\bR$ is obtained by eliminating zero rows of $rref(\bA)$. And this is actually a special case of \textbf{rank decomposition} of matrix $\bA$. However, CR decomposition is so special that it involves the reduced row echelon form so that we introduce it here particularly.

$\bR$ has a remarkable form whose $r$ columns containing the pivots form an $r\times r$ identity matrix. Note again that we can just remove the zero rows from the row reduced echelon form to obtain this matrix $\bR$. In \citep{strang2021every}, the authors give a specific notation for the row reduced echelon form without removing the zero rows as $\bR_0$:
$$
\bR_0 = rref(\bA)=
\begin{bmatrix}
	\bR \\
	\bzero
\end{bmatrix}=
\begin{bmatrix}
	\bI_r & \bF \\
	\bzero & \bzero
\end{bmatrix}\bP, \footnote{Permutation matrix $\bP$ in the right side of a matrix is to permute the column of that matrix.
}
$$
where the $n\times n$ permutation matrix $\bP$ puts the columns of $r\times r$ identity matrix $\bI_r$ into the correct positions, matching the first $r$ linearly independent columns of the original matrix $\bA$.

Previously we proved the important theorem in linear algebra that the row rank equals the column rank of any matrix by the UTV framework (Theorem~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}).
The CR decomposition also reveals the theorem.
\begin{proof}[\textbf{of Theorem~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}, A Second Way}]
For CR decomposition of matrix $\bA=\bC\bR$, we have $\bR = [\bI_r, \bF ]\bP$, where $\bP$ is an $n\times n$ permutation to put the columns of the $r\times r$ identity matrix $\bI_r$ into the correct positions as shown above. It can be easily verified that the $r$ rows of $\bR$ are linearly independent of the submatrix of $\bI_r$ (since $\bI_r$ is nonsingular) such that the row rank of $\bR$ is $r$.

Firstly, from the definition of the CR decomposition, the $r$ columns of $\bC$ are from $r$ linearly independent columns of $\bA$, the column rank of $\bA$ is $r$. Further,

$\bullet$ Since $\bA=\bC\bR$, all rows of $\bA$ are combinations of the rows of $\bR$. That is, the row rank of $\bA$ is no larger than the row rank of $\bR$;

$\bullet$ From $\bA=\bC\bR$, we also have $(\bC^\top\bC)^{-1}\bC^\top\bC\bR = (\bC^\top\bC)^{-1}\bC^\top\bA$, that is $\bR = (\bC^\top\bC)^{-1}\bC^\top\bA$. $\bC^\top\bC$ is nonsingular since it has full column rank $r$. Then all rows of $\bR$ are also combinations of the rows of $\bA$. That is, the row rank of $\bR$ is no larger than the row rank of $\bA$;

$\bullet$ By ``sandwiching", the row rank of $\bA$ is equal to the row rank of $\bR$ which is $r$.

Therefore, both the row rank and column rank of $\bA$ are equal to $r$ from which the result follows.
\end{proof}

In the proof above, we use CR decomposition to show that the row rank of a matrix is equal to its column rank. An elementary proof without using CR decomposition or Gaussian elimination is provided in Appendix~\ref{append:row-equal-column} (p.~\pageref{append:row-equal-column}). Moreover, we also discuss the special form of pseudo-inverse from CR decomposition in Appendix~\ref{appendix:pseudo-inverse} (p.~\pageref{appendix:pseudo-inverse}).

\section{Computing the CR Decomposition via the Gaussian Elimination}
The central step to compute CR decomposition is to find the row reduced echelon form of the matrix $\bA$. Suppose $\bA$ is of size $m\times n$:

$\bullet$ Get row echelon form (REF)\index{Row echelon form}:

A. Use row 1 of $\bA$ to make the values below $\bA_{11}$ to be 0 in the first column (permutation involved if $\bA_{11}=0$), and put the result into $\bR$, ($2(m-1)n+(m-1)$ flops);

B. Use row 2 of the result matrix $\bR$ to make the values under $\bR_{22}$ to be 0 (permutation involved if $\bR_{22}=0$), ($2(m-2)(n-1)+(m-2)$ flops);

C. Continue this process until we get the row echelon form.

$\bullet$ Get row reduced echelon form (RREF)\index{Row reduced echelon form}:

D. Use the last row to make the values above this last pivot to be zero (ignore if the row is all zero);

E. Use the penultimate row to make the values above this second last pivot to be zero (ignore if the row is all zero);

F. Continue this process until we get the row reduced echelon form and divide each row to make the pivots to be 1. Note that there are $m-1$ such steps if $m\leq n$ and $n-1$ such steps if $m>n$;

This process is formulated in Algorithm~\ref{alg:cr-decomposition-alg}.
\begin{algorithm}[H]
	\caption{CR Decomposition via the Gaussian Elimination}
	\label{alg:cr-decomposition-alg}
	\begin{algorithmic}[1]
		\Require
		rank-$r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ with size $m\times n $;
		\State Initially set $\bR=\bA$;
		\State Set $\bb_1= \ba_1$, (Suppose the first column is nonzero); \Comment{$0$ flops}
		\For{$k=1$ to $m-1$}
		\State Use row $k$ of $\bR$ to make the values under $\bR_{kk}$ to be 0 (permutation involved if $\bR_{kk}=0$);
		\EndFor
		\For{($k=m$ to $2$ if $m\leq n$), or ($k=n$ to $2$ if $m > n$)}
		\State Use row $k$ to make the values above this last pivots to be zero (ignore if the row is all zero);
		\EndFor
		\State Make the pivot to be 1;
		\State Select the rows from $\bA$ corresponding to the rows of $\bU$ into $\bR$.
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: CR via the Gaussian Elimination\index{Gaussian elimination complexity}]\label{theorem:cr-decomposition-alg}
Algorithm~\ref{alg:cr-decomposition-alg} requires
$$
\text{cost}=\left\{
\begin{aligned}
	&\sim 2m^2n-m^3 \text{ flops}, \qquad  &\text{if }&m\leq n; \\
	&\sim mn^2 \text{ flops},\qquad &\text{if }&m>n.
\end{aligned}
\right.
$$
to compute the CR decomposition of an $m\times n$ matrix.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:cr-decomposition-alg} ]
Procedure A in the above discussion needs $m-1$ divisions to get the multipliers and it takes $m-1$ times (rows) $n$ multiplications with the multipliers. To make the values under $\bA_{11}$ to be zero, it involves $m-1$ times $n$ subtractions. As a result, procedure A costs \fbox{$2(m-1)n+(m-1)$} flops.

Procedure B needs $m-2$ divisions to get the multipliers and it takes $m-2$ times (rows) $n-1$ multiplications with the multipliers. To make the values under $\bR_{22}$ to be zero, it involves $m-2$ times $n-1$ subtractions. As a result, procedure B costs \fbox{$2(m-2)(n-1)+(m-2)$} flops.

The procedure can go on, and we can thus summarize the costs for each loop of step 4 in the following table:
\begin{table}[H]
	\begin{tabular}{l|l|l|l}
		$k$      & Get multipliers & Multipliers multiply each row & Rows subtraction \\ \hline
		1        & $2:m=m-1$ rows      & $(m-1)(n)$                      & $(m-1)(n)$         \\
		2        & $3:m=m-2$ rows      & $(m-2)(n-1)$                  & $(m-2)(n-1)$     \\
		3        & $4:m=m-3$ rows      & $(m-3)(n-2)$                  & $(m-3)(n-2)$     \\
		$\vdots$ & $\vdots$            & $\vdots$                      & $\vdots$         \\
		$k$      & $k+1:m=m-k$ rows    & $(m-k)(n-k+1)$                & $(m-k)(n-k+1)$   \\
		$\vdots$ & $\vdots$            & $\vdots$                      & $\vdots$         \\
		$m-1$    & $m:m=1$ row        & $(1)(n-m+2)$                  & $(1)(n-m+2)$
	\end{tabular}
\end{table}
We notice that the $n-m+2$ in the last row of the above table may not be positive, so that we separate it into two cases to discuss the complexity of computing the row echelon form (REF).

\textbf{Get REF, case 1}: $m\leq n$, the procedure can go on until the loop $m-1$.
Thus in step 4 to get the row echelon form, we need
$$\sum_{i=1}^{m-1}\left[ 2(m-i)(n-i+1) +(m-i)\right] =\sum_{i=1}^{m-1} \left[2i^2-(2m+2n+3)i+(2mn+3m)\right],
$$
or \fbox{$m^2n-\frac{1}{3}m^3$} flops if keep only the leading terms.

\textbf{Get REF, case 2}: $m>n$, the procedure should stop at loop $n$, otherwise, $n-k+1$ will be zero if $k=n+1$.  Thus, in step 4 to get the row echelon form, we need
$$
\sum_{i=1}^{n} \left[2(m-i)(n-i+1) +(m-i)\right]=\sum_{i=1}^{n} \left[2i^2-(2m+2n+3)i+(2mn+3m)\right],
$$
or \fbox{$mn^2-\frac{1}{3}n^3$} flops if keep only the leading terms.

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[RREF for $m\leq n$]{\label{fig:cr-crcomputation1}
		\includegraphics[width=0.47\linewidth]{./imgs/crcomputation1.pdf}}
	\quad
	\subfigure[RREF for $m> n$]{\label{fig:cr-crcomputation2}
		\includegraphics[width=0.47\linewidth]{./imgs/crcomputation2.pdf}}
	\caption{Get RREF from row echelon form.}
	\label{fig:cr-crcomputation12}
\end{figure}
To get row reduced echelon form (RREF) from row echelon form (REF), we can also separate it into two cases, $m\leq n$ and $m>n$ as shown in Figure~\ref{fig:cr-crcomputation12} where blank entries indicate zeros, blue entries indicate values that are not necessarily zero, pivots are on the diagonal in the ``worst" case.

\textbf{Get RREF, case 1}: $m\leq n$

Procedure D needs $m-1$ divisions to get the multipliers and it takes $m-1$ times $n-m+1$ multiplications with the multipliers. Then to get the zeros above the pivots, it involves $m-1$ times $n-m+1$ subtractions. As a results, procedure D costs \fbox{$2(m-1)(n-m+1)+m-1$} flops.

Procedure E needs $m-2$ divisions to get the multipliers and it takes $m-2$ times $n-m+2$ multiplications with the multipliers. Then to get the zeros above the pivots, it involves $m-2$ times $n-m+2$ subtractions. As a results, procedure E costs \fbox{$2(m-2)(n-m+2)+m-2$} flops.

The procedure can go on, and we can thus again summarize the cost for each loop in the following table:
\begin{table}[H]
	\begin{tabular}{l|l|l|l}
		$k$      & Get multipliers & Multipliers multiply each row & Rows subtraction   \\ \hline
		$m$      & $1:m-1=m-1$ rows      & $(m-1)(n-m+1)$                & $(m-1)(n-m+1)$     \\
		$m-1$    & $1:m-2=m-2$ rows      & $(m-2)(n-m+2)$                & $(m-2)(n-m+2)$     \\
		$m-2$    & $1:m-3=m-3$ rows      & $(m-3)(n-m+3)$                & $(m-3)(n-m+3)$     \\
		$\vdots$ & $\vdots$            & $\vdots$                      & $\vdots$           \\
		$k$      & $1:k-1=k-1$ rows    & $(k-1)(n-k+1)$            & $(k-1)(n-k+1)$ \\
		$\vdots$ & $\vdots$            & $\vdots$                      & $\vdots$           \\
		$2$      & $1:1=1$ row        & $(1)(n-2+1)$                  & $(1)(n-2+1)$
	\end{tabular}
\end{table}

In the second loop to get the row reduced echelon form from the row echelon form, we need
$$
\begin{aligned}
\sum_{k=2}^{m} \left[2(k-1)(n-k+1)+k-1\right]&=\sum_{k=2}^{m} \left[2(k-1)(n-k+1)+k-1\right]\\
&=\sum_{i=1}^{m-1} \left[2(i)(n-i)+i\right], \text{ ($i=k-1$)} \\
&= \sum_{i=1}^{m-1} \left[-2i^2+(2n+1)i\right],
\end{aligned}
$$
or \fbox{$-\frac{2}{3}m^3+m^2n$} flops if we keep only the leading terms.

\textbf{Get RREF, case 2}: $m>n$

Procedure D needs $n-1$ divisions to get the multipliers and it takes $n-1$ times $1$ multiplications with the multipliers. Then to get the zeros above the pivots, it involves $n-1$ times $1$ subtractions. As a result, procedure D costs \fbox{$2(n-1)(1)+n-1$} flops.

Procedure E needs $n-2$ divisions to get the multipliers and it takes $n-2$ times $2$ multiplications with the multipliers. Then to get the zeros above the pivots, it involves $n-2$ times $2$ subtractions. As a result, procedure E costs \fbox{$2(n-2)(2)+n-2$} flops.

The procedure can go on, and we can thus again summarize the cost for each loop in the following table:
\begin{table}[H]
	\begin{tabular}{l|l|l|l}
		$k$      & Get multipliers & Multipliers multiply each row & Rows subtraction   \\ \hline
		$n$      & $1:n-1=n-1$ rows      & $(n-1)(1)$                & $(n-1)(1)$      \\
		$n-1$    & $1:n-2=n-2$ rows      & $(n-2)(2)$                & $(n-2)(2)$        \\
		$n-2$    & $1:n-3=n-3$ rows      & $(n-3)(3)$                & $(n-3)(3)$           \\
		$\vdots$ & $\vdots$            & $\vdots$                      & $\vdots$           \\
		$k$      & $1:k-1=k-1$ rows    & $(k-1)(n-k+1)$            & $(k-1)(n-k+1)$ \\
		$\vdots$ & $\vdots$            & $\vdots$                      & $\vdots$           \\
		$2$      & $1:1=1$ row        & $(1)(n-2+1)$                  & $(1)(n-2+1)$
	\end{tabular}
\end{table}

In the second loop to get the row reduced echelon form from the row echelon form, we need
$$
\begin{aligned}
	\sum_{k=2}^{n} \left[2(k-1)(n-k+1)+k-1\right]&=\sum_{k=2}^{n} \left[2(k-1)(n-k+1)+k-1\right]\\
	&=\sum_{i=1}^{n-1} \left[2(i)(n-i)+i\right], \text{ ($i=k-1$)} \\
	&= \sum_{i=1}^{n-1} \left[-2i^2+(2n+1)i\right],
\end{aligned}
$$
or \fbox{$\frac{1}{3}n^3$} flops if we keep only the leading terms.

\textbf{Total cost:}
Step 9 involves $mn$ flops (divisions) at most to make the pivots to be 1's. So the total cost is:

$\bullet$ the total cost for $m\leq n$ is then $m^2n-\frac{1}{3}m^3-\frac{2}{3}m^3+m^2n = $\fbox{$2m^2n-m^3$} flops if we keep only the leading terms.

$\bullet$ the total cost for $m> n$ is then $mn^2-\frac{1}{3}n^3+\frac{1}{3}n^3 = $\fbox{$mn^2$} flops if we keep only the leading term.

To conclude, the total cost is
$$
\text{cost}=\left\{
\begin{aligned}
	&2m^2n-m^3 \text{ flops}, \qquad  &\text{if }m\leq n; \\
	&mn^2 \text{ flops},\qquad &\text{if }m>n.
\end{aligned}
\right.
$$
And this completes the proof.
\end{proof}

In CUR decomposition, we will use the Gram-Schmidt process to find the linearly independent columns of $\bA$ which shares similar complexity with this Gaussian elimination for row reduced echelon form. But the Gram-Schmidt process is more clear from the perspective of equations to be computed, i.e., we can have specific mathematical forms for the entries of the matrices.

\section{Rank Decomposition}
We previously mentioned that the CR decomposition is a special case of rank decomposition. Formally, we prove the existence of the rank decomposition rigorously in the following theorem.

\begin{theoremHigh}[Rank Decomposition]\label{theorem:rank-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA }= \underset{m\times r}{\bD}\gap  \underset{r\times n}{\bF},
$$
where $\bD \in \real^{m\times r}$ has rank $r$, and $\bF \in \real^{r\times n}$ also has rank $r$, i.e., $\bD,\bF$ have full rank $r$.

The storage for the decomposition is then reduced or potentially increased from $mn$ to $r(m+n)$.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:rank-decomposition}]
By ULV decomposition in Theorem~\ref{theorem:ulv-decomposition} (p.~\pageref{theorem:ulv-decomposition}), we can decompose $\bA$ by
$$
\bA = \bU \begin{bmatrix}
	\bL & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV.
$$
Let $\bU_0 = \bU_{:,1:r}$ and $\bV_0 = \bV_{1:r,:}$, i.e., $\bU_0$ contains only the first $r$ columns of $\bU$, and $\bV_0$ contains only the first $r$ rows of $\bV$. Then, we still have $\bA = \bU_0 \bL\bV_0$ where $\bU_0 \in \real^{m\times r}$ and $\bV_0\in \real^{r\times n}$. This is also known as the reduced ULV decomposition as shown in Figure~\ref{fig:ulv-comparison}. Let \{$\bD = \bU_0\bL$ and $\bF =\bV_0$\}, or \{$\bD = \bU_0$ and $\bF =\bL\bV_0$\}, we find such rank decomposition.
\end{proof}
The rank decomposition is not unique. Even by elementary transformations, we have
$$
\bA =
\bE_1
\begin{bmatrix}
	\bZ & \bzero \\
	\bzero & \bzero
\end{bmatrix}
\bE_2,
$$
where $\bE_1 \in \real^{m\times m}, \bE_2\in \real^{n\times n}$ represent elementary row and column operations, $\bZ\in \real^{r\times r}$. The transformation is rather general, and there are dozens of these $\bE_1,\bE_2,\bZ$. Similar construction on this decomposition as shown in the above proof, we can recover another rank decomposition.

Analogously, we can find such $\bD,\bF$ by SVD, URV, CR, CUR, and many other decompositional algorithms. However, we may connect the different rank decompositions by the following lemma.
\begin{lemma}[Connection Between Rank Decompositions]\label{lemma:connection-rank-decom}
For any two rank decompositions of $\bA=\bD_1\bF_1=\bD_2\bF_2$, there exists a nonsingular matrix $\bP$ such that
$$
\bD_1 = \bD_2\bP
\qquad
\text{and}
\qquad
\bF_1 = \bP^{-1}\bF_2.
$$
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:connection-rank-decom}]
Since $\bD_1\bF_1=\bD_2\bF_2$, we have $\bD_1\bF_1\bF_1^\top=\bD_2\bF_2\bF_1^\top$. It is trivial that $rank(\bF_1\bF_1^\top)=rank(\bF_1)=r$ such that $\bF_1\bF_1^\top$ is a square matrix with full rank and thus is nonsingular. This implies $\bD_1=\bD_2\bF_2\bF_1^\top(\bF_1\bF_1^\top)^{-1}$. Let $\bP=\bF_2\bF_1^\top(\bF_1\bF_1^\top)^{-1}$, we have $\bD_1=\bD_2\bP$ and $\bF_1 = \bP^{-1}\bF_2$.
\end{proof}

To end up this section, we write another form of the rank decomposition that will be useful to make connection to the tensor decomposition (Theorem~\ref{theorem:cp-decomp}, p.~\pageref{theorem:cp-decomp}).
\begin{theoremHigh}[Rank Decomposition, An Alternative Form]\label{theorem:rank-decomposition-alternative}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA }= \underset{m\times r}{\bD}\gap  \underset{n\times r}{\bE^\top},
$$
where $\bD=[\bd_1, \bd_2, \ldots, \bd_r] \in \real^{m\times r}$ has rank $r$, and $\bE=[\be_1, \be_2, \ldots, \be_r] \in \real^{n\times r}$ also has rank $r$, i.e., $\bD,\bF$ have full rank $r$. Equivalently, the rank decomposition can be written as
$$
\bA = \sum_{i=1}^{r} \bd_i \be_i^\top.
$$
\end{theoremHigh}

\section{Application: Rank and Trace of an Idempotent Matrix}
The CR decomposition is quite useful to prove the rank of an idempotent matrix. See also the orthogonal projection in Appendix~\ref{appendix:orthogonal}.
\begin{lemma}[Rank and Trace of an Idempotent Matrix\index{Trace}]\label{lemma:rank-of-symmetric-idempotent2_tmp}
	For any $n\times n$ idempotent matrix $\bA$ (i.e., $\bA^2 = \bA$), the rank of $\bA$ equals the trace of $\bA$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-of-symmetric-idempotent2_tmp}]
Any $n\times n$  rank-$r$ matrix $\bA$ has CR decomposition $\bA = \bC\bR$, where $\bC\in\real^{n\times r}$ and $\bR\in \real^{r\times n}$ with $\bC, \bR$ having full rank $r$.
Then,
$$
\begin{aligned}
	\bA^2 &= \bA, \\
	\bC\bR\bC\bR &= \bC\bR, \\
	\bR\bC\bR &=\bR, \\
	\bR\bC &=\bI_r,
\end{aligned}
$$
where $\bI_r$ is an $r\times r$ identity matrix. Thus
$$
trace(\bA) = trace(\bC\bR) =trace(\bR\bC) = trace(\bI_r) = r,
$$
which equals the rank of $\bA$. The equality above is from the invariant of cyclic permutation of trace.
\end{proof}

\section{Other Applications}
The CR decomposition or rank decomposition is essential for the proof of many important theorems, such as the existence of pseudo-inverse in Lemma~\ref{lemma:existence-of-pseudo-inverse} (p.~\pageref{lemma:existence-of-pseudo-inverse}), finding the basis of the four subspaces in the fundamental theorem of linear algebra in Appendix~\ref{appendix:cr-decomposition-four-basis} (p.~\pageref{appendix:cr-decomposition-four-basis}).
 The CR factorization can also be used for data interpretation
or to solve computational problems, such as least squares where a reduced linear system can be considered to remove redundant variables.
Readers will find the usage of the CR decomposition throughout the text.

\chapter{Eigenvalue Problem}\label{section:eigenvalue-problem}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Background}
The decompositional methods discussed in the above sections are all related to the eigenvalues and eigenvectors of matrices. Thus, the eigenvalue problem merits independent consideration.
In this section, we present some classical eigenvalue algorithms that will be often useful for computing eigenvalue-related decompositions. And we simplify the discussion by considering only matrices that are real and symmetric. When $\bA$ is real and symmetric, it can be factored into (by spectral theorem, Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem})
\begin{equation*}
	\bA = \bQ \bLambda \bQ^\top = \bQ \bLambda \bQ^{-1},
\end{equation*}
where the columns of $\bQ = [\bq_1, \bq_2, \ldots, \bq_n]$ are eigenvectors of $\bA$ whilst mutually orthonormal, the entries of $\bLambda=diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$ are the corresponding eigenvalues of $\bA$, which are real. We suppose further the eigenvalues are ordered in magnitude such that
$$
|\lambda_1| \geq |\lambda_2| \geq \ldots \geq |\lambda_n| \geq 0,
$$
where in some discussions the equality will not be satisfied with special considerations. One thing to note further is that since $\bA$ is diagonalizable, the eigenvectors $\{\bq_1, \bq_2, \ldots, \bq_n\}$ span the whole space $\real^n$ such that any vector $\bv$ in space $\real^n$ can be written as a combination of the eigenvectors
$$
\bv = x_1\bq_1 +x_2\bq_2+\ldots+x_n\bq_n, \qquad \text{for all}\gap \bv\in\real^n.
$$

\section{Rate of Convergence}
Before we talk about specific algorithms to compute the eigenvalues, we need criteria to evaluate how fast the algorithms will be as most of them are iterative methods. We define the convergence of a sequence as follows. Note that the $k$-th element in a sequence is denoted by a superscript in parentheses, e.g., $\bA^{(k)}$ denotes the $k$-th matrix in a sequence, and $\ba^{(k)}$ denote the $k$-th vector in a sequence. \index{Rate of convergence}
\begin{definition}[Convergence of a Sequence]
	Let $\alpha^{(1)}, \alpha^{(2)}, \ldots \in \real$ be an infinite sequence of scalars. Then $\alpha^{(k)}$ is said to converge to $\alpha^\star$ if
	$$
	\mathop{\lim}_{k\rightarrow \infty} |\alpha^{(k)} - \alpha^\star| = 0.
	$$
	Similarly, let $\balpha^{(1)}, \balpha^{(2)}, \ldots \in \real^n$ be an infinite sequence of vectors. Then $\balpha^{(k)}$ is said to converge to $\balpha^\star$ if
	$$
	\mathop{\lim}_{k\rightarrow \infty} ||\balpha^{(k)} - \balpha^\star|| = 0.
	$$
\end{definition}
The convergence of a sequence of vectors or matrices rely on the norm. One should note that by the equivalence of vector norms (Theorem~\ref{theorem:equivalence-vector-norm}, p.~\pageref{theorem:equivalence-vector-norm}), if a sequence of vectors converges in one norm, then it converges in all norms. This will be proved important for the analysis of the convergence of eigenvectors in the sequel.

\begin{definition}[Linear Convergence]\label{definition:linear-convergence}
	A sequence $\alpha^{(k)}$ with limit $\alpha^\star$ is \textit{linearly convergence} if there exists a constant \textcolor{blue}{$c\in (0,1)$} such that
	$$
	|\alpha^{(k+1)} - \alpha^\star| \leq c |\alpha^{(k)} - \alpha^\star|.
	$$
	In other words, the \textit{linearly convergent sequence} has the following property:
	$$
	\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|} = c \in (0,1).
	$$
\end{definition}
For example, the sequence $\alpha^{(k)} = 4+ (1/4)^k$ converges linearly to $\alpha^\star = 4$ since
$$
\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|} = \frac{1}{4} \in (0,1).
$$
\begin{definition}[Superlinear Convergence]
	A sequence $\alpha^{(k)}$ with limit $\alpha^\star$ is \textit{superlinearly convergence} if there exists a constant \textcolor{blue}{$c_k >0$ with $c_k \rightarrow 0$}  such that
	$$
	|\alpha^{(k+1)} - \alpha^\star| \leq c_k |\alpha^{(k)} - \alpha^\star|.
	$$
	In other words, the \textit{superlinearly convergent sequence} has the following property:
	$$
	\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|} =0.
	$$
\end{definition}
For example, the sequence $\alpha^{(k)} = 4+\left(\frac{1}{k+4}\right)^{k+3}$ converges superlinearly to $\alpha^\star = 4$ since
$$
\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|}=
\left(\frac{k+4}{k+5}\right)^{k+3}  \frac{1}{k+5}
= 0.
$$

\begin{definition}[Quadratic Convergence]\label{definition:quadratic-convergence}
	A sequence $\alpha^{(k)}$ with limit $\alpha^\star$ is \textit{quadratically convergence} if there exists a constant \textcolor{blue}{$c>0$} such that
	$$
	|\alpha^{(k+1)} - \alpha^\star| \leq c |\alpha^{(k)} - \alpha^\star|^2.
	$$
	In other words, the \textit{quadratically convergent sequence} has the following property:
	$$
	\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|^2} = c .
	$$
\end{definition}
For example, the sequence $\alpha^{(k)} = 4+ (1/4)^{2^k}$ converges quaeratically to $\alpha^\star = 4$ since
$$
\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|^2} = 1.
$$

\begin{definition}[Cubic Convergence]
	A sequence $\alpha^{(k)}$ with limit $\alpha^\star$ is \textit{cubically convergence} if there exists a constant \textcolor{blue}{$c>0$} such that
	$$
	|\alpha^{(k+1)} - \alpha^\star| \leq c |\alpha^{(k)} - \alpha^\star|^3.
	$$
	In other words, the \textit{cubically convergent sequence} has the following property:
	$$
	\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|^3} = c .
	$$
\end{definition}
For example, the sequence $\alpha^{(k)} = 4+ (1/4)^{3^k}$ converges cubically to $\alpha^\star = 4$ since
$$
\mathop{\lim}_{k\rightarrow \infty}  \frac{|\alpha^{(k+1)} - \alpha^\star|}{|\alpha^{(k)} - \alpha^\star|^3} = 1.
$$

\section{Eigenvalues as Optimization}
For real and symmetric matrix $\bA$, we consider the following constrained optimization
$$
\mathop{\max}_{\bx \in \real^n} \bx^\top\bA\bx, \qquad s.t., \quad ||\bx||_2=1.
$$
By forming the Lagrangian, the optimization can be transformed into
$$
\mathcal{L}(\bx, \lambda)= \bx^\top\bA\bx - \lambda\bx^\top\bx,
$$
where $\lambda$ is called the \textit{Lagrange multiplier}\index{Lagrange multiplier}. To find the solution, the gradient of the Lagrangian has to be zero at $\bx^\star$:
$$
\Delta_{\bx} \mathcal{L}(\bx, \lambda) = 2\bA\bx-2\lambda\bx = \bzero.
$$
This implies $\bA\bx = \lambda\bx$ and and shows that the optimal points indicating $\lambda$ and $\bx^\star$ are eigenvalue and eigenvector of $\bA$.

\section{Rayleigh Quotient}\index{Rayleigh quotient}
The Rayleigh quotient of a vector $\bx\in \real^n$ associated with the matrix $\bA$ is the scalar given by the quadratic form:
$$
r(\bx) = \frac{\bx^\top\bA\bx}{\bx^\top\bx} = \left(\frac{\bx}{||\bx||}\right)^\top \bA \left(\frac{\bx}{||\bx||}\right),
$$
where $\left(\frac{\bx}{||\bx||}\right)$ is a normalized vector. When $\bx$ is an eigenvector of $\bA$, $r(\bx)$ is the eigenvalue $\lambda$ associated with $\bx$. To see this, suppose $\bA\bx=\lambda\bx$, it follows that $$\bx^\top\bA\bx = \lambda\bx^\top\bx \leadto \lambda = \frac{\bx^\top\bA\bx}{\bx^\top\bx}=r(\bx).
$$

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Function ``density" of $r(\bx)$]{\label{fig:project-rayleigh1}
		\includegraphics[width=0.47\linewidth]{./imgs/rayleigh1.pdf}}
	\quad
	\subfigure[Function ``density" of $\nabla r(\bx)$]{\label{fig:project-rayleigh2}
		\includegraphics[width=0.47\linewidth]{./imgs/rayleigh2.pdf}}
	\caption{Function ``density" and contour plots (\textcolor{bluepigment}{blue}=low, \textcolor{canaryyellow}{yellow}=high) where in Figure~\ref{fig:project-rayleigh1}, the upper graph is the ``density", and the lower one is the projection of it (i.e., contour). The example is drawn by setting $\bA=\begin{bmatrix}
			6 & 0\\ 0 & 2
		\end{bmatrix}$ where the input $\bx=[x_1,x_2]^\top$ lies in $\real^2$. The eigenvalues are $6$ and $2$, where the corresponding eigenvectors lie in the lines of
		$z\cdot \begin{bmatrix}
			1 \\ 0
		\end{bmatrix}$ and
		$z\cdot \begin{bmatrix}
			0 \\ 1
		\end{bmatrix}$ with $z\in (-\infty, \infty)$ being a scalar.}
	\label{fig:rayleigh-projection}
\end{figure}

Whereas, the most important property of the Rayleigh quotient is that when $\bx$ is not an eigenvector of $\bA$, the scalar $r(\bx)$ acts most like an eigenvalue in the sense that the squared norm $||\bA\bx-r(\bx)\bx||^2$ is minimized. To see this, suppose we want to find $\lambda$ such that $||\bA\bx-\lambda\bx||^2$ is minimized. Write out the equation with respect to $\lambda$
$$
||\bA\bx-r(\bx)\bx||^2 = \bx^\top\bx \lambda^2 - 2\bx^\top\bA\bx \lambda + \bx^\top\bA^\top\bA\bx.
$$
Since $\bx^\top\bx \geq 0$, the above equation is minimized by setting the gradient to be zero which leads to $\lambda=r(\bx)$, i.e., the Rayleigh quotient of $\bx$. Therefore, the Rayleigh quotient is a natural eigenvalue estimate to consider if $\bx$ is close to, but not necessarily equal to, an eigenvector. To see this, it is reasonable to take the vector $\bx\in \real^n$ as an input variable, and $r(\bx)$ as an output. Let $a = \bx^\top\bA\bx, b=\bx^\top\bx$, the gradient of $r(\bx)$ with respect to $\bx$ is given by
$$
\Delta r(\bx) = \frac{a^\prime b - ab^\prime}{b^2} = \frac{\bx^\top\bx (\bA+\bA^\top)\bx - 2\bx^\top\bA\bx \bx}{||\bx||^4}. \qquad (\text{$\bA$ is any square matrix})
$$
If we further restrict $\bA$ to be symmetric as we will consider mostly in this section, the gradient of the Rayleigh quotient reduces to
$$
\nabla r(\bx)  = \frac{2\bx^\top\bx \bA\bx - 2\bx^\top\bA\bx \bx}{||\bx||^4}
=\frac{2}{||\bx||^2}\left(\bA\bx - r(\bx)\bx\right).
 \qquad (\text{$\bA$ is symmetric})
$$
We observe that the gradient is a zero vector if and only if $\bx$ is an eigenvector of $\bA$ in which case $r(\bx)$ is the corresponding eigenvalue. This finding has a geometric meaning, when viewing $\bx$ as the input variable, the \textit{stationary points} (also known as the \textit{saddle points}) of function $r(\bx)$ are eigenvectors of $\bA$ and the function output is the corresponding eigenvalues. An example of these stationary points is shown in Figure~\ref{fig:rayleigh-projection} where the $r(\bx)$ and $\nabla r(\bx)$ are draw from the matrix $\bA=\begin{bmatrix}
	6 & 0\\ 0 & 2
\end{bmatrix}$.

\section{Power Method, Inverse Power Method, and Rayleigh Quotient Method}
We start computing the eigenvalues by several partial methods, which compute the extremal eigenvalues of $\bA$, i.e., the one having maximum and minimum magnitude.
\subsection{The Power Method}
The power method will produce a sequence $\bv^{(k)}$ that converges linearly to an eigenvector corresponding to the largest eigenvalue of $\bA$. To obtain the corresponding eigenvalue, the Rayleigh quotient can be utilized afterwards.
The first attempt to find the eigenvalue of $\bA$ will be using the value of the largest eigenvalue (in magnitude) $\lambda_1$. We will see in this and the sections to follow, this ``theoretical" algorithms will shed light on the convergence analysis of the algorithms.

\begin{algorithm}[H]
	\caption{Power Iteration (A Theoretical but Impossible One: For Convergence Analysis Only)}
	\label{alg:power-iteration-theoretical}
	\begin{algorithmic}[1]
		\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
		\State $\bv^{(0)}$= some vector with $||\bv^{(0)}||=1$;
		\For{$k=1,2,\ldots$}
		\State $\bw = \bA \bv^{(k-1)}$;
		\State $\bv^{(k)} = \bw / \textcolor{blue}{\lambda_1}$;
		\State $\lambda^{(k)} = \frac{(\bv^{(k)})^\top\bA\bv^{(k)}}{(\bv^{(k)})^\top\bv^{(k)}}$; \Comment{i.e., Rayleigh quotient}
		\EndFor
	\end{algorithmic}
\end{algorithm}
Suppose the eigenvalue $\lambda_1$ with largest magnitude is given up front, Algorithm~\ref{alg:power-iteration-theoretical} provide an iterative way to find the corresponding eigenvector.
Write $\bv^{(0)}$ as a linear combination of the orthonormal eigenvectors $\{\bq_1, \bq_2, \ldots, \bq_n\}$ (since they span the whole space $\real^n$):
\begin{equation}\label{equation:power-inprac-eq1}
\begin{aligned}
	\bv^{(0)} = x_1 \bq_1 + x_2\bq_2 + \ldots + x_n\bq_n = \bQ\bx  \\
	\leadto \bx = \bQ^{-1}\bv^{(0)},
\end{aligned}
\end{equation}
where $\bx = [x_1, x_2, \ldots, x_n]^\top$.
Since matrix $\bA\in \real^{n\times n}$ is real and symmetric, the $k$-th power of $\bA$ is given by
\begin{equation}\label{equation:power-inprac-eq2}
\bA^k = \bQ \bLambda^k \bQ^{-1} \leadto
\left\{\begin{aligned}
	\bA^k \bQ &= \bQ\bLambda^k;\\
	\bA^k \bq_i &= \lambda_i^k \bq_i,\\
\end{aligned}
\right.
\end{equation}
where $\bA = \bQ \bLambda \bQ^{-1}$ is the spectral decomposition of $\bA$ (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}; Remark~\ref{remark:power-spectral}, p.~\pageref{remark:power-spectral}).
Clearly, we can obtain the $k$-th element $\bv^{(k)}$ in the sequence by
$$
\begin{aligned}
\bv^{(k)} &= \bA\bv^{(k-1)} /\lambda_1 = \bA^k\bv^{(0)} / \lambda_1^k\\
&=1/ \lambda_1^k(x_1 \lambda_1^k\bq_1 + x_2\lambda_2^k\bq_2 + \ldots + x_n\lambda_n^k\bq_n) \qquad \text{(Equation~\eqref{equation:power-inprac-eq1} and \eqref{equation:power-inprac-eq2})} \\
&=x_1 \bq_1 + x_2\left(\frac{\lambda_2}{\lambda_1}\right)^k\bq_2 + \ldots + x_n\left(\frac{\lambda_n}{\lambda_1}\right)^k\bq_n.
\end{aligned}
$$
Then, it follows that
$$
\begin{aligned}
	\bv^{(k)} - x_1\bq_1 &= \bA^k\bv^{(0)} / \lambda_1^k- x_1\bq_1\\
	&= (\bQ \bLambda^k \bQ^{-1}) \bv^{(0)}/\lambda_1^k - x_1\bq_1 \\
	&\stackrel{\bx = \bQ^{-1}\bv^{(0)}}{=\joinrel=} \bQ
	\begin{bmatrix}
		1 & & & \\
		& \left(\frac{\lambda_2}{\lambda_1}\right)^k & & \\
		& & \ddots & \\
		& & & \left(\frac{\lambda_n}{\lambda_1}\right)^k
	\end{bmatrix}
	\bx- x_1\bq_1
	= \bQ
	\begin{bmatrix}
		0 & & & \\
		& \left(\frac{\lambda_2}{\lambda_1}\right)^k & & \\
		& & \ddots & \\
		& & & \left(\frac{\lambda_n}{\lambda_1}\right)^k
	\end{bmatrix}
	\bx.
\end{aligned}
$$
Therefore,
$$
\bQ^{-1}(\bv^{(k)} - x_1\bq_1) =
\begin{bmatrix}
	0 & & & \\
	& \left(\frac{\lambda_2}{\lambda_1}\right)^k & & \\
	& & \ddots & \\
	& & & \left(\frac{\lambda_n}{\lambda_1}\right)^k
\end{bmatrix}
\bx,
$$
and
$$
\bQ^{-1}(\bv^{(k+1)} - x_1\bq_1) =
\begin{bmatrix}
	0 & & & \\
	& \left(\frac{\lambda_2}{\lambda_1}\right)^{k+1} & & \\
	& & \ddots & \\
	& & & \left(\frac{\lambda_n}{\lambda_1}\right)^{k+1}
\end{bmatrix}
\bx
=
\begin{bmatrix}
	0 & & & \\
	& \left(\frac{\lambda_2}{\lambda_1}\right) & & \\
	& & \ddots & \\
	& & & \left(\frac{\lambda_n}{\lambda_1}\right)
\end{bmatrix}
\bQ^{-1}(\bv^{(k)} - x_1\bq_1).
$$
Define $||\bB||_{\bQ^{-1}}$ by $||\bQ^{-1} \bB||_2$ for matrix $\bB$,
this results in
\begin{equation}\label{equation:power-iteration-analysis}
	\begin{aligned}
		\bigg\Vert\bv^{(k+1)} - x_1\bq_1\bigg\Vert_{\bQ^{-1}} &= \bigg\Vert\bQ^{-1}(\bv^{(k+1)} - x_1\bq_1)\bigg\Vert_2\\
		&\leq \left|\frac{\lambda_2}{\lambda_1}\right| \cdot \bigg\Vert\bQ^{-1}(\bv^{(k)} - x_1\bq_1)\bigg\Vert_2
		= \left|\frac{\lambda_2}{\lambda_1}\right| \cdot \bigg\Vert\bv^{(k)} - x_1\bq_1\bigg\Vert_{\bQ^{-1}},
	\end{aligned}
\end{equation}
where the inequality is from the matrix-vector inequality (Remark~\ref{remark:2norm-properties}, p.~\pageref{remark:2norm-properties}).
The above deduction shows that if we can prove that $||\bv||_{\bX^{-1}} = ||\bX^{-1}\bv||_2$ is a reasonable vector norm that satisfies the three criteria of vector norm (Definition~\ref{definition:matrix-norm}, p.~\pageref{definition:matrix-norm}), then by
the equivalence of vector norms (Theorem~\ref{theorem:equivalence-vector-norm}, p.~\pageref{theorem:equivalence-vector-norm}), we prove $\bv^{(k)}$ in Algorithm~\ref{alg:power-iteration-theoretical} converges to $x_1\bq_1$ linearly (Definition~\ref{definition:linear-convergence}, p.~\pageref{definition:linear-convergence}). This is actually the case.

However, the problem of Algorithm~\ref{alg:power-iteration-theoretical} is that the $\lambda_1$ is unknown to us making it impractical for us to compute the eigenvector. To tame this problem, we scale it to be of unit length at each iteration. The algorithm is then formulated in Algorithm~\ref{alg:power-iteration} where the difference is shown in \textcolor{blue}{blue} text.

\begin{algorithm}[H]
	\caption{Power Iteration (A Practical One, Compare to Algorithm~\ref{alg:power-iteration-theoretical})}
	\label{alg:power-iteration}
	\begin{algorithmic}[1]
		\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
		\State $\bv^{(0)}$= some vector with $||\bv^{(0)}||=1$;
		\For{$k=1,2,\ldots$}
		\State $\bw = \bA \bv^{(k-1)}$;
		\State $\bv^{(k)} = \bw / \textcolor{blue}{||\bw||}$;
		\State $\lambda^{(k)} = (\bv^{(k)})^\top\bA\bv^{(k)}$;  \Comment{i.e., Rayleigh quotient}
		\EndFor
	\end{algorithmic}
\end{algorithm}
\paragraph{Convergence Analysis}
Clearly, $\bv^{(k)}$ is still a multiple of $\bA^k \bv^{(0)}$ such that  $\bv^{(k)} = c_k \bA^k \bv^{(0)} = \frac{\bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||}$. We have
\begin{equation}\label{equation:converge-equa-of-power-ite}
	\begin{aligned}
		\bv^{(k)} &=  c_k \bA^k\bv^{(0)} \\
		&=c_k (x_1 \lambda_1^k\bq_1 + x_2\lambda_2^k\bq_2 + \ldots + x_n\lambda_n^k\bq_n) \\
		&= c_k \lambda_1^k\left(x_1 \bq_1 + \underbrace{x_2\left(\frac{\lambda_2}{\lambda_1}\right)^k\bq_2 + \ldots + x_n\left(\frac{\lambda_n}{\lambda_1}\right)^k\bq_n}_{\by^{(k)}}\right).
	\end{aligned}
\end{equation}
The sequence $\by^{(k)}$ in the above equation vanishes as $k\rightarrow \infty$ if $|\lambda_1| \textcolor{blue}{>} |\lambda_2| \geq |\lambda_3| \geq \ldots \geq |\lambda_n| \geq 0$. Further, if $x_1\neq 0$, then as $k\rightarrow  \infty$, the vector sequence $\bv^{(k)}$ converges to $\pm \bq_1$ where the sign is decided by $\lambda_1^k x_1$.

Now since we know $\bv^{(k)}$ converges to $\pm \bq_1$ such that $c_k\lambda_1^k x_1 \rightarrow \pm 1$. Following from the discussion in \citep{quarteroni2010numerical}, we here provide an improved result on the convergence of the eigenvector $\bv^{(k)}$ with a lower bound such that a form in the linear convergence can be obtained (Definition~\ref{definition:linear-convergence}, p.~\pageref{definition:linear-convergence}).
This observation proceeds the following theorem.
\begin{theorem}[Convergence of Power Iteration: Eigenvector]\label{theorem:power-convergence}
Suppose $\bA\in \real^{n\times n}$ is real and symmetric. Suppose further that the following two conditions are met
\begin{itemize}
	\item  $|\lambda_1| \textcolor{blue}{>} |\lambda_2| \geq |\lambda_3| \geq \ldots \geq |\lambda_n| \geq 0$, i.e., $\lambda_1$ has algebraic multiplicity being equal to 1;
	\item  $\bq_1^\top \bv^{(0)} \neq 0$, i.e., the initial guess $\bv^{(0)}$ has a component in the direction of the eigenvector $\bq_1$ associated with the eigenvalue $\lambda_1$.
\end{itemize}
Then there exists a constant $c$ such that
$$
||\widetildebv^{(k)}-\bq_1|| \leq c\cdot \left|\frac{\lambda_2}{\lambda_1}\right|^k,
$$
where the sequence $\widetildebv^{(k)} = \frac{\bv^{(k)}}{c_k\lambda_1^k x_1} = \bq_1+ \sum_{i=2}^n \frac{x_i}{x_1}\left(\frac{\lambda_i}{\lambda_1}\right)^k \bq_i $.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:power-convergence}]
We notice that $||\widetildebv^{(k)}-\bq_1|| = \left\Vert\sum_{i=2}^n \frac{x_i}{x_1}\left(\frac{\lambda_i}{\lambda_1}\right)^k \bq_i\right\Vert$. Let $\bz=\left[0, \frac{x_2}{x_1}\left(\frac{\lambda_2}{\lambda_1}\right)^k, \frac{x_3}{x_1}\left(\frac{\lambda_3}{\lambda_1}\right)^k, \ldots, \frac{x_n}{x_1}\left(\frac{\lambda_n}{\lambda_1}\right)^k\right]$, $\bQ=[\bq_1,\bq_2, \bq_3, \ldots, \bq_n]$, it follows that
$$
\begin{aligned}
||\widetildebv^{(k)}-\bq_1|| &= ||\bQ\bz||  \stackrel{\star}{=}  ||\bz|| \\
&=\left(\sum_{i=2}^{n}\left(\frac{x_i}{x_1}\right)^2\left(\frac{\lambda_i}{\lambda_1}\right)^{2k} \right)^{1/2}
\leq \left|\frac{\lambda_2}{\lambda_1}\right|^k
\left(\sum_{i=2}^{n}\left(\frac{x_i}{x_1}\right)^2 \right)^{1/2},
\end{aligned}
$$
where the inequality comes from the Matrix-vector product inequality (Remark~\ref{remark:2norm-properties}, p.~\pageref{remark:2norm-properties}, or simply by the assumption $|\lambda_2| \geq |\lambda_3| \geq \ldots \geq |\lambda_n|$) and the equality ($\star$) comes from the length is preserved under orthogonal transformation (similar to Lemma~\ref{lemma:frobenius-orthogonal-equi}, p.~\pageref{lemma:frobenius-orthogonal-equi}). Let $c=\left(\sum_{i=2}^{n}\left(\frac{x_i}{x_1}\right)^2 \right)^{1/2}$, the result follows. Note in the discussion of \citep{quarteroni2010numerical}, the equality ($\star$) is replace by a inequality from Matrix-vector product inequality (Remark~\ref{remark:2norm-properties}, p.~\pageref{remark:2norm-properties} such that $||\bQ\bz||\leq ||\bQ||\cdot ||\bz||$.) This has a problem that we cannot find the lower bound on the above equation. The above equation also tells us (by the assumption $|\lambda_2| \geq |\lambda_3| \geq \ldots \geq |\lambda_n|$):
$$
\begin{aligned}
	||\widetildebv^{(k)}-\bq_1|| &= ||\bQ\bz||  \stackrel{\star}{=}  ||\bz|| \\
	&=\left(\sum_{i=2}^{n}\left(\frac{x_i}{x_1}\right)^2\left(\frac{\lambda_i}{\lambda_1}\right)^{2k} \right)^{1/2}\geq  \left|\frac{\lambda_n}{\lambda_1}\right|^k
	\left(\sum_{i=2}^{n}\left(\frac{x_i}{x_1}\right)^2 \right)^{1/2}.
\end{aligned}
$$
Therefore, a bound on the convergence is given by
$$
\left.
\begin{aligned}
||\widetildebv^{(k+1)}-\bq_1||
&\leq c\cdot \left|\frac{\lambda_2}{\lambda_1}\right|^{k+1}\\
||\widetildebv^{(k)}-\bq_1||
&\geq c\cdot \left|\frac{\lambda_n}{\lambda_1}\right|^k
\end{aligned}
\right\}
\leadto
\frac{||\widetildebv^{(k+1)}-\bq_1||}{||\widetildebv^{(k)}-\bq_1||}
\leq
\left|
\frac{\lambda_2^{k+1}}{\lambda_1 \cdot \lambda_n^k}
\right|.
$$
However, since we do not know the value of $\lambda_n, \lambda_2, \lambda_1$, it happens that the above bound can explode. Only when $|\lambda_1| \gg |\lambda_2|$, it will be tight.
\end{proof}

Going further and following from the discussion in \citep{golub2013matrix}, we here provide an improved result on the convergence of the eigenvalue $\lambda^{(k)}$ with a tighter bound such that the sequence $\lambda^{(k)}$ converges to $\lambda_1$ \textbf{quadratically} with respect to the ratio $\left|\frac{\lambda_2}{\lambda_1}\right|$.
\begin{theorem}[Convergence of Power Iteration: Eigenvalue]\label{Theorem:power-convergence-eigenvalue}
(Under the same condition as Theorem~\ref{theorem:power-convergence}) Suppose $\bA\in \real^{n\times n}$ is real and symmetric. Suppose further that the following two conditions are met
\begin{itemize}
	\item  $|\lambda_1| \textcolor{blue}{>} |\lambda_2| \geq |\lambda_3| \geq \ldots \geq |\lambda_n| \geq 0$, i.e., $\lambda_1$ has algebraic multiplicity being equal to 1;
	\item  $\bq_1^\top \bv^{(0)} \neq 0$, i.e., the initial guess $\bv^{(0)}$ has a component in the direction of the eigenvector $\bq_1$ associated with the eigenvalue $\lambda_1$.
\end{itemize}
Then define $\theta_k \in [0, \pi/2]$ by
$$
c_k = \cos\theta_k = |\bq_1^\top \bv^{(k)}|.
$$
The $c_k$ is well defined since $||\bq_1||=||\bv^{(k)}||=1 \rightarrow 0< |\bq_1^\top \bv^{(k)}|\leq1$.
Then the sequence $s_k=\sin \theta_k, t_k = \tan \theta_k$ follows
\begin{itemize}
\item Convergence of $s_k$:
\begin{itemize}
\item $|s_k| \leq \textcolor{blue}{s_0} \left|\frac{\lambda_2}{\lambda_1}\right|^k$;
\item $\text{or  }  |s_k| \leq \textcolor{blue}{t_0} \left|\frac{\lambda_2}{\lambda_1}\right|^k$;
\end{itemize}
\item Convergence of $\lambda^{(k)}$:
\begin{itemize}
\item
$
\begin{aligned}
&|\lambda^{(k)} - \lambda_1|  \leq  \mathop{\max}_{2\leq i \leq n} |\lambda_1-\lambda_i| \cdot  \textcolor{blue}{s_0^2} \left( \frac{\lambda_i}{\lambda_1}\right)^{2k};
\end{aligned}
$
\item
$
\text{or  }  |\lambda^{(k)} - \lambda_1|  \leq  \mathop{\max}_{2\leq i \leq n} |\lambda_1-\lambda_i| \cdot  \textcolor{blue}{t_0^2} \left( \frac{\lambda_i}{\lambda_1}\right)^{2k}.
$
\end{itemize}
\end{itemize}
\end{theorem}
\begin{proof}[of Theorem~\ref{Theorem:power-convergence-eigenvalue}]
Since $s_k^2 = 1-c_k^2 = 1- (\bq_1^\top \bv^{(k)})^2 = 1-  \left(\frac{\bq_1^\top \bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||} \right)^2$ by Equation~\eqref{equation:converge-equa-of-power-ite} where $\bv^{(0)}$ can be written as the linear combination of the orthonormal eigenvectors:
$$
\bv^{(0)} = x_1 \bq_1 + x_2\bq_2 + \ldots + x_n\bq_n = \bQ\bx  \leadto \bx = \bQ^{-1}\bv^{(0)}.
$$
Since $||\bx||^2 = ||\bQ^{-1}\bv^{(0)}||^2$ and $\bv^{(0)}$ is of length 1. By equivalence under orthogonal (similar to Lemma~\ref{lemma:frobenius-orthogonal-equi}, p.~\pageref{lemma:frobenius-orthogonal-equi}), this indicates
$$
x_1^2+x_2^2+\ldots+x_n^2=||\bx||^2=1.
$$
And we have also shown in Equation~\eqref{equation:converge-equa-of-power-ite} that
$$
\bA^k \bv^{(0)} = x_1 \lambda_1^k\bq_1 + x_2\lambda_2^k\bq_2 + \ldots + x_n\lambda_n^k\bq_n.
$$
The above findings imply that
\begin{equation}\label{equation:power-lambda-1}
	\begin{aligned}
		s_k^2 &=1-  \left(\frac{\bq_1^\top \bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||} \right)^2
		=1-  \left(\frac{x_1^2 \lambda_1^{2k}}{\sum_{i=1}^{n} x_i^2 \lambda_i^{2k}} \right)\\
		&= \frac{\sum_{i=2}^{n} x_i^2 \lambda_i^{2k}}{\sum_{i=1}^{n} x_i^2 \lambda_i^{2k}}
		\leq \textcolor{blue}{\frac{\sum_{i=2}^{n} x_i^2 \lambda_i^{2k}}{x_1^2 \lambda_1^{2k}}} \\
		&=\frac{1}{x_1^2} \sum_{i=2}^{n} x_i^2 \left( \frac{\lambda_i}{\lambda_1}\right)^{2k}
		\leq \frac{1}{x_1^2} \left( \sum_{i=2}^{n} x_i^2 \right) \left( \frac{\lambda_i}{\lambda_1}\right)^{2k}\\
		&= \frac{1-x_1^2}{x_1^2}\left( \frac{\lambda_i}{\lambda_1}\right)^{2k} = t_0^2 \left( \frac{\lambda_i}{\lambda_1}\right)^{2k},
	\end{aligned}
\end{equation}
where the last equality follows from the definition of $c_k$, when $k=0$, we have $c_0 = |\bq_1^\top\bv^{(0)}|=|x_1| \neq 0$.
Therefore, the first result follows:
$$
|s_k| \leq t_0 \left|\frac{\lambda_2}{\lambda_1}\right|^k.
$$
The above result is exactly what have done in \citep{golub2013matrix}, however, the blue text above is a loose bound since $0<x_1^2 \leq 1$ under our assumption. An improved result is shown as follows (where the difference is made into blue text):
\begin{equation}\label{equation:power-lambda-2}
\begin{aligned}
	s_k^2 &=1-  \left(\frac{\bq_1^\top \bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||} \right)^2
	=1-  \left(\frac{x_1^2 \lambda_1^{2k}}{\sum_{i=1}^{n} x_i^2 \lambda_i^{2k}} \right)\\
	&= \frac{\sum_{i=2}^{n} x_i^2 \lambda_i^{2k}}{\sum_{i=1}^{n} x_i^2 \lambda_i^{2k}}
	\leq \textcolor{blue}{\frac{\sum_{i=2}^{n} x_i^2 \lambda_i^{2k}}{\sum_{i=1}^{n} x_i^2 \lambda_1^{2k}}} =\frac{\sum_{i=2}^{n} x_i^2 \lambda_i^{2k}}{ \lambda_1^{2k}}\\
	&= \sum_{i=2}^{n} x_i^2 \left( \frac{\lambda_i}{\lambda_1}\right)^{2k}
	\leq  \left( \sum_{i=2}^{n} x_i^2 \right) \left( \frac{\lambda_i}{\lambda_1}\right)^{2k}\\
	&= (1-x_1^2)\left( \frac{\lambda_i}{\lambda_1}\right)^{2k} =s_0^2 \left( \frac{\lambda_i}{\lambda_1}\right)^{2k}.
\end{aligned}
\end{equation}
Therefore, the result follows that
$$
|s_k| \leq s_0 \left|\frac{\lambda_2}{\lambda_1}\right|^k.
$$
Further, since $\bv^{(k)}=\frac{\bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||}$ from Equation~\eqref{equation:converge-equa-of-power-ite}, we have
$$
\begin{aligned}
\lambda^{(k)} &= \bv^{(k)\top}\bA\bv^{(k)} =
\left(\frac{\bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||}\right)^\top
\bA
\left(\frac{\bA^k \bv^{(0)}}{||\bA^k \bv^{(0)}||}\right)\\
&=
\frac{\bv^{(0)\top} \bA^{2k+1}\bv^{(0)}}
{\bv^{(0)\top} \bA^{2k}\bv^{(0)}}
=
\frac{\sum_{i=1}^{n}x_i^2 \lambda_i^{2k+1}}{\sum_{i=1}^{n}x_i^2 \lambda_i^{2k}}.
\end{aligned}
$$
Therefore,
$$
\begin{aligned}
|\lambda^{(k)} - \lambda_1|
&=
\left|
\frac{\sum_{i=\textcolor{blue}{2}}^{n}x_i^2 \lambda_i^{2k}(\lambda_i-\lambda_1)}
{\sum_{i=1}^{n}x_i^2 \lambda_i^{2k}}
\right|
\leq \mathop{\max}_{2\leq i \leq n} |\lambda_1-\lambda_i|
\left(
\frac{\sum_{i=\textcolor{blue}{2}}^{n}x_i^2 \lambda_i^{2k}}
{\sum_{i=1}^{n}x_i^2 \lambda_i^{2k}}
\right)\\
&\leq \mathop{\max}_{2\leq i \leq n} |\lambda_1-\lambda_i| \cdot  s_0^2 \left( \frac{\lambda_i}{\lambda_1}\right)^{2k},
\end{aligned}
$$
where the last inequality comes from Equation~\eqref{equation:power-lambda-1} or Equation~\eqref{equation:power-lambda-2} shown above.
\end{proof}
From the above deduction, we conclude the following convergence result:
\begin{theorem}[Convergence of Power Iteration]\label{theorem:convergence-power-iteration}
(Under the same condition as Theorem~\ref{theorem:power-convergence}) Suppose $\bA\in \real^{n\times n}$ is real and symmetric. Suppose further that the following two conditions are met
	\begin{itemize}
		\item  $|\lambda_1| \textcolor{blue}{>} |\lambda_2| \geq |\lambda_3| \geq \ldots \geq |\lambda_n| \geq 0$, i.e., $\lambda_1$ has algebraic multiplicity being equal to 1;
		\item  $\bq_1^\top \bv^{(0)} \neq 0$, i.e., the initial guess $\bv^{(0)}$ has a component in the direction of the eigenvector $\bq_1$ associated to the eigenvalue $\lambda_1$.
	\end{itemize}
	Then the iterates of Algorithm~\ref{alg:power-iteration} satisfy
	$$
	||\bv^{(k)} - (\pm \bq_1)||= O\left(\left|\frac{\lambda_2}{\lambda_1}\right|^k\right), \qquad |\lambda^{(k)}-\lambda_1|=O\left(\left|\frac{\lambda_2}{\lambda_1}\right|^{2k}\right)
	$$
	as $k\rightarrow \infty$.
\end{theorem}
The $\pm$ sign of $\bq_1$ means that, one of $\bq_1$ and $-\bq_1$ is to be taken into the result. And this shows $\bv^{(k)}$ converges to $\pm\bq_1$ linearly in Algorithm~\ref{alg:power-iteration}.

\paragraph{Eigenvalue Assumptions}
The assumption $\bq_1^\top \bv^{(0)} \neq 0$ in the above algorithm is to assume the initial guess of the eigenvector has a component in the direction of the eigenvector $\bq_1$ we want to find. Otherwise, it will not converge to the $\pm \bq_1$.

We also carefully notice that, the equality sign does not appear in the assumption $|\lambda_1|>|\lambda_2|$, in which case $\lambda_1$ is also known as the \textit{dominant} eigenvalue of matrix $\bA$. Nevertheless, when \{$|\lambda_1|\textcolor{blue}{=|}\lambda_2| \textcolor{blue}{>} |\lambda_3| \geq |\lambda_4| \geq \ldots $, $\bq_1^\top \bv^{(0)} \neq 0$, and $\bq_2^\top \bv^{(0)} \neq 0 $\}, $\bv^{(k)}$ will converge to a multiple of $x_1\bq_1 \pm x_2\bq_2$, i.e., lies in the subspace spanned by $\{\bq_1, \bq_2\}$. To see this,
\begin{enumerate}
\item $\lambda_1=\lambda_2$, i.e., the two dominant eigenvalues are coincident. Equation~\eqref{equation:converge-equa-of-power-ite} shows $\bv^{(k)}, \lambda^{(k)}$ converges to
$$
\left\{
\begin{aligned}
\bv^{(k)} &\stackrel{k\rightarrow \infty}{\longrightarrow} \bbeta_1= \frac{ x_1 \bq_1 +  x_2\bq_2}{|| x_1 \bq_1 +  x_2\bq_2||}\in span\{\bq_1, \bq_2\}
\qquad \text{i.e.,} \gap
\bA\bbeta_1 =  \lambda_1 \bbeta_1,\\
\lambda^{(k)} &\stackrel{k\rightarrow \infty}{\longrightarrow} \bbeta_1^\top \bA\bbeta_1 = \lambda_1.
\end{aligned}
\right.
$$
Therefore, the vector sequence  $\bv^{(k)}$ still converges to an eigenvector of $\bA$, which lies in the space spanned by $\{\bq_1,\bq_2\}$, and the scalar sequence $\lambda^{(k)}$ still converges to $\lambda_1=\lambda_2$. We carefully notice that when $\lambda_1, \lambda_2$, any vector in $span\{\bq_1, \bq_2\}$ will be an eigenvector of $\bA$.

\item $\lambda_1=-\lambda_2$, i.e., the two dominant eigenvalues are opposite. Equation~\eqref{equation:converge-equa-of-power-ite} shows $\bv^{(k)}, \lambda^{(k)}$ converges to
$$
\begin{aligned}
	\bv^{(k)} &\stackrel{k\rightarrow \infty}{\longrightarrow} \bbeta_2= \frac{ \lambda_1^k x_1 \bq_1 +  \lambda_2^k x_2\bq_2}{||\lambda_1^k x_1 \bq_1 +  \lambda_2^k x_2\bq_2||}\in span\{\bq_1, \bq_2\}.
\end{aligned}
$$
Then, we have
$$
\left\{
\begin{aligned}
\bA\bbeta_2 &=  \lambda_1 \frac{x_1\bq_1+x_2\bq_2}{||x_1\bq_1-x_2\bq_2||}, \qquad \text{when $k$ is odd;}\\
\bA\bbeta_2 &=  \lambda_1 \frac{x_1\bq_1-x_2\bq_2}{||x_1\bq_1+x_2\bq_2||}, \qquad \text{when $k$ is even.}
\end{aligned}
\right.
$$
Therefore, $\bv^{(k)}$ will not converge to an eigenvector of $\bA$, neither $\lambda^{(k)}$ do. In this case, we observe that $\bA\bx=\lambda\bx \rightarrow \bA^2\bx=\lambda^2\bx$ such that the eigenvalues of $\bA^2$ will be nonnegative and $\lambda_1^2=\lambda_2^2$ are the same eigenvalue of $\bA^2$ if they are eigenvalues of $\bA$. Then apply the power method on $\bA^2$ will lead to the convergence to the eigenvector and eigenvalue of $\bA^2$ which is the same as analyzed in case (1).

\item $\lambda_1=\bar{\lambda}_2$, i.e., the two dominant eigenvalues are complex conjugate. The power method is not convergent  \citep{wilkinson1971algebraic, quarteroni2010numerical} and we shall not give the details.
\end{enumerate}
\paragraph{What if $\bq_1^\top \bv^{(0)} = 0$?} All the convergence results are made under the assumption that $\bq_1^\top \bv^{(0)} \neq 0$. But since we do not know $\bq_1$ up front, the requirement is not able to fulfill. When it happens that $\bq_1^\top \bv^{(0)} = 0$, from Equation~\eqref{equation:converge-equa-of-power-ite} again, the vector sequence $\bv^{(k)}$ converges to $\bq_2$ such that $\lambda^{(k)}\rightarrow \lambda_2$. Therefore, the requirement on the initial guess will not harm the convergence of the power method, but the net result will be different.

\paragraph{What do we start from the ``theoretical" one?} In the theoretical power method Algorithm~\ref{alg:power-iteration-theoretical}, we assume the $\lambda_1$ is known, and shows the vector sequence converges linearly to the eigenvector in Equation~\eqref{equation:power-iteration-analysis}. This result matches the convergence of the ``practical" power method Algorithm~\ref{alg:power-iteration} as shown in Theorem~\ref{theorem:convergence-power-iteration}. The ``theoretical" one thus can be employed to find a first analysis on the power method, and we shall shortly find its counterpart in the \textit{inverse power method} in the next section.

\subsection{The Inverse Power Method}
The \textit{power method} homes in on an eigenvector associated with the largest eigenvalue  (in magnitude). Nevertheless, the \textit{inverse power method} homes in on an eigenvector associated with the smallest
eigenvalue (in magnitude). To see this, we first provide the lemma about the eigenpair of the inverse of matrices as follows.
\begin{lemma}[Eigenpair of Inverse Matrix]\label{lemma:inverse-eigenpair}
	Suppose matrix $\bA^{n\times n}$ is nonsingular, and $(\lambda, \bx)$ is an eigenpair of $\bA$. Then $(1/\lambda, \bx)$ is an eigenpair of $\bA^{-1}$.
\end{lemma}

Furthermore, we assume previously that the matrix $\bA$ is real and symmetric. We have shown in Section~\ref{section:spectral-decomposition} that the rank of the real and symmetric matrix is the number of nonzero eigenvalues. The inverse power method involves the inverse of the eigenvalues such that we suppose further the matrix involved in this section is nonsingular, i.e., all the eigenvalues are nonzero and the matrix is invertible.

\begin{algorithm}[h]
	\caption{Inverse Power Iteration (Compare to Algorithm~\ref{alg:power-iteration})}
	\label{alg:inverse-power-iteration}
	\begin{algorithmic}[1]
		\Require \textcolor{blue}{nonsingular} matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
		\State $\bv^{(0)}$= some vector with $||\bv^{(0)}||=1$;
		\For{$k=1,2,\ldots$}
		\State $\bw = \textcolor{blue}{\bA^{-1}} \bv^{(k-1)}$;
		\State $\bv^{(k)} = \bw / ||\bw||$;
		\State $\lambda^{(k)} = (\bv^{(k)})^\top\bA\bv^{(k)}$;  \Comment{i.e., Rayleigh quotient
		\EndFor
	\end{algorithmic}
\end{algorithm}
The idea behind the algorithm is that an eigenvector associated with the smallest eigenvalue (in magnitude) of $\bA$ is an eigenvector
associated with the largest eigenvalue (in magnitude)  of $\bA^{-1}$ by Lemma~\ref{lemma:inverse-eigenpair}.
From the power method, it is trivial that the sequence $\bv^{(k)}$ in Algorithm~\ref{alg:inverse-power-iteration} converges linearly to the eigenvector of $\bA$ corresponding to the smallest eigenvalue $\lambda_n$ (or converges linearly to the eigenvector of $\bA^{-1}$ corresponding to the largest eigenvalue $1/\lambda_n$).

To abuse the analysis of the convergence of the inverse power method, we can carefully employ a theoretical (but impossible) way  as shown in Algorithm~\ref{alg:power-iteration-theoretical} where we assume $\lambda_1$ is known and the convergence result is shown in Equation~\eqref{equation:power-iteration-analysis}. Come back to the inverse power method, suppose $\lambda_n$ is known and a ``theoretical" inverse power method is induced, the convergence is thus given by (analogous to Equation~\eqref{equation:power-iteration-analysis}):
\begin{equation}\label{equation:-inverse-power-iteration-analysis1}
	\begin{aligned}
		||\bv^{(k+1)} - x_n \bq_n||_{\bQ^{-1}} &= ||\bQ^{-1}(\bv^{(k+1)} - x_n\bq_n)||_2\\
		&\leq \left|\frac{\lambda_n}{\lambda_{n-1}}\right| \cdot ||\bQ^{-1}(\bv^{(k)} - x_n\bq_n)||_2
		= \left|\frac{\lambda_n}{\lambda_{n-1}}\right| \cdot ||\bv^{(k)} - x_n\bq_n||_{\bQ^{-1}},
	\end{aligned}
\end{equation}
where we use the fact that if the spectral decomposition of $\bA$ is $\bA=\bQ\bLambda\bA^{-1}$, then the spectral decomposition of $\bA^{-1}$ is $\bA^{-1} =\bQ\bLambda^{-1} \bQ^{-1}$. Since $\lambda_n, \lambda_{n-1}$ are the smallest two eigenvalues of $\bA$ (in magnitude). It is possible that the two eigenvalues are very close to each other and thus the bound $\left|\frac{\lambda_n}{\lambda_{n-1}}\right| $ is close to 1.

\subsection{The Shifted Inverse Power Method}
Now we suppose that we know that one of the eigenvalues of $\bA$ is close to a value $\mu\in \real$. \textcolor{blue}{For any value $\mu$ that is not an eigenvalue of $\bA$, $\bA-\mu\bI$ is nonsingular even if $\bA$ is singular}. \textit{The matrix $\bA-\mu\bI$ is referred to as the matrix $\bA$ that has been ``shifted" by $\mu$}, and $\mu$ is called a \textit{shift}.

The following lemma reveals that a shifted version of the inverse power method can be employed to find the eigenvector associated with the eigenvalue that is closest to $\mu$.

\begin{lemma}[Eigenpair of Shifted Matrix]\label{lemma:shifted-eigenpair}
	Suppose $(\lambda, \bx)$ is an eigenpair of $\bA\in \real^{n\times n}$ and $\mu\in \real $ is not an eigenvalue of $\bA$. Then $(\lambda-\mu, \bx)$ is an eigenpair of $\bA-\mu\bI$.

	\item Notice that the eigenvectors of $\bA-\mu\bI$ are the same as those of $\bA$ since $\bA\bx =\lambda\bx \rightarrow (\bA-\mu\bI)\bx=(\lambda-\mu)\bx$.
\end{lemma}

\begin{algorithm}[h]
	\caption{Shifted Inverse Power Iteration (Compare to Algorithm~\ref{alg:inverse-power-iteration})}
	\label{alg:inverse-power-iteration-shifted}
	\begin{algorithmic}[1]
		\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric, \textcolor{blue}{$\mu$ is not an eigenvalue of $\bA$};
		\State $\bv^{(0)}$= some vector with $||\bv^{(0)}||=1$;
		\For{$k=1,2,\ldots$}
		\State $\bw = \textcolor{blue}{(\bA-\mu\bI)^{-1}} \bv^{(k-1)}$;
		\State $\bv^{(k)} = \bw / ||\bw||$;
		\State $\lambda^{(k)} = (\bv^{(k)})^\top\bA\bv^{(k)}$;  \Comment{i.e., Rayleigh quotient}
		\EndFor
	\end{algorithmic}
\end{algorithm}

The procedure is again formulated in Algorithm~\ref{alg:inverse-power-iteration-shifted}. To abuse the analysis of convergence again, suppose $\mu$ is closest to the smallest eigenvalue $\lambda_n$ in magnitude, the convergence is (again, analogous to Equation~\eqref{equation:power-iteration-analysis}):
\begin{equation}\label{equation:-inverse-power-iteration-analysis-shifted}
	\begin{aligned}
		||\bv^{(k+1)} - x_n \bq_n||_{\bQ^{-1}} &= ||\bQ^{-1}(\bv^{(k+1)} - x_n\bq_n)||_2\\
		&\leq \left|\frac{\lambda_n\textcolor{blue}{-\mu}}{\lambda_{n-1}\textcolor{blue}{-\mu}}\right| \cdot ||\bQ^{-1}(\bv^{(k)} - x_n\bq_n)||_2
		= \left|\frac{\lambda_n \textcolor{blue}{-\mu}}{\lambda_{n-1}\textcolor{blue}{-\mu}}\right| \cdot ||\bv^{(k)} - x_n\bq_n||_{\bQ^{-1}}.
	\end{aligned}
\end{equation}
When $\mu$ is close to $\lambda_n$, the bound $\left|\frac{\lambda_n \textcolor{blue}{-\mu}}{\lambda_{n-1}\textcolor{blue}{-\mu}}\right|$ is small such that the convergence is faster than the (naive) inverse power method (see Equation~\eqref{equation:-inverse-power-iteration-analysis1}), although it is still linearly convergent.

The formal convergence results is shown in the following theorem (one can follow the deduction as in Theorem~\ref{theorem:convergence-power-iteration} to obtain the result).

\begin{theorem}[Convergence of Shifted Inverse Power Iteration]
	Suppose $\lambda_J$ is the closest eigenvalue to $\mu$ and $\lambda_K$ is the second closest. Moreover, $\bq_J^\top \bv^{(0)}\neq 0$. Then the iterates of Algorithm~\ref{alg:inverse-power-iteration-shifted} satisfy
	$$
	||\bv^{(k)} - (\pm \bq_J)||= O\left(\left|\frac{\lambda_J-\mu}{\lambda_K-\mu}\right|^k\right), \qquad |\lambda^{(k)}-\lambda_J|=O\left(\left|\frac{\lambda_J-\mu}{\lambda_K-\mu}\right|^{2k}\right)
	$$
	as $k\rightarrow \infty$.
\end{theorem}
And this shows $\bv^{(k)}$ converges to $\pm\bq_J$ linearly for Algorithm~\ref{alg:inverse-power-iteration-shifted}.

\subsection{The Rayleigh Quotient Method}
We know that the \textit{inverse power iteration} converges to the eigenvector corresponding to the smallest eigenvalue of $\bA$, and the \textit{shifted inverse power iteration} converges to the eigenvector corresponding to the eigenvalue closest to $\mu$ with possible faster convergence. Fortunately, both of the two methods are \textit{inverse} power iteration in some sense. If we can combine the ideas behind the two algorithms, i.e., use the Rayleigh quotient of the estimated eigenvector in each iteration as the estimate of the eigenvalue, we can get a faster algorithm.

\begin{algorithm}[h]
	\caption{Rayleigh Quotient Iteration (Compare to Algorithm~\ref{alg:inverse-power-iteration-shifted})}
	\label{alg:rayleigh-quotient-iteration2}
	\begin{algorithmic}[1]
		\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
		\State $\bv^{(0)}$= some vector with $||\bv^{(0)}||=1$;
		\State $\lambda^{(0)}=(\bv^{(0)})^\top\bA\bv^{(0)}$; \Comment{i.e., Rayleigh quotient}
		\For{$k=1,2,\ldots$}
		\State $\bw = \textcolor{blue}{(\bA-\lambda^{(k-1)}\bI)^{-1}} \bv^{(k-1)}$;
		\State $\bv^{(k)} = \bw / ||\bw||$;
		\State $\lambda^{(k)} = (\bv^{(k)})^\top\bA\bv^{(k)}$;  \Comment{i.e., Rayleigh quotient}
		\EndFor
	\end{algorithmic}
\end{algorithm}

The Rayleigh quotient iteration finds an eigenvector, but with which eigenvalue it is associated
is not clear from the start.
\section{QR Algorithm}
The QR algorithm for computing the eigenvalues and eigenvectors of matrices has been named as one of the ten most important algorithms of the twentieth century \citep{dongarra2000guest, cipra2000best} which is published by John G. F. Francis in the work of \citep{francis1961qr, francis1962qr} that is quoted as one of the jewels of numerical analysis \citep{trefethen1997numerical}. We will introduce the QR algorithm from the most simple case to a shifted version with implicit calculation.

The QR algorithm goes further by \textit{simultaneously} calculating the eigenvalues of a given matrix $\bA$. The central idea is to reduce matrix $\bA$ by a set of \textit{similarity transformations} (Definition~\ref{definition:similar-matrices}, p.~\pageref{definition:similar-matrices}) from which the eigenvalues are kept unmodified (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices})
and in the meantime the eigenvalue is somewhat easier to calculate. The net result turns to be simple to handle, all we do in the QR algorithm is take a QR decomposition, multiply the computed factors $\bQ$ and $\bR$ together in the reverse order $\bR\bQ$, and repeat the procedure. Hence the name \textit{QR algorithm}.

\subsection{Preliminary: Power Iteration with Eigenvector Known}
We firstly show how it evolves from the power iteration algorithm.
Now, let's put a subscript for all the $\bv^{(k)}$'s to emphasize it converges to the eigenvector associated with $\lambda_i$: e.g., $\bv^{(k)}_{\textcolor{blue}{i}}$ will be proven to converge to $\lambda_{\textcolor{blue}{i}}$. And suppose further that we know the normalized eigenvector $\bq_1$ associated with $\lambda_1$ up front. A second initial vector, $\bv_2^{(0)}$, does not have a component in the direction of $\bq_1$. This can be met if it is orthogonal
to $\bq_1$ which is constructed by projection introduced in the Gram-Schmidt process (Section~\ref{section:project-onto-a-vector}, p.~\pageref{section:project-onto-a-vector}):
$$
\bv_2^{(k+1)} \leftarrow \bv_2^{(k)} -\bq_1^\top \bv_2^{(k)}\bq_1
$$
such that $\bq_1^\top \bv_2^{(k+1)}=0$ (recall $\bq_1^\top \bv_2^{(k)}\bq_1$ above is the component of $ \bv_2^{(k)}$ in the direction of $\bq_1$ since $\bq_1$ has unit length). With $\bq_1$ known beforehand, we consider the method in Algorithm~\ref{alg:power-iteration-in-qr-preliminary1}.
\begin{algorithm}[H]
\caption{Power Iteration ($\bq_1$ is Known Up Front)}
\label{alg:power-iteration-in-qr-preliminary1}
\begin{algorithmic}[1]
\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
\State \textcolor{blue}{$\bq_1$ is known up front};
\State $\bv_2^{(0)}$= some vector in $\real^n$;
\State $\bv_2^{(0)} = \bv_2^{(0)} -\textcolor{blue}{\bq_1}^\top \bv_2^{(0)} \textcolor{blue}{\bq_1}$; \Comment{i.e., project along $\bq_1$: $\bq_1^\top \bv_2^{(0)}=0$}
\State $\bv_2^{(0)} = \bv_2^{(0)} /||\bv_2^{(0)}||$;  \Comment{normalize to have length one}
\For{$k=1,2,\ldots$}
\State $\bv_2^{(k)}= \bA \bv_2^{(k-1)}$;
\State $\bv_2^{(k)} = \bv_2^{(k)} -\textcolor{blue}{\bq_1}^\top \bv_2^{(k)} \textcolor{blue}{\bq_1}$; \Comment{make sure $\bv_2^{(k)}$ is orthogonal to $\bq_1$}
\State $\bv_2^{(k)} = \bv_2^{(k)}/ ||\bv_2^{(k)}||$;
\State $\lambda_2^{(k)} = (\bv_2^{(k)})^\top\bA\bv_2^{(k)}$;  \Comment{i.e., Rayleigh quotient}
\EndFor
\end{algorithmic}
\end{algorithm}
Write again $\bv_2^{(0)}$ as a linear combination of the orthonormal eigenvectors $\bq_i$:
\begin{equation}\label{equation:power-qr-premi-1}
\bv_2^{(0)} = x_1 \bq_1 + x_2\bq_2 + \ldots + x_n\bq_n.
\end{equation}
Since in step 3 of above algorithm, we project along the $\bv_2^{(0)}$ along $\bq_1$, then the component $x_1$ of Equation~\eqref{equation:power-qr-premi-1} is equal to 0.
Similarly, $\bv_2^{(k)}$ is a multiple of $\bA^k \bv_2^{(0)}$ such that  $\bv_2^{(k)} = c_{2k} \bA^k \bv_2^{(0)} = \frac{\bA^k \bv_2^{(0)}}{||\bA^k \bv_2^{(0)}||}$. We have
$$
\begin{aligned}
\bv_2^{(k)} &=  c_{2k} \bA^k\bv_2^{(0)} \\
&=c_{2k} (x_1 \lambda_1^k\bq_1 + x_2\lambda_2^k\bq_2+x_3\lambda_3^k\bq_3 + \ldots + x_n\lambda_n^k\bq_n) \\
&=c_{2k} (x_2\lambda_2^k\bq_2 + x_3\lambda_3^k\bq_3+ \ldots + x_n\lambda_n^k\bq_n) \\
&= c_{2k} \lambda_2^k\left(x_2\bq_2 + x_3\left(\frac{\lambda_3}{\lambda_2}\right)^k\bq_3 + \ldots + x_n\left(\frac{\lambda_n}{\lambda_1}\right)^k\bq_n\right).
\end{aligned}
$$
Therefore, following from Theorem~\ref{theorem:convergence-power-iteration}, if we assume $|\lambda_2| \textcolor{blue}{>} |\lambda_3| \geq |\lambda_3| \geq \ldots \geq |\lambda_n| \geq 0$ and $\bq_2^\top \bv_2^{(0)}\neq 0$, $\bv_2^{(k)}$ will converge linearly to $\pm \bq_2$, and so $\lambda_2^{(k)}$ will converge to $\lambda_2$ by Algorithm~\ref{alg:power-iteration-in-qr-preliminary1}.

Analogously, when \{$|\lambda_2|\textcolor{blue}{=|}\lambda_3| \textcolor{blue}{>} |\lambda_4| \geq |\lambda_5| \ldots $, $\bq_1^\top \bv^{(0)} \neq 0$, and $\bq_2^\top \bv^{(0)} \neq 0$\}, $\bv^{(k)}$ will converge to a multiple of $x_2\bq_2 \pm x_3\bq_3$, i.e., lies in the space spanned by $\{\bq_2, \bq_3\}$.

\subsection{Preliminary: Power Iteration with Eigenvector Unknown}
However, the method presented in Algorithm~\ref{alg:power-iteration-in-qr-preliminary1} is not practical since we usually do not know $\bq_1$ up front. But since we have a trivial power iteration that can compute the $\bv_1^{(0)}$ converging to $\pm \bq_1$ (Algorithm~\ref{alg:power-iteration}, p.~\pageref{alg:power-iteration}), a simultaneous algorithms can be constructed. In lieu of project $\bv_2^{(k)}$ along $\bq_1$, one can project it along $\bv_1^{(k)}$ instead. Therefore, a method to find both $\bq_1, \bq_2$ simultaneously can be constructed in Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-q1}.

\begin{algorithm}[H]
	\caption{Power Iteration ($\bq_1$ is Unknown Up Front, Compare to Algorithm~\ref{alg:power-iteration-in-qr-preliminary1})}
	\label{alg:power-iteration-in-qr-preliminary2-unknown-q1}
	\begin{algorithmic}[1]
		\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
		\State $\bv_1^{(0)}, \bv_2^{(0)}$= two random vectors in $\real^n$;
		\State $\bv_2^{(0)} = \bv_2^{(0)} -\textcolor{blue}{\bv_1^{(0)}}^\top \bv_2^{(0)} \textcolor{blue}{\bv_1^{(k)}}$; \Comment{i.e., project along $\bv_1^{(0)}$: \textcolor{blue}{$\bv_1^{(0)\top} \bv_2^{(0)}=0$}}
		\State $\bv_1^{(0)} = \bv_1^{(0)} /||\bv_1^{(0)}||$;  \Comment{normalize to have length one}
		\State $\bv_2^{(0)} = \bv_2^{(0)} /||\bv_2^{(0)}||$;  \Comment{normalize to have length one}
		\For{$k=1,2,\ldots$}
		\State // update $\bv_1^{(k)}$
		\State $\bv_1^{(k)}  = \bA\bv_1^{(k-1)}$;
		\State $\bv_1^{(k)} = \bv_1^{(k)} / ||\bv_1^{(k)} ||$;
		\State // update $\bv_2^{(k)}$
		\State $\bv_2^{(k)} = \bA \bv_2^{(k-1)}$;
		\State $\bv_2^{(k)} = \bv_2^{(k)} -\textcolor{blue}{\bv_1^{(k)}}^\top \bv_2^{(k)} \textcolor{blue}{\bv_1^{(k)}}$; \Comment{make sure $\bv_2^{(k)}$ is orthogonal to \textcolor{blue}{$\bv_1^{(k)}$}}
		\State $\bv_2^{(k)} = \bv_2^{(k)}/ ||\bv_2^{(k)}||$;
		\State // compute the corresponding eigenvalues
		\State $\lambda_1^{(k)} = (\bv_1^{(k)})^\top\bA\bv_1^{(k)}$;  \Comment{i.e., Rayleigh quotient}
		\State $\lambda_2^{(k)} = (\bv_2^{(k)})^\top\bA\bv_2^{(k)}$;  \Comment{i.e., Rayleigh quotient}
		\EndFor
	\end{algorithmic}
\end{algorithm}
Combine the findings above, the iterates of Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-q1} satisfies that (when $\bq_1^\top \bv_1^{(0)} \neq 0$ and $\bq_2^\top \bv_2^{(0)}\neq 0$)
\begin{itemize}
\item If $|\lambda_1| > |\lambda_2|$, the vector sequence $\bv_1^{(k)}$ will converge linearly to $\pm \bq_1$ at a rate of $|\frac{\lambda_2}{\lambda_1}|$;
\item If $|\lambda_1| > |\lambda_2|> |\lambda_3|$, the vector sequence $\bv_2^{(k)}$ will converge linearly to $\pm \bq_2$  at a rate of $|\frac{\lambda_3}{\lambda_2}|$;
\item If $|\lambda_1| = |\lambda_2| > |\lambda_3|$, the vector sequence $\bv_1^{(k)} $ will converge to a multiple of $x_1\bq_1 \pm x_2\bq_2$, and the vector sequence $\bv_2^{(k)}$ will converge linearly to $\pm \bq_2$. I.e., the span of $\{\bq_1, \bq_2\}$ can be approximated by the span of $\{\bv_1^{(k)}, \bv_2^{(k)}\}$.
\end{itemize}

\subsection{Preliminary: Power Iteration with Eigenvector Unknown and QR Decomposition}\label{section:power-eigen-unknown-prelimise}
We carefully notice that that step 2 to step 4 Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-q1} is equivalent to apply a QR decomposition on a $\real^{n\times 2}$ matrix $[\bv_1^{(0)}, \bv_2^{(0)}]$, that is
$$
\underbrace{[\bv_1^{(0)}, \bv_2^{(0)}] }_{\widehat{\bV}^{(0)}}, \bR \leftarrow QR([\bv_1^{(0)}, \bv_2^{(0)}]),
$$
where $QR(\bA)$ is defined as the function to obtain the QR decomposition of matrix $\bA$.
Moreover, step 6 to step 12 of Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-q1} can also be rephrased as an QR decomposition on the $n\times 2$ matrix $\bA[\bv_1^{(k-1)}, \bv_2^{(k-1)}]$. A further simplification on the form is to obtain the eigenvalues $\lambda_1^{(k)}, \lambda_2^{(k)} $ as follows:
$$
\underbrace{\begin{bmatrix}
		\lambda_1^{(k)}  & 0\\
		0 & \lambda_2^{(k)}
\end{bmatrix}}_{\bA^{(k)}}
=
\underbrace{\begin{bmatrix}
		\bv_1^{(k)} & \bv_1^{(k)}
	\end{bmatrix}^\top }_{\widehat{\bV}^{(k)\top}}
\bA
\underbrace{\begin{bmatrix}
		\bv_1^{(k)} & \bv_1^{(k)}
\end{bmatrix}}_{\widehat{\bV}^{(k)}}
$$

\begin{algorithm}[H]
	\caption{Power Iteration (On 2 Vectors, Equivalent to Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-q1})}
	\label{alg:power-iteration-in-qr-preliminary2-unknown-qr-function}
	\begin{algorithmic}[1]
		\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
		\State $\widehat{\bV}^{(0)}= [\bv_1^{(0)}, \bv_2^{(0)} ]\in \real^{n\times 2}$= two random vectors in $\real^n$;
		\State $\widehat{\bV}^{(0)}, \bR= QR(\widehat{\bV}^{(0)})$;
		\For{$k=1,2,\ldots$}
		\State $\widehat{\bV}^{(k)}, \bR= QR(\bA\widehat{\bV}^{(k-1)})$;
		\State $\widehat{\bA}^{(k)}=\widehat{\bV}^{(k)\top} \bA \widehat{\bV}^{(k)}$; \Comment{ compute the corresponding eigenvalues}
		\EndFor
	\end{algorithmic}
\end{algorithm}

Note that the $\widehat{\text{widehat}}$ above the matrices in Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-qr-function} will be proven useful to differentiate the ones in the QR algorithms.
With hindsight, it is natural to extent the algorithm not only for two vectors, but for $p\leq n$ vectors. The full algorithm is formulated in Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-qr-function-mn}.
\begin{algorithm}[H]
\caption{Power Iteration (On $p$ Vectors, Compare to Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-qr-function})}
\label{alg:power-iteration-in-qr-preliminary2-unknown-qr-function-mn}
\begin{algorithmic}[1]
\Require matrix $\bA\in \real^{n\times n}$ that is real and symmetric;
\State $\widehat{\bV}^{(0)}$= random matrix in $\real^{n\times \textcolor{blue}{p}}$;
\State $\widehat{\bV}^{(0)}, \bR= QR(\widehat{\bV}^{(0)})$;
\For{$k=1,2,\ldots$}
\State $\widehat{\bV}^{(k)}, \bR= QR(\bA\widehat{\bV}^{(k-1)})$;
\State $\widehat{\bA}^{(k)}=\widehat{\bV}^{(k)\top} \bA \widehat{\bV}^{(k)}$; \Comment{ compute the corresponding eigenvalues}
\EndFor
\end{algorithmic}
\end{algorithm}
Again, we observe that, when $\bq_i^\top \bv_i^{(0)} \neq 0$ for all $i \in \{1,2,\ldots, p\}$, \footnote{That is, the initial guess $\bv_i^{(0)}$ is not orthogonal to eigenvector $\bq_i$, and has a component in the direction of the eigenvector.} the iterates of Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-qr-function-mn} satisfies that
\begin{enumerate}
\item If $|\lambda_1| > |\lambda_2| > \ldots > |\lambda_p| > |\lambda_{p+1}| \geq |\lambda_{p+2}| \geq \ldots $, then each column $i$ of $\widehat{\bV}^{(k)}$ (i.e., $\bv_i^{(k)}$) will converge linearly to $\pm \bq_i$ where the rate of removing from the component in the direction of $\bq_j$ is recorded as $|\frac{\lambda_i}{\lambda_j}|$ ($0<i\leq p $ and $p<j\leq n$);
\item If some of the eigenvalues have equal magnitude, then the subspace spanned by the corresponding columns of $\widehat{\bV}^{(k)}$ will approximate the subspace spanned by the corresponding eigenvectors associated with those eigenvalues;
\item If $p=n$, and $|\lambda_1| > |\lambda_2| > \ldots >|\lambda_n|$, then we will find all the eigenvectors of $\bA$.
\end{enumerate}

\subsection{A Simple QR Algorithm from Power Iteration: without Shifts}\label{section:qralg-without-shifts}
So far, we have transferred power iteration into an algorithm finding all the eigenvectors (under mild conditions) which employs QR decomposition.

Now, let's consider the power iteration with slight modification in Algorithm~\ref{alg:qr-algorithm-simple1} that cares the $\bR$ from the QR decomposition as a sequence of vectors, and $\widehat{\bA}^{(0)}$ initialized to be matrix $\bA$ (where we ignored the value indexed by 0 previously). Compare the two Algorithm~\ref{alg:qr-algorithm-simple1} and \ref{alg:qr-algorithm-simple2}.

\noindent
\begin{minipage}[t]{0.495\linewidth}
\begin{algorithm}[H]
\caption{Power Iteration}
\label{alg:qr-algorithm-simple1}
\begin{algorithmic}[1]
\Require $\bA\in \real^{n\times n}$ is real and symmetric;
\State Same as Algorithm~\ref{alg:power-iteration-in-qr-preliminary2-unknown-qr-function-mn};
\State $\widehat{\bA}^{(0)} = \bA$;
\State $\widehat{\bV}^{(0)}=\bI_n$; \Comment{initial eigenvector guess}
\State $\widehat{\bR}^{(0)} = \bI_n$; \Comment{compensate the sequence}
\For{$k=1,2,\ldots$}
\State $\widehat{\bV}^{(k)}, \widehat{\bR}^{(k)}= QR(\bA\widehat{\bV}^{(k-1)})$;
\State $\widehat{\bA}^{(k)}=\widehat{\bV}^{(k)\top} \bA \widehat{\bV}^{(k)}$;
\EndFor
\end{algorithmic}
\end{algorithm}
\end{minipage}%
\hfil
\begin{minipage}[t]{0.495\linewidth}
\begin{algorithm}[H]
	\caption{Simple QR Algorithm}
	\label{alg:qr-algorithm-simple2}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{n\times n}$ is real and symmetric;
		\State $\bA^{(0)} = \bA$;
		\State $\bV^{(0)}=\bI_n$;  \Comment{initial eigenvector guess}
		\State $\bR^{(0)} = \bI_n$; \Comment{compensate the sequence}
		\State $\bQ^{(0)} = \bI_n$; \Comment{compensate the sequence}
		\For{$k=1,2,\ldots$}
		\State $\bQ^{(k)}, \bR^{(k)}= QR(\bA^{(k-1)})$;
		\State $\bA^{(k)}=\bR^{(k)} \bQ^{(k)}$;
		\State $\bV^{(k)} = \bV^{(k-1)}\bQ^{(k)} $;
		\EndFor
	\end{algorithmic}
\end{algorithm}
\end{minipage}

We firstly show the equivalence of the two algorithms by the following lemma.
\begin{lemma}[QR Algorithm from Power Iteration]\label{lemma:qr-algo-from-power}
We can show Algorithm~\ref{alg:qr-algorithm-simple1} and Algorithm~\ref{alg:qr-algorithm-simple2} are equivalent in the sense that for all iterates $k\in \{0,1,2,\ldots\}$, we have
$$
\left\{
\begin{aligned}
	\widehat{\bA}^{(k)} &= \bA^{(k)};  \qquad \text{(diagonals that will converge to eigenvalues)}\\
	\widehat{\bR}^{(k)} &= \bR^{(k)}; \\
	\widehat{\bV}^{(k)} &= \bV^{(k)}. \qquad \text{(columns that will converge to eigenvectors)}
\end{aligned}
\right.
$$
\end{lemma}
For clarity, the proof is delayed in Section~\ref{section:proofs-qralgorithms}.
You might well ask: how could we find the reduction of Algorithm~\ref{alg:qr-algorithm-simple1} to Algorithm~\ref{alg:qr-algorithm-simple2}. The answer is unglamorous! It was by trial and error.

\paragraph{What's in the QR algorithm}
All we do in Algorithm~\ref{alg:qr-algorithm-simple2} is to take a QR decomposition, multiply the computed factors $\bQ$ and $\bR$
together in the reverse order $\bR\bQ$, and repeat. For convergence to diagonal form to be useful for finding eigenvalues, of course, the operations from $\bA^{(k-1)}$ to $\bA^{(k)}$ should be similarity transformations (Definition~\ref{definition:similar-matrices}, p.~\pageref{definition:similar-matrices}).
By Algorithm~\ref{alg:qr-algorithm-simple2}, it can be shown that
\begin{tcolorbox}[title={Simple QR Algorithm Property 1}]
\begin{equation}\label{equation:simple-qr-find}
\text{(SQR 1)} \qquad \bA^{(k)} = \bQ^{(k)\top } \bA^{(k-1)} \bQ^{(k)},
\end{equation}
\end{tcolorbox}
\noindent since $\bA^{(k)}=\bR^{(k)} \bQ^{(k)} = (\bQ^{(k)\top} \bA^{(k-1)}) \bQ^{(k)} $. Therefore, $\bA^{(k)}$ is a \textit{similarity transformation}  \footnote{Here, when the nonsingular matrix is orthogonal, it is also known as \textit{orthogonal similarity transformation}.} of $\bA^{(k-1)}$, which is also similar transformation from $\bA$. This results in that $\bA^{(k)}$ and $\bA$ have \textit{same eigenvalues, trace, and rank} (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). Going to the root of the induction in  Equation~\eqref{equation:simple-qr-find}, it follows that
\begin{tcolorbox}[title={Simple QR Algorithm Property 2}]
\begin{equation}\label{equation:simple-qr-find-root}
\begin{aligned}
\text{(SQR 2)} \qquad \bA^{(k)} &= \bQ^{(k)\top } \bA^{(k-1)} \bQ^{(k)} \\
&=\bQ^{(k)\top } \left(\bQ^{(k-1)\top } \bA^{(k-2)} \bQ^{(k-1)} \right) \bQ^{(k)} \\
&=\ldots \\
&=
\underbrace{\bQ^{(k)\top } \bQ^{(k-1)\top }\ldots \bQ^{(0)\top } }_{\bV^{(k)\top}}
   \bA
\underbrace{\bQ^{(0)} \ldots  \bQ^{(k-1)}  \bQ^{(k)} }_{\bV^{(k)}}
\\
&=\bV^{(k)\top}\bA \bV^{(k)}.
\end{aligned}
\end{equation}
\end{tcolorbox}

The above lemma tells that the simple QR algorithms is equivalent to the power iteration algorithms such that the diagonals of $\bA^{(k)}$ converge to the eigenvalues of $\bA$ (under mild conditions).
$\bA^{(k)} =\bV^{(k)\top}\bA \bV^{(k)} $ is not only a similarity transformation, but also an \textit{orthogonal similarity transformation} since $\bV^{(k)}$ is orthogonal. This is particular important for the stability of the iterative method as the condition of $\bA^{(k)}$ is not worse than that of the original matrix $\bA$.
Then, the equivalence indicates that the $i$-th column of $\bV^{(k)}$ will converge linearly to the $i$-th eigenvector of $\bA$, i.e., $\pm \bq_i$.
A delve into the process, it shows by Algorithm~\ref{alg:qr-algorithm-simple1} and~\ref{alg:qr-algorithm-simple2} that
\begin{tcolorbox}[title={Simple QR Algorithm Property 3}]
\begin{equation}\label{equation:simple-qr-find2}
\begin{aligned}
&\left.
\begin{aligned}
	\bV^{(k)} &= \bV^{(0)}\bQ^{(1)}\bQ^{(2)}\ldots \bQ^{(k)}    \\
	\bA^{\textcolor{blue}{k}}&=\bV^{(k)} \bR^{(k)} \bR^{(k-1)} \ldots \bR^{(0)}
\end{aligned}
\right\}
\leadtosmall \\
&\text{(SQR 3)} \qquad \bA^k= \underbrace{\bQ^{(0)}\bQ^{(1)}\bQ^{(2)}\ldots \bQ^{(k)}}_{=\bV^{(k)}, \text{orthogonal}}
\underbrace{\bR^{(k)} \bR^{(k-1)} \ldots \bR^{(0)}}_{=\bU^{(k)}, \text{upper triangular}}
,
\end{aligned}
\end{equation}
\end{tcolorbox}
\noindent where the left hand side of the above equation is delayed to be proved in Section~\ref{section:proofs-qralgorithms} and where $\bQ^{(0)} = \bV^{(0)} = \bI$ for simplicity.
That is, $\bA^{k}$ can be expressed as a QR decomposition by $\bA^{k}=\bV^{(k)}\bU^{(k)}$.\footnote{Note the different between the notation $\bA^k$ and $\bA^{(k)}$ where $\bA^k$ here is the $k$-th power of $\bA$.}

The equivalence of the simultaneous power method and the simple QR algorithm (Algorithm~\ref{alg:qr-algorithm-simple1} and \ref{alg:qr-algorithm-simple2}) tells us a lot about convergence.
Same as that in Section~\ref{section:power-eigen-unknown-prelimise}, it follows that
\begin{enumerate}
\item If $|\lambda_1| > |\lambda_2| > \ldots > |\lambda_p| > |\lambda_{p+1}| \geq |\lambda_{p+2}| \geq \ldots \geq |\lambda_n|$, then each column $i$ of $\bV^{(k)}$ (i.e., $\bv_i^{(k)}$) will converge linearly to $\pm \bq_i$ where the rate of removing from the component in the direction of $\bq_j$ is recorded as $|\frac{\lambda_i}{\lambda_j}|$ ($0<i\leq p $ and $p<j\leq n$);
\item If some of the eigenvalues have equal magnitude, then the subspace spanned by the corresponding columns of $\bV^{(k)}$ will approximate the subspace spanned by the corresponding eigenvectors associated with those eigenvalues;
\item If $p=n$, and $|\lambda_1| > |\lambda_2| > \ldots >|\lambda_n|$, then we will find all the eigenvectors of $\bA$. Specifically, suppose $$
\bA^{(k)} =
\begin{bmatrix}
\lambda_1^{(k)} & a_{12} & a_{13} & \ldots & a_{1n}\\
a_{12} & \lambda_2^{(k)} & a_{23} & \ldots & a_{2n}\\
\vdots & \vdots & \ddots & \ddots & \vdots\\
a_{1,n-1} &  a_{2,n-1} &a_{3,n-1}  & \ddots & a_{n-1,n}\\
a_{1n} &  a_{2n} &a_{3n}  & \ldots & \lambda_n^{(k)}\\
\end{bmatrix},
$$
which is symmetric, and the convergence rate is
$$
|a_{i,i-1}| = O\left(\left|\frac{\lambda_i}{\lambda_{i-1}}\right|^k\right).
$$
\end{enumerate}
Under different conditions, say, $|\lambda_1|\geq |\lambda_2| \geq \ldots \geq |\lambda_p| > |\lambda_{p+1}| > \ldots |\lambda_{n}|$, then
\begin{enumerate}
\item The span of the first $p$ columns of $\bV^{(k)}$ will converge to the subspace spanned by the first $p$ orthonormal eigenvectors $span\{\bq_1, \bq_2, \ldots, \bq_p\}$;
\item The span of the last $n-p$ columns of $\bV^{(k)}$ will converge to the subspace spanned by the last $n-p$ orthonormal eigenvectors $span\{\bq_{p+1}, \bq_{p+2}, \ldots, \bq_n\}$;
\item Since $\bA$ is assumed to be real and symmetric, it follows that
 $$span\{\bq_1, \bq_2, \ldots, \bq_p\} \perp span\{\bq_{p+1}, \bq_{p+2}, \ldots, \bq_n\} .$$
Therefore, the first $p$ columns and last $n-p$ columns of $\bV^{(k)}$ lie in a mutually orthogonal complement space. And the rate of convergence with which the two subspace become orthogonal to each other is linear
with a constant $|\lambda_{p+1}/\lambda_p|$;
\item Furthermore, when the above $p=n$, it follows that the last column of $\bV^{(k)}$ converges linearly to $\pm \bq_n$ with a constant $|\lambda_n/\lambda_{n-1}|$.
\end{enumerate}
By the finding in the shifted inverse power method (Algorithm~\ref{alg:inverse-power-iteration-shifted}), a shift by the estimate of the smallest eigenvalue (in magnitude) in each iteration can accelerate the convergence of the algorithm, and this reveals a ``practical" QR algorithm.

\begin{remark}[Asymmetric Matrix $\bA$]
In the above discussions, we assume $\bA$ is real and symmetric. If $\bA$ is asymmetric with real eigenvalues that are distinct in module , it can be shown that the $\bA^{(k)}$ in the QR algorithm converges to an upper triangular matrix where the eigenvalues lie on the diagonal (see the second form of Schur decomposition, Corollary~\ref{corollary:schur-second-form}, p.~\pageref{corollary:schur-second-form}). Fore more general matrices $\bA$, the sequence converges to an upper \textit{quasi-triangular} matrix. See \citep{quarteroni2010numerical, golub2013matrix} for more details.
\end{remark}

\paragraph{LU Algorithm} We shall only briefly discuss the LU algorithm here. Instead of factor the matrix in each iteration by a QR decomposition, the LU decomposition can be applied as well. To see this, we consider the procedure in Algorithm~\ref{alg:lu-algorithm-simple2}, and suppose further $\bA$ is nonsingular (such that its LU decomposition has nonsingular factors). We then have
\begin{equation}\label{equation:simple-qr-find-lu-simple}
\boxed{\begin{aligned}
		\text{(SLU 1)} &\qquad \bA^{(k)} = \bU^{(k)}\bL^{(k)}
		=((\bL^{(k)})^{-1} \bL^{(k)})\bU^{(k)}\bL^{(k)}=(\bL^{(k)})^{-1}\bA^{(k-1)}\bL^{(k)}, \\
		\text{(SLU 2)} &\qquad \bA^{(k)} =
		(\bL^{(k) } )^{-1} (\bL^{(k-1) })^{-1}\ldots (\bL^{(0) } )^{-1}
		\bA
		\bL^{(0)} \ldots  \bL^{(k-1)}  \bL^{(k)} .
\end{aligned}}
\end{equation}
From (SLU 1), the update from $\bA^{(k-1)}$ to $\bA^{(k)}$ is a similarity transformation (not orthogonal similarity transformation now). Therefore, the accuracy depends on the condition of each $\bL^{(k) }$ that may be out of control. This results in the rare use of the LU algorithm. See \citep{rutishauser1958solution, francis1961qr} fore more details (which is named as LR algorithm in the original papers).

\begin{algorithm}[H]
\caption{Simple LU Algorithm (Compare to Algorithm~\ref{alg:qr-algorithm-simple2})}
\label{alg:lu-algorithm-simple2}
\begin{algorithmic}[1]
\Require $\bA\in \real^{n\times n}$ is real and symmetric;
\State $\bA^{(0)} = \bA$;
\State $\bV^{(0)}=\bI_n$;  \Comment{initial eigenvector guess}
\State $\textcolor{blue}{\bU}^{(0)} = \bI_n$; \Comment{compensate the sequence}
\State $\textcolor{blue}{\bL}^{(0)} = \bI_n$; \Comment{compensate the sequence}
\For{$k=1,2,\ldots$}
\State $\textcolor{blue}{\bL}^{(k)}, \textcolor{blue}{\bU}^{(k)}= \textcolor{blue}{LU}(\bA^{(k-1)})$;
\State $\bA^{(k)}=\textcolor{blue}{\bU}^{(k)} \textcolor{blue}{\bL}^{(k)}$;
\State $\bV^{(k)} = \bV^{(k-1)}\textcolor{blue}{\bL}^{(k)} $;
\EndFor
\end{algorithmic}
\end{algorithm}

\subsection{A Practical QR Algorithm: with Shifts}
From the above discussion,
we could compute the Rayleigh quotient of the last column of $\bV^{(k-1)}$. Or recall that the matrix $\bA^{(k-1)}$ estimates the eigenvalues in the diagonal such that $a_{nn}^{(k-1)}$ (the last diagonal of $\bA^{(k-1)}$) can be applied as a shift. The procedure is shown in Algorithm~\ref{alg:qr-algorithm-practical}.

\begin{algorithm}[H]
\caption{Practical QR Algorithm (The Final Algorithm! Compare to Algorithm~\ref{alg:qr-algorithm-simple2})}
\label{alg:qr-algorithm-practical}
\begin{algorithmic}[1]
\Require $\bA\in \real^{n\times n}$ is real and symmetric;
\State $\bA^{(0)} = \bA$;
\State $\bV^{(0)}=\bI_n$;\Comment{initial eigenvector guess}
\State $\bR^{(0)} = \bI_n$;  \Comment{compensate the sequence}
\State $\bQ^{(0)} = \bI_n$; \Comment{compensate the sequence}
\For{$k=1,2,\ldots$}
\State Pick a shift $\mu^{(k)}$; \Comment{e.g., $\mu^{(k)}=a_{nn}^{(k-1)}$}
\State $\bQ^{(k)}, \bR^{(k)}= QR\left(\bA^{(k-1)} - \textcolor{blue}{\mu^{(k)}\bI}\right)$; \Comment{QR decomposition}
\State $\bA^{(k)}=\bR^{(k)} \bQ^{(k)} + \textcolor{blue}{\mu^{(k)}\bI}$;  \Comment{diagonals converge to eigenvalues}
\State $\bV^{(k)} = \bV^{(k-1)}\bQ^{(k)} $; \Comment{columns converge to eigenvectors}
\EndFor
\end{algorithmic}
\end{algorithm}

Similar observation can be applied:
\begin{tcolorbox}[title={Practical QR Algorithm Property 1}]
\begin{equation}\label{equation:practical-qr-find1}
\left.
\begin{aligned}
	\bR^{(k)}&= \bQ^{(k)\top} \left(\bA^{(k-1)} - \mu^{(k)}\bI\right)\\
	\bA^{(k)}&=\bR^{(k)} \bQ^{(k)} + \mu^{(k)}\bI
\end{aligned}
\right\} \leadtosmall
\underbrace{\bA^{(k)} = \bQ^{(k)\top } \bA^{(k-1)} \bQ^{(k)}}_{\text{(PQR 1)=(SQR 1)}},
\end{equation}
\end{tcolorbox}
\noindent which is the same \textit{similar transformation} as the simple QR algorithm shown in Equation~\eqref{equation:simple-qr-find}.
Similar to Equation~\eqref{equation:simple-qr-find-root}, the same observation can be applied:
\begin{tcolorbox}[title={Practical QR Algorithm Property 2}]
\begin{equation}\label{equation:practical-qr-find-root}
	\begin{aligned}
		\bA^{(k)} =\bV^{(k)\top}\bA \bV^{(k)} \leadto \underbrace{\bA =\bV^{(k)}  \bA^{(k)} \bV^{(k)\top}}_{\text{(PQR 2)=(SQR 2)}},
	\end{aligned}
\end{equation}
\end{tcolorbox}
\noindent where $\bV^{(k)}=\bQ^{(0)}\bQ^{(1)}\bQ^{(2)}\ldots \bQ^{(k)}$ (same as (SQR 2)).
Again, $\bA^{(k)} =\bV^{(k)\top}\bA \bV^{(k)} $ is an \textit{orthogonal similarity transformation} since $\bV^{(k)}$ is orthogonal. The condition of $\bA^{(k)}$ is still not worse than that of the original matrix $\bA$.
Further, the third property of the practical QR algorithm is slightly different to that of the simple QR algorithm:
\begin{tcolorbox}[title={Practical QR Algorithm Property 3}]
\begin{equation}\label{equation:practical-qr-find2}
\begin{aligned}
&\text{(PQR 3)} \neq  \text{(SQR 3)}: \\
&(\bA-\mu^{(k)}\bI)(\bA-\mu^{(k-1)}\bI)\ldots (\bA-\mu^{(1)}\bI)= \underbrace{\bQ^{(0)}\bQ^{(1)}\bQ^{(2)}\ldots \bQ^{(k)}}_{=\bV^{(k)}, \text{orthogonal}}
\underbrace{\bR^{(k)} \bR^{(k-1)} \ldots \bR^{(0)}}_{=\bU^{(k)}, \text{upper triangular}}.
\end{aligned}
\end{equation}
\end{tcolorbox}
\noindent Again, for clarity, the proof of Equation~\eqref{equation:practical-qr-find2} is delayed in Section~\ref{section:proofs-qralgorithms}. Similar to the analysis in the simple QR algorithm (Section~\ref{section:qralg-without-shifts}), assume first $\mu^{(k)}=\mu$ is fixed in Algorithm~\ref{alg:qr-algorithm-practical}, and the eigenvalues are ordered such that $|\lambda_1-\mu| > |\lambda_2-\mu| > \ldots >|\lambda_n-\mu|$, then the $(i,i-1)$-th entry in $\bA^{(k)}$ converges linearly to zero with a constant
$$
\left(\left|\frac{\lambda_i-\mu}{\lambda_{i-1}-\mu}\right|^k\right).
$$
This implies if $\mu^{(k)} = a_{nn}^{(k-1)}$, $|\lambda_n - \mu^{(k)}|$ tends to be much smaller than $|\lambda_i - \mu^{(k)}|$ for $i\in \{1,2,\ldots, n-1\}$:
$$
|\lambda_n - \mu^{(k)}| \ll |\lambda_i - \mu^{(k)}|, \gap i\in \{1,2,\ldots, n-1\}.
$$
This will make the last column of $\bA^{(k)}$ converge to the eigenvalue of $\bA$ rapidly (i.e., $a_{nn}^{(k-1)}$ converges to the eigenvalue rapidly).

\section{Apply the Practical QR Algorithm to Tridiagonal Matrices}
We observe that the practical QR algorithm will result in the eigengectors and eigenvalues via a sequence of \textit{orthogonal similarity transformations} by the property (PQR 1) in Equation~\eqref{equation:practical-qr-find1}: $\bA^{(k)} = \bQ^{(k)\top } \bA^{(k-1)} \bQ^{(k)}$. When $\bA$ is symmetric, a tridiagonal decomposition $\bA=\bQ^{(0)\top}\bT^{(0)}\bQ^{(0)}$ exists (Theorem~\ref{theorem:tridiagonal-decom}, p.~\pageref{theorem:tridiagonal-decom}) which is an orthogonal similarity transformation as well. Such decomposition can be employed as a phase 1 of the QR algorithm since the tridiagonal matrix $\bT^{(0)}$ is close to a diagonal form and easier to converge to the diagonal eigenvalue matrix. The procedure is shown in Algorithm~\ref{alg:qr-algorithm-practical-tridiagonal}.
\subsection{Explicit Shifted QR Algorithm}
\begin{algorithm}[H]
	\caption{Practical QR Algorithm (Two Phases, Compare to Algorithm~\ref{alg:qr-algorithm-practical})}
	\label{alg:qr-algorithm-practical-tridiagonal}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{n\times n}$ is real and symmetric;
		\State \textcolor{blue}{$\bA=\bQ^{(0)}\bT^{(0)}\bQ^{(0)\top}$}; \Comment{tridiagonal decomposition of $\bA$}

		\State $\bV^{(0)}=\textcolor{blue}{\bQ^{(0)}}$;    \Comment{previously $\bV^{(0)}=\bI_n$}
		\State $\bR^{(0)} = \bI_n$;
		\For{$k=1,2,\ldots$}
		\State Pick a shift $\mu^{(k)}$; \Comment{e.g., $\mu^{(k)}=\textcolor{blue}{t}_{nn}^{(k-1)}$}
		\State $\bQ^{(k)}, \bR^{(k)}= QR\left(\textcolor{blue}{\bT}^{(k-1)} - \mu^{(k)}\bI\right)$;
		\State $\textcolor{blue}{\bT}^{(k)}=\bR^{(k)} \bQ^{(k)} + \mu^{(k)}\bI$;
		\State $\bV^{(k)} = \bV^{(k-1)}\bQ^{(k)} $;
		\EndFor
	\end{algorithmic}
\end{algorithm}

The properties of the practical QR algorithm still hold that
\begin{equation}\label{equation:tpqr-1-2}
\boxed{
	\begin{aligned}
		\text{(TPQR 1)} \qquad 	\bT^{(k)} &= \bQ^{(k)\top } \bT^{(k-1)} \bQ^{(k)};\\
		\text{(TPQR $2^\prime$)} \qquad 	\bT^{(k)}
		&= \bQ^{(k)\top } \bQ^{(k-1)\top }\ldots \bQ^{(1)\top } \bT^{(0)}\bQ^{(1)} \ldots  \bQ^{(0)}  \bQ^{(k)} \bQ^{(k)};\\
		&= \underbrace{\bQ^{(k)\top } \bQ^{(k-1)\top }\ldots \bQ^{(1)\top } }_{\bV^{(k)\top}}

		\bA

		\underbrace{\bQ^{(1)} \ldots  \bQ^{(0)}  \bQ^{(k)} \bQ^{(k)}}_{\bV^{(k)}}.
\end{aligned}}
\end{equation}
Except now that the $\bT^{(k)}$'s are tridiagonal matrices (see discussion in the next section).
Suppose $\bT^{(k-1)}$ has the following form where we want to decide the shift $\mu^{(k)}$:
\begin{equation}\label{equation:implicit-matrix-qr}
	\bT^{(k-1)} =
	\begin{bmatrix}
		a_1 & b_1 & & \ldots & 0 \\
		b_1 & a_2 & \ddots & & \vdots \\
		& \ddots & \ddots & \ddots & \\
		\vdots & & \ddots & a_{n-1} & b_{n-1} \\
		0 & \ldots & & b_{n-1} & a_n
	\end{bmatrix}.
\end{equation}
As we have discussed, one reasonable choice for the shift is $\mu^{(k)} = a_n$. And \citep{wilkinson1968global} shows that a more effective choice is to shift by the eigenvalue of
$$
\bT^{(k-1)}_{n-1:n, n-1:n}=
\begin{bmatrix}
	a_{n-1} & b_{n-1}\\
	b_{n-1} & a_n
\end{bmatrix}.
$$
The above matrix has two possible eigenvalues where the one closer to $a_n$ is chosen and it is given by
$$
\mu^{(k)} = \frac{a_{n-1}+a_n   }{2} + \text{sign}(d) \frac{1}{2}\sqrt{(a_{n-1}-a_n)^2+4b_{n-1}^2},
$$
where $d=a_n-a_{n-1}$. This is known as the \textit{Wilkinson shift}. \citep{wilkinson1968global} shows the algorithm converges \textit{cubically} to the eigenvectors of $\bA$ with either shift strategy and the Wilkinson shift is preferred for some heuristic reasons.

\subsection{Implicit Shifted QR Algorithm}\label{section:implifit-shift-qr}
It happens that the shift $\mu^{(k)}$ in Algorithm~\ref{alg:qr-algorithm-practical-tridiagonal} is much larger than some of the diagonal value $a_i$'s in $\bT^{(k-1)}$. Thus it is more reasonable to favor an implicit update on the $\bT^{(k)}$ from $\bT^{(k-1)}$ in which case there is no such sequence of $\bV^{(k)}$ converging to the eigenvectors of $\bA$ and it needs an extra ``explicit" computation on the eigenvectors. To see this, we conclude some observations that are necessary for the implicit shift QR algorithm (make sure to recap the properties of the tridiagonal decomposition in Section~\ref{section:tridiagonal-decomposition}, p.~\pageref{section:tridiagonal-decomposition} before going forward).
\begin{itemize}
\item \textit{Preservation of Form in Simple QR Algorithm.} The tridiagonal decomposition of a tridiagonal decomposition may not be a tridiagonal decomposition, i.e., $\bT_+ = \bQ\bT\bQ^\top$ may not be tridiagonal even if $\bT$ is tridiagonal. However, in our case, if $\bT=\bQ\bR$ is the QR decomposition of a symmetric and tridiagonal matrix $\bT\in \real^{n\times n}$, then $\bQ$ has lower bandwidth 1 and $\bR$ has upper bandwidth 2 (Definition~\ref{defin:matrix-bandwidth}, p.~\pageref{defin:matrix-bandwidth}). Then the reverse QR update $\bT_+ = \bR\bQ = \bQ^\top \bQ(\bR\bQ) = \bQ^\top \bT \bQ$ is also symmetric and tridiagonal.

\item \textit{Positive Lower Subdiagonals of the Tridiagonal Matrix.} The QR decomposition is not unique (Section~\ref{section:nonunique-qr}, p.~\pageref{section:nonunique-qr}). However, when restricting the diagonals of $\bR$ to be positive, the QR decomposition is unique (Corollary~\ref{corollary:unique-qr}, p.~\pageref{corollary:unique-qr}). Then, if $\bT$ has \textit{positive lower subdiagonals} (which implies $\bT$ is \textit{unreduced} \footnote{See Definition~\ref{definition:tridiagonal-hessenbert}, p.~\pageref{definition:tridiagonal-hessenbert}.}, i.e., nonzero lower subdiagonals, and further the lower subdiagonals are \textcolor{blue}{positive}), then $\bT_+=\bQ^\top\bT\bQ$ also has positive lower subdiagonals.\footnote{This is an important claim that is usually ignored in many text.}

\item \textit{Preservation of Form in Practical QR Algorithm.} If $\mu\in \real$, and $\bT-\mu\bI=\bQ\bR$ is the QR decomposition of shifted $\bT$, then $\bT_+=\bR\bQ +\mu\bI$ is also symmetric and tridiagonal.

\item \textit{Implicit Q Theorem.} We observe that if we restrict the elements in the lower sub-diagonal of the tridiagonal matrix $\bT$ to be \textcolor{blue}{positive} (if possible), i.e., \textit{unreduced} with positive lower subdiagonals, then the tridiagonal decomposition $\bT_+=\bQ\bT\bQ^\top$ is uniquely determined by $\bT$ and the first column of $\bQ$ (Theorem~\ref{theorem:implicit-q-tridiagonal}, p.~\pageref{theorem:implicit-q-tridiagonal}); \index{Implicit Q theorem}
\item \textit{Tridiagonal Update.} By condition (TPQR 1) in Equation~\eqref{equation:tpqr-1-2}, when the QR algorithm applied to a tridiagonal matrix, the update of the ``eigenvalue matrix" forms a tridiagonal decomposition: $\bT^{(k)} = \bQ^{(k)\top } \bT^{(k-1)} \bQ^{(k)}\rightarrow  \bT^{(k-1)}=\bQ^{(k) }\bT^{(k)}\bQ^{(k)\top } $, i.e., the tridiagonal decomposition of $\bT^{(k-1)}$ is given by $\bT^{(k-1)}=\bQ^{(k) }\bT^{(k)}\bQ^{(k)\top } $. We notice that if we assume $\bT^{(k-1)}$ is \textit{unreduced with positive lower subdiagonals}, then $\bT^{(k)}$ is also \textit{unreduced with positive lower subdiagonals}. By the implicit Q theorem above, the tridiagonal decomposition is \textbf{uniquely} determined by $\bT^{(k-1)}$ itself and the first column of $\bQ^{(k) }$.

\item If $\bT^{(k-1)}$ has positive lower subdiagonals, $\bT^{(k)}$ will also has positive lower subdiagonals which in turn result in the positive lower subdiagonals in $\bT^{(k+1)}$. However, at some point, they will converge to zero.

\item \textit{Connection to the ``simple" QR algorithm.} We notice that when the ``simple" QR algorithm without shifts (Algorithm~\ref{alg:qr-algorithm-simple2}, p.~\pageref{alg:qr-algorithm-simple2}) applied to the tridiagonal matrix also has this tridiagonal update, however, the difference relies on the first column of $\bQ^{(k) }$.
\end{itemize}

We then introduce the implicit update on the practical QR algorithm.
\paragraph{Step 1: Introducing the bulge}
To see the implicit shift algorithm, a $5\times 5$ example is given step by step. Suppose at the $(k-1)$-th iteration, the elements in $\bT^{(k-1)}$ is given by Equation~\eqref{equation:implicit-matrix-qr}. Find the $2$ by 2 Givens rotation $\widetildebG_1^\top$ where $c=\cos(\theta), s=\sin(\theta)$ are computed such that\index{Bulge}
$$
\underbrace{\begin{bmatrix}
		c & s \\
		-s & c
\end{bmatrix}}_{\widetildebG_1^\top}
\begin{bmatrix}
a_1 - \mu^{(k)}\\
b_1
\end{bmatrix}
=
\begin{bmatrix}
\boxtimes \\
0
\end{bmatrix}.
$$
And an $n$ by $n$ Givens rotation constructed by
$$
\bG_{12}^\top =
\begin{bmatrix}
\widetildebG_1^\top & \\
 & \bI_{n-2}
\end{bmatrix},
$$
where $n=5$ in our example and the subscript ``$12$" of $\bG_{12}^\top$ denotes the position where the rotation happens (Definition~\ref{definition:givens-rotation-in-qr}, p.~\pageref{definition:givens-rotation-in-qr}). Till now, this is equivalent to what we did in the first step of the QR decomposition of ($\bT^{(k-1)}-\mu^{(k)}\bI$) via the Givens rotation (Section~\ref{section:qr-givens}, p.~\pageref{section:qr-givens}).
For the $5\times 5$ example, we realize that $\bG_{12}^\top$ working on $(\bT^{(k-1)}-\textcolor{blue}{\mu^{(k)}\bI})$ will introduce a zero in entry (2,1) and destroy the zero in entry $(1,3)$. And the process is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed:
$$
\begin{aligned}
	\begin{sbmatrix}{(\bT^{(k-1)}-\textcolor{blue}{\mu^{(k)}\bI})}
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
		0 & \boxtimes & \boxtimes & \boxtimes & 0\\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}^\top\times }{\rightarrow}
	\begin{sbmatrix}{\bG_{12}^\top(\bT^{(k-1)}-\textcolor{blue}{\mu^{(k)}\bI})}
		\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}} & \bm{0} & \bm{0}\\
		\textcolor{blue}{\bm{0}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0}\\
		0 & \boxtimes & \boxtimes & \boxtimes & 0 \\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}
\end{aligned}
$$
And $\bG_{12}^\top$ working on $\bT^{(k-1)}$ will destroy the zero in entry (1,3):
$$
\begin{aligned}
	\begin{sbmatrix}{\bT^{(k-1)}}
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
		0 & \boxtimes & \boxtimes & \boxtimes & 0\\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}^\top\times }{\rightarrow}
	\begin{sbmatrix}{\bG_{12}^\top\bT^{(k-1)}}
		\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  & \bm{0} & \bm{0}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0}\\
		0 & \boxtimes & \boxtimes & \boxtimes & 0 \\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}
&\stackrel{\times\bG_{12}}{\rightarrow}
\mathop{\left[\begin{array}{ccccc}
		\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  & \bm{0} & \bm{0}\\\cline{1-1}
		\multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0}\\
		\multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes & \boxtimes & 0 \\\cline{1-1}
		\bm{0} & \bm{0} & \boxtimes & \boxtimes & \boxtimes\\
		\bm{0} & \bm{0} & 0 & \boxtimes & \boxtimes
	\end{array}\right]}_{\textstyle\mathstrut{\bG_{12}^\top\bT^{(k-1)}\bG_{12}}},
\end{aligned}
$$
where we find the Givens $\bG_{12}^\top$ multiplying on the left of a matrix will modify the first two rows of it, and $\bG_{12}$ multiplying on the right will modify the first two columns of it. The blue $\textcolor{blue}{\bm{\boxtimes}}$'s indicate the zero entries in tridiagonal matrix destroyed by the orthogonal similarity transformation and is known as ``\textit{introducing the bulge}".

\paragraph{Step 2: Chasing the bulge}
Now the problem becomes ``\textit{chasing the bulge}" that will introduce the ``bulge" back to zero. And in the meantime, as the final tridiagonal decomposition is decided by the first column. The second Givens rotation can be constructed by calculating $c=\cos(\theta), s=\sin(\theta)$ such that
$$
\underbrace{\begin{bmatrix}
		c & s \\
		-s & c
\end{bmatrix}}_{\widetildebG_2^\top}
\underbrace{(\bG_{12}^\top\bT^{(k-1)}\bG_{12})_{1:2,1}}_{\text{the vector in the \fbox{box} of above matrix}}
=
\begin{bmatrix}
	\boxtimes \\
	0
\end{bmatrix}.
$$
And a Givens rotation constructed by
$$
\bG_{23}^\top =
\begin{bmatrix}
	1 & & \\
	& \widetildebG_2^\top & \\
& 	& \bI_{n-3}
\end{bmatrix}.
$$
Following the above example, we have
$$
\begin{aligned}
	\begin{sbmatrix}{\bG_{12}^\top\bT^{(k-1)}\bG_{12}}
		\boxtimes & \boxtimes & \textcolor{blue}{\boxtimes}  & 0 & 0\\
		\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
		\textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes & \boxtimes & 0 \\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{23}^\top\times }{\rightarrow}
	\begin{sbmatrix}{\bG_{23}^\top(\bG_{12}^\top\bT^{(k-1)}\bG_{12})}
		\boxtimes & \boxtimes & \textcolor{blue}{\boxtimes}  & 0 & 0\\
		\bm{\boxtimes} & \bm{\boxtimes} &\bm{\boxtimes}  & \textcolor{blue}{\bm{\boxtimes} } & \bm{0}\\
		\textcolor{brown}{\bm{0}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} \\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}
	&\stackrel{\times\bG_{23} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
		\boxtimes & \bm{\boxtimes} & \textcolor{brown}{\bm{0}}  & 0 & 0\\
		\bm{\boxtimes} & \bm{\boxtimes} &\bm{\boxtimes}  & \textcolor{blue}{\bm{\boxtimes} } & \bm{0}\\\cline{2-2}
		\textcolor{brown}{\bm{0}} & \multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} \\
		0 & \multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes & \boxtimes\\\cline{2-2}
		0 & \bm{0} & \bm{0} & \boxtimes & \boxtimes
\end{array}\right]}_{\textstyle\mathstrut{\bG_{23}^\top(\bG_{12}^\top\bT^{(k-1)}\bG_{12})\bG_{23}}},
\end{aligned}
$$
where we find the Givens $\bG_{23}^\top$ multiplying on the left of a matrix will modify the rows $2,3$ of it, and $\bG_{23}$ multiplying on the right will modify the first columns $2,3$ of it. Now we introduce back zero for the entries (3,1) and (1,3) from step 1, however, introducing ``bulge" for entries (4,2) and (2,4) again.

Same process can go on, the example as a whole can be shown as follows where the blue $\textcolor{blue}{\bm{\boxtimes}}$ indicates the bulge introduced, the brown $\textcolor{brown}{\bm{0}}$ indicates the bulge was chased out, and \textbf{boldface} indicates the value has just been changed:
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10, frametitle={A Complete Example of Implicit QR Algorithm}]
\begin{equation}\label{equation:tridia-update-implicit}
\begin{aligned}
\begin{sbmatrix}{\bT^{(k-1)}}
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
		0 & \boxtimes & \boxtimes & \boxtimes & 0\\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
			\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  & \bm{0} & \bm{0}\\\cline{1-1}
			\multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0}\\
			\multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes & \boxtimes & 0 \\\cline{1-1}
			\bm{0} & \bm{0} & \boxtimes & \boxtimes & \boxtimes\\
			\bm{0} & \bm{0} & 0 & \boxtimes & \boxtimes
		\end{array}\right]}_{\textstyle\mathstrut{\bG_{12}^\top(\cdot)\bG_{12}}}
	\stackrel{\bG_{23} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
			\boxtimes & \bm{\boxtimes} & \textcolor{brown}{\bm{0}}  & 0 & 0\\
			\bm{\boxtimes} & \bm{\boxtimes} &\bm{\boxtimes}  & \textcolor{blue}{\bm{\boxtimes} } & \bm{0}\\\cline{2-2}
			\textcolor{brown}{\bm{0}} & \multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} \\
			0 & \multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes & \boxtimes\\\cline{2-2}
			0 & \bm{0} & \bm{0} & \boxtimes & \boxtimes
		\end{array}\right]}_{\textstyle\mathstrut{\bG_{23}^\top(\cdot)\bG_{23}}}\\
\end{aligned}
\end{equation}
$$
\begin{aligned}
\qquad \qquad \qquad  \gap &\stackrel{\bG_{34} }{\rightarrow}
\mathop{\left[\begin{array}{ccccc}
		\boxtimes & \boxtimes & \bm{0}   & \bm{0} & 0\\
		\boxtimes & \boxtimes &\bm{\boxtimes}  & \textcolor{brown}{\bm{0} }& 0\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} &\bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes} } \\\cline{3-3}
		\bm{0} & \textcolor{brown}{\bm{0} } & \multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes  \\\cline{3-3}
	\end{array}\right]}_{\textstyle\mathstrut{\bG_{34}^\top(\cdot)\bG_{34}}}
\stackrel{\bG_{45} }{\rightarrow}
\mathop{\left[\begin{array}{ccccc}
		\boxtimes & \boxtimes & 0 & \bm{0} &\bm{0}\\
		\boxtimes & \boxtimes &\boxtimes  & \bm{0} & \bm{0}\\
		0 & \boxtimes & \boxtimes & \bm{\boxtimes} & \textcolor{brown}{\bm{0} } \\
		\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \textcolor{brown}{\bm{0} } & \bm{\boxtimes} & \bm{\boxtimes}  \\
	\end{array}\right]}_{\textstyle\mathstrut{\bT^{(k)}=\bG_{45}^\top(\cdot)\bG_{45}}}.
\end{aligned}
$$
\end{mdframed}
For general matrix with shape $n\times n$, one can compute rotations $\bG_{12}, \bG_{23}, \ldots, \bG_{n-1,n}$ with the property that if $\bZ = \bG_{12}\bG_{23} \ldots\bG_{n-1,n}$, then $\bT^{(k-1)} = \bZ \bT^{(k)}\bZ^\top$ is the tridiagonal decomposition where the first column of $\bZ$ is given by $\bZ\be_1 = \bG_{12}\be_1 = \bQ^{(k)}$. We realize that the first column of $\bZ$ and $\bQ^{(k)}$ are identical and therefore $\bZ=\bQ^{(k)}$ by implicit Q theorem under the following conditions:
\begin{itemize}
\item $\bT^{(k-1)}$ is unreduced with positive lower subdiagonals;
\item QR decomposition method used in the QR algorithm is uniquely in that the diagonals of the upper triangular matrix have positive diagonals;
\end{itemize}

\paragraph{The complete algorithm}
For simplicity, we denote the construction of $\widetildebG_i$ such that $\widetildebG_i^\top \bx =\widetildebG_i^\top
\begin{bmatrix}
	x_1\\x_2
\end{bmatrix}= \begin{bmatrix}
	\boxtimes \\ 0
\end{bmatrix}$ by
$$
\widetildebG_i^\top =\text{givens}(x_1,x_2).
$$
In all iterations, $\widetildebG_i^\top $ will be of size $2\times 2$. And the construction of $n\times n$ Givens rotation $\bG_{i,i+1}^\top$ by
$$
\bG_{i,i+1}^\top=
G(\widetildebG_i^\top )=
\begin{bmatrix}
\bI_{i-1} & & \\
& \widetildebG_i^\top& \\
& & \bI_{n-i-1}
\end{bmatrix}.
$$
For further simplicity, we will denote $\bG_{i,i+1}^\top$ by $\bG_i^\top$ which multiplies on the left of another matrix will modify the $i$-th and $i+1$-th rows implicitly. The full procedure is then formulated in Algorithm~\ref{alg:qr-algorithm-practical-tridiagonal-implicit-shift} where $t_{ij}^{(k-1)}$ is the $(i,j)$-th entry of $\bT^{(k-1)}$.

\begin{algorithm}[H]
\caption{Practical QR Algorithm with Implicit Shift}
\label{alg:qr-algorithm-practical-tridiagonal-implicit-shift}
\begin{algorithmic}[1]
\Require $\bA\in \real^{n\times n}$ is real and symmetric;
\State $\bA=\bQ^{(0)}\bT^{(0)}\bQ^{(0)\top}$; \Comment{tridiagonal decomposition of $\bA$}
\For{$k=1,2,\ldots$}
	\State Pick a shift $\mu^{(k)}$; \Comment{e.g., $\mu^{(k)}=t_{nn}^{(k-1)}$}
	\State $x_1 = t_{11}-\mu^{(k)}, x_2=t_{21}$; \Comment{$t_{ij} = t^{(k-1)}_{ij}$}
	\State $\bT^{(k)} = \bT^{(k-1)}$; \Comment{initialize $\bT^{(k)} $}
	\For{$i=1:n-1$}
		\State $\widetildebG_i^\top = \text{givens}(x_1,x_2)$;
		\State $\bG_{i}^\top = G(\widetildebG_i^\top)$;
		\State $\bT^{(k)} = \bG_{i}^\top\bT^{(k)}\bG_{i}$;
		\If{$i<n-1$}
			\State $x_1=t_{i+1,i}, x_2=t_{i+2,i}$;\Comment{$t_{ij} = t^{(k-1)}_{ij}$}
		\EndIf
	\EndFor
	\State $\bQ^{(k)\top} = \bG_{n-1}^\top\ldots\bG_1^\top$; \Comment{this results in $\bT^{(k)} =\bQ^{(k)\top} \bT^{(k-1)}\bQ^{(k)} $}

\EndFor
\end{algorithmic}
\end{algorithm}
Suppose for iteration $p$, $\bT^{(p)}$ converges to a diagonal matrix (within the machine error). Then write out the updates in each iteration:
$$
\left.
\begin{aligned}
	\bT^{(p)}&=\bQ^{(p)\top}\bT^{(p-1)} \bQ^{(p)}\\
	\bT^{(p-1)}&=\bQ^{(p-1)\top}\bT^{(p-2)} \bQ^{(p-1)}\\
	\vdots &= \vdots \\
	\bT^{(1)}&=\bQ^{(1)\top}\bT^{(0)} \bQ^{(1)}\\
 \bT^{(0)}&=\bQ^{(0)\top}\bA\bQ^{(0)}
\end{aligned}
\right\}
\leadtosmall
\bA =
\underbrace{\bQ^{(0)}\ldots \bQ^{(p)}}_{\bQ}
\bT^{(p)}
\underbrace{(\bQ^{(0)}\ldots \bQ^{(p)})^\top}_{\bQ^\top}
$$
is the approximated spectral decomposition of real symmetric $\bA$ where $\bQ$ is orthogonal containing the eigenvectors of $\bA$ and $\bT^{(p)}$ is diagonal containing eigenvalues of $\bA$ (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}).

\paragraph{Decouple}
However, we can assure that the $\bT^{(k-1)}$ has nonnegative lower subdiagonals (the specific QR decomposition favored as mentioned above), whereas it happens that it is \textit{reduced}, i.e., some lower subdiagonals are 0. In this case, the eigenproblem splits into a pair of smaller problems. For example, when $\bT^{(k-1)}_{k+1,k}=0$, then, a ``practical" QR algorithm can be applied to the submatrices:
$$
\bT^{(k-1)}_{1:k,1:k}   \qquad \text{and} \qquad \bT^{(k-1)}_{k+1:n,k+1:n}.
$$
And the eigenvalues can be obtained by
$$
\Lambda\left(\bT^{(k-1)} \right) = \Lambda\left(\bT^{(k-1)}_{1:k,1:k}  \right) \cup \Lambda\left(\bT^{(k-1)}_{k+1:n,k+1:n}\right),
$$
where $\Lambda(\cdot)$ is the spectrum of a matrix (Definition~\ref{definition:spectrum}, p.~\pageref{definition:spectrum}).

\section{Jacobi's Method}\label{section:jacobi-spectral}
Jacobi's method is one of the oldest method to compute the eigenvalues of a matrix which was introduced in 1846 by \citep{jacobi1846theory}. The idea is to diagonalize a small submatrix at each time such that the full matrix can be diagonalized eventually. The mathematical measure for the quantity of the reduction is defined by \textit{off-diagonal norm}\index{Jacobi's rotation}\index{Off-diagonal norm}
$$
\off(\bA) = \sqrt{\sum_{i=1}^{n} \sum_{j=1, j\neq i}^{n} a_{ij}^2 },
$$
i.e., the Frobenius norm of the off-diagonal entries.
The idea behind the method is to iteratively reduce the off-diagonal quantity and relies on the Jacobi's rotation
$$
\begin{blockarray}{cccccccccccc}
	\begin{block}{c[cccccccccc]c}
		&1 &          &   &  &   &   & &  & & &\\
		&& \ddots  &  &  &  & && & & &\\
		&&      & 1 &  & & &  && & & \\
		&&      &  & c &  &  &  & s & & &k\\
		&&& &   & 1 & & && & &\\
		\bJ_{kl}=&&& &   &   &\ddots &  && & &\\
		&&& &  &   &  & 1&& & &\\
		&&& & -s &  &  & &c& & & l\\
		&&& & &  &  & & &1 & &\\
		&&& & &  &  & & & &\ddots &\\
	\end{block}
	&&      &  & k &  &  &  & l & & &\\
\end{blockarray},
$$
which is the same as the Givens rotation (Definition~\ref{definition:givens-rotation-in-qr}, p.~\pageref{definition:givens-rotation-in-qr}). The difference relies on the usage of them where in the Jacobi's rotation, the $\theta$ for $s = \cos \theta$ and $c=\cos \theta$ is chosen to make the submatrix of $\bJ^\top \bA\bJ$ to be diagonal.

\subsection{The 2 by 2 Case}
To see how this Jacobi's rotation works, suppose we are looking at the 2 by 2 submatrix of a symmetric matrix
$$
\bA(k,l):=
\begin{bmatrix}
	a_{kk} & a_{kl} \\
	a_{lk} & a_{ll}
\end{bmatrix}
=
\begin{bmatrix}
	a_{kk} & a_{kl} \\
	a_{kl} & a_{ll}
\end{bmatrix}.
$$
Then $\theta$ can be computed such that
$$
\begin{aligned}
\bJ^\top \bA(k,l)\bJ
&=
\begin{bmatrix}
	c & -s \\
	s & c
\end{bmatrix}
\begin{bmatrix}
	a_{kk} & a_{kl} \\
	a_{kl} & a_{ll}
\end{bmatrix}
\begin{bmatrix}
	c & s \\
	-s & c
\end{bmatrix}\\
&=
\begin{bmatrix}
	(c^2-cs) a_{kk}+s^2 a_{ll} -cs \cdot a_{kl}  & (c^2-s^2)a_{kl}+cs (a_{kk}-a_{ll}) \\
	(c^2-s^2)a_{kl} +cs (a_{kk}-a_{ll}) & s^2 a_{kk} +c^2 a_{ll}+2cs \cdot a_{kl}
\end{bmatrix}
=
\begin{bmatrix}
	\neq 0 & 0 \\
	0 & \neq 0
\end{bmatrix}.
\end{aligned}
$$
If $a_{kl}=0$, then we can just set $c=1,s=0$, the submatrix $\bA(k,l)$ remains unchanged. Otherwise, it is trivial that both $c\neq$ and $s\neq 0$. Divide the off-diagonal element $\{(c^2-s^2)a_{kl} +cs (a_{kk}-a_{ll}) \}$ by $c^2$ and $a_{kl}$, it follows that
$$
(1- \frac{s^2}{c^2} + \frac{s}{c} \frac{a_{kk}-a_{ll}}{a_{kl}}) = 0
$$
Let
$$
\tan \theta = t = \frac{\sin \theta}{\cos \theta} = \frac{s}{c}
\qquad
\text{and}
\qquad
\tau = \frac{a_{ll}-a_{kk}}{2a_{kl}},
$$
it suffices to solve the equation
$$
t^2+2\tau t -1=0.
$$
This gives
$$
t = -\tau \pm \sqrt{\tau^2+1}.
$$
We notice that
$$
||\bA - \bJ_{kl}^\top \bA\bJ_{kl}||_F^2 = 4(1-c) \sum_{i\neq k,l}(a_{ik}^2+a_{il}^2) + 2a_{kl}^2 /c^2,
$$
and
$$
c = \frac{1}{\sqrt{1+t^2}}.
$$
Therefore, the larger the $c$, the smaller the $||\bA - \bJ_{kl}^\top \bA\bJ_{kl}||_F^2$. This implies we should choose a $t$ with smaller magnitude:
$$
t_{min}=
\left\{
\begin{aligned}
&-\tau + \sqrt{\tau^2+1}, \qquad \text{if $\tau\geq 0$};\\
&-\tau - \sqrt{\tau^2+1}, \qquad \text{if $\tau< 0$}.\\
\end{aligned}
\right.
$$
From the discussion above, we can define the \textit{ComputeJacobiRotation} function that computes the Jacobi's rotation from the submatrix of $\bA$.
\begin{algorithm}[H]
	\caption{Compute Jacobi Rotation Given the Submatrix}
	\label{alg:jacobi-submatrix}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{n\times n}$ is real and symmetric, ($k,l$) such that $1\leq k< l\leq n$;
		\Function{computeJacobiRotation}{$\bA$, $k,l$}
		\If{$A_{kl}\neq 0$}
		\State $\tau = (a_{ll} - a_{kk})/(2a_{kl})$
		\If{$\tau \geq 0$}
			\State $t=-\tau + \sqrt{\tau^2+1}$;
		\Else
			\State $t=-\tau - \sqrt{\tau^2+1}$;
		\EndIf
		\State $c=1/\sqrt{1+t^2}, s=tc$;
		\Else
		\State $c=1,s=0$;
		\EndIf
		\State Output $c,s$;
		\EndFunction
	\end{algorithmic}
\end{algorithm}

\subsection{The Complete Jacobi's Method}
The name of the ``complete" comes from the terminology of the complete pivoting that searches the largest element in the matrix to pivot (Section~\ref{section:complete-pivoting}, p.~\pageref{section:complete-pivoting}). At each iteration, we need to decide the submatrix $\bA(k,l)$ to diagonalize. Then in the complete Jacobi's method, we choose $(k,l)$ such that the $(k,l)$-th entry,  $a_{kl}^2$, is maximal hoping that the reduction of the off-diagonal quantity to be maximal. The complete search for the largest magnitude defines the Algorithm~\ref{alg:jacobi-complete-alg}.
\begin{algorithm}[H]
	\caption{Complete Jacobi's Method}
	\label{alg:jacobi-complete-alg}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{n\times n}$ is real and symmetric, a positive tolerance $tol$ such that $\delta = tol \cdot ||\bA||_F$;
		\State $\bQ = \bI_n$;
		\State $i=0$;
		\State $\bLambda = \bA$;
		\While{$\off(\bLambda)> \delta$}
		\State Choose $(k,l)$ so that $a_{kl}^2 = \arg\max \bLambda_{ij}^2, \forall 1\leq i,j \leq n$;
		\State Compute $c,s$ from \textit{computeJacobiRotation}($\bLambda, k,l$);
		\State Decide the $n\times n$ Jacobi's rotation $\bJ_{kl}$ by $c,s$;
		\State $\bLambda  = \bJ_{kl}^\top \bLambda \bJ_{kl}$;
		\State $\bQ = \bQ\bJ_{kl}$;
		\State Compute $\off(\bLambda^{(i)})^2 $; \Comment{which we will show converges linearly to 0.}
		\State $i=i+1$;
		\EndWhile
		\State Output the sequence $\off(\bLambda^{(i)})^2 $,  approximated diagonal $\bLambda$ and orthogonal $\bQ$;
	\end{algorithmic}
\end{algorithm}
The algorithm computes the spectral decomposition by outputting $\bA = \bQ\bLambda\bQ^\top$.
It is not hard to see that, at each iteration, it requires $O(n^2)$ flops to search for the largest magnitude entry $(k,l)$ and $O(n)$ flops to update the iteration. We shall notice that although a symmetric pair of zeros is introduced into the matrix at each iteration,
the previous zeros are destroyed. However, what matters is the off-diagonal quantity reduces steadily.

Suppose the index $(k,l)$ is chosen and the Jacobi's rotation $\bJ_{kl}$ is constructed such that
$$
\bLambda_+ = \bJ_{kl}^\top\bLambda \bJ_{kl}.
$$
Since the Jacobi's rotation is orthogonal, it follows that
$$
||\bLambda_+||_F^2 = ||\bLambda||_F^2
$$
and
$$
\off(\bLambda_+)^2 = \off(\bLambda)^2 -2a_{kl}^2.
$$
As $a_{kl}$ has largest magnitude in $\bLambda$, we also have
$$
\off(\bLambda)^2 \leq n(n-1)a_{kl}^2.
$$
Therefore,
$$
\off(\bLambda_+)^2= \off(\bLambda)^2 -2a_{kl}^2 \leq \off(\bLambda)^2 - \frac{2}{n(n-1)} \off(\bLambda)^2 = \left(1-\frac{2}{n(n-1)}\right)\off(\bLambda)^2.
$$
By induction, this implies, the Jacobi's method reduces the off-diagonal quantity after $k$ iterations by
$$
\off(\bLambda^{(i)})^2 \leq \left(1-\frac{2}{n(n-1)}\right)^2 \off(\bLambda^{(0)}).
$$
and
$$
\mathop{\lim}_{k\rightarrow \infty}  \frac{|\off(\bLambda^{(i)})^2 - 0|}{|\off(\bLambda^{(i-1)})^2 -0|} = 1-\frac{2}{n(n-1)} \in (0,1),
$$
such that the sequence $\off(\bLambda^{(i)})^2 $ converges linearly to 0 (Definition~\ref{definition:linear-convergence}, p.~\pageref{definition:linear-convergence}). However, in practice, \citep{henrici1958speed, schonhage1964quadratic, van1966quadratic} show that a quadratic convergence (Definition~\ref{definition:quadratic-convergence}, p.~\pageref{definition:quadratic-convergence}) can be obtained such that
$$
\mathop{\lim}_{k\rightarrow \infty}  \frac{|\off(\bLambda^{(i+\frac{n(n-1)}{2})})^2|}{|\off(\bLambda^{(i)})^2|^2} = c,
$$
where $c$ is a constant. We shall not give details.

\subsection{The Cyclic-by-Row Jacobi's Method}
We mentioned in the last section that most cost of the complete Jacobi's method is from the search for the element with largest magnitude which takes $O(n^2)$ flops.
To tame the explosion, it is reasonable to update in a row-by-row fashion that iteratively updates the $n(n-1)$ upper triangular entries. This is known as the \textit{cyclic-by-row} algorithm and the procedure is shown in Algorithm~\ref{alg:jacobi-cyclicrow-alg}.
\begin{algorithm}[H]
	\caption{Cyclic-by-Row Jacobi's Method}
	\label{alg:jacobi-cyclicrow-alg}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{n\times n}$ is real and symmetric, a positive tolerance $tol$ such that $\delta = tol \cdot ||\bA||_F$;
		\State $\bQ = \bI_n$;
		\State $i=0$;
		\State $\bLambda=\bA$;
		\While{$\off(\bLambda)> \delta$}
		\For{$k=1:n-1$}
		\For{$l=k+1:n$}
		\State Compute $c,s$ from \textit{computeJacobiRotation}($\bLambda, k,l$);
		\State Decide the $n\times n$ Jacobi's rotation $\bJ_{kl}$ by $c,s$;
		\State $\bLambda  = \bJ_{kl}^\top \bLambda \bJ_{kl}$;
		\State $\bQ = \bQ\bJ_{kl}$;
		\State $\off(\bLambda^{(i)})^2 $;
		\State $i=i+1$;
		\EndFor
		\EndFor
		\EndWhile
		\State Output the sequence $\off(\bLambda^{(i)})^2 $,  approximated diagonal $\bLambda$ and orthogonal $\bQ$;
	\end{algorithmic}
\end{algorithm}
Similarly, \citep{wilkinson1962note, van1966quadratic} show that the cyclic-by-row algorithm converges \textit{quadratically}. And in that it does not need to compare the element in the matrix, this reduces the complexity of calculation.

\subsection{Other Issues}
In practice, when computing on a $p$-processor computer, it is reasonable to do the Jacobi's algorithm in a block way such that a parallelizable method can be favored. We refer the issue to \citep{bischof1986two, shroff1989convergence, golub2013matrix} and also its counterpart in SVD computation \citep{van1985block}.

\section{Computing the SVD}
\subsection{Implicit Shifted QR Algorithm}
We previously have shown that any symmetric matrix can be reduced to tridiagonal form via a sequence of Householder reflectors that are applied on the left and the right to the matrix, whilst it is a special case of the Hessenberg decomposition (Section~\ref{section:compute-tridiagonal}, p.~\pageref{section:compute-tridiagonal}). This can reduce the cost of the QR algorithm applied to computing the spectral decomposition of a matrix. This two-phase computation is not unique but has its counterpart. Since the work of Golub, Kahan, and others on the bidiagonalization in the 1960s (Theorem~\ref{theorem:Golub-Kahan-Bidiagonalization-decom}, p.~\pageref{theorem:Golub-Kahan-Bidiagonalization-decom}), an analogous two-phase approach has been standard for the SVD. For computing the SVD, the matrix is firstly reduced to bidiagonal form \footnote{The bidiagonal matrix we discuss about in this section implicitly means an upper bidiagonal matrix. We will suppress the terminology for simplicity.} and then the bidiagonal matrix is diagonalized:
$$
\begin{bmatrix}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{bmatrix}
\mathop{\longrightarrow}^{\text{phase 1} }_{\text{bidiagonalize} }
\begin{bmatrix}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	0 & \boxtimes & \boxtimes & 0 & 0\\
	0 & 0 & \boxtimes & \boxtimes & 0\\
	0 & 0 & 0 & \boxtimes & \boxtimes\\
	0 & 0 & 0 & 0 & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
	0 & 0 & 0 & 0 & 0
\end{bmatrix}
\mathop{\longrightarrow}^{\text{phase 2} }_{\text{diagonalize} }
\begin{bmatrix}
	\boxtimes & 0 & 0 & 0 & 0\\
	0 & \boxtimes & 0 & 0 & 0\\
	0 & 0 & \boxtimes & 0 & 0\\
	0 & 0 & 0 & \boxtimes & 0\\
	0 & 0 & 0 & 0 & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
	0 & 0 & 0 & 0 & 0
\end{bmatrix}.
$$
The bidiagonalization is different from the tridiagonalization in that it does not require the two orthogonal on the left and on the right to be the transpose of the same matrix, and even the shape of the two orthogonal matrices can be different. This is exactly what we want for the SVD. For the second phase of the SVD computation, we will directly go to the solution via the implicit shift QR algorithm, the development to this final procedure is similar from what we have developed for the spectral decomposition.

\paragraph{Phase 2 of SVD}
Following the fact that $\bT=\bB^\top\bB$ is tridiagonal if $\bB$ is bidiagonal (Lemma~\ref{lemma:construct-triangular-from-bidia}, p.~\pageref{lemma:construct-triangular-from-bidia}). Following the tridiagonal update in QR algorithm (Section~\ref{section:implifit-shift-qr}), suppose in the $k$-th iteration, we have the bidiagonal matrix $\bB^{(k-1)}$ and its tridiagonal companion \textcolor{blue}{$\bT^{(k-1)} = \bB^{(k-1)\top}\bB^{(k-1)}$}, for $k=1,2,\ldots$. The tridiagonal update is followed by a set of Givens rotations on the left and right iteratively as shown in Equation~\eqref{equation:tridia-update-implicit} by a $5\times 5$ example where the blue $\textcolor{blue}{\bm{\boxtimes}}$ indicates the bulge introduced and the \fbox{boxed} vector indicates how the Givens matrix constructed:
$$
\begin{aligned}
	\bT^{(k-1)}=	\begin{sbmatrix}{\bT^{(k-1)}}
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
		0 & \boxtimes & \boxtimes & \boxtimes & 0\\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
			\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  & \bm{0} & \bm{0}\\\cline{1-1}
			\multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0}\\
			\multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes & \boxtimes & 0 \\\cline{1-1}
			\bm{0} & \bm{0} & \boxtimes & \boxtimes & \boxtimes\\
			\bm{0} & \bm{0} & 0 & \boxtimes & \boxtimes
		\end{array}\right]}_{\textstyle\mathstrut{\bG_{12}^\top(\cdot)\bG_{12}}}
	\stackrel{\bG_{23} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
			\boxtimes & \bm{\boxtimes} & \textcolor{brown}{\bm{0}}  & 0 & 0\\
			\bm{\boxtimes} & \bm{\boxtimes} &\bm{\boxtimes}  & \textcolor{blue}{\bm{\boxtimes} } & \bm{0}\\\cline{2-2}
			\textcolor{brown}{\bm{0}} & \multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} \\
			0 & \multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes & \boxtimes\\\cline{2-2}
			0 & \bm{0} & \bm{0} & \boxtimes & \boxtimes
		\end{array}\right]}_{\textstyle\mathstrut{\bG_{23}^\top(\cdot)\bG_{23}}}\\
	&\stackrel{\bG_{34} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
			\boxtimes & \boxtimes & \bm{0}   & \bm{0} & 0\\
			\boxtimes & \boxtimes &\bm{\boxtimes}  & \textcolor{brown}{\bm{0} }& 0\\
			\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} &\bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes} } \\\cline{3-3}
			\bm{0} & \textcolor{brown}{\bm{0} } & \multicolumn{1}{|c|}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes}\\
			0 & 0 & \multicolumn{1}{|c|}{\textcolor{blue}{\bm{\boxtimes}}} & \bm{\boxtimes} & \boxtimes  \\\cline{3-3}
		\end{array}\right]}_{\textstyle\mathstrut{\bG_{34}^\top(\cdot)\bG_{34}}}
	\stackrel{\bG_{45} }{\rightarrow}
	\mathop{\left[\begin{array}{ccccc}
			\boxtimes & \boxtimes & 0 & \bm{0} &\bm{0}\\
			\boxtimes & \boxtimes &\boxtimes  & \bm{0} & \bm{0}\\
			0 & \boxtimes & \boxtimes & \bm{\boxtimes} & \textcolor{brown}{\bm{0} } \\
			\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
			\bm{0} & \bm{0} & \textcolor{brown}{\bm{0} } & \bm{\boxtimes} & \bm{\boxtimes}  \\
		\end{array}\right]}_{\textstyle\mathstrut{\bG_{45}^\top(\cdot)\bG_{45}}}=\bT^{(k)}.
\end{aligned}
$$
That is,
$$
\begin{aligned}
\bT^{(k)} &=
(\bG_{45}^\top\bG_{34}^\top\bG_{23}^\top\bG_{12}^\top)
\textcolor{blue}{\bT^{(k-1)}}
(\bG_{12}\bG_{23}\bG_{34}\bG_{45})\\
&=
(\bG_{45}^\top\bG_{34}^\top\bG_{23}^\top\bG_{12}^\top)
\textcolor{blue}{(\bB^{(k-1)\top}\bB^{(k-1)})}
(\bG_{12}\bG_{23}\bG_{34}\bG_{45}).
\end{aligned}
$$
If we can find $\bT^{(k)}$ can also be decomposed into a product of bidiagonal matrices, then the ``tridiagonal update" can be replaced by  a ``bidiagonal update". Then two problems must be tamed:
\begin{itemize}
\item Decompose $\bT^{(k)}$ into the bidiagonal form if $\bT^{(k-1)}$ has the bidiagonal form: $\bT^{(k-1)} = \bB^{(k-1)\top}\bB^{(k-1)}$;
\item Find the Givens rotations from the bidiagonal matrix $\bB^{(k-1)}$ rather than the tridiagonal one $\bT^{(k-1)}$.
\end{itemize}
It is not hard to see that what we want to prove is either
$$
\text{Choice 1:} \qquad \bB^{(k)\top} = (\bG_{45}^\top\bG_{34}^\top\bG_{23}^\top\bG_{12}^\top)\bB^{(k-1)\top}
$$
or
$$
\text{Choice 2:} \qquad \bB^{(k)\top} = (\bG_{45}^\top\bG_{34}^\top\bG_{23}^\top\bG_{12}^\top)\bB^{(k-1)\top}
\textcolor{blue}{\bV_{12}\bV_{23}\bV_{34}\bV_{45}}.
$$
I.e., one of the above choices is also lower bidiagonal if $ \bB^{(k-1)\top}$ is lower bidiagonal. The $\bV_{i,i+1}$'s are orthogonal such that they will cancel out in $\bB^{(k)\top} \bB^{(k)} $. We will see second form of $\bB^{(k)\top} $ will be used (to chase the bulge).
Let
$$
\bT^{(k-1)} =
\begin{bmatrix}
	a_1 & b_1 & & \ldots & 0 \\
	b_1 & a_2 & \ddots & & \vdots \\
	& \ddots & \ddots & \ddots & \\
	\vdots & & \ddots & \ddots & b_{4} \\
	0 & \ldots & & b_{4} & a_5
\end{bmatrix}=
\begin{sbmatrix}{\bB^{(k-1)\top}}
	c_1 & 0 & & \ldots & 0 \\
	d_1 & c_2 & \ddots & & \vdots \\
	& \ddots & \ddots & \ddots & \\
	\vdots & & \ddots & \ddots & 0 \\
	0 & \ldots & & d_{4} & c_5
\end{sbmatrix}
\begin{sbmatrix}{\bB^{(k-1)} }
	c_1 & d_1 & & \ldots & 0 \\
	0 & c_2 & \ddots & & \vdots \\
	& \ddots & \ddots & \ddots & \\
	\vdots & & \ddots & \ddots & d_{4} \\
	0 & \ldots & & 0 & c_5
\end{sbmatrix}.
$$

\paragraph{Introducing and chasing the bulge in the bidiagonal update}
Following the Givens rotations constructed in Section~\ref{section:implifit-shift-qr}, it follows that
a Givens rotation constructed by
$$
\bG_{12}^\top =
\begin{bmatrix}
	\widetildebG_1^\top & \\
	& \bI_{n-2}
\end{bmatrix},
\qquad
\text{where}
\qquad
\underbrace{\begin{bmatrix}
		c & s \\
		-s & c
\end{bmatrix}}_{\widetildebG_1^\top }
\begin{bmatrix}
	c_1^2 - \mu^{(k)}\\
	c_1d_1
\end{bmatrix}
=
\begin{bmatrix}
	\boxtimes \\
	0
\end{bmatrix}.
$$
where $\mu^{(k)}$ is the shift in the $k$-th iteration, $n=5$ and $a_1 = c_1^2, b_1=c_1d_1$ in our example, this will \textit{introduce a bulge} in $\bB^{(k-1)\top}$:
$$
\begin{aligned}
	\begin{sbmatrix}{\bB^{(k-1)\top}}
		\boxtimes & 0 & 0 & 0 & 0\\
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		0 & \boxtimes & \boxtimes & 0 & 0\\
		0 & 0 & \boxtimes & \boxtimes & 0\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}^\top\times }{\rightarrow}
	\begin{sbmatrix}{\bG_{12}^\top\bB^{(k-1)\top}}
		\bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  &\bm{0} & \bm{0} & \bm{0}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0}\\
		0 & \boxtimes & \boxtimes & 0 & 0 \\
		0 & 0 & \boxtimes & \boxtimes & 0\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}.
\end{aligned}
$$
Then, we observe that $\bG_{23}$ rotating on the left will not help chase out the bulge \footnote{$\bG_{23}$ working on the left of $\bG_{12}\bB^{(k-1)}$ will modify the rows 2 and 3 of it which will not introduce zero back in the first row.}, a right Givens rotations must be constructed  such that
$$
\bV_{12}^\top =
\begin{bmatrix}
	\widetildebV_1^\top & \\
	& \bI_{n-2}
\end{bmatrix},
\qquad
\text{where}
\qquad
\underbrace{\begin{bmatrix}
		c & s \\
		-s & c
\end{bmatrix}}_{\widetildebV_1^\top}
\begin{bmatrix}
	\Big(\bG_{12}^\top\bB^{(k-1)\top}\Big)_{11}\\
\Big(\bG_{12}^\top\bB^{(k-1)\top}\Big)_{12}
\end{bmatrix}
=
\begin{bmatrix}
	\boxtimes \\
	0
\end{bmatrix}.
$$
This results in
$$
\begin{aligned}
	\begin{sbmatrix}{\bB^{(k-1)\top}}
		\boxtimes & 0 & 0 & 0 & 0\\
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		0 & \boxtimes & \boxtimes & 0 & 0\\
		0 & 0 & \boxtimes & \boxtimes & 0\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}^\top\times }{\rightarrow}
	\begin{sbmatrix}{\bG_{12}^\top\bB^{(k-1)\top}}
		\bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  &\bm{0} & \bm{0} & \bm{0}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0}\\
		0 & \boxtimes & \boxtimes & 0 & 0 \\
		0 & 0 & \boxtimes & \boxtimes & 0\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}
&\stackrel{\times\bV_{12} }{\rightarrow}
\begin{sbmatrix}{\bG_{12}^\top\bB^{(k-1)\top} \bV_{12}}
	\bm{\boxtimes} & \textcolor{brown}{\bm{0}}  &\bm{0} & \bm{0} & \bm{0}\\
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0}\\
	\textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \boxtimes & 0 & 0 \\
	\bm{0} & \bm{0} & \boxtimes & \boxtimes & 0\\
	\bm{0} & \bm{0}& 0 & \boxtimes & \boxtimes
\end{sbmatrix}.
\end{aligned}
$$
As long as the $\bG_{12}^\top$ is constructed in the same way as it in the ``tridiagonal update", then
$\bG_{12}^\top \bT^{{k-1}}\bG_{12}$
is equal to
$\bG_{12}^\top \bB^{{k-1}\top}\underbrace{\bV_{12}\bV_{12}^\top}_{\bI} \bB^{(k-1)}\bG_{12}$.

\paragraph{Step 2} The step 2 is different to what we have did in the ``tridiagonal update". In the ``tridiagonal update", we construct a Givens matrix $\bG_{23}$ by chasing the bulge in $\bT^{(k-1)}$. Apply the implicit Q theorem again (Theorem~\ref{theorem:implicit-q-tridiagonal}, p.~\pageref{theorem:implicit-q-tridiagonal}), we can decide the Givens rotation by $\bB^{(k-1)}$ directly (\textit{need not to construct $\bT^{(k-1)}=\bB^{(k-1)\top}\bB^{(k-1)}$ explicitly}), and now denoted by $\bU_{23}$:
$$
\bU_{23}^\top =
\begin{bmatrix}
	1 & & \\
	& \widetildebU_2^\top & \\
	& & \bI_{n-3}
\end{bmatrix},
\qquad
\text{where}
\qquad
\underbrace{\begin{bmatrix}
		c & s \\
		-s & c
\end{bmatrix}}_{\widetildebU_2^\top }
\begin{bmatrix}
	\Big(\bG_{12}^\top\bB^{(k-1)\top} \bV_{12}\Big)_{21}\\
	\Big(\bG_{12}^\top\bB^{(k-1)\top} \bV_{12}\Big)_{31}
\end{bmatrix}
=
\begin{bmatrix}
	\boxtimes \\
	0
\end{bmatrix}.
$$

The process can go on, and the full example for chasing the bulge can be shown as follows where the blue $\textcolor{blue}{\bm{\boxtimes}}$ indicates the bulge introduced, the brown $\textcolor{brown}{\bm{0}}$ indicates the bulge was chased out, and \textbf{boldface} indicates the value has just been changed:
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10, frametitle={A Complete Example of Implicit QR Algorithm for SVD}]
$$
\begin{aligned}
	\begin{sbmatrix}{\bB^{(k-1)\top}}
		\boxtimes & 0 & 0 & 0 & 0\\
		\boxtimes & \boxtimes & 0 & 0 & 0\\
		0 & \boxtimes & \boxtimes & 0 & 0\\
		0 & 0 & \boxtimes & \boxtimes & 0\\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}^\top\times }{\rightarrow}
	\begin{sbmatrix}{\bG_{12}^\top\bB^{(k-1)\top}}
		\bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  &\bm{0} & \bm{0} & \bm{0}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0}\\
		0 & \boxtimes & \boxtimes & 0 & 0 \\
		0 & 0 & \boxtimes & \boxtimes & 0\\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}
	&\stackrel{\times\bV_{12} }{\rightarrow}
	\begin{sbmatrix}{\bG_{12}^\top\bB^{(k-1)\top} \bV_{12}}
		\bm{\boxtimes} & \textcolor{brown}{\bm{0}}  &\bm{0} & \bm{0} & \bm{0}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0}\\
		\textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \boxtimes & 0 & 0 \\
		\bm{0} & \bm{0} & \boxtimes & \boxtimes & 0\\
		\bm{0} & \bm{0}& 0 & \boxtimes & \boxtimes
	\end{sbmatrix}\\
&\stackrel{\bU_{23}^\top \times}{\rightarrow}
\begin{sbmatrix}{\bU_{23}^\top (\cdot) }
	\boxtimes & 0  & 0 & 0 & 0\\
	\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  & \bm{0} & \bm{0}\\
	\textcolor{brown}{\bm{0}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} \\
	0 & 0 & \boxtimes & \boxtimes & 0\\
	0 & 0& 0 & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\times\bV_{23} }{\rightarrow}
\begin{sbmatrix}{\bU_{23}^\top(\cdot ) \bV_{23}}
	\boxtimes & \bm{0}  & \bm{0} & 0 & 0\\
	\bm{\boxtimes} & \bm{\boxtimes} & \textcolor{brown}{\bm{0}}  & \bm{0} & \bm{0}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0} & \bm{0} \\
	0 & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \boxtimes & 0\\
	0 &  \bm{0}&  \bm{0}& \boxtimes & \boxtimes
\end{sbmatrix}\\
\end{aligned}
$$
$$
\begin{aligned}
\qquad \qquad \qquad\qquad\,\,\,\,\,&\stackrel{\bU_{34}^\top \times}{\rightarrow}
\begin{sbmatrix}{\bU_{34}^\top(\cdot)}
	\boxtimes & 0  & 0 & 0 & 0\\
	\boxtimes & \boxtimes & 0  & 0 & 0\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}  & \bm{0} \\
	\bm{0} & \textcolor{brown}{\bm{0}} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0}\\
	0 & 0&  0& \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\times\bV_{34} }{\rightarrow}
\begin{sbmatrix}{\bU_{34}^\top(\cdot )\bV_{34}}
	\boxtimes & 0  & \bm{0} & \bm{0} & 0\\
	\boxtimes & \boxtimes & \bm{0}  & \bm{0} & 0\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \textcolor{brown}{\bm{0}} & \bm{0} \\
	\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{0}\\
	0 & 0&  \textcolor{blue}{\bm{\boxtimes}}& \bm{\boxtimes}  & \boxtimes
\end{sbmatrix}\\
&\stackrel{\bU_{45}^\top \times}{\rightarrow}
\begin{sbmatrix}{\bU_{45}^\top(\cdot)}
	\boxtimes & 0  & 0 & 0& 0\\
	\boxtimes & \boxtimes & 0  & 0 & 0\\
	0 & \boxtimes & \boxtimes& 0 & 0\\
	\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \textcolor{blue}{\bm{\boxtimes}}\\
	\bm{0} & \bm{0} &  \textcolor{brown}{\bm{0}} & \bm{\boxtimes}  & \bm{\boxtimes}
\end{sbmatrix}
&\stackrel{\times\bV_{45} }{\rightarrow}
\begin{sbmatrix}{\bU_{45}^\top(\cdot) \bV_{45}}
	\boxtimes & 0  & 0  & \bm{0}& \bm{0}\\
	\boxtimes & \boxtimes & 0 & \bm{0}& \bm{0}\\
	0 & \boxtimes & \boxtimes& \bm{0}& \bm{0}\\
	\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \textcolor{brown}{\bm{0}}\\
	\bm{0} & \bm{0} &  \bm{0} & \bm{\boxtimes}  & \bm{\boxtimes}
\end{sbmatrix},
\end{aligned}
$$
where $\bG_{12}$ is used in the first step to indicate the same construction as ``tridiagonal update", and $\bU_{i,i+1}$'s are used to indicate the different construction via the implicit Q theorem.

For clarity, we will denote $\bG_{12}$ by $\bU_{12}$ and $\widetildebG_1$ by $\widetildebU_1$ in the following algorithm.
\end{mdframed}

\paragraph{The complete algorithm}
Again, for simplicity, we denote the construction of $\widetildebU_i$ such that $\widetildebU_i^\top \bx =\widetildebU_i^\top
\begin{bmatrix}
	x_1\\x_2
\end{bmatrix}= \begin{bmatrix}
	\boxtimes \\ 0
\end{bmatrix}$ by
$$
\widetildebU_i^\top =\text{givens}(x_1,x_2).
$$
In all iterations, $\widetildebU_i^\top $ will be of size $2\times 2$. And the construction of the $n\times n$ Givens rotation $\bG_{i,i+1}^\top$ by
$$
\bU_{i,i+1}^\top=
G(\widetildebU_i^\top )=
\begin{bmatrix}
	\bI_{i-1} & & \\
	& \widetildebU_i^\top& \\
	& & \bI_{n-i-1}
\end{bmatrix}.
$$
For further simplicity, we will denote $\bU_{i,i+1}^\top$ by $\bU_i^\top$ with multiplying on the left of another matrix will modify the $i$-th and $i+1$-th rows implicitly. The procedure is then shown in Algorithm~\ref{alg:qr-svd-implifit} where the shift $\mu^{(k)} = t_{nn}^{(k-1)}$ (i.e., the last diagonal of $\bT^{(k-1)}$) can be obtained via the last diagonal of the multiplication of $\bB^{(k-1)\top}\bB^{(k-1)}$, i.e., $\mu^{(k)} =b_{n,n-1}^2+b_{nn}^2$.

\begin{algorithm}[H]
	\caption{Golub-Kahan SVD: Practical QR Algorithm with Implicit Shift (Compare to Algorithm~\ref{alg:qr-algorithm-practical-tridiagonal-implicit-shift})}
	\label{alg:qr-svd-implifit}
\begin{algorithmic}[1]
\Require $\bA\in \real^{n\times n}$ is real and symmetric;
\State \textcolor{blue}{$\bA^\top=\bV^{(0)}\bB^{(0)}\bQ^{(0)\top}$}; \Comment{bidiagonal decomposition of $\bA^\top$}
\For{$k=1,2,\ldots$}
\State Pick a shift $\mu^{(k)}$; \Comment{e.g., $\mu^{(k)}=t_{nn}^{(k-1)}=b_{n,n-1}^2+b_{nn}^2$}
\State $x_1 = t_{11}-\mu^{(k)}, x_2=t_{21}$; \Comment{$t_{ij} = t^{(k-1)}_{ij}$}
\State $\bB^{(k)} = \bB^{(k-1)}$; \Comment{initialize $\bB^{(k)} $}
\For{$i=1:n-1$}
\State $\bU_{i}^\top = G(\widetildebU_i^\top)$, where $\widetildebU_i^\top = \text{givens}(x_1,x_2)$;
\State $\bB^{(k)\top} = \bU_{i}^\top\bB^{(k)\top}$; \Comment{left update}
\State $x_1 = b_{ii}, x_2=b_{i,i+1}$; \Comment{$b_{ij} = b^{(k)\top}_{ij}$}
\State $\bV_{i}\textcolor{blue}{^\top} = G(\widetildebV_i^\top)$ where $\widetildebV_i\textcolor{blue}{^\top} = \text{givens}(x_1,x_2) \leadtosmall \bV_i = (\bV_i^\top)^\top$;
\State $\bB^{(k)\top} = \bU_{i}^\top\bB^{(k)\top}\bV_i$; \Comment{right update}
\If{$i<n-1$}
\State $x_1=b_{i+1,i}, x_2=b_{i+2,i}$; \Comment{$b_{ij} = b^{(k)\top}_{ij}$}
\EndIf
\EndFor
\State $\bU^{(k)\top} = \bU_{n-1}^\top\ldots \bU_1^\top$;
\State  $\bV^{(k)\top} = \bV_{1}\ldots\bU_{n-1}$; \Comment{this results in $\bB^{(k)^\top} =\bU^{(k)\top} \bB^{(k-1)^\top}\bV^{(k)} $}
\EndFor
\end{algorithmic}
\end{algorithm}

Again, suppose for iteration $p$, $\bB^{(p)}$ converges to a diagonal matrix (within the machine error). Then write out the updates in each iteration:
$$
\left.
\begin{aligned}
	\bB^{(p)\top}&=\bU^{(p)\top}\bB^{(p-1)\top} \bV^{(p)}\\
	\bB^{(p-1)\top}&=\bU^{(p-1)\top}\bB^{(p-2)\top} \bV^{(p-1)}\\
	\vdots &= \vdots \\
	\bB^{(1)\top}&=\bU^{(1)\top}\bB^{(0)\top} \bV^{(1)}\\
	\bB^{(0)\top}&=\bU^{(0)\top}\bA\bV^{(0)}
\end{aligned}
\right\}
\longleftrightarrow
\left\{
\begin{aligned}
	\bT^{(p)}&=\bB^{(p)\top}\bB^{(p)}&=&\bU^{(p)\top}\bT^{(p-1)} \bU^{(p)}\\
	\bT^{(p-1)}&=\bB^{(p-1)\top}\bB^{(p-1)}&=&\bU^{(p-1)\top}\bT^{(p-2)} \bU^{(p-1)}\\
	\vdots &= \vdots \\
	\bT^{(1)}&=\bB^{(1)\top}\bB^{(1)}&=&\bU^{(1)\top}\bT^{(0)} \bU^{(1)}\\
	\bT^{(0)}&=\bB^{(0)\top}\bB^{(0)}&=&\bU^{(0)\top}\textcolor{blue}{\bA\bA^\top} \bU^{(0)}
\end{aligned}
\right.
$$
This yields
$$
\bB^{(p)\top} \approx \bB^{(p)} =
\underbrace{\bU^{(p)\top} \ldots \bU^{(0)\top}}_{\bU^\top}
 \bA
 \underbrace{\bV^{(0)}\bV^{(p)}}_{\bV}
 \leadto
 \left\{
 \begin{aligned}
 	\bA &=\bU \bB^{(p)}\bV^\top\\
 	\bA\bA^\top &= \bU \bB^{(p)}\bB^{(p)}\bU^\top
 \end{aligned}
\right.
$$
which approximates the SVD of $\bA$.

\subsection{Jacobi's SVD Method}
We have discussed the Jacobi's method for computing the spectral decomposition of a matrix where a pair of orthogonal matrices are applied on the left and right iteratively to reduce the off-diagonal quantity. The orthogonal matrices applied on the left and right are equal (but transposed: $\bQ$ and $\bQ^\top$). It is then straightforward to apply the Jacobi's method by two different sequences of orthogonal matrices hoping the off-diagonal quantity can also be reduced. The problem remains
$$
\begin{aligned}
	\bJ_1^\top \bA(k,l)\bJ_2
	&=
	\begin{bmatrix}
		c_1 & -s_1 \\
		s_1 & c_1
	\end{bmatrix}
	\begin{bmatrix}
		a_{kk} & a_{kl} \\
		a_{kl} & a_{ll}
	\end{bmatrix}
	\begin{bmatrix}
		c_2 & s_2 \\
		-s_2 & c_2
	\end{bmatrix}
	=
	\begin{bmatrix}
		\neq 0 & 0 \\
		0 & \neq 0
	\end{bmatrix}.
\end{aligned}
$$
Such problem can be decomposed into two parts:
\begin{itemize}
\item Find a Jacobi's rotation such that $\widetilde{\bJ}_1\bA(k,l)$ is symmetric;
\item Apply the Jacobi's method on the symmetric $\widetilde{\bJ}_1\bA(k,l)$: $\bJ_2^{\top}(\widetilde{\bJ}_1\bA(k,l)) \bJ_2$ is diagonal;
\end{itemize}
Therefore, let $\bJ_1^\top = \bJ_2^\top\widetilde{\bJ}_1 $, we diagonalize the submatrix inside $\bA$. We shall not discuss the details.

\section{Proof of Results}\label{section:proofs-qralgorithms}
\begin{proof}[of Lemma~\ref{lemma:qr-algo-from-power}]
	It is trivial the iteration $0$ of the three sequences are equal by same initialization. We will show that if \{$\widehat{\bA}^{(k-1)} = \bA^{(k-1)};
	\widehat{\bR}^{(k-1)} = \bR^{(k-1)};
	\widehat{\bV}^{(k-1)} = \bV^{(k-1)}$\}, if this is also true for the $k$-th iteration that
	\{$\widehat{\bA}^{(k)} = \bA^{(k)};
	\widehat{\bR}^{(k)} = \bR^{(k)};
	\widehat{\bV}^{(k)} = \bV^{(k)}$\}. Then we complete the proof by induction.

By Algorithm~\ref{alg:qr-algorithm-simple1},
$$
\left\{
\begin{aligned}
	\widehat{\bV}^{(k)}, \widehat{\bR}^{(k)}= QR(\bA\widehat{\bV}^{(k-1)}) &\leadto
	\widehat{\bV}^{(k)}\widehat{\bR}^{(k)} = \bA\widehat{\bV}^{(k-1)}. \\
	\widehat{\bA}^{(k-1)}=\widehat{\bV}^{(k-1)\top} \underbrace{\bA \widehat{\bV}^{(k-1)}}_{\widehat{\bV}^{(k)}\widehat{\bR}^{(k)}} &\leadto
	\widehat{\bA}^{(k-1)}= \underbrace{\widehat{\bV}^{(k-1)\top} \widehat{\bV}^{(k)}}_{\text{orthogonal}}\widehat{\bR}^{(k)}.
\end{aligned}
\right.
$$
	Since $\widehat{\bV}^{(k-1)\top} \widehat{\bV}^{(k)}$ is a multiplication of two orthogonal matrices, it is an orthogonal matrix as well. Therefore, \textcolor{blue}{$\widehat{\bA}^{(k-1)}= \widehat{\bV}^{(k-1)\top} \widehat{\bV}^{(k)}\widehat{\bR}^{(k)}$} is the QR decomposition of $\widehat{\bA}^{(k-1)}$.

	By Algorithm~\ref{alg:qr-algorithm-simple2},
	$$
	\bQ^{(k)}, \bR^{(k)}= QR(\bA^{(k-1)}) \leadto  \bA^{(k-1)}=\bQ^{(k)} \bR^{(k)}.
	$$
	Therefore, a second QR decomposition of $\bA^{(k-1)}$ is given by \textcolor{blue}{$\bA^{(k-1)}=\bQ^{(k)} \bR^{(k)}$}. Although, the QR decomposition is not unique, if we take the diagonal of the upper triangular matrix to be nonnegative, the QR decomposition is unique (Corollary~\ref{corollary:unique-qr}, p.~\pageref{corollary:unique-qr}). Provided $\widehat{\bA}^{(k-1)} = \bA^{(k-1)}$,  this shows
	$$
	\widehat{\bR}^{(k)} = \bR^{(k)}
	$$
	and
	$$
	\begin{aligned}
		\bQ^{(k)} = \widehat{\bV}^{(k-1)\top} \widehat{\bV}^{(k)} \leadto \widehat{\bV}^{(k-1)}\bQ^{(k)} &=  \widehat{\bV}^{(k)} \\
		\underbrace{\bV^{(k-1)}\bQ^{(k)}}_{\bV^{(k)}} &=  \widehat{\bV}^{(k)} .
	\end{aligned}
	$$
	Therefore, it follows that
	$$
	\widehat{\bV}^{(k)} = \bV^{(k)}.
	$$
	To see $\widehat{\bA}^{(k)} = \bA^{(k)} $, it follows that
	$$
	\begin{aligned}
		\widehat{\bA}^{(k)} &=\widehat{\bV}^{(k)\top} \bA \widehat{\bV}^{(k)}  &\text{(By Algorithm~\ref{alg:qr-algorithm-simple1})}\\
		&=\bV^{(k)\top} \bA \widehat{\bV}^{(k)} &\text{($\widehat{\bV}^{(k)} = \bV^{(k)}$)} \\
		&=(\bV^{(k-1)}\bQ^{(k)} )^\top \bA (\bV^{(k-1)}\bQ^{(k)} ) &\text{(By Algorithm~\ref{alg:qr-algorithm-simple2})}\\
		&= \bQ^{(k)\top} \widehat{\bA}^{(k-1)} \bQ^{(k)}=\bQ^{(k)\top} \bA^{(k-1)} \bQ^{(k)}  &\text{(By Algorithm~\ref{alg:qr-algorithm-simple1})}\\
		&=\bA^{(k)}.    &\text{(By Equation~\eqref{equation:simple-qr-find})}
	\end{aligned}
	$$
	This completes the proof.
\end{proof}

\begin{proof}[of Equation~\eqref{equation:simple-qr-find2}]
The first equality of the LHS of Equation~\eqref{equation:simple-qr-find2} comes from the step 7 of Algorithm~\ref{alg:qr-algorithm-simple2}.
The second equation is from the fact of Algorithm~\ref{alg:qr-algorithm-simple1} that
$$
\left.
\begin{aligned}
	\bA &= \widehat{\bV}^{(k)} \widehat{\bR}^{(k)} \widehat{\bV}^{(k-1)}
	=\bV^{(k)} \bR^{(k)} \bV^{(k-1)} \\
	&= \widehat{\bV}^{(k-1)} \widehat{\bR}^{(k-1)} \widehat{\bV}^{(k-2)}
	=\bV^{(k-1)} \bR^{(k-1)} \bV^{(k-2)} \\
	& \ldots
\end{aligned}
\right\} \leadtosmall
\bA^{k}=\bV^{(k)} \bR^{(k)} \bR^{(k-1)} \ldots \bR^{(0)} .
$$
This completes the proof.
\end{proof}

\begin{proof}[of Equation~\eqref{equation:practical-qr-find2}]
	For $k=1$, it is trivial to see $(\bA-\mu^{(1)}\bI) = \bQ^{(1)}\bR^{(1)} = \bQ^{(0)}\bQ^{(1)}\bR^{(1)} \bR^{(0)}$. Suppose it is true for $k-1$ that
	$$
	(\bA-\mu^{(k-1)}\bI)(\bA-\mu^{(k-2)}\bI)\ldots (\bA-\mu^{(1)}\bI)= \underbrace{\bQ^{(0)}\bQ^{(1)}\bQ^{(2)}\ldots \bQ^{(k-1)}}_{=\bV^{(k-1)}, \text{orthogonal}}
	\underbrace{\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}}_{=\bU^{(k-1)}, \text{upper triangular}}.
	$$
	If we can prove it is also true for $k$, then we complete the proof by induction. To see this, we have
	$$
	\begin{aligned}
		&\gap (\bA-\mu^{(k)}\bI)(\bA-\mu^{(k-1)}\bI)\ldots (\bA-\mu^{(1)}\bI)\\
		&=(\bA-\mu^{(k)}\bI)\bV^{(k-1)}\left(\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}\right)\\
		&=\left(\underbrace{\bV^{(k) } \bA^{(k)} \bV^{(k)\top}}_{\text{Equation}~\eqref{equation:practical-qr-find-root}}    -\mu^{(k)}\bI\right) \bV^{(k-1)}\left(\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}\right)\\
		&=\left(\bV^{(k) } \bA^{(k)} \bV^{(k)\top} -\mu^{(k)}  \underbrace{\bV^{(k)}\bV^{(k)\top} }_{\bI}  \right) \bV^{(k-1)}\left(\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}\right)\\
		&=\bV^{(k) }\left( \bA^{(k)} -\mu^{(k)}  \bI   \right)\bV^{(k)\top}  \bV^{(k-1)}\left(\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}\right)\\
		&=\bV^{(k)}\left( \underbrace{\bR^{(k)}\bQ^{(k)}  }_{\text{step 7}}\right)
		\underbrace{( \bV^{(k-1)}\bQ^{(k)} )^\top}_{\text{step 8}}
		\bV^{(k-1)}\left(\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}\right)\\
		&=\bV^{(k)}\left(\bR^{(k)}\bR^{(k-1)} \bR^{(k-2)} \ldots \bR^{(0)}\right).
	\end{aligned}
	$$
	This completes the proof.
\end{proof}

\part{Eigenvalue Problem}

\newpage
\chapter{Eigenvalue and Jordan Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Eigenvalue Decomposition}
\begin{theoremHigh}[Eigenvalue Decomposition]\label{theorem:eigenvalue-decomposition}
Any square matrix $\bA\in \real^{n\times n}$ with linearly independent eigenvectors can be factored as
$$
\bA = \bX\bLambda\bX^{-1},
$$
where $\bX$ contains the eigenvectors of $\bA$ as columns, and $\bLambda$ is a diagonal matrix $diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$ and $\lambda_1, \lambda_2, \ldots, \lambda_n$ are eigenvalues of $\bA$.
\end{theoremHigh}

Eigenvalue decomposition is also known as to diagonalize the matrix $\bA$. When no eigenvalues of $\bA$ are repeated, the eigenvectors are sure to be linearly independent. Then $\bA$ can be diagonalized. Note here without $n$ linearly independent eigenvectors, we cannot diagonalize. In Section~\ref{section:otherform-spectral} (p.~\pageref{section:otherform-spectral}), we will further discuss conditions under which the matrix has linearly independent eigenvectors.

\section{Existence of the Eigenvalue Decomposition}
\begin{proof}[of Theorem~\ref{theorem:eigenvalue-decomposition}]
Let $\bX=[\bx_1, \bx_2, \ldots, \bx_n]$ as the linearly independent eigenvectors of $\bA$. Clearly, we have
$$
\bA\bx_1=\lambda_1\bx_1,\qquad \bA\bx_2=\lambda_2\bx_2, \qquad \ldots, \qquad\bA\bx_n=\lambda_n\bx_n.
$$
In the matrix form,
$$
\bA\bX = [\bA\bx_1, \bA\bx_2, \ldots, \bA\bx_n] = [\lambda_1\bx_1, \lambda_2\bx_2, \ldots, \lambda_n\bx_n] = \bX\bLambda.
$$
Since we assume the eigenvectors are linearly independent, then $\bX$ has full rank and is invertible. We obtain
$$
\bA = \bX\bLambda \bX^{-1}.
$$
This completes the proof.
\end{proof}

We will discuss some similar forms of eigenvalue decomposition in the spectral decomposition section, where the matrix $\bA$ is required to be symmetric, and the $\bX$ is not only nonsingular but also orthogonal. Or, the matrix $\bA$ is required to be a \textit{simple matrix}, that is, the algebraic multiplicity and geometric multiplicity are the same for $\bA$, and $\bX$ will be a trivial nonsingular matrix that may not contain the eigenvectors of $\bA$. The decomposition also has a geometric meaning, which we will discuss in Section~\ref{section:coordinate-transformation} (p.~\pageref{section:coordinate-transformation}).

A matrix decomposition in the form of $\bA =\bX\bLambda\bX^{-1}$ has a nice property that we can compute the $m$-th power efficiently.

\begin{remark}[$m$-th Power]\label{remark:power-eigenvalue-decom}
The $m$-th power of $\bA$ is $\bA^m = \bX\bLambda^m\bX^{-1}$ if the matrix $\bA$ can be factored as $\bA=\bX\bLambda\bX^{-1}$.
\end{remark}

We notice that we require $\bA$ have linearly independent eigenvectors to prove the existence of the eigenvalue decomposition. Under specific conditions, the requirement is intrinsically satisfied.
\begin{lemma}[Different Eigenvalues]\label{lemma:diff-eigenvec-decompo}
Suppose the eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_n$ of $\bA\in \real^{n\times n}$ are all different. Then the corresponding eigenvectors are automatically independent. In another word, any square matrix with different eigenvalues can be diagonalized.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:diff-eigenvec-decompo}]
Suppose the eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_n$ are all different, and the eigenvectors $\bx_1,\bx_2, \ldots, \bx_n$ are dependent. That is, there exists a nonzero vector $\bc = [c_1,c_2,\ldots,c_{n-1}]^\top$ such that
$$
\bx_n = \sum_{i=1}^{n-1} c_i\bx_{i}.
$$
Then we have
$$
\begin{aligned}
\bA \bx_n &= \bA (\sum_{i=1}^{n-1} c_i\bx_{i}) \\
&=c_1\lambda_1 \bx_1 + c_2\lambda_2 \bx_2 + \ldots + c_{n-1}\lambda_{n-1}\bx_{n-1}.
\end{aligned}
$$
and
$$
\begin{aligned}
	\bA \bx_n &= \lambda_n\bx_n\\
	&=\lambda_n (c_1\bx_1 +c_2\bx_2+\ldots +c_{n-1} \bx_{n-1}).
\end{aligned}
$$
Combine above two equations, we have
$$
\sum_{i=1}^{n-1} (\lambda_n - \lambda_i)c_i \bx_i = \bzero .
$$
This leads to a contradiction since $\lambda_n \neq \lambda_i$ for all $i\in \{1,2,\ldots,n-1\}$, from which the result follows.
\end{proof}

\begin{remark}[Limitation of Eigenvalue Decomposition]
The limitation of eigenvalue decomposition is that:

$\bullet$ The eigenvectors in $\bX$ are usually not orthogonal and there are not always enough eigenvectors (i.e., some eigenvalues are equal).

$\bullet$ To compute the eigenvalues and eigenvectors $\bA\bx = \lambda\bx$ requires $\bA$ to be square. Rectangular matrices cannot be diagonalized by eigenvalue decomposition.
\end{remark}

\section{Computing the Eigenvalue Decomposition}
The computation of eigenvalues and eigenvectors involves the computation of a polynomial. Some algorithms like Rayleigh quotient iterations can solve this problem and we will further discuss algorithms to find the eigenvalues and eigenvectors of a matrix in Section~\ref{section:eigenvalue-problem} (p.~\pageref{section:eigenvalue-problem}).

As long as we have the eigenvalues and eigenvectors, we just need to compute the inverse of the nonsingular matrix $\bX$. As shown in Theorem~\ref{theorem:inverse-by-lu2} (p.~\pageref{theorem:inverse-by-lu2}), the complexity is $2n^3$ flops.

\section{Jordan Decomposition}
In eigenvalue decomposition, we suppose matrix $\bA$ has $n$ linearly independent eigenvectors. However, this is not necessarily true for all square matrices. We introduce further a generalized version of eigenvalue decomposition which is called the Jordan decomposition named after Camille Jordan \citep{jordan1870traite}.

We first introduce the definitin of Jordan blocks and Jordan form for the further description of Jordan decomposition.
\begin{definition}[Jordan Block]
	An $m\times m$ upper triangular matrix $B(\lambda, m)$ is called a Jordan block provided all $m$ diagonal elements are the same eigenvalue $\lambda$ and all upper sub-digonal elements are all ones:
	$$
	B(\lambda, m)=
	\begin{bmatrix}
		\lambda & 1        & 0 & \ldots & 0 & 0 & 0 \\
		0       & \lambda  & 1 & \ldots & 0 & 0 & 0 \\
		0       & 0  & \lambda & \ldots & 0 & 0 & 0 \\
		\vdots  & \vdots   & \vdots & \vdots & \vdots & \vdots\\
		0 & 0 & 0 & \ldots & \lambda & 1& 0\\
		0 & 0 & 0 & \ldots & 0 &\lambda & 1\\
		0 & 0 & 0 & \ldots & 0 & 0 & \lambda
	\end{bmatrix}_{m\times m}
	$$
\end{definition}
\begin{definition}[Jordan Form\index{Jordan block}]
Given an $n\times n$ matrix $\bA$, a Jordan form $\bJ$ for $\bA$ is a block diagonal matrix defined as
$$
\bJ=diag(B(\lambda_1, m_1), B(\lambda_2, m_2), \ldots B(\lambda_k, m_k))
$$
where $\lambda_1, \lambda_2, \ldots, \lambda_k$ are eigenvalues of $\bA$ (duplicates possible) and $m_1+m_2+\ldots+m_k=n$.
\end{definition}

Then, the Jordan decomposition follows:
\begin{theoremHigh}[Jordan Decomposition]\label{theorem:jordan-decomposition}
Any square matrix $\bA\in \real^{n\times n}$ can be factored as
$$
\bA = \bX\bJ\bX^{-1},
$$
where $\bX$ is a nonsingular matrix containing the generalized eigenvectors of $\bA$ as columns, and $\bJ$ is a Jordan form matrix $diag(\bJ_1, \bJ_2, \ldots, \bJ_k)$ where
$$
\bJ_i =
\begin{bmatrix}
	\lambda_i & 1        & 0 & \ldots & 0 & 0 & 0 \\
	0       & \lambda_i  & 1 & \ldots & 0 & 0 & 0 \\
	0       & 0  & \lambda_i & \ldots & 0 & 0 & 0 \\
	\vdots  & \vdots   & \vdots & \vdots & \vdots & \vdots\\
	0 & 0 & 0 & \ldots & \lambda_i & 1& 0\\
	0 & 0 & 0 & \ldots & 0 &\lambda_i & 1\\
	0 & 0 & 0 & \ldots & 0 & 0 & \lambda_i
\end{bmatrix}_{m_i\times m_i}
$$
is an $m_i\times m_i$ square matrix with $m_i$ being the number of repetitions of eigenvalue $\lambda_i$ and $m_1+m_2+\ldots +m_k = n$. $\bJ_i$'s are referred to as Jordan blocks.

Further, nonsingular matrix $\bX$ is called the \textbf{matrix of generalized eigenvectors} of $\bA$.
\end{theoremHigh}

As an example, a Jordan form can have the following structure:
$$
\begin{aligned}
	\bJ&=diag(B(\lambda_1, m_1), B(\lambda_2, m_2), \ldots, B(\lambda_k, m_k))\\
	&=
	\begin{bmatrix}
		\begin{bmatrix}
			\lambda_1 & 1        & 0 \\
			0       & \lambda_1  & 1 \\
			0       & 0  & \lambda_1
		\end{bmatrix} &  & & &\\
		& \begin{bmatrix}
			\lambda_2
		\end{bmatrix} & & &\\
		&  &\begin{bmatrix}
			\lambda_3 & 1 \\
			0 & \lambda_3
		\end{bmatrix} & &\\
		&  &   & \ddots & &\\
		&  &   &  & &\begin{bmatrix}
			\lambda_k & 1 \\
			0 & \lambda_k
		\end{bmatrix}\\
	\end{bmatrix}.
\end{aligned}
$$
\textbf{Decoding a Jordan Decomposition}: Note that zeros can appear on the upper sub-diagonal of $\bJ$ and in each block, the first column is always a diagonal containing only eigenvalues of $\bA$. Take out one block to decode, without loss of generality, we take out the first block $\bJ_1$. We shall show the columns $1, 2, \ldots, m_1$ of $\bA\bX = \bX\bJ$ with $\bX=[\bx_1, \bx_2, \ldots, \bx_n]$:
$$
\begin{aligned}
\bA\bx_1 &= \lambda_1 \bx_1 \\
\bA\bx_2 &= \lambda_1 \bx_2 + \bx_1	\\
\vdots &= \vdots \\
\bA\bx_{m_1} &= \lambda_1 \bx_{m_1} + \bx_{m_1-1}.
\end{aligned}
$$
For more details about Jordan decomposition, please refer to \citep{gohberg1996simple, hales1999jordan}.

The Jordan decomposition is not particularly interesting in practice as it is extremely sensitive to perturbation. Even with the smallest random change to a matrix
, the matrix can be made diagonalizable \citep{van2020advanced}. As a result, there is no practical mathematical software library or tool that computes it. And the proof takes dozens of pages to discuss. For this reason, we leave the proof to interesting readers.

\section{Application: Computing Fibonacci Numbers}
We use eigenvalue decomposition to compute the Fibonacci number. This example is drawn from \citep{strang1993introduction}. Every new Fibonacci number $F_{k+2}$ is the sum of the two previous Fibonacci numbers $F_{k+1}+F_{k}$. The sequence is $0, 1, 1, 2, 3, 5, 8, \ldots$. Now, the problem is what is the number of $F_{100}$?\index{Fibonacci number}

Let $\bu_{k}=\begin{bmatrix}
	F_{k+1}\\
	F_k
\end{bmatrix}$.
Then $\bu_{k+1}=\begin{bmatrix}
	F_{k+2}\\
	F_{k+1}
\end{bmatrix}=
\begin{bmatrix}
	1&1\\
	1&0
\end{bmatrix}
\bu_k
$ by the rule that $F_{k+2}=F_{k+1}+F_k$ and $F_{k+1}=F_{k+1}$.

Let $\bA=\begin{bmatrix}
	1&1\\
	1&0
\end{bmatrix}$
, we then have the equation that is $\bu_{100} = \bA^{100}\bu_0$ where $\bu_0=
\begin{bmatrix}
	1\\
	0
\end{bmatrix}
$.

We will see in Lemma~\ref{lemma:determinant-intermezzo} that $\det(\bA-\lambda\bI)=0$ where $\lambda$ is the eigenvalue of $\bA$. Simple calculation shows that $\det(\bA-\lambda\bI) = \lambda^2-\lambda+1=0$ and
$$
\lambda_1 = \frac{1+\sqrt{5}}{2}, \qquad \lambda_2 = \frac{1-\sqrt{5}}{2}.
$$
The corresponding eigenvectors are
$$
\bx_1 =
\begin{bmatrix}
	\lambda_1\\
	1
\end{bmatrix}, \qquad
\bx_2 =
\begin{bmatrix}
	\lambda_2\\
	1
\end{bmatrix}.
$$
By Remark~\ref{remark:power-eigenvalue-decom}, $\bA^{100} = \bX\bLambda^{100}\bX^{-1} = \bX
\begin{bmatrix}
	\lambda_1^{100}&0\\
	0&\lambda_2^{100}
\end{bmatrix}\bX^{-1}$ where $\bX^{-1}$ can be easily calculated as $\bX^{-1} =
\begin{bmatrix}
	\frac{1}{\lambda_1-\lambda_2} & \frac{-\lambda_2}{\lambda_1-\lambda_2} \\
	-\frac{1}{\lambda_1-\lambda_2} & \frac{\lambda_1}{\lambda_1-\lambda_2}
\end{bmatrix}
=\begin{bmatrix}
\frac{\sqrt{5}}{5} & \frac{5-\sqrt{5}}{10} \\
-\frac{\sqrt{5}}{5} & \frac{5+\sqrt{5}}{10}
\end{bmatrix}$. We notice that $\bu_{100} = \bA^{100}\bu_0$ is just the first column of $\bA^{100}$ which is
 $$
\bu_{100} =
\begin{bmatrix}
F_{101}\\
F_{100}
\end{bmatrix}=
\begin{bmatrix}
	\frac{\lambda_1^{101}-\lambda_2^{101}}{\lambda_1-\lambda_2}\\
	\frac{\lambda_1^{100}-\lambda_2^{100}}{\lambda_1-\lambda_2}
\end{bmatrix}.
$$
Check by calculation, we have $F_{100}=3.542248481792631e+20$. Or more generally,
$$
\bu_{K} =
\begin{bmatrix}
	F_{K+1}\\
	F_{K}
\end{bmatrix}=
\begin{bmatrix}
	\frac{\lambda_1^{K+1}-\lambda_2^{K+1}}{\lambda_1-\lambda_2}\\
	\frac{\lambda_1^{K}-\lambda_2^{K}}{\lambda_1-\lambda_2}
\end{bmatrix},
$$
where the general form of $F_K$ is given by $F_K=\frac{\lambda_1^{K}-\lambda_2^{K}}{\lambda_1-\lambda_2}$.

\newpage
\chapter{Schur Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Schur Decomposition}
\begin{theoremHigh}[Schur Decomposition]\label{theorem:schur-decomposition}
Any square matrix $\bA\in \real^{n\times n}$ with real eigenvalues can be factored as
$$
\bA = \bQ\bU\bQ^\top,
$$
where $\bQ$ is an orthogonal matrix, and $\bU$ is an upper triangular matrix. That is, all square matrix $\bA$ with real eigenvalues can be triangularized.
\end{theoremHigh}

\paragraph{A close look at Schur decomposition} The first column of $\bA\bQ$ and $\bQ\bU$ are $\bA\bq_1$ and  $\bU_{11}\bq_1$. Then, $\bU_{11}, \bq_1$ are eigenvalue and eigenvector of $\bA$. But other columns of $\bQ$ need not be eigenvectors of $\bA$.

\paragraph{Schur decomposition for symmetric matrices} Symmetric matrix $\bA=\bA^\top$ leads to $\bQ\bU\bQ^\top = \bQ\bU^\top\bQ^\top$. Then $\bU$ is a diagonal matrix. And this diagonal matrix actually contains eigenvalues of $\bA$. All the columns of $\bQ$ are eigenvectors of $\bA$. We conclude that all symmetric matrices are diagonalizable even with repeated eigenvalues.

\section{Existence of the Schur Decomposition}
To prove Theorem~\ref{theorem:schur-decomposition}, we need to use the following lemmas.
\begin{lemma}[Determinant Intermezzo]\label{lemma:determinant-intermezzo}
We have the following properties for determinant of matrices:

$\bullet$ The determinant of multiplication of two matrices is $\det(\bA\bB)=\det (\bA)\det(\bB)$;

$\bullet$ The determinant of the transpose is $\det(\bA^\top) = \det(\bA)$;

$\bullet$ Suppose matrix $\bA$ has eigenvalue $\lambda$, then $\det(\bA-\lambda\bI) =0$;

$\bullet$ Determinant of any identity matrix is $1$;

$\bullet$ Determinant of an orthogonal matrix $\bQ$:
$$
\det(\bQ) = \det(\bQ^\top) = \pm 1, \qquad \text{since  } \det(\bQ^\top)\det(\bQ)=\det(\bQ^\top\bQ)=\det(\bI)=1;
$$

$\bullet$ Any square matrix $\bA$, we then have an orthogonal matrix $\bQ$:

$$
\det(\bA) = \det(\bQ^\top) \det(\bA)\det(\bQ) =\det(\bQ^\top\bA\bQ);
$$
\end{lemma}

\begin{lemma}[Submatrix with Same Eigenvalue]\label{lemma:submatrix-same-eigenvalue}
Suppose square matrix $\bA_{k+1}\in \real^{(k+1)\times (k+1)}$ has real eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_{k+1}$. Then we can construct a $k\times k$ matrix $\bA_{k}$ with eigenvalues $\lambda_2, \lambda_3, \ldots, \lambda_{k+1}$ by
$$
\bA_{k} =
\begin{bmatrix}
-\bp_2^\top- \\
-\bp_3^\top- \\
\vdots \\
-\bp_{k+1}^\top-
\end{bmatrix}
\bA_{k+1}
\begin{bmatrix}
\bp_2 & \bp_3 &\ldots &\bp_{k+1}
\end{bmatrix},
$$
where $\bp_1$ is a eigenvector of $\bA_{k+1}$ with norm 1 corresponding to eigenvalue $\lambda_1$, and $\bp_2, \bp_3, \ldots, \bp_{k+1}$ are any orthonormal vectors orthogonal to $\bp_1$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:submatrix-same-eigenvalue}]
Let $\bP_{k+1} = [\bp_1, \bp_2, \ldots, \bp_{k+1}]$. Then $\bP_{k+1}^\top\bP_{k+1}=\bI$, and
$$
\bP_{k+1}^\top \bA_{k+1} \bP_{k+1} =
\begin{bmatrix}
	\lambda_1 & \bzero \\
	\bzero  & \bA_{k}
\end{bmatrix}.
$$
For any eigenvalue $\lambda = \{\lambda_2, \lambda_3, \ldots, \lambda_{k+1}\}$, by Lemma~\ref{lemma:determinant-intermezzo}, we have
$$
\begin{aligned}
\det(\bA_{k+1} -\lambda\bI) &= \det(\bP_{k+1}^\top (\bA_{k+1}-\lambda\bI)  \bP_{k+1}) \\
&=\det(\bP_{k+1}^\top \bA_{k+1}\bP_{k+1} - \lambda\bP_{k+1}^\top\bP_{k+1}) \\
&= \det\left(
\begin{bmatrix}
	\lambda_1-\lambda & \bzero \\
	\bzero & \bA_k - \lambda\bI
\end{bmatrix}
\right)\\
&=(\lambda_1-\lambda)\det(\bA_k-\lambda\bI).
\end{aligned}
$$
Where the last equality is from the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bE & \bF \\
		\bG & \bH
	\end{bmatrix}$, then $\det(\bM) = \det(\bE)\det(\bH-\bG\bE^{-1}\bF)$.
Since $\lambda$ is an eigenvalue of $\bA$ and $\lambda \neq \lambda_1$, then $\det(\bA_{k+1} -\lambda\bI) = (\lambda_1-\lambda)\det(\bA_{k}-\lambda\bI)=0$ means $\lambda$ is also an eigenvalue of $\bA_{k}$.
\end{proof}

We then prove the existence of the Schur decomposition by induction.
\begin{proof}[\textbf{of Theorem~\ref{theorem:schur-decomposition}: Existence of Schur Decomposition}]
We note that the theorem is trivial when $n=1$ by setting $Q=1$ and $U=A$. Suppose the theorem is true for $n=k$ for some $k\geq 1$. If we prove the theorem is also true for $n=k+1$, then we complete the proof.

Suppose for $n=k$, the theorem is true for $\bA_k =\bQ_k \bU_k \bQ_k^\top$.

Suppose further \textit{$\bP_{k+1}$ contains orthogonal vectors $\bP_{k+1} = [\bp_1, \bp_2, \ldots, \bp_{k+1}]$ as constructed in Lemma~\ref{lemma:submatrix-same-eigenvalue} where $\bp_1$ is an eigenvector of $\bA_{k+1}$ corresponding to eigenvalue $\lambda_1$ and its norm is 1, $\bp_2, \ldots, \bp_{k+1}$ are orthonormal to $\bp_1$}. Let the other $k$ eigenvalues of $\bA_{k+1}$ be $\lambda_2, \lambda_3, \ldots, \lambda_{k+1}$. Since we suppose for $n=k$, the theorem is true, we can find a matrix $\bA_{k}$ with eigenvalues $\lambda_2, \lambda_3, \ldots, \lambda_{k+1}$. So we have the following property by Lemma~\ref{lemma:submatrix-same-eigenvalue}:
$$
\bP_{k+1}^\top \bA_{k+1} \bP_{k+1} =
\begin{bmatrix}
	\lambda &\bzero \\
	\bzero & \bA_k
\end{bmatrix} \qquad
\text{and} \qquad
\bA_{k+1} \bP_{k+1} =
\bP_{k+1}
\begin{bmatrix}
	\lambda_1 &\bzero \\
	\bzero & \bA_k
\end{bmatrix}.
$$
Let
$
\bQ_{k+1} = \bP_{k+1}
\begin{bmatrix}
1 &\bzero \\
\bzero & \bQ_k
\end{bmatrix}.
$
Then, it follows that
$$
\begin{aligned}
\bA_{k+1} \bQ_{k+1} &=
\bA_{k+1}
\bP_{k+1}
\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix}\\
&=
\bP_{k+1}
\begin{bmatrix}
	\lambda_1 &\bzero \\
	\bzero & \bA_k
\end{bmatrix}
\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix} \\
&=
\bP_{k+1}
\begin{bmatrix}
\lambda_1 & \bzero \\
\bzero & \bA_k\bQ_k
\end{bmatrix}\\
&=
\bP_{k+1}
\begin{bmatrix}
	\lambda_1 & \bzero \\
	\bzero & \bQ_k \bU_k
\end{bmatrix}\qquad &\text{(By the assumption for $n=k$)} \\
&=\bP_{k+1}
\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix}
\begin{bmatrix}
	\lambda_1 &\bzero \\
	\bzero & \bU_k
\end{bmatrix}\\
&=\bQ_{k+1}\bU_{k+1}. \qquad  &\text{($\bU_{k+1} = \begin{bmatrix}
		\lambda_1 &\bzero \\
		\bzero & \bU_k
	\end{bmatrix}$)}
\end{aligned}
$$
We then have $\bA_{k+1} = \bQ_{k+1}\bU_{k+1}\bQ_{k+1}^\top$, where $\bU_{k+1}$ is an upper triangular matrix, and $\bQ_{k+1}$ is an orthogonal matrix since $\bP_{k+1}$ and
$\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix}$ are both orthogonal matrices and this orthogonal form is decoded in Appendix~\ref{appendix:orthogonal-matrix} (p.~\pageref{appendix:orthogonal-matrix}).
\end{proof}

\section{Other Forms of the Schur Decomposition}\label{section:other-form-schur-decom}

From the proof of the Schur decomposition, we obtain the upper triangular matrix $\bU_{k+1}$ by appending the eigenvalue $\lambda_1$ to $\bU_k$. From this process, the values on the diagonal are always eigenvalues. Therefore, we can decompose the upper triangular into two parts.

\begin{corollaryHigh}[Form 2 of Schur Decomposition]\label{corollary:schur-second-form}
Any square matrix $\bA\in \real^{n\times n}$ with real eigenvalues can be factored as
$$
\bQ^\top\bA\bQ = \bLambda +\bT, \qquad \textbf{or} \qquad \bA = \bQ(\bLambda +\bT)\bQ^\top,
$$
where $\bQ$ is an orthogonal matrix, $\bLambda=diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$ is a diagonal matrix containing the eigenvalues of $\bA$, and $\bT$ is a \textit{strictly upper triangular} matrix (with zeros on the diagonal).
\end{corollaryHigh}
A strictly upper triangular matrix is an upper triangular matrix having 0's along the diagonal as well as the lower portion. Another proof for this decomposition is that $\bA$ and $\bU$ (where $\bU = \bQ^\top\bA\bQ$) are similar matrices so that they have the same eigenvalues (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). And the eigenvalues of any upper triangular matrices are on the diagonal.
To see this, for any upper triangular matrix $\bR \in \real^{n\times n}$ where the diagonal values are $r_{ii}$ for all $i\in \{1,2,\ldots,n\}$. We have
$$
\bR \be_i = r_{ii}\be_i,
$$
where $\be_i$ is the $i$-th basis vector in $\real^n$, i.e., $\be_i$ is the $i$-th column of the $n\times n$ identity matrix $\bI_n$.
So we can decompose $\bU$ into $\bLambda$ and $\bT$. Apparently, the first part of Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}) can prove the existence of the second form of the Schur decomposition.

A final observation on the second form of the Schur decomposition is shown as follows. From $\bA\bQ = \bQ(\bLambda +\bT)$, it follows that
$$
\bA \bq_k = \lambda_k\bq_k + \sum_{i=1}^{k-1}t_{ik}\bq_i,
$$
where $t_{ik}$ is the ($i,k$)-th entry of $\bT$. The form is quite close to the eigenvalue decomposition. Nevertheless, the columns become orthonormal bases and the orthonormal bases are correlated.

\part{Reduction to Hessenberg, Tridiagonal, and Bidiagonal Form}
\section*{Introduction}
In real applications, we often want to factor matrix $\bA$ into two orthogonal matrices $\bA =\bQ\bLambda\bQ^\top$ where $\bLambda$ is diagonal or upper triangular, e.g., eigen analysis via the Schur decomposition and principal component analysis (PCA) via the spectral decomposition. This can be computed via a sequence of \textit{orthogonal similarity transformations}:
$$
\underbrace{\bQ_k^\top\ldots \bQ_2^\top \bQ_1^\top}_{\bQ^\top} \bA \underbrace{\bQ_1\bQ_2\ldots\bQ_k}_{\bQ}
$$
which converges to $\bLambda$. However, this will always be very hard to handle in practice, for example, via the Householder reflectors. Following from the QR decomposition via the Householder example, \footnote{where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed. } the sequence of orthogonal similarity transformation can be constructed via the Householder reflectors:
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bH_1\times }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times \bH_1^\top }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix},
\end{aligned}
$$
where the left Householder introduces zeros in the first column below the main diagonal (see Section~\ref{section:qr-via-householder}), and unfortunately the right Householder will destroy the zeros introduced by the left Householder.

However, if we are less ambitious to modify the algorithms into two phases, where the first phase transforms into a Hessenberg matrix (Definition~\ref{definition:upper-hessenbert}, p.~\pageref{definition:upper-hessenbert}) or a tridiagonal matrix (Definition~\ref{definition:tridiagonal-hessenbert}, p.~\pageref{definition:tridiagonal-hessenbert}). And if we find a second phase algorithm to transform the results from the first one to the goal we want to find, then we complete the algorithm:
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bH_1\times }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times \bH_1^\top }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix},
\end{aligned}
$$
where the left Householder will not influence the first row, and the right Householder will not influence the first column.
A phase 2 \footnote{which is usually an iterative algorithm.} algorithm to find the triangular matrix is shown as follows:
$$
\begin{aligned}
	\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top \bH_3^\top}
		\boxtimes & \boxtimes  & \boxtimes  & \boxtimes & \boxtimes  \\
		\boxtimes & \boxtimes  & \boxtimes  & \boxtimes & \boxtimes  \\
		0 & \boxtimes  & \boxtimes  & \boxtimes & \boxtimes  \\
		0 & 0 & \boxtimes  & \boxtimes & \boxtimes  \\
		0 & 0 & 0 & \boxtimes  & \boxtimes
	\end{sbmatrix}
	\stackrel{\text{Phase 2} }{\longrightarrow}
	\begin{sbmatrix}{\bLambda}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & \bm{0}  & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & \bm{0}  & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & 0 & \bm{0}  & \bm{\boxtimes}
	\end{sbmatrix}
\end{aligned}
$$

From the discussion above, to compute the spectral decomposition, Schur decomposition, or singular value decomposition (SVD), we usually reach a compromise to calculate the Hessenberg, tridiagonal, or bidiagonal form in the first phase and leave the second phase to finish the rest \citep{van2012families, van2014restructuring, trefethen1997numerical}.

\newpage
\chapter{Hessenberg Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Hessenberg Decomposition}
We firstly give the rigorous definition of the upper Hessenberg matrix.
\begin{definition}[Upper Hessenberg Matrix\index{Hessenbert matrix}]\label{definition:upper-hessenbert}
An \textit{upper Hessenberg matrix} is a square matrix where all the entries below the first diagonal (i.e., the ones below the \textit{main diagonal}) (a.k.a., \textit{lower subdiagonal}) are zeros. Similarly, a lower Hessenberg matrix is a square matrix where all the entries above the first diagonal (i.e., the ones above the main diagonal) are zeros.

 The definition of the upper Hessenberg can also be extended to rectangular matrices, and the form can be implied from the context.

 In matrix language, for any matrix $\bH\in \real^{n\times n}$, and the entry ($i,j$) denoted by $h_{ij}$ for all $i,j\in \{1,2,\ldots, n\}$. Then $\bH$ with $h_{ij}=0$ for all $i\geq j+2$ is known as an Hessenberg matrix.

 Let $i$ denote the smallest positive integer for which $h_{i+1, i}=0$ where $i\in \{1,2,\ldots, n-1\}$, then $\bH$ is \textit{unreduced} if $i=n-1$.
\end{definition}

Take a $5\times 5$ matrix as an example, the lower triangular below the lower sub-diagonal are zero in the upper Hessenberg matrix:
$$
\begin{sbmatrix}{possibly\,\, unreduced}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}
\qquad
\text{or}
\qquad
\begin{sbmatrix}{reduced}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	0 & 0 & \textcolor{blue}{0} & \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}.
$$
Then we have the following Hessenberg decomposition:
\begin{theoremHigh}[Hessenberg Decomposition]\label{theorem:hessenberg-decom}
Every $n\times n$ square matrix $\bA$ can be factored as
$$
\bA = \bQ\bH\bQ^\top \qquad \text{or} \qquad \bH = \bQ^\top \bA\bQ,
$$
where $\bH$ is an upper Hessenberg matrix, and $\bQ$ is an orthogonal matrix.
\end{theoremHigh}
It's not hard to find that a lower Hessenberg decomposition of $\bA^\top$ is given by $\bA^\top = \bQ\bH^\top\bQ^\top$ if $\bA$ has the Hessenberg decomposition $\bA = \bQ\bH\bQ^\top$. The Hessenberg decomposition shares a similar form as the QR decomposition in that they both reduce a matrix into a sparse form where the lower parts of both are zero.
\begin{remark}[Why Hessenberg Decomposition]
We will see that the zeros introduced into $\bH$ from $\bA$ is accomplished by the left orthogonal matrix $\bQ$ (same as the QR decomposition) and the right orthogonal matrix $\bQ^\top$ here does not transform the matrix into any better or simple form. Then why do we want the Hessenberg decomposition rather than just a QR decomposition which has a simpler structure in that it even has zeros in the lower sub-diagonal? The answer is given in the previous section that the Hessenberg decomposition is usually used by other algorithms as a phase 1 step to find a decomposition that factor the matrix into two orthogonal matrices, e.g., SVD, UTV, and so on. And if we employ an aggressive algorithm that even favors zeros in the lower sub-diagonal (again, as in the QR decomposition), the right orthogonal transform $\bQ^\top$ will destroy the zeros that can be seen very shortly.

On the other hand, the form $\bA = \bQ\bH\bQ^\top$ on $\bH$ is known as the \textit{orthogonal similarity transformation} (Definition~\ref{definition:similar-matrices}, p.~\pageref{definition:similar-matrices}) on $\bA$ such that the eigenvalues, rank and trace of $\bA$ and $\bH$ are the same (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). Then if we want to study the properties of $\bA$, exploration on $\bH$ can be a relatively simpler task than that on the original matrix $\bA$.
\end{remark}

\section{Similarity Transformation and Orthogonal Similarity Transformation}
As mentioned previously, the Hessenberg decomposition introduced in this section, the tridiagonal decomposition in the next section, the Schur decomposition (Theorem~\ref{theorem:schur-decomposition}, p.~\pageref{theorem:schur-decomposition}), and the spectral decomposition (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}) share a similar form that transforms the matrix into a similar matrix. We now give the rigorous definition of similar matrices and similarity transformations. \index{Similar matrices}\index{Similarity tansformation}
\begin{definition}[Similar Matrices and Similarity Transformation]\label{definition:similar-matrices}
	$\bA$ and $\bB$ are called \textit{similar matrices} if there exists a nonsingular matrix $\bP$ such that $\bB = \bP\bA\bP^{-1}$.

	In words, for any nonsingular matrix $\bP$, the matrices $\bA$ and $\bP\bA\bP^{-1}$ are similar matrices.
	And in this sense, given the nonsingular matrix $\bP$, $\bP\bA\bP^{-1}$
	is called a \textit{similarity transformation} applied to matrix $\bA$.

	Moreover, when $\bP$ is orthogonal, then $\bP\bA\bP^\top$ is also known as the \textit{orthogonal similarity transformation} of $\bA$. The orthogonal similarity transformation is important in the sense that the condition number of the transformed matrix $\bP\bA\bP^\top$ is not worse than that of the original matrix $\bA$.
\end{definition}
The difference between the similarity transformation and orthogonal similarity transformation is partly explained in the sense of coordinate transformation (Section~\ref{section:coordinate-transformation}, p.~\pageref{section:coordinate-transformation}). Now we prove the important properties of similar matrices in the following lemma.
\begin{lemma}[Eigenvalue, Trace and Rank of Similar Matrices\index{Trace}]\label{lemma:eigenvalue-similar-matrices}
	Any eigenvalue of $\bA$ is also an eigenvalue of $\bP\bA\bP^{-1}$. The converse is also true that any eigenvalue of $\bP\bA\bP^{-1}$ is also an eigenvalue of $\bA$. I.e., $\Lambda(\bA) = \Lambda(\bB)$, where $\Lambda(\bX)$ is the spectrum of matrix $\bX$ (Definition~\ref{definition:spectrum}, p.~\pageref{definition:spectrum}).

	And also the trace and rank of $\bA$ are equal to those of matrix $\bP\bA\bP^{-1}$ for any nonsingular matrix $\bP$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:eigenvalue-similar-matrices}]
	For any eigenvalue $\lambda$ of $\bA$, we have $\bA\bx =\lambda \bx$. Then $\lambda \bP\bx = \bP\bA\bP^{-1} \bP\bx$ such that $\bP\bx$ is an eigenvector of $\bP\bA\bP^{-1}$ corresponding to $\lambda$.

	Similarly, for any eigenvalue $\lambda$ of $\bP\bA\bP^{-1}$, we have $\bP\bA\bP^{-1} \bx = \lambda \bx$. Then $\bA\bP^{-1} \bx = \lambda \bP^{-1}\bx$ such that $\bP^{-1}\bx$ is an eigenvector of $\bA$ corresponding to $\lambda$.

	For the trace of $\bP\bA\bP^{-1}$, we have $trace(\bP\bA\bP^{-1}) = trace(\bA\bP^{-1}\bP) = trace(\bA)$, where the first equality comes from the fact that
	trace of a product is invariant under cyclical permutations of the factors:
	\begin{equation}
		trace(\bA\bB\bC) = trace(\bB\bC\bA) = trace(\bC\bA\bB), \nonumber
	\end{equation}
	if all $\bA\bB\bC$, $\bB\bC\bA$, and $\bC\bA\bB$ exist.

	For the rank of $\bP\bA\bP^{-1}$, we separate it into two claims as follows.
	\paragraph{Rank claim 1: $rank(\bZ\bA)=rank(\bA)$ if $\bZ$ is nonsingular}
	We will first show that $rank(\bZ\bA)=rank(\bA)$ if $\bZ$ is nonsingular. For any vector $\bn$ in the null space of $\bA$, that is $\bA\bn = \bzero$. Thus, $\bZ\bA\bn = \bzero$, that is, $\bn$ is also in the null space of $\bZ\bA$. And this implies $\nspace(\bA)\subseteq \nspace(\bZ\bA)$.

	Conversely, for any vector $\bmm$ in the null space of $\bZ\bA$, that is $\bZ\bA\bmm = \bzero$, we have $\bA\bmm = \bZ^{-1} \bzero=\bzero$. That is, $\bmm$ is also in the null space of $\bA$. And this indicates $\nspace(\bZ\bA)\subseteq \nspace(\bA)$.

	By ``sandwiching",  the above two arguments imply
	$$
	\nspace(\bA) = \nspace(\bZ\bA)\quad  \longrightarrow \quad rank(\bZ\bA)=rank(\bA).
	$$

	\paragraph{Rank claim 2: $rank(\bA\bZ)=rank(\bA)$ if $\bZ$ is nonsingular}
	We notice that the row rank is equal to the column rank of any matrix (Corollary~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}). Then $rank(\bA\bZ) = rank(\bZ^\top\bA^\top)$. Since $\bZ^\top$ is nonsingular, by claim 1, we have $rank(\bZ^\top\bA^\top) = rank(\bA^\top) = rank(\bA)$ where the last equality is again from the fact that the row rank is equal to the column rank of any matrix. This results in $rank(\bA\bZ)=rank(\bA)$ as claimed.

	Since $\bP, \bP^{-1}$ are nonsingular, we then have $rank(\bP\bA\bP^{-1}) = rank(\bA\bP^{-1}) = rank(\bA)$ where the first equality is from claim 1 and the second equality is from claim 2. We complete the proof.
\end{proof}
The lemma above will be proved very useful in the sequel (see Lemma~\ref{lemma:rank-of-symmetric-idempotent}, p.~\pageref{lemma:rank-of-symmetric-idempotent} to prove the trace and rank of symmetric idempotent matrices).

\section{Existence of the Hessenberg Decomposition}
We will prove that any $n\times n$ matrix can be reduced to Hessenberg form via a sequence of Householder transformations that are applied from the left and the right to the matrix.
Previously, we utilized a Householder reflector to triangularize matrices and introduce zeros below the diagonal to obtain the QR decomposition. A similar approach can be applied to introduce zeros below the subdiagonal.  To see this, you have to revisit the ideas behind the Householder reflector in Definition~\ref{definition:householder-reflector} (p.~\pageref{definition:householder-reflector}).

Before introducing the mathematical construction of such decomposition, we emphasize the following remark which will be very useful in the finding of the decomposition.
\begin{remark}[Left and Right Multiplied by a Matrix with Block Identity]\label{remark:left-right-identity}
For square matrix $\bA\in \real^{n\times n}$, and a matrix
$$
\bB = \begin{bmatrix}
	\bI_k &\bzero \\
	\bzero & \bB_{n-k}
\end{bmatrix},
$$
where $\bI_k$ is a $k\times k$ identity matrix. Then $\bB\bA$ will not change the first $k$ rows of $\bA$, and $\bA\bB$ will not change the first $k$ columns of $\bA$.
\end{remark}
The proof of this remark is trivial.

\subsubsection*{\textbf{First Step: Introduce Zeros for the First Column}}
Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ be the column partitions of $\bA$, and each $\ba_i \in \real^{n}$. Suppose $\bar{\ba}_1, \bar{\ba}_2, \ldots, \bar{\ba}_n \in \real^{n-1}$ are vectors removing the first component in $\ba_i$'s. Let
$$
 r_1 = ||\bar{\ba}_1||, \qquad \bu_1 = \frac{\bar{\ba}_1 - r_1 \be_1}{||\bar{\ba}_1 - r_1 \be_1||}, \qquad \text{and}\qquad \widetilde{\bH}_1 = \bI - 2\bu_1\bu_1^\top \in \real^{(n-1)\times (n-1)},
$$
where $\be_1$ here is the first basis for $\real^{n-1}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \real^{n-1}$. To introduce zeros below the sub-diagonal and operate on the submatrix $\bA_{2:n,1:n}$, we append the Householder reflector into
$$
\bH_1 = \begin{bmatrix}
	1 &\bzero \\
	\bzero & \widetilde{\bH}_1
\end{bmatrix},
$$
in which case, $\bH_1\bA$ will introduce zeros in the first column of $\bA$ below entry (2,1). The first row of $\bA$ will not be affected at all and kept unchanged by Remark~\ref{remark:left-right-identity}. And we can easily verify that both $\bH_1$ and $\widetilde{\bH}_1$ are orthogonal matrices and they are symmetric (from the definition of Householder reflector). To have the form in Theorem~\ref{theorem:hessenberg-decom}, we multiply $\bH_1\bA$ on the right by $\bH_1^\top$ which results in $\bH_1\bA\bH_1^\top$. The $\bH_1^\top$ on the right will not change the first column of $\bH_1\bA$ and thus keep the zeros introduced in the first column.

An example of a $5\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	\stackrel{\bH_1\times}{\rightarrow}
	&\begin{sbmatrix}{\bH_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_1^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}\\
\end{aligned}
$$

\subsubsection*{\textbf{Second Step: Introduce Zeros for the Second Column}}
Let $\bB = \bH_1\bA\bH_1^\top$, where the entries in the first column below entry (2,1) are all zeros. And the goal is to introduce zeros in the second column below entry (3,2). Let $\bB_2 = \bB_{2:n,2:n}=[\bb_1, \bb_2, \ldots, \bb_{n-1}]$. Suppose again $\bar{\bb}_1, \bar{\bb}_2, \ldots, \bar{\bb}_{n-1} \in \real^{n-2}$ are vectors removing the first component in $\bb_i$'s. We can again construct a Householder reflector
\begin{equation}\label{equation:householder-qr-lengthr}
r_1 = ||\bar{\bb}_1||,  \qquad \bu_2 = \frac{\bar{\bb}_1 - r_1 \be_1}{||\bar{\bb}_1 - r_1 \be_1||}, \qquad \text{and}\qquad \widetilde{\bH}_2 = \bI - 2\bu_2\bu_2^\top\in \real^{(n-2)\times (n-2)},
\end{equation}
where $\be_1$ now is the first basis for $\real^{n-2}$. To introduce zeros below the sub-diagonal and operate on the submatrix $\bB_{3:n,1:n}$, we append the Householder reflector into
$$
\bH_2 = \begin{bmatrix}
	\bI_2 &\bzero \\
	\bzero & \widetilde{\bH}_2
\end{bmatrix},
$$
where $\bI_2$ is a $2\times 2$ identity matrix. We can see that $\bH_2\bH_1\bA\bH_1^\top$ will not change the first two rows of $\bH_1\bA\bH_1^\top$, and as the Householder cannot reflect a zero vector such that the zeros in the first column will be kept. Again, putting $\bH_2^\top$ on the right of $\bH_2\bH_1\bA\bH_1^\top$ will not change the first 2 columns so that the zeros will be kept.

Following the example of a $5\times 5$ matrix, the second step is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		0 & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		0 & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		0 & \boxtimes & \boxtimes & \boxtimes &  \boxtimes
	\end{sbmatrix}
	\stackrel{\bH_2\times}{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_2^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
		\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
\end{aligned}
$$

Same process can go on, and there are $n-2$ such steps. We will finally triangularize by
$$
\bH = \bH_{n-2} \bH_{n-3}\ldots\bH_1 \bA\bH_1^\top\bH_2^\top\ldots\bH_{n-2}^\top.
$$
And since $\bH_i$'s are symmetric and orthogonal, the above equation can be simply reduced to
$$
\bH =\bH_{n-2} \bH_{n-3}\ldots\bH_1 \bA\bH_1\bH_2\ldots\bH_{n-2}.
$$
Note here only $n-2$ such stages exist rather than $n-1$ or $n$. We will verify this number of steps by the example below.
The example of a $5\times 5$ matrix as a whole is shown as follows where again $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10, frametitle={A Complete Example of Hessenberg Decomposition}]
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bH_1\times}{\rightarrow}
&\begin{sbmatrix}{\bH_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bH_1^\top}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
	\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\stackrel{\bH_2\times}{\rightarrow}
&\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bH_2^\top}{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
	\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\stackrel{\bH_3\times}{\rightarrow}
&\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bH_3^\top}{\rightarrow}
\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top\bH_3^\top}
	\boxtimes & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix},
\end{aligned}
$$
\end{mdframed}
where we find
\begin{itemize}
\item when multiplying by $\bH_1$ on the left, we operate on a $(5-1)\times (5-1+1)$ submatrix; \item when multiplying by $\bH_2$ on the left, we operate on a $(5-2)\times (5-2+1)$ submatrix;  \item and when multiplying by $\bH_3$ on the left, we operate on a $(5-3)\times(5-3+1)$ submatrix.
\end{itemize}
Similarly,
\begin{itemize}
\item when multiplying by $\bH_1^\top$ on the right, we operate on a $5\times (5-1)$ submatrix; \item when multiplying by $\bH_2^\top$ on the right, we operate on a $5\times (5-2)$ submatrix;
\item and when multiplying by $\bH_3^\top$ on the right, we operate on a $5\times (5-3)$ submatrix.
\end{itemize}
The above two findings are important for the computation and reduction in the complexity of the Hessenberg decomposition.

\section{Computing the Hessenberg Decomposition}
Matrix $\bA\in \real^{n\times n}$, and for left multiplied by $\bH_i$'s in each step $i\in \{1, 2, \ldots, n-2\}$, we operate on a submatrix of size $(n-i)\times(n-i+1)$. Furthermore, if we take the first column of this $(n-i)\times(n-i+1)$ matrix, we can set the first column by the reflected vector to obtain the Householder reflector such that we can operate on an $(n-i)\times(n-i)$ submatrix instead.

For multiplied by $\bH_i^\top$'s on the right in each step $i\in \{1, 2, \ldots, n-2\}$, we operate on a submatrix of size $n\times(n-i)$.
After the Householder transformation, we get the final triangular matrix $\bR$, and the process is shown in Algorithm~\ref{alg:hessenbert-decomposition-householder}.

Similar to computing the QR decomposition via Householder reflectors, in Algorithm~\ref{alg:hessenbert-decomposition-householder}, to compute $\bH = \bH_{n-2} \bH_{n-3}\ldots\bH_1 \bA\bH_1^\top\bH_2^\top\ldots\bH_{n-2}^\top$, we write out the equation
$$
\begin{aligned}
\bH &= \cleft[black](\bH_{n-2} \ldots\cleft[green](\bH_3\cleft[red](\bH_2\cleft[blue](\bH_1 \bA\bH_1\cright[blue])\bH_2\cright[red])\bH_3\cright[green])\ldots\bH_{n-2}\cright[black]) \\
&=
\begin{bmatrix}
	\bI_{n-2}& \bzero \\
	\bzero& \bI - 2\bu_{n-2}\bu_{n-2}^\top
\end{bmatrix}
\ldots
\begin{bmatrix}
	\bI_3& \bzero \\
	\bzero& \bI - 2\bu_3\bu_3^\top
\end{bmatrix}
\begin{bmatrix}
	\bI_2& \bzero \\
	\bzero& \bI - 2\bu_2\bu_2^\top
\end{bmatrix}
\begin{bmatrix}
	\bI_1 &\bzero \\
	\bzero & \bI - 2\bu_1\bu_1^\top
\end{bmatrix}\\
&\bA
\begin{bmatrix}
		\bI_1 &\bzero \\
		\bzero & \bI - 2\bu_1\bu_1^\top
	\end{bmatrix}
	\begin{bmatrix}
		\bI_2& \bzero \\
		\bzero& \bI - 2\bu_2\bu_2^\top
	\end{bmatrix}
	\begin{bmatrix}
		\bI_3& \bzero \\
		\bzero& \bI - 2\bu_3\bu_3^\top
	\end{bmatrix}
	\ldots
	\begin{bmatrix}
		\bI_{n-2}& \bzero \\
		\bzero& \bI - 2\bu_{n-2}\bu_{n-2}^\top
	\end{bmatrix},
\end{aligned}
$$
where the different colors for the parentheses indicate the computational order, and
\begin{itemize}
\item the upper-left of $\bH_1$ is a $1\times 1$ identity matrix, multiplying on the left will not change the \textbf{first row} of $\bA$, then multiplying on the right will not change the \textbf{first column} of $\bH_1\bA$;
\item the upper-left of $\bH_2$ is a $2\times 2$ identity matrix, multiplying on the left will not change the \textbf{first 2 rows} of $\bH_1\bA\bH_1$, then multiplying on the right will not change the \textbf{first 2 columns} of $\bH_2\bH_1\bA\bH_1$;
\item the upper-left of $\bH_3$ is a $3\times 3$ identity matrix, multiplying on the left will not change the \textbf{first 3 rows} of $\bH_2\bH_1\bA\bH_1\bH_2$, then multiplying on the right will not change the \textbf{first 3 columns} of $\bH_3\bH_2\bH_1\bA\bH_1\bH_2$;
\item the process can go on, and this property yields the step 8 and 9 in the algorithm.
\end{itemize}

Similarly, to get the final orthogonal matrix $\bQ=\bH_1 \bH_2\ldots\bH_{n-2}$, we write out the equation:
$$
\begin{aligned}
	\bQ
	&=\bH_1 \bH_2\bH_3\ldots\bH_{n-2} \\
	&= \begin{bmatrix}
		\bI_1 &\bzero \\
		\bzero & \bI - 2\bu_1\bu_1^\top
	\end{bmatrix}
	\begin{bmatrix}
		\bI_2& \bzero \\
		\bzero& \bI - 2\bu_2\bu_2^\top
	\end{bmatrix}
	\begin{bmatrix}
		\bI_3& \bzero \\
		\bzero& \bI - 2\bu_3\bu_3^\top
	\end{bmatrix}
	\ldots
	\begin{bmatrix}
		\bI_{n-2}& \bzero \\
		\bzero& \bI - 2\bu_{n-2}\bu_{n-2}^\top
	\end{bmatrix},
\end{aligned}
$$
where the upper-left of $\bH_1$ is a $1\times 1$ identity matrix, the upper-left of $\bH_2$ is a $2\times 2$ identity matrix, and it will not change the first 2 columns of $\bH_1$; and the upper-left of $\bH_3$ is a $3\times 3$ identity matrix which will not modify the first 3 columns of $\bH_1\bH_2$; $\ldots$. This property yields the step 15 in the algorithm.

\begin{algorithm}[h]
	\caption{Hessenberg Decomposition via the Householder Reflector}
	\label{alg:hessenbert-decomposition-householder}
	\begin{algorithmic}[1]
		\Require matrix $\bA$ with size $n\times n $;
		\State Initially set $\bH = \bA$; (\textcolor{blue}{note $\bH$ is the Hessenberg, $\bH_i$'s are Householders})
		\For{$i=1$ to $n-2$}
		\State $\ba = \bH_{i+1:n,i}$, i.e., first column of $\bH_{i+1:n,i:n}\in \real^{(n-i)\times(n-i+1)}$;
		\State $r = ||\ba||$;\Comment{$2(n-i)$ flops}
		\State $\bu_i = \ba-r\be_1$;, \Comment{1 flop}
		\State $\bu_i = \bu_i / ||\bu_i||$; \Comment{$3(n-i)$ flops}
		\State $\bH_{i+1,i}=r$, $\bH_{i+2:n,i}=\bzero$, i.e., set the value of first column of $\bH_{i+1:n,i:n}$; \Comment{0 flops}
		\State Left: set the value of columns $2$ to $n$ of $\bH_{i+1:n,i:n}$,
		$$
		\begin{aligned}
			\bH_{i+1:n,i+1:n} &= (\bI-2\bu_i\bu_i^\top)\bH_{i+1:n,i+1:n} \\
			&= \bH_{i+1:n,i+1:n} - 2\bu_i\bu_i^\top\bH_{i+1:n,i+1:n} \in \real^{(n-i)\times(n-i)}\\
			&\text{\qquad\qquad($4(n-i)^2$ flops);}
		\end{aligned}
		$$
		\State Right:
		$$
		\begin{aligned}
			\bH_{1:n,i+1:n} &= \bH_{1:n,i+1:n}(\bI-2\bu_i\bu_i^\top) \\
			&=\bH_{1:n,i+1:n} - \bH_{1:n,i+1:n} 2\bu_i\bu_i^\top\in \real^{n\times(n-i)}\\
			&\text{\qquad\,\,\,\,\,($4n(n-i)-n$ flops);}
		\end{aligned}
		$$
		\EndFor
		\State Output $\bH$ as the upper Hessenberg matrix;
		\State Get $\bQ=\bH_1\bH_2\ldots\bH_{n-2}$, where $\bH_i$'s are Householder reflectors.
		\State Initially set $\bQ = \bH_1$;
		\For{$i=1$ to $n-3$}
		\State Compute $\bQ$:
		$$\begin{aligned}
				\bQ_{1:n,i+2:n} &= \bQ_{1:n,i+2:n}\bH_{i+1}\\
			&= \bQ_{1:n,i+2:n}(\bI - 2\bu_{i+1}\bu_{i+1}^\top)\\
			&=\bQ_{1:n,i+2:n}-\bQ_{1:n,i+2:n}2\bu_{i+1}\bu_{i+1}^\top \in \real^{n\times (n-i-1)}\\
			&\text{\qquad\,\,\,\,\,\gap($4n(n-i-1) - n$ flops);}
		\end{aligned}$$
		\EndFor
		\State Output $\bQ$ as the orthogonal matrix.
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: Hessenberg via Householder]\label{theorem:hessenbert-householder}
	Algorithm~\ref{alg:hessenbert-decomposition-householder} requires $\sim \frac{10}{3}n^3$ flops to compute a Hessenberg decomposition of an $n\times n$ square matrix. Further, if $\bQ$ is needed explicitly, additional $\sim 2n^3$ flops required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:hessenbert-householder}] We separate the proof into obtaining the Hessenberg matrix and the orthogonal matrix.
\paragraph{To obtain the Hessenberg matrix: }
For loop $i$, $\bH_{i+1:n,i:n}\in \real^{(n-i)\times(n-i+1)}$. Thus $\ba$ is in $\real^{n-i}$.

In step 4, to compute $r = ||\ba||$ involves $n-i$ multiplications, $n-i-1$ additions, and 1 square root operation which is \fbox{$2(n-i)$} flops.

In step 5, $\bu_i = \ba-r\be_1$ requires 1 subtraction which is \fbox{$1$} flop as the special structure of $\be_1$;

In step 6, same to step 4, it requires $2(n-i)$ flops to compute the norm and $(n-i)$ additional divisions which is \fbox{$3(n-i)$} flops.

In step 8, suppose in loop $i$, $\bu_i^\top \bH_{i+1:n,i+1:n}$ requires $n-i$ times ($n-i$ multiplications and $n-i-1$ additions) which is \fbox{$(n-i)(2(n-i)-1)$} flops. $2\bu_i$ requires \fbox{$n-i$} multiplications. Further $2\bu_i (\bu_i^\top \bH_{i+1:n,i+1:n})$ requires \fbox{$(n-i)^2$} multiplications to make an $(n-i)\times (n-i)$ matrix. The final matrix subtraction needs \fbox{$(n-i)^2$} subtractions. Thus the total complexity for step 7 if loop $i$ is \fbox{$4(n-i)^2$} flops for each iteration $i$.

In step 9, the computation of $2\bu_i$ needs 0 flops since it has already been calculated in step 8. Similarly, $\bH_{1:n,i+1:n} 2\bu_i$ involves $n$ times ($n-i$ multiplications and $n-i-1$ additions) which is \fbox{$n(2(n-i)-1)$} flops. $\bH_{1:n,i+1:n} 2\bu_i\bu_i^\top$ takes \fbox{$n(n-i)$} multiplications to make an $n\times (n-i)$ matrix. The final matrix subtraction needs additional \fbox{$n(n-i)$} subtractions. This makes the complexity of step 9 to be \fbox{$4n(n-i)-n$} flops for each iteration $i$.

So for loop $i$ the total complexity is $4i^2 - (12n+4)i + (8n^2+3n+2)$ flops. Let $f(i)=4i^2 - (12n+4)i + (8n^2+3n+2)$, the complexity for step 2 to step 10 can be obtained by
$$
\mathrm{cost}=f(1)+f(2)+\ldots+f(n-2).
$$
Simple calculation will show the sum of $n-2$ loops is $\frac{10}{3}n^3$ flops if we keep only the leading term.

\paragraph{To obtain the orthogonal matrix: }
For the additional computation of $\bQ$ in step 15, the situation is similar to step 9. the computation of $2\bu_{i+1}$ needs 0 flops since it has already been calculated in step 8. $\bQ_{1:n,i+2:n}2\bu_{i+1}$ involves $n$ times ($n-i-1$ multiplications and $n-i-2$ additions) which is \fbox{$n(2(n-i-1)-1)$} flops. $\bQ_{1:n,i+2:n}2\bu_{i+1}\bu_{i+1}^\top$ takes \fbox{$n(n-i-1)$} to make an $n\times (n-i-1)$ matrix. The final matrix subtraction needs additional \fbox{$n(n-i-1)$} subtractions. This makes the complexity of step 15 to be \fbox{$4n(n-i-1) - n$} flops for each iteration $i$. Let $g(i)=4n(n-i-1) - n$, the complexity for step 14 to step 16 is given by
$$
\mathrm{cost}=g(1)+g(2)+\ldots +g(n-3).
$$
Trivial calculations can show that the complexity is $2n^3$ flops if we keep only the leading term.
\end{proof}

\section{Properties of the Hessenberg Decomposition}\label{section:hessenberg-decomposition}
The Hessenberg decomposition is not unique in the different ways to construct the Householder reflectors (say Equation~\eqref{equation:householder-qr-lengthr}, p.~\pageref{equation:householder-qr-lengthr}). However, under mild conditions, we can claim a similar structure in different decompositions.

\begin{theorem}[Implicit Q Theorem for Hessenberg Decomposition\index{Implicit Q theorem}]\label{theorem:implicit-q-hessenberg}
Suppose two Hessenberg decompositions of matrix $\bA\in \real^{n\times n}$ are given by $\bA=\bU\bH\bU^\top=\bV\bG\bV^\top$ where $\bU=[\bu_1, \bu_2, \ldots, \bu_n]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU,\bV$. Suppose further that $k$ is the smallest positive integer for which $h_{k+1,k}=0$ where $h_{ij}$ is the entry $(i,j)$ of $\bH$. Then
\begin{itemize}
\item If $\bu_1=\bv_1$, then $\bu_i = \pm \bv_i$ and $|h_{i,i-1}| = |g_{i,i-1}|$ for $i\in \{2,3,\ldots,k\}$.
\item When $k=n-1$, the Hessenberg matrix $\bH$ is known as \textit{unreduced} (Definition~\ref{definition:upper-hessenbert}, p.~\pageref{definition:upper-hessenbert}). However, if $k<n-1$, then $g_{k+1,k}=0$.
\end{itemize}
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:implicit-q-hessenberg}]
Define the orthogonal matrix $\bQ=\bV^\top\bU$ and we have
$$
\left.
\begin{aligned}
\bG\bQ &= \bV^\top\bA\bV \bV^\top\bU = \bV^\top\bA\bU \\
\bQ\bH &= \bV^\top\bU \bU^\top\bA\bU = \bV^\top\bA\bU
\end{aligned}
\right\}
\leadto
\bG\bQ = \bQ\bH,
$$
the $(i-1)$-th column of each can be represented as
$$
\bG\bq_{i-1} = \bQ\bh_{i-1},
$$
where $\bq_{i-1}$ and $\bh_{i-1}$ are the $(i-1)$-th column of $\bQ$ and $\bH$ respectively. Since $h_{l,i-1}=0$ for $l\geq i+1$ (by the definition of upper Hessenberg matrices), $\bQ\bh_{i-1}$ can be represented as
$$
\bQ\bh_{i-1} = \sum_{j=1}^{i} h_{j,i-1} \bq_j = h_{i,i-1}\bq_i + \sum_{j=1}^{i-1} h_{j,i-1} \bq_j.
$$
Combine the two findings above, it follows that
$$
h_{i,i-1}\bq_i  = \bG\bq_{i-1} - \sum_{j=1}^{i-1} h_{j,i-1} \bq_j.
$$
A moment of reflexion reveals that $[\bq_1, \bq_2, \ldots,\bq_k]$ is upper triangular. And since $\bQ$ is orthogonal, it must be diagonal and each value on the diagonal is in $\{-1, 1\}$ for $i\in \{2,\ldots, k\}$. Then, $\bq_1=\be_1$ and $\bq_i = \pm \be_i$ for $i\in \{2,\ldots, k\}$. Further, since $\bq_i =\bV^\top\bu_i$ and $h_{i,i-1}=\bq_i^\top (\bG\bq_{i-1} - \sum_{j=1}^{i-1} h_{j,i-1} \bq_j)=\bq_i^\top \bG\bq_{i-1}$. For $i\in \{2,\ldots,k\}$, $\bq_i^\top \bG\bq_{i-1}$ is just $\pm g_{i,i-1}$. It follows that
$$
\begin{aligned}
|h_{i,i-1}| &= |g_{i,i-1}|, \qquad &\forall i\in \{2,\ldots,k\},\\
\bu_i &= \pm\bv_i, \qquad &\forall i\in \{2,\ldots,k\}.
\end{aligned}
$$
This proves the first part. For the second part, if $k<n-1$,
$$
\begin{aligned}
g_{k+1,k} &= \be_{k+1}^\top\bG\be_{k} = \pm \be_{k+1}^\top\underbrace{\bG\bQ}_{\bQ\bH} \be_{k} =  \pm \be_{k+1}^\top\underbrace{\bQ\bH \be_{k}}_{\text{$k$-th column of $\bQ\bH$}} \\
&= \pm \be_{k+1}^\top \bQ\bh_k = \pm \be_{k+1}^\top \sum_{j}^{k+1} h_{jk}\bq_j
=\pm \be_{k+1}^\top \sum_{j}^{\textcolor{blue}{k}} h_{jk}\bq_j=0,
\end{aligned}
$$
where the penultimate equality is from the assumption that $h_{k+1,k}=0$. This completes the proof.
\end{proof}
We observe from the above theorem, when two Hessenberg decompositions of matrix $\bA$ are both unreduced and have the same first column in the orthogonal matrices, then the Hessenberg matrices $\bH, \bG$ are similar matrices such that $\bH = \bD\bG\bD^{-1}$ where $\bD=\diag(\pm 1, \pm 1, \ldots, \pm 1)$. \textit{Moreover, and most importantly, if we restrict the elements in the lower sub-diagonal of the Hessenberg matrix $\bH$ to be positive (if possible), then the Hessenberg decomposition $\bA=\bQ\bH\bQ^\top$ is uniquely determined by $\bA$ and the first column of $\bQ$.} This is similar to what we have claimed on the uniqueness of the QR decomposition (Corollary~\ref{corollary:unique-qr}, p.~\pageref{corollary:unique-qr}).

The next finding involves a Krylov matrix defined as follows:
\begin{definition}[Krylov Matrix\index{Krylov matrix}]\label{definition:krylov-matrix}
	Given matrix $\bA\in \real^{n\times n}$, a vector $\bq\in \real^n$, and a scalar $k$, the \textit{Krylov matrix} is defined to be
	$$
	\bK(\bA, \bq, k) =
	\begin{bmatrix}
		\bq & \bA\bq & \ldots & \bA^{k-1}\bq
	\end{bmatrix}
	\in \real^{n\times n}.
	$$
\end{definition}

\begin{theorem}[Reduced Hessenberg]\label{theorem:implicit-q-hessenberg-v2}
	Suppose there exists an orthogonal matrix $\bQ$ such that $\bA\in \real^{n\times n}$ can be factored as $\bA = \bQ\bH\bQ^\top$. Then $\bQ^\top\bA\bQ=\bH$ is an unreduced upper Hessenberg matrix if and only if $\bR=\bQ^\top \bK(\bA, \bq_1, n)$ is nonsingular and upper triangular where $\bq_1$ is the first column of $\bQ$.

		\item If $\bR$ is singular and $k$ is the smallest index so that $r_{kk}=0$, then $k$ is also the smallest index that $h_{k,k-1}=0$.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:implicit-q-hessenberg-v2}] We prove by forward implication and converse implication separately as follows:
\paragraph{Forward implication}
Suppose $\bH$ is unreduced, write out the following matrix
$$
\bR = \bQ^\top \bK(\bA, \bq_1, n) = [\be_1, \bH\be_1, \ldots, \bH^{n-1}\be_1],
$$
where $\bR$ is upper triangular with $r_{11}=1$ obviously. Observe that $r_{ii} = h_{21}h_{32}\ldots h_{i,i-1}$ for $i\in \{2,3,\ldots, n\}$. When $\bH$ is unreduced, $\bR$ is nonsingular as well.
\paragraph{Converse implication}
Now suppose $\bR$ is upper triangular and nonsingular, we observe that $\br_{k+1} = \bH\br_{k}$ such that the $(k+2:n)$-th rows of $\bH$ are zero and $h_{k+1,k}\neq 0$ for $k\in \{1,2,\ldots, n-1\}$. Then $\bH$ is unreduced.

If $\bR$ is singular and $k$ is the smallest index so that $r_{kk}=0$, then
$$
\left.
\begin{aligned}
r_{k-1,k-1}&=h_{21}h_{32}\ldots h_{k-1,k-2}&\neq 0 \\
r_{kk}&=h_{21}h_{32}\ldots h_{k-1,k-2} h_{k,k-1}&= 0
\end{aligned}
\right\}
\leadto
h_{k,k-1} =0,
$$
from which the result follows.
\end{proof}

\newpage
\chapter{Tridiagonal Decomposition: Hessenberg in Symmetric Matrices}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Tridiagonal Decomposition}
We firstly give the formal definition of the tridiagonal matrix.
\begin{definition}[Tridiagonal Matrix\index{Tridiagonal matrix}]\label{definition:tridiagonal-hessenbert}
	A tridiagonal matrix is a square matrix where all the entries below the lower sub-diagonal and the entries above the upper sub-diagonal are zeros. I.e., the tridiagonal matrix is a \textit{band matrix}.

	 The definition of the tridiagonal matrix can also be extended to rectangular matrices, and the form can be implied from the context.

	 In matrix language, for any matrix $\bT\in \real^{n\times n}$, and the entry ($i,j$) denoted by $t_{ij}$ for all $i,j\in \{1,2,\ldots, n\}$. Then $\bT$ with $t_{ij}=0$ for all $i\geq j+2$ and $i \leq j-2$ is known as a tridiagonal matrix.

	 Let $i$ denote the smallest positive integer for which $h_{i+1, i}=0$ where $i\in \{1,2,\ldots, n-1\}$, then $\bT$ is \textit{unreduced} if $i=n-1$.
\end{definition}

Take a $5\times 5$ matrix as an example, the lower triangular below the lower sub-diagonal and upper triangular above the upper sub-diagonal are zero in the tridiagonal matrix:
$$
\begin{sbmatrix}{possibly\,\, unreduced}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
	0 & \boxtimes & \boxtimes & \boxtimes & 0\\
	0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}
\qquad
\begin{sbmatrix}{reduced}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
	0 & \boxtimes & \boxtimes & \boxtimes & 0\\
	0 & 0 & \textcolor{blue}{0} & \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}.
$$
Obviously, a tridiagonal matrix is a special case of an upper Hessenberg matrix.
Then we have the following tridiagonal decomposition:
\begin{theoremHigh}[Tridiagonal Decomposition]\label{theorem:tridiagonal-decom}
	Every $n\times n$ symmetric matrix $\bA$ can be factored as
	$$
	\bA = \bQ\bT\bQ^\top \qquad \text{or} \qquad \bT = \bQ^\top \bA\bQ,
	$$
	where $\bT$ is a \textit{symmetric} tridiagonal matrix, and $\bQ$ is an orthogonal matrix.
\end{theoremHigh}
The existence of the tridiagonal matrix is trivial by applying the Hessenberg decomposition to symmetric matrix $\bA$.

\section{Computing the Tridiagonal Decomposition}\label{section:compute-tridiagonal}
Since zeros are now introduced in rows as well as columns by symmetry, additional
arithmetic can be avoided by ignoring these additional zeros.

An example of a $5\times 5$ matrix is shown as follows where $\boxtimes$ or a letter represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	\stackrel{\bH_1\times}{\rightarrow}
	&\begin{sbmatrix}{\bH_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\bm{a} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_1^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \bm{a} & \bm{0} & \bm{0} & \bm{0} \\
		a & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}\\
	\stackrel{\bH_2\times}{\rightarrow}
	&\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top}
		\boxtimes & a & 0 & 0 & 0 \\
		a & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \bm{b} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_2^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
		\boxtimes & a & 0 & 0 & 0 \\
		a & \boxtimes & \bm{b} & \bm{0} & \bm{0} \\
		0 & b & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}\\
	\stackrel{\bH_3\times}{\rightarrow}
	&\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
		\boxtimes & a & 0 & 0 & 0 \\
		a & \boxtimes & b & 0 & 0 \\
		0 & b & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \bm{c} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_3^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top\bH_3^\top}
		\boxtimes & a & 0 & 0 & 0 \\
		a & \boxtimes & b & 0 & 0 \\
		0 & b & \boxtimes & \bm{c} & \bm{0}\\
		0 & 0 & c & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix},
\end{aligned}
$$
\begin{algorithm}[h]
	\caption{Tridiagonal Decomposition via the Householder Reflector}
	\label{alg:tridiagonal-decomposition-householder}
	\begin{algorithmic}[1]
		\Require matrix $\bA$ with size $n\times n $;
		\State Initially set $\bT = \bA$;
		\For{$i=1$ to $n-2$}
		\State $\ba = \bT_{i+1:n,i}$, i.e., first column of $\bT_{i+1:n,i:n}\in \real^{(n-i)\times(n-i+1)}$;
		\State $r = ||\ba||$; \Comment{$2(n-i)$ flops}
		\State $\bu_i = \ba-r\be_1$; \Comment{1 flop}
		\State $\bu_i = \bu_i / ||\bu_i||$; \Comment{$3(n-i)$ flops}
		\State $\bT_{i+1,i}=r$, $\bT_{i+2:n,i}=\bzero$, i.e., set the value of first column of $\bT_{\textcolor{blue}{i+1}:n,i:n}$; \Comment{0 flops}
		\State  $\bT_{i,i+1}=r$, $\bT_{i,i+2:n}=\bzero$,\gap \, i.e., set the value of first row of $\bT_{i:n,\textcolor{blue}{i+1}:n}$; \Comment{0 flops}

		\State Left and Right: let $\bZ = \bT_{i+1:n,i+1:n} \in \real^{(n-i)\times (n-i)}$,
$$
\begin{aligned}
\bZ &= \bH_i \bZ \bH_i   \qquad \text{($\bH_i$ is the $i$-th Householder reflector)}\\
&= (\bI-2\bu_i\bu_i^\top)\bZ (\bI-2\bu_i\bu_i^\top)\\
&= \bZ - \bZ2\bu_i\bu_i^\top  - 2\bu_i\bu_i^\top\bZ+ 2\bu_i\bu_i^\top\bZ 2\bu_i\bu_i^\top \\
\end{aligned}
$$
\Comment{$4(n-i)^2$ flops}
		\EndFor
		\State Output $\bT$ as the tridiagonal matrix;
		\State Get $\bQ=\bH_1\bH_2\ldots\bH_{n-2}$, where $\bH_i$'s are Householder reflectors.
		\State Initially set $\bQ = \bH_1$;
		\For{$i=1$ to $n-3$}
		\State Compute $\bQ$:
		$$\begin{aligned}
			\bQ_{1:n,i+2:n} &= \bQ_{1:n,i+2:n}\bH_{i+1}\\
			 &= \bQ_{1:n,i+2:n}(\bI - 2\bu_{i+1}\bu_{i+1}^\top)\\
			&=\bQ_{1:n,i+2:n}-\bQ_{1:n,i+2:n}2\bu_{i+1}\bu_{i+1}^\top \in \real^{n\times (n-i-1)}\\
		\end{aligned}$$
	\Comment{$4n(n-i-1) - n$ flops}

		\EndFor
		\State Output $\bQ$ as the orthogonal matrix.
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: Tridiagonalization via Householder]\label{theorem:tridiagonal-householder}
	Algorithm~\ref{alg:tridiagonal-decomposition-householder} requires $\sim \frac{4}{3}n^3$ flops to compute a Tridiagonal decomposition of an $n\times n$ symmetric matrix. Further, if $\bQ$ is needed explicitly, additional $\sim 2n^3$ flops required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:tridiagonal-householder}]
The complexity of step 9 needs close scrutiny. It follows from the left Householder and right Householder updates. Since we have computed the first column of $\bT_{\textcolor{blue}{i+1}:n,i:n}$ and the first row of $\bT_{i:n,\textcolor{blue}{i+1}:n}$ explicitly, the left and right Householder updates have the following form (where the \textcolor{red}{red}-colored text is the difference from the one in computing Hessenberg decomposition for non-symmetric matrices):
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={Where Does the Step 9 Come From: A Splitting Way}]
 Left: update columns $2$ to $n$ of $\bT_{\textcolor{blue}{i+1}:n,i:n}$, i.e., working on $\bT_{i+1:n,i+1:n}\in \real^{(n-i)\times (n-i)}$:
$$
\begin{aligned}
	\bT_{i+1:n,i+1:n} &= (\bI-2\bu_i\bu_i^\top)\bT_{i+1:n,i+1:n} \\
	&= \bT_{i+1:n,i+1:n} - 2\bu_i\bu_i^\top\bT_{i+1:n,i+1:n} \in \real^{(n-i)\times(n-i)}\\
	&\text{\qquad\qquad($4(n-i)^2$ flops);}
\end{aligned}
$$
Right: update rows $2$ to $n$ of $\bT_{i:n,\textcolor{blue}{i+1}:n}$, i.e., working on $\bT_{i+1:n,i+1:n}\in \real^{(n-i)\times (n-i)}$:
$$
\begin{aligned}
	\bT_{\textcolor{red}{i+1}:n,i+1:n} &= \bT_{\textcolor{red}{i+1}:n,i+1:n}(\bI-2\bu_i\bu_i^\top) \\
	&=\bT_{\textcolor{red}{i+1}:n,i+1:n} - \bT_{\textcolor{red}{i+1}:n,i+1:n} 2\bu_i\bu_i^\top\in \real^{n\times(n-i)}\\
	&\text{\qquad\,\,\,\,\,($4n(n-i)-n$ flops);}
\end{aligned}
$$
\end{mdframed}
Since the two updates now working on the same submatrix $\bT_{i+1:n,i+1:n}$, we can put them together which results in the step 9 of the algorithm. Let $\bZ = \bT_{i+1:n,i+1:n} \in \real^{(n-i)\times (n-i)}$. A delve into the reduction of the complexity in step 9 goes as follows:
$$
\begin{aligned}
\bZ &\leftarrow  \bH_i \bZ \bH_i   \qquad  \text{($\bH_i$ is the $i$-th Householder reflector)}\\
&= (\bI-2\bu_i\bu_i^\top)\bZ (\bI-2\bu_i\bu_i^\top)\\
&= (\bZ - 2\bu_i\underbrace{\bu_i^\top \bZ}_{\by^\top })(\bI-2\bu_i\bu_i^\top)  \\
&= (\bZ - 2\bu_i\by^\top )(\bI-2\bu_i\bu_i^\top)  & \text{(let $\by^\top = \bu_i^\top \bZ\rightarrow \by = \bZ\bu_i$)}\\
&= \bZ - 2\bu_i\by^\top - 2\bZ\bu_i\bu_i^\top + 4\beta \bu_i\bu_i^\top & \text{(let $\beta = \by^\top\bu_i$)}\\
&= \bZ - (2\bu_i\by^\top+2\beta \bu_i\bu_i^\top)  - (2\underbrace{\bZ\bu_i}_{\by}\bu_i^\top -2\beta \bu_i\bu_i^\top )  \\
&= \bZ - (2\bu_i\by^\top+2\beta \bu_i\bu_i^\top)  - (2\by\bu_i^\top -2\beta \bu_i\bu_i^\top )  &\text{($\by = \bZ\bu_i$)}\\
&= \bZ - \{\bu_i\underbrace{(2\by^\top+ 2\beta \bu_i^\top)}_{\bx^\top }  +\underbrace{ (2\by^\top+2\beta \bu_i^\top)^\top}_{\bx } \bu_i^\top   \}\\
&= \bZ - \{\bu_i\bx^\top  +( \bu_i\bx^\top)^\top   \}   &(\text{let $\bx=(2\by+ 2\beta \bu_i)$})
\end{aligned}
$$
The costs come from
\begin{itemize}
\item $\by = \bZ\bu_i$: $[2(n-i)-1](n-i) = 2(n-i)^2 - (n-i)$ flops from Lemma~\ref{lemma:matrix-multi-complexity} (on the complexity of a matrix multiplication);
\item $\beta = \by^\top\bu_i$: $2(n-i)-1$ flops;
\item  $\bx=(2\by+ 2\beta \bu_i)$: $(n-i)+1+(n-i)=2(n-i)+1$ flops;
\item $\bu_i\bx^\top$: $(n-i)^2$ flops;
\item $( \bu_i\bx^\top)^\top $: 0 flops;
\item $\bu_i\bx^\top  +( \bu_i\bx^\top)^\top$: $1+2+\ldots+(n-i)= \frac{(n-i)^2+(n-i)}{2}$ additions since it results in a symmetric matrix;
\item $\underbrace{\bZ}_{\text{symmetric}} - \underbrace{\{\bu_i\bx^\top  +( \bu_i\bx^\top)^\top   \}}_{\text{symmetric}} $: $1+2+\ldots+(n-i)= \frac{(n-i)^2+(n-i)}{2}$ subtractions since both matrices are symmetric;
\end{itemize}
If we keep only the leading terms of step 4 to step 9, the total complexity for loop $i$ is given by
$f(i) = 4(n-i)^2$ flops. By summation, we find the final cost
$$
\text{cost=} f(1)+f(2)+\ldots +f(n-2)=~ \frac{4}{3}n^3 \text{ flops}.
$$
This completes the proof.
\end{proof}

\section{Properties of the Tridiagonal Decomposition}\label{section:tridiagonal-decomposition}
Similarly, the tridiagonal decomposition is not unique. However, and most importantly, if we restrict the elements in the lower sub-diagonal of the tridiagonal matrix $\bT$ to be positive (if possible), then the tridiagonal decomposition $\bA=\bQ\bT\bQ^\top$ is uniquely determined by $\bA$ and the first column of $\bQ$.
\begin{theorem}[Implicit Q Theorem for Tridiagonal\index{Implicit Q theorem}]\label{theorem:implicit-q-tridiagonal}
	Suppose two Tridiagonal decompositions of symmetric matrix $\bA\in \real^{n\times n}$ are given by $\bA=\bU\bT\bU^\top=\bV\bG\bV^\top$ where $\bU=[\bu_1, \bu_2, \ldots, \bu_n]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU,\bV$. Suppose further that $k$ is the smallest positive integer for which $t_{k+1,k}=0$ where $t_{ij}$ is the entry $(i,j)$ of $\bT$. Then
	\begin{itemize}
		\item if $\bu_1=\bv_1$, then $\bu_i = \pm \bv_i$ and $|t_{i,i-1}| = |g_{i,i-1}|$ for $i\in \{2,3,\ldots,k\}$.
		\item When $k=n-1$, the tridiagonal matrix $\bT$ is known as unreduced. However, if $k<n-1$, then $g_{k+1,k}=0$.
	\end{itemize}
\end{theorem}
From the above theorem, we observe that  if we restrict the elements in the lower sub-diagonal of the tridiagonal matrix $\bT$ to be positive (if possible), i.e., \textit{unreduced}, then the tridiagonal decomposition $\bA=\bQ\bT\bQ^\top$ is uniquely determined by $\bA$ and the first column of $\bQ$. This again is similar to what we have claimed on the uniqueness of the QR decomposition (Corollary~\ref{corollary:unique-qr}, p.~\pageref{corollary:unique-qr}).

Similarly, a reduced tridiagonal decomposition can be obtained from the implication of the Krylov matrix (Definition~\ref{definition:krylov-matrix}, p.~\pageref{definition:krylov-matrix}).
\begin{theorem}[Reduced Tridiagonal]\label{theorem:implicit-q-tridiagonal-v2}
	Suppose there exists an orthogonal matrix $\bQ$ such that $\bA\in \real^{n\times n}$ can be factored as $\bA = \bQ\bT\bQ^\top$. Then $\bQ^\top\bA\bQ=\bT$ is an unreduced tridiagonal matrix if and only if $\bR=\bQ^\top \bK(\bA, \bq_1, n)$ is nonsingular and upper triangular where $\bq_1$ is the first column of $\bQ$.

	\item If $\bR$ is singular and $k$ is the smallest index so that $r_{kk}=0$, then $k$ is also the smallest index that $t_{k,k-1}=0$.
\end{theorem}
The proofs of the above two theorems are the same as those in Theorem~\ref{theorem:implicit-q-hessenberg} and~\ref{theorem:implicit-q-hessenberg-v2} (p.~\pageref{theorem:implicit-q-hessenberg} and p.~\pageref{theorem:implicit-q-hessenberg-v2}).

\newpage
\chapter{Bidiagonal Decomposition}\label{section:bidiagonal-decompo}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Bidiagonal Decomposition}
We firstly give the rigorous definition of the upper Bidiagonal matrix as follows:
\begin{definition}[Upper Bidiagonal Matrix\index{Bidiagonal matrix}]\label{definition:bidiagonal-matrix}
	An upper bidiagonal matrix is a square matrix which  is a banded matrix with non-zero entries along the \textit{main diagonal} and the \textit{upper subdiagonal} (i.e., the ones above the main diagonal). This means there are exactly two nonzero diagonals in the matrix.

	 Furthermore, when the diagonal below the main diagonal has the non-zero entries, the matrix is lower bidiagonal.

	 The definition of bidigonal matrices can also be extended to rectangular matrices, and the form can be implied from the context.
\end{definition}

Take a $7\times 5$ matrix as an example, the lower triangular below the main diagonal and the upper triangular above the upper subdiagonal are zero in the upper bidiagonal matrix:
$$
\begin{bmatrix}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	0 & \boxtimes & \boxtimes & 0 & 0\\
	0 & 0 & \boxtimes & \boxtimes & 0\\
	0 & 0 & 0 & \boxtimes & \boxtimes\\
	0 & 0 & 0 & 0 & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
	0 & 0 & 0 & 0 & 0
\end{bmatrix}.
$$
Then we have the following bidiagonal decomposition:
\begin{theoremHigh}[Bidiagonal Decomposition]\label{theorem:Golub-Kahan-Bidiagonalization-decom}
	Every $m\times n$ matrix $\bA$ can be factored as
	$$
	\bA = \bU\bB\bV^\top \qquad \text{or} \qquad \bB= \bU^\top \bA\bV,
	$$
	where $\bB$ is an upper bidiagonal matrix, and $\bU, \bV$ are orthogonal matrices.
\end{theoremHigh}
We will see the bidiagonalization resembles the form of a singular value decomposition where the only difference is the values of $\bB$ in bidiagonal form has nonzero entries on the upper sub-diagonal such that it will be shown to play an important role in the calculation of the singular value decomposition.

\section{Existence of the Bidiagonal Decomposition: Golub-Kahan Bidiagonalization}
Previously, we utilized a Householder reflector to triangularize matrices and introduce zeros below the diagonal to obtain the QR decomposition, and introduce zeros below the sub-diagonal to obtain the Hessenberg decomposition. A similar approach can be employed to find the bidiagonal decomposition. To see this, you have to recap the ideas behind the Householder reflector in Definition~\ref{definition:householder-reflector} (p.~\pageref{definition:householder-reflector}).\index{Golub-Kahan}

\subsubsection*{\textbf{First Step 1.1: Introduce Zeros for the First Column}}
Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ be the column partitions of $\bA$, and each $\ba_i \in \real^{m}$.
We can construct the Householder reflector as follows:
$$
r_1 = ||\ba_1||, \qquad \bu_1 = \frac{\ba_1 - r_1 \be_1}{||\ba_1 - r_1 \be_1||} ,\qquad \text{and}\qquad \bH_1 = \bI - 2\bu_1\bu_1^\top \in \textcolor{blue}{\real^{m\times m}},
$$
where $\be_1$ here is the first basis for $\textcolor{blue}{\real^{m}}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{m}}$.
In this case, $\bH_1\bA$ will introduce zeros in the first column of $\bA$ below entry (1,1), i.e., reflect $\ba_1$ to $r_1\be_1$.
We can easily verify that both $\bH_1$ is a symmetric and orthogonal matrix (from the definition of Householder reflector).

An example of a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bH_1\times}{\rightarrow}
&\begin{sbmatrix}{\bH_1\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}.
\end{aligned}
$$
Till now, this is exactly what we have done in the QR decomposition via the Householder reflector in Section~\ref{section:qr-via-householder} (p.~\pageref{section:qr-via-householder}).
Going further, to introduce zeros above the upper sub-diagonal of $\bH_1\bA$ is equivalent to introducing zeros below the lower subdiagonal of $(\bH_1\bA)^\top$.

\subsubsection*{\textbf{First Step 1.2: Introduce Zeros for the First Row}}

Now suppose we are looking at the \textit{transpose} of $\bH_1\bA$, that is $(\bH_1\bA)^\top =\bA^\top\bH_1^\top \in \real^{n\times m}$ and the column partition is given by $\bA^\top\bH_1^\top = [\bz_1, \bz_2, \ldots, \bz_m]$ where each $\bz_i \in \real^n$.
Suppose $\bar{\bz}_1, \bar{\bz}_2, \ldots, \bar{\bz}_m \in \real^{n-1}$ are vectors removing the first component in $\bz_i$'s.
Let
$$
r_1 = ||\bar{\bz}_1||, \qquad \bv_1 = \frac{\bar{\bz}_1 - r_1 \be_1}{||\bar{\bz}_1 - r_1 \be_1||},  \qquad \text{and}\qquad \widetilde{\bL}_1 = \bI - 2\bv_1\bv_1^\top \in\textcolor{blue}{\real^{(n-1)\times (n-1)}},
$$
where $\be_1$ now is the first basis for $\textcolor{blue}{\real^{n-1}}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{n-1}}$. To introduce zeros below the sub-diagonal and operate on the submatrix $(\bA^\top\bH_1^\top)_{2:n,1:m}$, we append the Householder reflector into
$$
\bL_1 = \begin{bmatrix}
	1 &\bzero \\
	\bzero & \widetilde{\bL}_1
\end{bmatrix},
$$
in which case, $\bL_1(\bA^\top\bH_1^\top)$ will introduce zeros in the first column of $(\bA^\top\bH_1^\top)$ below entry (2,1), i.e., reflect $\bar{\bz}_1$ to $r_1\be_1$. The first row of $(\bA^\top\bH_1^\top)$ will not be affected at all and kept unchanged by Remark~\ref{remark:left-right-identity} (p.~\pageref{remark:left-right-identity}) such that the zeros introduced in Step 1.1) will be kept. And we can easily verify that both $\bL_1$ and $\widetilde{\bL}_1$ are orthogonal matrices and they are symmetric (from the definition of Householder reflector).

Come back to the original \textit{untransposed} matrix $\bH_1\bA$, multiply on the right by $\bL_1^\top$ is to introduce zeros in the first row to the right of entry (1,2).
Again, following the example above, a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bA^\top\bH_1^\top}
	\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{sbmatrix}
\stackrel{\bL_1\times}{\rightarrow}
\begin{sbmatrix}{\bL_1 \bA^\top\bH_1^\top}
\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}
\stackrel{(\cdot)^\top}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA\bL_1^\top }
	\boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0} \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}.
\end{aligned}
$$

In short, $\bH_1\bA\bL_1^\top$ finishes the first step to introduce zeros for the first column and the first row of $\bA$.

\subsubsection*{\textbf{Second Step 2.1: Introduce Zeros for the Second Column}}
Let $\bB = \bH_1\bA\bL_1^\top$, where the entries in the first column below entry (1,1) are all zeros and the entries in the first row to the right of entry (1,2) are all zeros as well.
And the goal is to introduce zeros in the second column below entry (2,2).
Let $\bB_2 = \bB_{2:m,2:n}=[\bb_1, \bb_2, \ldots, \bb_{n-1}] \in \real^{(m-1)\times (n-1)}$.
We can again construct a Householder reflector
$$
r_1 = ||\bb_1||,\qquad	\bu_2 = \frac{\bb_1 - r_1 \be_1}{||\bb_1 - r_1 \be_1||}, \qquad  \text{and}\qquad \widetilde{\bH}_2 = \bI - 2\bu_2\bu_2^\top\in \textcolor{blue}{\real^{(m-1)\times (m-1)}},
$$
where $\be_1$ now is the first basis for $\textcolor{blue}{\real^{m-1}}$ i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{m-1}}$. To introduce zeros below the main diagonal and operate on the submatrix $\bB_{2:m,2:n}$, we append the Householder reflector into
$$
\bH_2 = \begin{bmatrix}
	1 &\bzero \\
	\bzero & \widetilde{\bH}_2
\end{bmatrix},
$$
in which case, we can see that $\bH_2(\bH_1\bA\bL_1^\top)$ will not change the first  row of $(\bH_1\bA\bL_1^\top)$ by Remark~\ref{remark:left-right-identity} (p.~\pageref{remark:left-right-identity}), and as the Householder cannot reflect a zero vector such that the zeros in the first column will be kept as well.

Following the above example, a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bH_1\bA\bL_1^\top }
\boxtimes & \boxtimes & 0 & 0 & 0 \\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{sbmatrix}
\stackrel{\bH_2\times }{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top }
	\boxtimes & \boxtimes & 0& 0 & 0 \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}.
\end{aligned}
$$

\subsubsection*{\textbf{Second Step 2.2: Introduce Zeros for the Second Row}}
Same as step 1.2), now suppose we are looking at the \textit{transpose} of $\bH_2\bH_1\bA\bL_1^\top$, that is $(\bH_2\bH_1\bA\bL_1^\top)^\top =\bL_1\bA^\top\bH_1^\top\bH_2^\top \in \real^{n\times m}$ and the column partition is given by $\bL_1\bA^\top\bH_1^\top\bH_2^\top = [\bx_1, \bx_2, \ldots, \bx_m]$ where each $\bx_i \in \real^n$.
Suppose $\bar{\bx}_1, \bar{\bx}_2, \ldots, \bar{\bx}_m \in \real^{n-2}$ are vectors removing the first two components in $\bx_i$'s.
Construct the Householder reflector as follows:
$$
 r_1 = ||\bar{\bx}_1||,\qquad \bv_2 = \frac{\bar{\bx}_1 - r_1 \be_1}{||\bar{\bx}_1 - r_1 \be_1||}, \qquad \text{and}\qquad \widetilde{\bL}_2 = \bI - 2\bv_2\bv_2^\top \in \textcolor{blue}{\real^{(n-2)\times (n-2)}},
$$
where $\be_1$ now is the first basis for $\textcolor{blue}{\real^{n-2}}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{n-2}}$. To introduce zeros below the sub-diagonal and operate on the submatrix $(\bL_1\bA^\top\bH_1\bH_2)_{3:n,1:m}$, we append the Householder reflector into
$$
\bL_1 = \begin{bmatrix}
	\bI_2 &\bzero \\
	\bzero & \widetilde{\bL}_2
\end{bmatrix},
$$
where $\bI_2$ is a $2\times 2$ identity matrix.
In this case, $\bL_2(\bL_1\bA^\top\bH_1^\top\bH_2^\top)$ will introduce zeros in the second column of $(\bL_1\bA^\top\bH_1^\top\bH_2^\top)$ below entry (3,2). The first two rows of $(\bL_1\bA^\top\bH_1^\top\bH_2^\top)$ will not be affected at all and kept unchanged by Remark~\ref{remark:left-right-identity} (p.~\pageref{remark:left-right-identity}). \textbf{Further, the first column of it will be kept unchanged as well}. And we can easily verify that both $\bL_1$ and $\widetilde{\bL}_1$ are orthogonal matrices and they are symmetric (from the definition of Householder reflector).

Come back to the original \textit{untransposed} matrix $\bH_2\bH_1\bA\bL_1^\top$, multiply on the right by $\bL_2^\top$ is to introduce zeros in the second row to the right of entry (2,3).  Following the above example, a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bL_1\bA^\top\bH_1^\top\bH_2^\top }
	\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\boxtimes & \boxtimes & 0 & 0 & 0 & 0 & 0\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{sbmatrix}
	\stackrel{\bL_2\times }{\rightarrow}
	\begin{sbmatrix}{\bL_2\bL_1\bA^\top\bH_1^\top\bH_2^\top }
	\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\boxtimes & \boxtimes & 0 & 0 & 0 & 0 & 0\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\end{sbmatrix}
\stackrel{(\cdot)^\top }{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top\bL_2^\top}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	0 & \boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} \\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}.\\
\end{aligned}
$$

In short, $\bH_2(\bH_1\bA\bL_1^\top)\bL_2^\top$ finish the second step to introduce zeros for the second column and the second row of $\bA$.
Same process can go on, and we shall notice that there are $n$ such $\bH_i$ Householder reflectors on the left and $n-2$ such $\bL_i$ Householder reflectors on the right (suppose $m>n$ for simplicity). The interleaved Householder factorization is known as the \textit{Golub-Kahan Bidiagonalization} \citep{golub1965calculating}. We will finally bidiagonalize
$$
\bB = \bH_{n} \bH_{n-1}\ldots\bH_1 \bA\bL_1^\top\bL_2^\top\ldots\bL_{n-2}^\top.
$$
And since the $\bH_i$'s and $\bL_i$'s are symmetric and orthogonal, we have
$$
\bB =\bH_{n} \bH_{n-1}\ldots\bH_1 \bA\bL_1\bL_2\ldots\bL_{n-2}.
$$
A full example of a $7\times 5$ matrix is shown as follows where again $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={A Complete Example of Golub-Kahan Bidiagonalization}]
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bH_1\times}{\rightarrow}
&\begin{sbmatrix}{\bH_1\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}
\stackrel{\times\bL_1^\top}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA\bL_1^\top}
\boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0} \\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}\\
\stackrel{\bH_2\times}{\rightarrow}
&\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bL_2^\top}{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top\bL_2^\top}
\boxtimes & \boxtimes & 0 & 0 & 0\\
0 & \boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} \\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\stackrel{\bH_3\times}{\rightarrow}
&\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2^\top}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bL_3^\top}{\rightarrow}
\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2^\top\bL_3^\top}
\boxtimes & \boxtimes & 0 & 0 & 0\\
\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
0 & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{0}\\
0 & 0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\stackrel{\bH_4\times}{\rightarrow}
&\begin{sbmatrix}{\bH_4\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2\bL_3^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes& \boxtimes & \boxtimes\\
	0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{0} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{0} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{0} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bH_5\times}{\rightarrow}
\begin{sbmatrix}{\bH_5\bH_4\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2\bL_3^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes& \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes\\
	0 & 0 & 0 & 0 & \bm{\boxtimes}\\
	0 & 0 & 0 & 0 & \bm{0}\\
	0 & 0 & 0 & 0 & \bm{0}
\end{sbmatrix}.
\end{aligned}
$$
\end{mdframed}
We present in a way where a right Householder reflector $\bL_i$ follows from a left one $\bH_i$. However, a trivial error that might be employed is that we do the left ones altogether, and the right ones follow. That is, a bidiagonal decomposition is a combination of a QR decomposition and a Hessenberg decomposition. Nevertheless, this is problematic, the right Householder reflector $\bL_1$ will destroy the zeros introduced by the left ones. Therefore, the left and right reflectors need to be employed in an interleaved manner to introduce back the zeros.

\section{Computing the Bidiagonal Decomposition: Golub-Kahan Bidiagonalization}

\begin{algorithm}[H]
	\caption{Golub-Kahan Bidiagonal Decomposition}
	\label{alg:bidiagonal-decomposition-householder}
	\begin{algorithmic}[1]
		\Require matrix $\bA$ with size $m\times n $ with $m>n$;
		\State Initially set $\bB = \bA^\top$;
		\For{$i=1$ to $n-2$}
		\State \textcolor{winestain}{// do the left Householder reflector;}
		\If{$i<=n-2$}
		\State \textcolor{black}{$\bB = \bB^\top$ such that $\bB\in \textcolor{blue}{\real^{m\times n}}$;}
		\EndIf
		\State \textcolor{black}{$\ba = \bB_{i:m,i}$, i.e., first column of $\bB_{i:m,i:n} \in \real^{(m-i+1)\times(n-i+1)}$;}
		\State $r = ||\ba||$; \Comment{$2(m-i+1)$ flops}
		\State $\bu_i = \ba-r\be_1 \in \real^{m-i+1}$; \Comment{1 flop}
		\State $\bu_i = \bu_i / ||\bu_i||$; \Comment{$3(m-i+1)$ flops}
		\State $\bB_{i,i} = r$, $\bB_{i+1:m,i}=\bzero$; \Comment{update the first column of $\bB_{i:m,i:n}$, 0 flops}
		\State $\bB_{i:m,i+1:n} = \bB_{i:m,i+1:n} - 2\bu_i (\bu_i^\top \bB_{i:m,i+1:n})$; \Comment{update the rest columns of $\bB_{i:m,i:n}$, $4(m-i+1)(n-i) +(m-n+1)$ flops}
		\If{$i<=n-2$} \textcolor{winestain}{// do the right Householder reflector;}
		\State $\bB = \bB^\top$ such that $\bB\in \textcolor{blue}{\real^{n\times m}}$;
\State $\bz = \bB_{i+1:n,i}$, i.e., first column of $\bB_{i+1:n,i:\textcolor{red}{m}}\in \real^{(n-i)\times(\textcolor{red}{m}-i+1)}$;
\State $s = ||\bz||$; \Comment{$2(n-i)$ flops}
\State $\bv_i = \bz-s\be_1 \in \real^{n-i}$; \Comment{1 flop}
\State $\bv_i = \bv_i / ||\bv_i||$; \Comment{$3(n-i)$ flops}
\State $\bB_{i+1,i}=r$, $\bB_{i+2:n,i}=\bzero$; \Comment{update the first column of $\bB_{i+1:n,i:\textcolor{red}{m}}$, 0 flops}
\State update columns $i+1$ to $m$ of $\bB_{i+1:n,i:\textcolor{red}{m}}$:
$$
\begin{aligned}
	 \bB_{i+1:n,i+1:\textcolor{red}{m}} &= (\bI-2\bv_i\bv_i^\top)\bB_{i+1:n,i+1:\textcolor{red}{m}}\\
	 &= \bB_{i+1:n,i+1:\textcolor{red}{m}} - 2\bv_i(\bv_i^\top\bB_{i+1:n,i+1:\textcolor{red}{m}}) \in \real^{(n-i)\times(\textcolor{red}{m}-i)}\\
	 & \text{\gap($4(n-i)(m-i) +(n-m)$ flops)}
\end{aligned}
$$
\EndIf
		\EndFor
		\State Output $\bB$ as the bidiagonal matrix;
		\State Get $\bU=\bH_1 \bH_2\ldots\bH_n$;
		\State Initially set $\bU = \bH_1$;
		\For{$i=1$ to $n-1$}
		\State $\bU_{1:m,i+1:m} = \bU_{1:m,i+1:m}(\bI - 2\bu_{i+1}\bu_{i+1}^\top)=\bU_{1:m,i+1:m}-\bU_{1:m,i+1:m}2\bu_{i+1}\bu_{i+1}^\top$.
		\EndFor
		\State Initially set $\bV = \bL_1$;
		\For{$i=1$ to $n-3$}
		\State $\bV_{1:n,i+2:n} = \bV_{1:n,i+2:n}(\bI - 2\bv_{i+1}\bv_{i+1}^\top)=\bV_{1:n,i+2:n}-\bV_{1:n,i+2:n}2\bv_{i+1}\bv_{i+1}^\top \in \real^{n\times (n-i-1)}$;
		\EndFor
		\State Output $\bU$, $\bV$ as the orthogonal matrix;
	\end{algorithmic}
\end{algorithm}

From the QR decomposition and Hessenberg decomposition via the Householder reflector, it is trivial to obtain the procedure formulated in Algorithm~\ref{alg:bidiagonal-decomposition-householder} where the red-colored text is the difference between the right Householder reflectors
 in bidiagonal decomposition and the reflectors in Hessenberg decomposition (Algorithm~\ref{alg:hessenbert-decomposition-householder}).
\begin{theorem}[Algorithm Complexity: Golub-Kahan Bidiagonalization]\label{theorem:bidiagonal-full-householder}
	Algorithm~\ref{alg:bidiagonal-decomposition-householder} requires $\sim 4mn^2-\frac{4}{3}n^3$ flops to compute a bidiagonal decomposition of an $m\times n$ matrix with $m>n$. Further, if $\bU, \bV$ are needed explicitly, additional $\sim 4m^2n-2mn^2 + 2n^3$ flops are required.
\end{theorem}
The proof is trivial that the procedure shown above requires twice the complexity of QR decomposition via the Householder reflector as it resembles two Householder QR decomposition interleaved with one operating on the $m\times n$ matrix $\bA$ and the other on the $n\times m$ matrix $\bA^\top$. The complexity to obtain the orthogonal matrix $\bU$ is $4m^2n-2mn^2$ flops which is the same as that in QR decomposition. And similarly, the complexity to obtain the orthogonal matrix $\bV$ is $2n^3$ flops which is the same as that in Hessenberg decomposition.

\section{Computing the Bidiagonal Decomposition: LHC Bidiagonalization}
We mentioned in the previous section that the left Householder reflectors and the right reflectors are applied in an interleaved manner, otherwise, zeros introduced by the left reflectors will be destroyed. Nevertheless, when $m\gg n$, we can extract the square triangular matrix (i.e., the QR decomposition) and apply the Golub-Kahan diagonalization on the square $n\times n$ matrix. This is known as the \textit{Lawson-Hanson-Chan (LHC) bidiagonalization} \citep{lawson1995solving, chan1982improved} and the procedure is shown in Figure~\ref{fig:lhc-bidiagonal}.\index{LHC bidiagonalization}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\textwidth]{imgs/LHC-bidiagonal.pdf}
	\caption{Demonstration of LHC-bidiagonalization of a matrix}
	\label{fig:lhc-bidiagonal}
\end{figure}
The LHC bidiagonalization starts by computing the QR decomposition $\bA = \bQ\bR$. Then follows by applying the Golub-Kahan process such that $\widetilde{\bR} = \widetilde{\bU} \widetilde{\bB} \bV^\top$ where $\widetilde{\bR}$ is the square $n\times n$ triangular submatrix inside $\bR$. Append $\widetilde{\bU}$ into
$$
\bU_0 =
\begin{bmatrix}
	\widetilde{\bU} & \\
	& \bI_{m-n}
\end{bmatrix},
$$
which results in $\bR=\bU_0\bB \bV^\top$ and $\bA = \bQ\bU_0\bB \bV^\top$. Let $\bU=\bQ\bU_0$, we obtain the bidiagonal decomposition. The QR decomposition requires $2mn^2-\frac{2}{3}n^3$ flops and the Golub-Kahan process now requires $\frac{8}{3}n^3$ (operating on an $n\times n$ submatrix). Thus the total complexity to obtain the bidiagonal matrix $\bB$ is
$$
\text{LHC bidiagonalization:   } \sim 2mn^2 + 2n^3 \text{  flops}.
$$
 The LHC process creates zeros and then destroys them again in the lower triangle of the upper $n\times n $ square of $\bR$, but the zeros in the lower $(m-n)\times n$ rectangular matrix of $\bR$ will be kept. Thus when $m-n$ is large enough (or $m\gg n$), there is a net gain. Simple calculations will show the LHC bidiagonalization costs less when $m>\frac{5}{3}n$ compared to the Golub-Kahan bidiagonalization.

\section{Computing the Bidiagonal Decomposition: Three-Step Bidiagonalization}
The LHC procedure is advantageous only when $m>\frac{5}{3}n$. A further trick is to apply the QR decomposition not at the beginning of the computation, but at a suitable point in the middle \citep{trefethen1997numerical}. In particular, the procedure is shown in Figure~\ref{fig:lhc-bidiagonal2} where we apply the first $k$ steps of left and right Householder reflectors as in the Golub-Kahan process leaving the bottom-right $(m-k)\times(n-k)$ submatrix ``unreflected". Then follow up the same LHC process on the submatrix to obtain the final bidiagonal decomposition. By doing so, the complexity reduces when $n<m<2n$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\textwidth]{imgs/LHC-bidiagonal2.pdf}
	\caption{Demonstration of Three-Step bidiagonalization of a matrix}
	\label{fig:lhc-bidiagonal2}
\end{figure}

The complexity of the Three-Step bidiagonalization can be decomposed into three ones.
The complexity of $k$ loops in Algorithm~\ref{alg:bidiagonal-decomposition-householder} can be shown to be
$$
\text{Step 1:  } \qquad f_1 = 8mnk - (4m+4n)k^2 + \frac{8}{3}k^3 \text{  flops},
$$
which makes $4mn^2 - \frac{8}{3}n^3$ flops when $k=n$.
The complexity of the QR decomposition via the Householder reflector for $\widetilde{\bR} \in \real^{(m-k)\times (n-k)}$ is
$$
\text{Step 2:  }\qquad f_2 = 2(m-k)(n-k)^2 - \frac{2}{3} (n-k)^3\text{  flops}.
$$
And the complexity of the Golub-Kahan diagonalization for $\widetilde{\bT} \in \real^{(n-k)\times (n-k)}$ is
$$
\text{Step 3:  }\qquad f_3 = \frac{8}{3}(n-k)^3\text{  flops}.
$$
Thus, the total complexity of the three steps is given by
$$
g(k) =f_1 +f_2+f_3 = -\frac{4}{3}k^3 + (6n-2m)k^2  +(4mn-8n^2)k + 2mn^2+2n^3  .
$$
The problem now becomes finding a $k<n$ such that $g(k)$ is minimized. Taking the gradient of above function, we obtain
$$
g^\prime(k) = -4k^2 +(12n-4m)k + (4mn-8n^2),
$$
of which the root is $k=n$ or $2n-m$. We notice that $0<k<n$ such that when $2n-m>0$, the optimal value appears in one of $\{0, n, 2n-m\}$. Trivial calculation shows that
the optimal value for $k$ is $k=2n-m$ and the final complexity is now reduced to
$$
g(2n-m) =  2mn^2 + 2m^2n -\frac{2}{3}m^3 -\frac{2}{3}n^3 \,\, \text{ flops}, \qquad  \text{when $m<2n$}.
$$

An example of the complexity of the Three-Step method when $n=70, m=100$ is shown in Figure~\ref{fig:bidiagonal-gk-sample} where the roots of the gradient are found to be $2n-m=40$ and $n=70$ such that $g^\prime(40)=g^\prime(70)=0$. In this specific case, the function $g(k)$ is decreasing when $k\in (0, 2n-m]$ and increasing when $k\in (2n-m, n]$.

\noindent
\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[$g(k)$]{\label{fig:bidiagonal-gk-sample1}
		\includegraphics[width=0.475\linewidth]{./imgs/bidiagonal-gk-sample.pdf}}
	\quad
	\subfigure[$g^\prime(k)$]{\label{fig:bidiagonal-gk-sample2}
		\includegraphics[width=0.475\linewidth]{./imgs/bidiagonal-gk-sample2.pdf}}
	\caption{An example of the complexity when $n=70, m=100$.}
	\label{fig:bidiagonal-gk-sample}
\end{figure}

To conclude, the costs of the three methods are shown as follows:
$$
\left\{
\begin{aligned}
	&\text{Golub-Kahan: }  \sim 4mn^2-\frac{4}{3}n^3 \,\, \text{ flops},   \\
	&\text{LHC: } \sim 2mn^2 + 2n^3  \,\, \text{ flops},  \\
	&\text{Three-Step: } \sim 2mn^2 + 2m^2n -\frac{2}{3}m^3 -\frac{2}{3}n^3 \,\, \text{ flops} .
\end{aligned}
\right.
$$
When $m>2n$, LHC is preferred; when $n<m<2n$, the Three-Step method is preferred though the improvement is small enough as shown in Figure~\ref{fig:bidiagonal-loss-compare} where the operation counts for the three methods are plotted as a function of $\frac{m}{n}$.

\begin{SCfigure
	\centering
	\includegraphics[width=0.6\textwidth]{imgs/bidiagonal-loss.pdf}
	\caption{Comparison of the complexity among the three bidiagonal methods. When $m>2n$, LHC is preferred; when $n<m<2n$, the Three-Step method is preferred though the improvement is small enough.}
	\label{fig:bidiagonal-loss-compare}
\end{SCfigure}
Notice that the complexity discussed here does not involve the extra computation of $\bU, \bV$. We shall not discuss the issue for simplicity.

\section{Connection to Tridiagonal Decomposition}

We fist illustrate the connection by the following lemma that reveals how to construct a tridiagonal matrix from a bidiagonal one.
\begin{lemma}[Construct Tridiagonal From Bidiagonal]\label{lemma:construct-triangular-from-bidia}
Suppose $\bB\in \real^{n\times n}$ is upper bidiagonal, then $\bT_1=\bB^\top\bB$ and $\bT_2=\bB\bB^\top$ are \textit{symmetric} triangular matrices.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:construct-triangular-from-bidia}]
Suppose $\bB$ has the following form
$$
\bB=
\begin{bmatrix}
b_{11} & b_{12} & 0 & 0 &\ldots \\
0      & b_{22} & b_{23} & 0 & \ldots \\
0      & 0      & b_{33} & b_{34} & \ldots \\
\vdots & \vdots & \vdots & \ddots & \ddots \\
\ldots & \ldots & \ldots & \ldots & b_{nn}
\end{bmatrix}.
$$
Then $ \bT_1=\bB^\top \bB$ is given by
$$
\begin{aligned}
&\gap \bT_1=\bB^\top\bB =\\
&\begin{bmatrix}
	b_{11} & 0 & 0 & 0 &\ldots \\
	b_{12}     & b_{22} & 0  & 0 & \ldots \\
	0      & b_{23}     & b_{33} & 0 & \ldots \\
	0 & 0 & b_{34} & \vdots & \ddots \\
	\vdots & \vdots & \vdots & \vdots & \vdots
\end{bmatrix}
\begin{bmatrix}
	b_{11} & b_{12} & 0 & 0 &\ldots \\
	0      & b_{22} & b_{23} & 0 & \ldots \\
	0      & 0      & b_{33} & b_{34} & \ldots \\
	\vdots & \vdots & \vdots & \ddots & \ddots \\
	\ldots & \ldots & \ldots & \ldots & b_{nn}
\end{bmatrix}
=
\begin{bmatrix}
	b_{11}^2      & b_{11}b_{12} & 0  &\ldots \\
	b_{11}b_{12} &b_{12}^2+ b_{22}^2 & b_{22}b_{23} & \ldots \\
	0      & b_{22}b_{23}      & b_{23}^2+b_{33}^2 &  \ldots \\
	\vdots & \vdots & \ddots & \ddots  \\
\end{bmatrix},
\end{aligned}
$$
which is symmetric and tridiagonal as claimed. Similarly, we can prove $\bT_2=\bB\bB^\top$ is also symmetric and tridiagonal:
$$
\begin{aligned}
	&\gap \bT_2=\bB\bB^\top =\\
	&
\begin{bmatrix}
		b_{11} & b_{12} & 0 & 0 &\ldots \\
		0      & b_{22} & b_{23} & 0 & \ldots \\
		0      & 0      & b_{33} & b_{34} & \ldots \\
		\vdots & \vdots & \vdots & \ddots & \ddots \\
		\ldots & \ldots & \ldots & \ldots & b_{nn}
	\end{bmatrix}
\begin{bmatrix}
	b_{11} & 0 & 0 & 0 &\ldots \\
	b_{12}     & b_{22} & 0  & 0 & \ldots \\
	0      & b_{23}     & b_{33} & 0 & \ldots \\
	0 & 0 & b_{34} & \vdots & \ddots \\
	\vdots & \vdots & \vdots & \vdots & \vdots
\end{bmatrix}
	=
	\begin{bmatrix}
		b_{11}^2 +b_{12}^2     & b_{12}b_{22} & 0  &\ldots \\
		b_{12}b_{22} &b_{22}^2+ b_{23}^2 & b_{23}b_{33} & \ldots \\
		0      & b_{23}b_{33}      & b_{33}^2+b_{34}^2 &  \ldots \\
		\vdots & \vdots & \ddots & \ddots  \\
	\end{bmatrix},
\end{aligned}
$$
\end{proof}
The lemma above reveals an important property. Suppose $\bA=\bU\bB\bV^\top$ is the bidiagonal decomposition of $\bA$, then the symmetric matrix $\bA\bA^\top$ has a tridiagonal decomposition
$$
\bA\bA^\top=\bU\bB\bV^\top \bV\bB^\top\bU^\top = \bU\bB\bB^\top\bU^\top.
$$
And the symmetric matrix $\bA^\top\bA$ has a tridiagonal decomposition
$$
\bA^\top\bA=\bV\bB^\top\bU^\top \bU\bB\bV^\top=\bV\bB^\top\bB\bV^\top.
$$

As a final result in this section, we state a theorem giving the tridiagonal decomposition of a symmetric matrix with special eigenvalues.
\begin{theoremHigh}[Tridiagonal Decomposition for Nonnegative Eigenvalues]\label{theorem:tri-nonnegative-eigen}
Suppose $n\times n$ symmetric matrix $\bA$ has nonnegative eigenvalues, then there exists a matrix $\bZ$ such that
$$
\bA=\bZ\bZ^\top.
$$
Moreover, the tridiagonal decomposition of $\bA$ can be reduced to a problem to find the bidiagonal decomposition of $\bZ =\bU\bB\bV^\top$ such that the tridiagonal decomposition of $\bA$ is given by
$$
\bA = \bZ\bZ^\top = \bU\bB\bB^\top\bU^\top.
$$
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:tri-nonnegative-eigen}]
The eigenvectors of symmetric matrices can be chosen to be orthogonal (Lemma~\ref{lemma:orthogonal-eigenvectors}, p.~\pageref{lemma:orthogonal-eigenvectors}) such that symmetric matrix $\bA$ can be decomposed into $\bA=\bQ\bLambda\bQ^\top$ (spectral theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}) where $\bLambda$ is a diagonal matrix containing the eigenvalues of $\bA$. When eigenvalues are nonnegative, $\bLambda$ can be factored as $\bLambda=\bLambda^{1/2} \bLambda^{1/2}$. Let $\bZ = \bQ\bLambda^{1/2}$, $\bA$ can be factored as $\bA=\bZ\bZ^\top$. Thus, combining our findings yields the result.
\end{proof}

\chapter{Interpolative Decomposition (ID)}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Interpolative Decomposition}
The column interpolative decomposition (ID) factors a matrix as the product of two matrices, one of which contains selected columns from the original matrix, and the other of which has a subset of columns consisting of the identity matrix and all its values are no greater than 1 in absolute value. Formally, we have the following theorem describing the details of the column ID.
\begin{theoremHigh}[Column Interpolative Decomposition]\label{theorem:interpolative-decomposition}
	Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
	$$
	\underset{m \times n}{\bA} = \underset{m\times r}{\bC} \gap  \underset{r\times n}{\bW},
	$$
	where $\bC\in \real^{m\times r}$ is some $r$ linearly independent columns of $\bA$, $\bW\in \real^{r\times n}$ is the matrix to reconstruct $\bA$ which contains an $r\times r$ identity submatrix (under a mild column permutation). Specifically entries in $\bW$ have values no larger than 1 in magnitude:
	$$
	\max |w_{ij}|\leq 1, \,\, \forall \,\, i\in [1,r], j\in [1,n].
	$$
	\item The storage for the decomposition is then reduced or potentially increased from $mn$ floats to $mr$, $(n-r)r$ floats for storing $\bC, \bW$ respectively and extra $r$ integers are required to remember the position of each column of $\bC$ in that of $\bA$.
\end{theoremHigh}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\textwidth]{imgs/id-column.pdf}
	\caption{Demonstration of the column ID of a matrix where the \textcolor{canaryyellow}{yellow} vector denotes the linearly independent columns of $\bA$, white entries denote zero, and \textcolor{canarypurple}{purple} entries denote one.}
	\label{fig:column-id}
\end{figure}

The illustration of the column ID is shown in Figure~\ref{fig:column-id} where the \textcolor{canaryyellow}{yellow} vectors denote the linearly independent columns of $\bA$ and the \textcolor{canarypurple}{purple} vectors in $\bW$ form an $r\times r$ identity submatrix. The positions of the \textcolor{canarypurple}{purple} vectors inside $\bW$ are exactly the same as the positions of the corresponding \textcolor{canaryyellow}{yellow} vectors inside $\bA$. The column ID is very similar to the CR decomposition (Theorem~\ref{theorem:cr-decomposition}, p.~\pageref{theorem:cr-decomposition}), both select $r$ linearly independent columns into the first factor and the second factor contains an $r\times r$ identity submatrix. The difference is in that the CR decomposition will exactly choose the first $r$ linearly independent columns into the first factor and the identity submatrix appears in the pivots (Definition~\ref{definition:pivot}, p.~\pageref{definition:pivot}). And more importantly, the second factor in the CR decomposition comes from the RREF (Lemma~\ref{lemma:r-in-cr-decomposition}, p.~\pageref{lemma:r-in-cr-decomposition}).
Therefore, the column ID can also be utilized in the applications of the CR decomposition, say proving the fact of rank equals trace in idempotent matrices (Lemma~\ref{lemma:rank-of-symmetric-idempotent2_tmp}, p.~\pageref{lemma:rank-of-symmetric-idempotent2_tmp}), and proving the elementary theorem in linear algebra that column rank equals row rank of a matrix (Corollary~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}).
Moreover, the column ID is also a special case of rank decomposition (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}) and is apparently not unique. The connection between different column IDs is given by Lemma~\ref{lemma:connection-rank-decom} (p.~\pageref{lemma:connection-rank-decom}).

\paragraph{Notations that will be extensively used in the sequel} Following again the Matlab-style notation, if $J_s$ is an index vector with size $r$ that contains the indices of columns selected from $\bA$ into $\bC$, then $\bC$ can be denoted as $\bC=\bA[:,J_s]$ (Definition~\ref{definition:matlabnotation}, p.~\pageref{definition:matlabnotation}).
The matrix $\bC$ contains ``skeleton" columns of $\bA$, hence the subscript $s$ in $J_s$. From the ``skeleton" index vector $J_s$, the $r\times r$ identity matrix inside $\bW$ can be recovered by
$$
\bW[:,J_s] = \bI_r \in \real^{r\times r}.
$$
Suppose further we put the remaining indices of $\bA$ into an index vector $J_r$ where
$$
J_s\cap J_r=\varnothing \qquad \text{and}\qquad J_s\cup J_r = \{1,2,\ldots, n\}.
$$
The remaining $n-r$ columns in $\bW$ consists of an $r\times (n-r)$ \textit{expansion matrix} since the matrix contains \textit{expansion coefficients} to reconstruct the columns of $\bA$ from $\bC$:
$$
\bE = \bW[:,J_r] \in \real^{r\times (n-r)},
$$
where the entries of $\bE$ are known as the \textit{expansion coefficients}. Moreover, let $\bP\in \real^{n\times n}$ be a (column) permutation matrix (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}) defined by $\bP=\bI_n[:,(J_s, J_r)]$ so that
$$
\bA\bP = \bA[:,(J_s, J_r)] = \left[\bC, \bA[:,J_r]\right],
$$
and
\begin{equation}\label{equation:interpolatibve-w-ep}
	\bW\bP = \bW[:,(J_s, J_r)] =\left[\bI_r, \bE \right] \leadto \bW = \left[\bI_r, \bE \right] \bP^\top.
\end{equation}
\section{Existence of the Column Interpolative Decomposition}\label{section:proof-column-id}
\paragraph{Cramer's rule}
The proof of the existence of the column ID relies on the Cramer's rule that we shall shortly discuss here. Consider a system of $n$ linear equations for $n$ unknowns, represented in matrix multiplication form as follows \index{Cramer's rule}:
$$
\bM \bx = \bl,
$$
where $\bM\in \real^{n\times n}$ is nonsingular and $\bx,\bl \in \real^n$. Then the theorem states that in this case, the system has a unique solution, whose individual values for the unknowns are given by:
$$
x_i = \frac{\det(\bM_i)}{\det(\bM)}, \qquad \text{for all}\gap i\in \{1,2,\ldots, n\},
$$
where $\bM_i$ is the matrix formed by replacing the $i$-th column of $\bM$ with the column vector $\bl$. In full generality, the Cramer's rule considers the matrix equation
$$
\bM\bX = \bL,
$$
where $\bM\in \real^{n\times n}$ is nonsingular and $\bX,\bL\in \real^{n\times m}$. Let $I=[i_1, i_2, \ldots, i_k]$ and $J=[j_1,j_2,\ldots, j_k]$ be two index vectors where $1\leq i_1\leq i_2\leq \ldots\leq i_k\leq n$ and $1\leq j_1\leq j_2\leq \ldots\leq j_k\leq n$. Then $\bX[I,J]$ is a $k\times k$ submatrix of $\bX$. Let further $\bM_{\bL}(I,J)$ be the $n\times n$ matrix formed by replacing the $i_s$ column of $\bM$ by $j_s$ column of $\bL$ for all $s\in \{1,2,\ldots, k\}$. Then
$$
\det(\bX[I,J]) = \frac{\det\left(\bM_{\bL}(I,J)\right)}{\det(\bM)}.
$$
When $I,J$ are of size 1, it follows that
\begin{equation}\label{equation:cramer-rule-general}
	x_{ij} = \frac{\det\left(\bM_{\bL}(i,j)\right)}{\det(\bM)}.
\end{equation}
Now we are ready to prove the existence of the column ID.
\begin{proof}[of Theorem~\ref{theorem:interpolative-decomposition}]
	We have mentioned above the proof relies on the Cramer's rule. If we can show the entries of $\bW$ can be denoted by the Cramer's rule equality in Equation~\eqref{equation:cramer-rule-general} and the numerator is smaller than the denominator, then we can complete the proof. However, we notice that the matrix in the denominator of Equation~\eqref{equation:cramer-rule-general} is a square matrix. Here comes the trick.

	\paragraph{Step 1: column ID for full row rank matrix}
	For a start, we first consider the full row rank matrix $\bA$ (which implies $r=m$, $m\leq n$, and $\bA\in \real^{r\times n}$ such that the matrix $\bC\in \real^{r\times r}$ is a square matrix in the column ID $\bA=\bC\bW$ that we want). Determine the ``skeleton" index vector $J_s$ by
	\begin{equation}\label{equation:interpolative-choose-js}
		\boxed{	J_s = \mathop{\arg\max}_{J} \left\{|\det(\bA[:,J])|: \text{$J$ is a subset of $\{1,2,\ldots, n\}$ with size $r=m$} \right\},}
	\end{equation}
	i.e., $J_s$ is the index vector that is determined by maximizing the magnitude of the determinant of $\bA[:,J]$. As we have discussed in the last section, there exists a (column) permutation matrix such that
	$$
	\bA\bP =
	\begin{bmatrix}
		\bA[:,J_s]&\bA[:,J_r]
	\end{bmatrix}.
	$$
	Since $\bC=\bA[:,J_s]$ has full column rank $r=m$, it is then nonsingular. The above equation can be rewritten as
	$$
	\begin{aligned}
		\bA
		&=\begin{bmatrix}
			\bA[:,J_s]&\bA[:,J_r]
		\end{bmatrix}\bP^\top\\
		&=
		\bA[:,J_s]
		\bigg[
		\bI_r \gap \bA[:,J_s]^{-1}\bA[:,J_r]
		\bigg]
		\bP^\top\\
		&= \bC
		\underbrace{\begin{bmatrix}
				\bI_r & \bC^{-1}\bA[:,J_r]
			\end{bmatrix}
			\bP^\top}_{\bW}
	\end{aligned},
	$$
	where the matrix $\bW$ is given by
	$
	\begin{bmatrix}
		\bI_r & \bC^{-1}\bA[:,J_r]
	\end{bmatrix}\bP^\top
	=
	\begin{bmatrix}
		\bI_r & \bE
	\end{bmatrix}\bP^\top
	$ by Equation~\eqref{equation:interpolatibve-w-ep}. To prove the claim that the magnitude of $\bW$ is no larger than 1 is equivalent to proving that entries in $\bE=\bC^{-1}\bA[:,J_r]\in \real^{r\times (n-r)}$ are no greater than 1 in absolute value.

	Define the index vector $[j_1,j_2,\ldots, j_n]$ as a permutation of $[1,2,\ldots, n]$ such that
	$$
	[j_1,j_2,\ldots, j_n] = [1,2,\ldots, n] \bP = [J_s, J_r].\footnote{Note here $[j_1,j_2,\ldots, j_n] $, $[1,2,\ldots, n]$, $J_s$, and $J_r$ are row vectors.}
	$$
	Thus, it follows from $\bC\bE=\bA[:,J_r]$ that
	$$
	\begin{aligned}
		\underbrace{	[\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]}_{=\bC=\bA[:,J_s]} \bE &=
		\underbrace{[\ba_{j_{r+1}}, \ba_{j_{r+2}}, \ldots, \ba_{j_n}]}_{=\bA[:,J_r]:=\bB},
	\end{aligned}
	$$
	where $\ba_i$ is the $i$-th column of $\bA$ and let $\bB=\bA[:,J_r]$.
	Therefore, by Cramer's rule in Equation~\eqref{equation:cramer-rule-general}, we have
	\begin{equation}\label{equation:column-id-expansionmatrix}
		\bE_{kl} =
		\frac{\det\left(\bC_{\bB}(k,l)\right)}
		{\det\left(\bC\right)},
	\end{equation}
	where $\bE_{kl}$ is the entry ($k,l$) of $\bE$ and $\bC_{\bB}(k,l)$ is the $r\times r$ matrix formed by replacing the $k$-th column of $\bC$ by the $l$-th column of $\bB$. For example,
	$$
	\begin{aligned}
		\bE_{11} &=
		\frac{\det\left([\textcolor{blue}{\ba_{j_{r+1}}}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)}
		{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)},
		\qquad
		&\bE_{12} &=
		\frac{\det\left([\textcolor{blue}{\ba_{j_{r+2}}}, \ba_{j_2},\ldots, \ba_{j_r}]\right)}
		{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)},\\
		\bE_{21} &=
		\frac{\det\left([\ba_{j_1},\textcolor{blue}{\ba_{j_{r+1}}}, \ldots, \ba_{j_r}]\right)}
		{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)},
		\qquad
		&\bE_{22} &=
		\frac{\det\left([\ba_{j_1},\textcolor{blue}{\ba_{j_{r+2}}}, \ldots, \ba_{j_r}]\right)}
		{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)}.
	\end{aligned}
	$$
	Since $J_s$ is chosen to maximize the magnitude of $\det(\bC)$ in Equation~\eqref{equation:interpolative-choose-js}, it follows that
	$$
	|\bE_{kl}|\leq 1, \qquad \text{for all}\gap k\in \{1,2,\ldots, r\}, l\in \{1,2,\ldots, n-r\}.
	$$

	\paragraph{Step 2: apply to general matrices}
	To summarize what we have proved above and to abuse the notation. For any matrix $\bF\in \real^{r\times n}$ with \textbf{full} rank $r\leq n$, the column ID exists that $\bF=\bC_0\bW$ where the values in $\bW$ are no greater than 1 in absolute value.

	Apply the finding to the full general matrix $\bA\in \real^{m\times n}$ with rank $r\leq \{m,n\}$, it is trivial that the matrix $\bA$ admits a rank decomposition (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}):
	$$
	\underset{m\times n}{\bA} = \underset{m\times r}{\bD}\gap \underset{r\times n}{\bF},
	$$
	where $\bD,\bF$ have full column rank $r$ and full row rank $r$ respectively. For the column ID of $\bF=\bC_0\bW$ where $\bC_0=\bF[:,J_s]$ contains $r$ linearly independent columns of $\bF$. We notice by $\bA=\bD\bF$ such that
	$$
	\bA[:,J_s]=\bD\bF[:,J_s],
	$$
	i.e., the columns indexed by $J_s$ of $(\bD\bF)$ can be obtained by $\bD\bF[:,J_s]$ which in turn are the columns of $\bA$ indexed by $J_s$. This makes
	$$
	\underbrace{\bA[:,J_s]}_{\bC}= \underbrace{\bD\bF[:,J_s]}_{\bD\bC_0},
	$$
	And
	$$
	\bA = \bD\bF =\bD\bC_0\bW = \underbrace{\bD\bF[:,J_s]}_{\bC}\bW=\bC\bW.
	$$
	This completes the proof.

\end{proof}

The above proof reveals an intuitive way to compute the optimal column ID of matrix $\bA$ as shown in Algorithm~\ref{alg:column-id-intuitive}. However, any algorithm that is guaranteed to find such an optimally-conditioned factorization must have combinatorial complexity \citep{martinsson2019randomized}. In the next sections, we will consider alternative ways to find a relatively well-conditioned factorization.
\begin{algorithm}[h]
	\caption{An \textcolor{blue}{Intuitive} Method to Compute the Column ID}
	\label{alg:column-id-intuitive}
	\begin{algorithmic}[1]
		\Require
		Rank-$r$ matrix $\bA$ with size $m\times n $;
		\State Compute the rank decomposition $\underset{m\times n}{\bA} = \underset{m\times r}{\bD}\gap \underset{r\times n}{\bF}$ (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}) such as from UTV (Section~\ref{section:ulv-urv-decomposition}, p.~\pageref{section:ulv-urv-decomposition});
		\State Compute column ID of $\bF$: $\bF=\bF[:,J_s]\bW = \widetildebC\bW$:
		$$
		\begin{aligned}
			2.1. &\left\{
			\begin{aligned}
				J_s &= \mathop{\arg\max}_{J} \left\{|\det(\bF[:,J])|: \text{$J$ is a subset of $\{1,2,\ldots, n\}$ with size $r$} \right\};&\\
				J_r &= \{1,2,\ldots, n\} /J_s;&\\
			\end{aligned}
			\right.\\
			2.2.&\left\{
			\begin{aligned}
				\widetildebC &= \bF[:,J_s]; \\
				\bM &= \bF[:,J_r];
			\end{aligned}
			\right.\\
			2.3. &\bF\bP = \bF[:,(J_s,J_r)] \text{ to obtain $\bP$};\\
			2.4. &\bE_{kl} =
			\frac{\det\left(\widetildebC_{\bM}(k,l)\right)}
			{\det\left(\widetildebC\right)},  \qquad \text{for all}\gap k\in [1, r], l\in [1,n-r] \text{  ~(Equation~\eqref{equation:column-id-expansionmatrix})};\\
			2.5. &\bW= [\bI_r, \bE]\bP^\top \text{ ~(Equation~\eqref{equation:interpolatibve-w-ep})}.
		\end{aligned}
		$$
		\State 	$\bC=\bA[:,J_s]$;
		\State Output the column ID $\bA=\bC\bW$;
	\end{algorithmic}
\end{algorithm}

\begin{example}[Compute the Column ID]\label{example:column-id-a}
	For matrix
	$$
	\bA=
	\begin{bmatrix}
		56 & 41 & 30\\
		32 & 23 & 18\\
		80 & 59 & 42
	\end{bmatrix}
	$$
	with rank 2. The trivial process for computing the column ID of $\bA$ is shown as follows. We first find a rank decomposition
	$$
	\bA = \bD\bF=
	\begin{bmatrix}
		1 & 0 \\
		0 & 1 \\
		2 &-1
	\end{bmatrix}
	\begin{bmatrix}
		56 & 41 & 30 \\
		32 & 23 & 18
	\end{bmatrix}.
	$$
	Since rank $r=2$, $J_s$ is one of $[1,2], [0,2], [0,1]$ where the absolute determinant of $\bF[:,J_s]$ are $48, 48, 24$ respectively. We proceed by choosing $J_s=[0,2]$:
	$$
	\begin{aligned}
		\widetildebC &= \bF[:,J_s]=
		\begin{bmatrix}
			56 & 30 \\
			32 & 18
		\end{bmatrix},\qquad
		\bM &= \bF[:,J_r]=\begin{bmatrix}
			41 \\
			23
		\end{bmatrix}.
	\end{aligned}
	$$
	And
	$$
	\bF\bP = \bF[:(J_s,J_r)] = \bF[:,(0,2,1)]
	\leadto
	\bP =
	\begin{bmatrix}
		1 &  & \\
		& &1\\
		& 1 &
	\end{bmatrix}.
	$$
	In this example, $\bE\in \real^{2\times 1}$:
	$$
	\begin{aligned}
		\bE_{11} &=
		\det\left(
		\begin{bmatrix}
			41 & 30 \\
			23 & 18
		\end{bmatrix}\right)\bigg/
		\det\left(
		\begin{bmatrix}
			56 & 30 \\
			32 & 18
		\end{bmatrix}\right)=1;\\
		\bE_{21} &=
		\det\left(
		\begin{bmatrix}
			56 & 41 \\
			32 & 23
		\end{bmatrix}\right)\bigg/
		\det\left(
		\begin{bmatrix}
			56 & 30 \\
			32 & 18
		\end{bmatrix}\right)=-\frac{1}{2}.
	\end{aligned}
	$$
	This makes
	$$
	\bE =
	\begin{bmatrix}
		1\\-\frac{1}{2}
	\end{bmatrix}
	\leadto
	\bW = [\bI_2, \bE]\bP^\top =
	\begin{bmatrix}
		1 & 1 & 0\\
		0 & -\frac{1}{2} & 1
	\end{bmatrix}.
	$$
	The final selected columns are
	$$
	\bC = \bA[:,J_s] =
	\begin{bmatrix}
		56 & 30\\
		32 & 18\\
		80 & 42
	\end{bmatrix}.
	$$
	The net result is given by
	$$
	\bA=\bC\bW =
	\begin{bmatrix}
		56 & 30\\
		32 & 18\\
		80 & 42
	\end{bmatrix}
	\begin{bmatrix}
		1 & 1 & 0\\
		0 & -\frac{1}{2} & 1
	\end{bmatrix},
	$$
	where entries of $\bW$ are no greater than 1 in absolute value as we want.
	\exampbar
\end{example}

To end up this section, we discuss where comes the non-uniqueness of the column ID.
\begin{remark}[Non-uniqueness of the Column ID]
	In the above specific example~\ref{example:column-id-a}, we notice the determinant for $\bF[:,(1,2)]$ and $\bF[:,(0,2)]$ both get maximal absolute determinant. Therefore, both of them can result in a column ID of $\bA$. Whilst, we only select $J_s$ from $[1,2], [0,2], [0,1]$. When the $J_s$ is fixed from the maximal absolute determinant search, any permute of it can also be selected, e.g., $J_s=[0,2]$ or $J_s=[2,0]$ are both good. The two choice on the selection of the column index search yield the non-uniqueness of the column ID.
\end{remark}

\section{Row ID and Two-Sided ID}
We term the decomposition above as column ID. This is no coincidence since it has its siblings:

\begin{theoremHigh}[The Whole Interpolative Decomposition]\label{theorem:interpolative-decomposition-row}
	Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
	$$
	\begin{aligned}
		\text{Column ID: }&\gap \underset{m \times n}{\bA} &=& \boxed{\underset{m\times r}{\bC}} \gap  \underset{r\times n}{\bW} ; \\
		\text{Row ID: } &\gap &=&\underset{m\times r}{\bZ} \gap  \boxed{\underset{r\times n}{\bR}}; \\
		\text{Two-Sided ID: } &\gap &=&\underset{m\times r}{\bZ} \gap \boxed{\underset{r\times r}{\bU}}  \gap  \underset{r\times n}{\bW}, \\
	\end{aligned}
	$$
	where
	\begin{itemize}
		\item $\bC=\bA[:,J_s]\in \real^{m\times r}$ is some $r$ linearly independent columns of $\bA$, $\bW\in \real^{r\times n}$ is the matrix to reconstruct $\bA$ which contains an $r\times r$ identity submatrix (under a mild column permutation): $\bW[:,J_s]=\bI_r$;
		\item $\bR=\bA[I_s,:]\in \real^{r\times n}$ is some $r$ linearly independent rows of $\bR$, $\bZ\in \real^{m\times r}$ is the matrix to reconstruct $\bA$ which contains an $r\times r$ identity submatrix (under a mild row permutation): $\bZ[I_s,:]=\bI_r$;
		\item Entries in $\bW, \bZ$ have values no larger than 1 in magnitude: $\max |w_{ij}|\leq 1$ and $\max |z_{ij}|\leq 1$;
		\item $\bU=\bA[I_s,J_s] \in \real^{r\times r}$ is the nonsingular submatrix on the intersection of $\bC,\bR$;
		\item The three matrices $\bC,\bR,\bU$ in the $\boxed{\text{boxed}}$ texts share same notation as the skeleton decomposition (Theorem~\ref{theorem:skeleton-decomposition}, p.~\pageref{theorem:skeleton-decomposition}) where they even have same meanings such that the three matrices make the skeleton decomposition of $\bA$: $\bA=\bC\bU^{-1}\bR$.
	\end{itemize}
\end{theoremHigh}
The proof of the row ID is just similar to that of the column ID. Suppose the column ID of $\bA^\top$ is given by $\bA^\top=\bC_0\bW_0$ where $\bC_0$ contains $r$ linearly independent columns of $\bA^\top$ (i.e., $r$ linearly independent rows of $\bA$). Let $\bR=\bC_0, \bZ=\bW_0$, the row ID is obtained by $\bA=\bZ\bR$.

For the two-sided ID, recall from the skeleton decomposition (Theorem~\ref{theorem:skeleton-decomposition}, p.~\pageref{theorem:skeleton-decomposition}). When $\bU$ is the intersection of $\bC, \bR$, it follows that $\bA=\bC\bU^{-1}\bR$. Thus $\bC\bU^{-1}=\bZ$ by the row ID. And this implies $\bC=\bZ\bU$. By column ID, it follows that $\bA=\bC\bW=\bZ\bU\bW$ which proves the existence of the two-sided ID.

\paragraph{Data storage} For the data storage of each ID, we summarize as follows
\begin{itemize}
	\item \textit{Column ID.} It requires $mr$ and $(n-r)r$ floats to store $\bC$ and $\bW$ respectively , and $r$ integers to store the indices of the selected columns in $\bA$;
	\item \textit{Row ID.} It requires $nr$ and $(m-r)r$ floats to store $\bR$ and $\bZ$ respectively, and $r$ integers to store the indices of the selected rows in $\bA$;
	\item \textit{Two-Sided ID.} It requires $(m-r)r$, $(n-r)r$, and $r^2$ floats to store $\bZ,\bW$, and $\bU$ respectively. And extra $2r$ integers are required to store the indices of the selected rows and columns in $\bA$.
\end{itemize}

\paragraph{Further reduction on the storage for two-sided ID for sparse matrix $\bA$}
Suppose the column ID of $\bA=\bC\bW$ where $\bC=\bA[:,J_s]$ and a good spanning rows index $I_s$ set of $\bC$ could be found:
$$
\bA[I_s,:] = \bC[I_s,:]\bW.
$$
We observe that $\bC[I_s,:] = \bA[I_s,J_s]\in \real^{r\times r}$ which is nonsingular (since full rank $r$ in the sense of both row rank and column rank). It follows that
$$
\bW = (\bA[I_s,J_s])^{-1} \bA[I_s,:].
$$
Therefore, there is no need to store the matrix $\bW$ explicitly. We only need to store $\bA[I_s,:]$ and $(\bA[I_s,J_s])^{-1}$. Or when we can compute the inverse of $\bA[I_s,J_s]$ on the fly, it only requires $r$ integers to store $J_s$ and recover $\bA[I_s,J_s]$ from $\bA[I_s,:]$. The storage of $\bA[I_s,:]$ is cheap if $\bA$ is sparse.

\section{Computing the Column ID via the CPQR}
The method used in the proof of the last section can be utilized to compute the ``optimal" column ID.  However, any algorithm
that is guaranteed to find such an optimally-conditioned factorization must have combinatorial complexity. An inexpensive alternative is to favor that $\bW$ is small in norm rather than bounding each entry in modulus by one.
Recall the column-pivoted QR decomposition (CPQR, Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}) such that for matrix $\bA\in \real^{m\times n}$, the \textit{reduced} CPQR is given by:
$$
\bA\bP = \bQ_r
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
\end{bmatrix}
=
\begin{bmatrix}
	\bQ_r\bR_{11} & \bQ_r\bR_{12} \\
\end{bmatrix},
$$
where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ_r\in \real^{m\times r}$ has orthonormal columns, and $\bP$ is a permutation matrix. The complexity of the CRPQ is $O(mnr)$ flops (Theorem~\ref{theorem:qr-reduced-rank-revealing}, p.~\pageref{theorem:qr-reduced-rank-revealing}). $\bA\bP$
is permuting the $r$ linearly independent columns into the first $r$ columns of $\bA\bP$:
$$
\bA\bP = \bA[:, (J_s, J_r)]=
\begin{bmatrix}
	\bQ_r\bR_{11} & \bQ_r\bR_{12} \\
\end{bmatrix}.
$$
In the ``practical" CPQR via CGS \footnote{As a recap, CGS is short for the classical Gram-Schmidt process.} we introduced in Section~\ref{section:practical-cpqr-cgs} (p.~\pageref{section:practical-cpqr-cgs}), $\bQ_r\bR_{11}$ contains the $r$ independent columns of $\bR$ with the largest norm and $\bQ_r\bR_{12}$ is small in norm. This is important to our aim that the column ID is \textit{well-conditioned} in that entries of $\bW$ are small in magnitude.
Therefore $\bQ_r \bR_{11}$ contains $r$ linearly independent columns of $\bA$. Let $\bC=\bQ_r\bR_{11}$, and solve the linear equation $\bC\bE=\bQ_r\bR_{12}$, the column ID then follows:
$$
\bA = \bC \underbrace{[\bI_r, \bE]\bP^\top}_{\bW}.
$$
\paragraph{Calculation of the linear system}
The linear system $\bC\bE=\bQ_r\bR_{12}$ is well defined and is not in a sense of least squares as we have shown in Section~\ref{section:application-ls-qr} (p.~\pageref{section:application-ls-qr}), since every column of $\bQ_r\bR_{12}$ is in the column space of $\bC$ (i.e., the column space of $\bA$). The solution can be obtained via the \textit{normal equation} \footnote{We shall briefly discuss in Definition~\ref{definition:normal-equation-als} (p.~\pageref{definition:normal-equation-als}).}: $\bE=(\bC^\top\bC)^{-1} \bC^\top\bQ_r\bR_{12}$.
An extra cost for computing the columnd ID thus comes from the calculation of the $\bE$. The calculation of $(\bC^\top\bC)^{-1}$ takes $\boxed{r^2(2m-1)+2r^3}$ flops where $2r^3$ comes from the computation of the inverse of an $r\times r$ matrix (Theorem~\ref{theorem:inverse-by-lu2}, p.~\pageref{theorem:inverse-by-lu2}). Let's see what's left:
$$
\text{step 2: }\qquad
\bE=\underbrace{(\bC^\top\bC)^{-1} }_{r\times r}
\gap
\underbrace{ \bC^\top}_{r\times m}
\gap
\underbrace{\bQ_r}_{m\times r}
\gap
\underbrace{ \bR_{12}}_{r\times (n-r)}.
$$
Since $r<m$, the $\bC^\top\bQ_r$ should be considered as the the next step to make a smaller $r\times r$ matrix which takes $\boxed{(2m-1)r^2}$ flops:
$$
\text{step 3: }\qquad
\bE=\underbrace{(\bC^\top\bC)^{-1} }_{r\times r}
\gap
\underbrace{ \bC^\top\bQ_r}_{r\times r}
\gap
\underbrace{ \bR_{12}}_{r\times (n-r)}.
$$
When $r<(n-r)$, the calculation of $(\bC^\top\bC)^{-1} \bC^\top\bQ_r$ in the above equation should be done firstly which makes the rest complexity be $\boxed{(2r-1)r^2+ (2r-1)r(n-r)}$. Otherwise, $\bC^\top\bQ_r\bR_{12}$ should be done firstly, which makes the rest complexity be $\boxed{2(2r-1)r(n-r)}$. To conclude, the final complexity from normal equation is summarized as follows:
$$
\text{cost=}
\left\{
\begin{aligned}
	&\{r^2(2m-1)+2r^3 + (2m-1)r^2\}+(2r-1)r^2+ (2r-1)r(n-r), \qquad &r&<(n-r); \\
	&\{r^2(2m-1)+2r^3 + (2m-1)r^2\}+ 2(2r-1)r(n-r), \qquad &r&\geq (n-r).\\
\end{aligned}
\right.
$$
The normal equation is just a trivial way to solve the above linear equation. Other iterative methods can be employed such as the gradient descent (Section~\ref{section:als-gradie-descent}, p.~\pageref{section:als-gradie-descent}).

\subsubsection*{\textbf{Partial factorization via the CPQR}}
The complexity of the algorithms for computing column ID of a matrix via the CPQR relies on the complexity of the CPQR decomposition. When the matrix has ``exact" rank $r$, the complexity of the CPQR is $O(mr^2)$ flops for matrix $\bA\in \real^{m\times n}$. However, when it comes to the matrix $\bA$ being rank deficient, the complexity will become $O(mn^2)$. The partial factorization CPQR via MGS \footnote{As a recap, MGS is short for the modified Gram-Schmidt process.}(Section~\ref{section:partial-cpqr-mgs}, p.~\pageref{section:partial-cpqr-mgs}) can be employed to attack this problem where the partial CPQR decomposition is given by
$$
\bA\bP =
\bQ\bR=
\bQ
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero & \bR_{22}
\end{bmatrix},
$$
where $\bR_{22}$ is small in norm. Such a partial factorization can either take a rank $k$ set up front, or a tolerance $\delta$ such that whenever $r_{kk}<\delta$ in the upper triangular matrix $\bR$ of the CPQR decomposition, we stop. This is similar to what we will introduce in the low-rank column ID via the rank-revealing QR (RRQR) decomposition. But in the partial CPQR, the norm on $\bR_{22}$ is not guaranteed to be minimal which is assured in RRQR.

\section{Low-Rank Column ID via the RRQR}

An approximate rank $\gamma$ ID of a matrix $\bA\in \real^{m\times n}$ is the approximate
factorization:
$$
\begin{aligned}
	\underset{m\times n}{\bA} &=& \underset{m\times \gamma}{\bC} &\gap\underset{ \gamma \times n}{ \bW} ;\\
	\bA \bP &=& \bC &\gap[\bI, \bE] ,\\
\end{aligned}
$$
where  the partial column skeleton $\bC\in \real^{m\times \gamma}$
is given by a subset of the columns of $\bA$, $\gamma$ is known as the \textit{numerical rank}, the entries of $\bE$ are known as the \textit{expansion coefficients} as we have shown previously,  and $\bW$
is well-conditioned in a sense that we will make precise shortly.

The low-rank approximation of column ID has been studied extensively in the context of rank-revealing QR decomposition
\citep{voronin2017efficient, martinsson2019randomized, martinsson2002randomized, halko2011finding}.
By rank-revealing QR (RRQR) (Equation~\eqref{equation:rankr-reval-qr}, p.~\pageref{equation:rankr-reval-qr}\index{Rank-revealing QR}), there exists a permutation $\bP$ such that the linearly independent columns of $\bA$ can be permuted in the left and
$$
\begin{aligned}
	\bA\bP =
	\bQ\bR
	&=
	\begin{bmatrix}
		\bQ_1 & \bQ_2
	\end{bmatrix}
	\begin{bmatrix}
		\bL & \bM \\
		\bzero & \bN
	\end{bmatrix}\\
	&=
	\begin{bmatrix}
		\bQ_1\bL & \bQ_1\bM+\bQ_2\bN
	\end{bmatrix}\\
	&=
	\bQ_1\bL
	\begin{bmatrix}
		\bI_{\gamma} & \bY
	\end{bmatrix},
\end{aligned}
$$
where $\bN \in \real^{(n-\gamma)\times (n-\gamma)}$ and $||\bN||$ is small in some norm. And $\bY$ is defined to be the solution of the linear system $(\bQ_1\bL) \bY = (\bQ_1\bM+\bQ_2\bN)$.
We observe that $\bQ_1 \bL$ is the fist $\gamma$ columns of $\bA\bP$.
Let $\bC=\bQ_1\bL$, $\bE=\bY$, then the low-rank column ID approximation can be found by solving the linear system
\begin{equation}\label{equation:rrqr-columnid-linear}
	(\bQ_1\bL) \bE = (\bQ_1\bM+\bQ_2\bN).
\end{equation}
Since $\bN$ is small in norm, it can be approximated to find the solution of
$$
\begin{aligned}
	\underset{\gamma\times \gamma}{\bL} & &\underset{\gamma \times (n-\gamma)}{\bE}= &\gap \underset{(n-\gamma)  \times (n-\gamma)  }{\bM} .\\
\end{aligned}
$$
The problem is well defined since $\bL$ is nonsingular (upper triangular with nonzero diagonals). If $\bL$ is singular,  then
one can show that $\bA$ must necessarily have rank $\gamma^\prime$
less than $\gamma$, and the bottom $\gamma-\gamma^\prime$
rows in the above linear system consist of all zeros, so there exists a solution in this case as well.

The approximation error of the column ID obtained via RRQR with pivoting is the same as that of
the RRQR:
$$
||\bA-\bC\bW| = ||\bQ_2\bN||,
$$
where $\bW=[\bI_{\gamma}, \bE]\bP^\top$.

\begin{remark}[Condition of the Linear System]
	Unfortunately, $\bL$ in the linear Equation~\eqref{equation:rrqr-columnid-linear} is typically quite ill-conditioned. However, the linear system in Equation~\eqref{equation:rrqr-columnid-linear} still has a solution $\bE$ whose entries are of moderate size. Informally, the directions
	where $\bL$ and $\bM$ point in are ``lined up" \citep{cheng2005compression}. We shall not give the details.
\end{remark}

\section{Computing the ID via Randomized Algorithm}\label{section:randomi-id}
Suppose matrix $\bA$ admits the rank decomposition:
$$
\underset{m\times n}{\bA} = \underset{m\times r}{\bD}\gap \underset{r\times n}{\bF}.
$$
Upon finding the rank decomposition, suppose further that the row ID of $\bD$ is obtained by
$$
\text{row ID of $\bD$: }\qquad \underset{m\times r}{\bD} = \underset{m\times r}{\bZ}\gap \underset{r\times r}{\bR_0}
=\bZ\bD[I_s,:].
$$
Similar to the proof of the column ID in Section~\ref{section:proof-column-id}, we observe that $Z \bA[I_s,:]$ is automatically a row ID of $\bA$: $\bA=\bZ\bA[I_s,:]$. This can be shown as follows:
\begin{equation}\label{equation:row-id-sub-d}
	\begin{aligned}
		\bZ\bA[I_s,:] &= \bZ (\bD[I_s,:]\bF) &\qquad &\text{(since $\bA=\bD\bF$)}\\
		&=\bD \bF &\qquad &\text{(since $\bD=\bZ\bD[I_s,:]$)}\\
		&=\bA.
	\end{aligned}
\end{equation}
We notice that $\bD$ spans the same column space as $\bA$: $\cspace(\bD)=\cspace(\bA)$.
The finding above tells us that as long as we find an $m\times r$ matrix $\bD$ spanning the same column space of $\bA$, a row ID can be applied to $\bD$ to find the row ID of $\bA$. This reveals the randomized algorithm for computing row ID of $\bA$. To show this, we need a few facts that will be useful.
\begin{lemma}[Subspace of $\bA^\top \bA$ and $\bA\bA^\top$]\label{lemma:rank-of-ttt}
	For matrix $\bA\in \real^{m\times n}$, we have
	\begin{itemize}
		\item The column space of $\bA^\top \bA$ is equal to the column space of $\bA^\top$ (i.e., row space of $\bA$): $\cspace(\bA^\top\bA)=\cspace(\bA^\top)$;
		\item The column space of $\bA\bA^\top$ is equal to the column space of $\bA$: $\cspace(\bA\bA^\top)=\cspace(\bA)$.
	\end{itemize}
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-of-ttt}]
	Let $\bx\in \nspace(\bA)$, we have
	$$
	\bA\bx  = \bzero \leadto \bA^\top\bA \bx =\bzero,
	$$
	i.e., $\bx\in \nspace(\bA) \leadtosmall \bx \in \nspace(\bA^\top \bA)$, therefore $\nspace(\bA) \subseteq \nspace(\bA^\top\bA)$.
	Further, let $\bx \in \nspace(\bA^\top\bA)$, we have
	$$
	\bA^\top \bA\bx = \bzero\leadtosmall \bx^\top \bA^\top \bA\bx = 0\leadtosmall ||\bA\bx||^2 = 0 \leadtosmall \bA\bx=\bzero,
	$$
	i.e., $\bx\in \nspace(\bA^\top \bA) \leadtosmall \bx\in \nspace(\bA)$, therefore $\nspace(\bA^\top\bA) \subseteq\nspace(\bA) $.

	As a result, by ``sandwiching", it follows that
	$$\nspace(\bA) = \nspace(\bA^\top\bA).
	$$
	By the fundamental theorem of linear algebra in Appendix~\ref{appendix:fundamental-rank-nullity} (p.~\pageref{appendix:fundamental-rank-nullity}),
	$$
	\cspace(\bA^\top)=\cspace(\bA^\top\bA).
	$$
	The second half of the lemma is just the same when applying the process on $\bA^\top$.
\end{proof}
The above lemma tells us that if a matrix $\bA\in \real^{m\times n}$ with rank $r$, a matrix $\bB\in \real^{m\times k}$ with $k>r$ and $\cspace(\bB)=\cspace(\bA)$. Then a moment of reflexion reveals $\bA\bA^\top\bB$ spans the same column space as $\bA$:
\begin{equation}\label{equation:random-row-id-column-space0}
	\cspace(\bA\bA^\top\bB)=\cspace(\bA), \qquad \text{if }\gap  \cspace(\bB)=\cspace(\bA).
\end{equation}
Further, by Lemma~\ref{lemma:column-basis-from-row-basis} (p.~\pageref{lemma:column-basis-from-row-basis}), for any matrix $\bA\in \real^{m\times n}$, suppose that $\{\bg_1, \bg_2, \ldots, \bg_r\}$ is a set of vectors in $\real^n$ which forms a basis for the row space, then $\{\bA\bg_1, \bA\bg_2, \ldots, \bA\bg_r\}$ is a basis for the column space of $\bA$:
\begin{equation}\label{equation:random-row-id-column-space-eq2}
	\cspace(\bA\bG) =\cspace(\bA).
\end{equation}
Then, the above matrix $\bB$ in Eequation~\eqref{equation:random-row-id-column-space0} can be constructed by $\bA\bG$ where the columns of $\bG$ contain row basis of $\bA$:
\begin{equation}\label{equation:random-row-id-column-space2}
	\cspace(\bA\bA^\top \bA\bG) =\cspace(\bA).
\end{equation}

\begin{algorithm}[h]
	\caption{A \textcolor{blue}{Randomized} Method to Compute the \textcolor{blue}{Row} ID}
	\label{alg:row-id-randomize1}
	\begin{algorithmic}[1]
		\Require
		Rank-$r$ matrix $\bA$ with size $m\times n $;
		\State Decide the over-sampling parameter $k$ (e.g., $k=10$)$\rightarrow$ let $z=r+k$;
		\State Decide the iteration number: $\eta$ (e.g., $\eta=0,1$ or $2$);
		\State Generate $r+k$ Gaussian random vectors in $\real^n$ into columns of matrix $\bG\in \real^{n\times (r+k)}$;\gap\Comment{i.e., probably contain the row basis of $\bA$}
		\State Initialize $\bD=\bA\bG\in \real^{m\times (r+k)}$; \Comment{i.e., probably $\cspace(\bD)=\cspace(\bA)$, $(2n-1)mz$ flops}
		\For{$i=1$ to $\eta$}
		\State $\bD = \bA\bA^\top\bD$; \Comment{$(2m-1)nz+(2n-1)mz$ flops}
		\EndFor
		\State Calc. the row ID of small matrix $\bD$: $\bD=\bZ\bR_0=\bZ\bD[I_s,:]$;\Comment{$O(mz^2)$ flops by CPQR}
		\State Output row ID of $\bA$: $\bA=\bZ\bA[I_s,:]$;
	\end{algorithmic}
\end{algorithm}

Normally, we could simply stop at $\bA\bG$ since $\cspace(\bA\bG) =\cspace(\bA)$, and the row ID of $\bA\bG$ would reveal the row ID of $\bA$ by Equation~\eqref{equation:row-id-sub-d}. Computing the row ID of $\bA\bG$ is a relatively simpler task compared to that of $\bA$ directly since $\bA\bG\in \real^{m\times r}$ and $r\ll n$.

However, in some situations, when matrix $\bA$ is not exactly rank-$r$, i.e., some singular values $\sigma_k$ (with $k>r$) of $\bA$ may be small enough to be regarded as zero (truncated). We will introduce the SVD in Section~\ref{section:SVD} (p.~\pageref{section:SVD}) such that any matrix $\bA$ admits factorization $\bA=\bU\bSigma\bV^\top$ where $\bU,\bV$ are orthogonal, and roughly speaking $\bSigma$ is a diagonal matrix containing singular values with the number of nonzero singular values being the rank of $\bA$. The columns of $\bU$ contain the column basis of $\bA$ and the columns of $\bV$ contain the row basis of $\bA$ (Lemma~\ref{section:four-space-svd}, p.~\pageref{section:four-space-svd}). Therefore, $\bA\bA^\top \bA\bG$ results in
$$
\bA\bA^\top \bA\bG = (\bU\bSigma\bV^\top) (\bV\bSigma\bU^\top) (\bU\bSigma\bV^\top)\bG=\bU\bSigma^3\bV^\top\bG,
$$
where $\bSigma^3$ will \textbf{shrink the small singular values towards 0}. This is reasonable in that they do not count in the \textit{numerical rank}. For matrices whose singular values decay slowly, we should consider the process twice or third times:
$$
\bA\bA^\top(\bA\bA^\top \bA\bG ) =\bU\bSigma^5\bV^\top\bG.
$$
The procedure to compute the row ID of a matrix is shown in Algorithm~\ref{alg:row-id-randomize1} where a parameter $k$ is used to assure that there is a high probability that $\bG$=$[\bg_1, \ldots,\bg_r, \bg_{r+1}, \ldots, \bg_{r+k}]$ contains the row basis of $\bA$ (The choice $k=10$ is often good) and each $\bg_i$ are generated by \textit{random Gaussian vector}. Iteration parameter $\eta$ is usually picked as 1 or 2 to improve accuracy (to shrink the small singular values). The complexity of the algorithm is shown in the comment of each step which makes it
\begin{equation}\label{equation:rnadom-rowid-complexity}
	\boxed{O(mn(r+k))} \qquad \text{ flops.}
\end{equation}

\begin{remark}[A Word on the Source of Row Basis]\label{remark:source-row-basis}
	In step 4 of Algorithm~\ref{alg:row-id-randomize1}, we desire $\bD$ contains linearly independent vectors that can span the column space of $\bA$, i.e., find a column basis. We just show one way that transform row basis matrix $\bG$ to the column basis matrix $\bD=\bA\bG$, and this is known as the \textit{random projection method}.
	The matrix $\bG$ formed by these columns is expected to be very close
	to $\bA$ in a sense that the basis of the range of $\bD$ covers the range of $\bA$ well. The probability of failure is negligible.
	However, further question can be posed, what if $\bG$ does not contain enough row basis vectors?
	The column basis matrix $\bD$ can be chosen in different ways: by \textit{subsampling of the input matrix} directly (in the sense of row basis or column basis that we will show in next paragraph). The orthonormal basis
	consisting of $r$ linearly independent vectors can be obtained using exact methods since the size of $\bD$ or $\bG$ is very small. These techniques are relatively insensitive to the quality of randomness and produce high accurate results.
\end{remark}

\paragraph{Subsampling method for constructing basis}
We have shown in Gram-Schmidt process (Section~\ref{section:project-onto-a-vector}, p.~\pageref{section:project-onto-a-vector}), the projection of a vector $\ba$ onto $\bb$ results in the projection $\hat{\ba} = \frac{\ba^\top\bb}{\bb^\top\bb}\bb$ which is in the direction of $\bb$ since it is a multiple of $\bb$. The component along $\bb$ thus can be obtained by $\ba-\hat{\ba} = \ba-\frac{\ba^\top\bb}{\bb^\top\bb}\bb$ which is $\bzero$ if $\ba$ and $\bb$ are dependent (i.e., in the same direction). When $\ba,\bb\in \real^n$, the complexity is $O(n)$ flops. Therefore, when we generate the $\bg_i$'s vector in Algorithm~\ref{alg:row-id-randomize1}, we can project it onto each row of $\bA$ to check it is in the row space of $\bA$, and this costs $O(mn)$ flops. And there are $\sim (r+k)$ such generation, which makes it $\boxed{O(mn(r+k))}$ flops to assure $\{\bg_1, \bg_2, \ldots, \bg_{r+k}\}$ can span the row space of $\bA$.
This is the same order of cost compared to the total cost of the algorithm in Equation~\eqref{equation:rnadom-rowid-complexity} when $n\leq m$.
Analogously, we can directly generate the columns of $\bD=[\bd_1, \bd_2, \ldots, \bd_{r+k}]$ via Gaussian random vector and check if each $\bd_i$ is in the column space of $\bA$. This costs $\boxed{O(mn(r+k))}$ that again is equal to the cost of the algorithm in Equation~\eqref{equation:rnadom-rowid-complexity}. So the complexity of the algorithm would not blow up.

\begin{algorithm}[h]
	\caption{A \textcolor{blue}{Randomized} Method to Compute the \textcolor{blue}{Column} ID}
	\label{alg:column-id-randomize1}
	\begin{algorithmic}[1]
		\Require
		Rank-$r$ matrix $\bA$ with size $m\times n $;
		\State Decide the over-sampling parameter $k$ (e.g., $k=10$)$\rightarrow$ let $z=r+k$;
		\State Decide the iteration number: $\eta$ (e.g., $\eta=0,1$ or $2$);
		\State Generate $r+k$ Gaussian random vectors in $\textcolor{blue}{\real^m}$ into columns of matrix $\bG\in \real^{\textcolor{blue}{m}\times (r+k)}$;\Comment{i.e., probably contain the column basis of $\bA$}
		\State Initialize $\bF=\textcolor{blue}{\bA^\top}\bG\in \real^{n\times (r+k)}$; \Comment{i.e., probably $\cspace(\bF)=\cspace(\bA^\top)$, $\textcolor{blue}{(2m-1)nz}$ flops}
		\For{$i=1$ to $\eta$}
		\State $\bF = \textcolor{blue}{\bA^\top\bA}\bF$;  \Comment{$(2m-1)nz+(2n-1)mz$ flops}
		\EndFor
		\State Calc. the \textcolor{blue}{column} ID of small $\bF$: $\bF=\bC_0\bW=\bF[:,J_s]\bW$;\Comment{$O\textcolor{blue}{(mz^2)}$ flops by CPQR}
		\State Output column ID of $\bA$: $\bA=\bA[:,J_s]\bW$;
	\end{algorithmic}
\end{algorithm}

\paragraph{Compute the column ID by randomized method}
Analogously, the above procedure on row ID can be easily applied to compute the column ID of matrix $\bA\in \real^{m\times n}$ by calculating the row ID of $\bA^\top$. Or
from the proof of the column ID. If $\bA$ admits a rank decomposition (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition})
$$
\underset{m\times n}{\bA} = \underset{m\times r}{\bD}\gap \underset{r\times n}{\bF},
$$
where $\bD,\bF$ have full column rank $r$ and full row rank $r$ respectively.
Upon finding the rank decomposition, suppose further that the column ID of $\bF$ is obtained by
$$
\text{column ID of $\bF$: }\qquad \underset{r\times n}{\bF} = \underset{r\times r}{\bC_0}\gap \underset{r\times n}{\bW}
=\bF[:,J_s]\bW.
$$
We observe that $ \bA[:,J_s]\bW$ is automatically a column ID of $\bA$: $\bA=\bA[:,J_s]\bW$. This can be shown as follows:
\begin{equation}\label{equation:column-id-sub-d}
	\begin{aligned}
		\bA[:,J_s]\bW &= (\bD\bF[:,J_s])\bW &\qquad &\text{(since $\bA=\bD\bF$)}\\
		&=\bD \bF &\qquad &\text{(since $\bF=\bF[:,J_s]\bW$)}\\
		&=\bA.
	\end{aligned}
\end{equation}
We notice that $\bF$ spans the same row space as $\bA$: $\cspace(\bF^\top)=\cspace(\bA^\top)$.
The finding above tells us that as long as we find an $r\times n$ matrix $\bD$ spanning the same row space of $\bA$, a column ID can be applied on $\bF$ to find the column ID of $\bA$. This reveals the randomized algorithm for computing column ID of $\bA$. The is actually the second part of the proof of column ID in Section~\ref{section:proof-column-id}.

We may further notice that suppose columns of $\bG$ contain the column basis of $\bA$, $\bA^\top\bG$ will contain row basis of $\bA$ by Lemma~\ref{lemma:column-basis-from-row-basis} (p.~\pageref{lemma:column-basis-from-row-basis}) again. $\bA^\top\bA\bA^\top\bG$ will span the row space of $\bA$ where $\bA^\top\bA$ on the left can help shrink small singular values:
$$
\bA^\top\bA\bA^\top\bG = (\bV\bSigma\bU^\top) (\bU\bSigma\bV^\top)(\bV\bSigma\bU^\top)\bG=\bV\bSigma^3\bU^\top\bG.
$$
The randomized procedure to compute column ID is formulated in Algorithm~\ref{alg:column-id-randomize1} where the difference compared to row ID is made in \textcolor{blue}{blue} texts.

\section*{Introduction and Background}
\addcontentsline{toc}{section}{Introduction and Background}

Matrix decomposition has become a core technology in statistics \citep{banerjee2014linear, gentle1998numerical}, optimization \citep{gill2021numerical}, recommender system \citep{symeonidis2016matrix}, and machine learning \citep{goodfellow2016deep, bishop2006pattern}, largely due to the development of back propagation algorithm in fitting a neural network. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, and Hilbert space. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields. Some excellent examples include \citep{householder2006principles, trefethen1997numerical, strang1993introduction, stewart2000decompositional, gentle2007matrix, higham2002accuracy, quarteroni2010numerical, golub2013matrix, beck2017first, gallier2017fundamentals, boyd2018introduction, strang2019linear, van2020advanced, strang2021every}. Moreover,
this survey will cover the calculation and complexity of the decompositional methods with details on the cost reduction. For a compact text of only rigorous proof, one can refer to \citep{lu2022matrix}.

A matrix decomposition is a way of reducing a complex matrix into its constituent parts which are in simpler forms.
The underlying principle of the decompositional approach to matrix computation is that it is not the
business of the matrix algorithmists to solve particular problems, but
it is an approach that can simplify more complex matrix operations which can be performed on the decomposed parts rather than on the original matrix itself. At a general level, a matrix decomposition task on matrix $\bA$ can be cast as
\begin{itemize}
\item $\bA=\bQ\bU$: where $\bQ$ is an orthogonal matrix that contains the same column space as $\bA$ and $\bU$ is a relatively simple and sparse matrix to reconstruct $\bA$.
\item $\bA=\bQ\bT\bQ^\top$: where $\bQ$ is orthogonal such that $\bA$ and $\bT$ are \textit{similar matrices} that share the same properties such as same eigenvalues, sparsity. Moreover, working on $\bT$ is an easier task compared to that of $\bA$.
\item $\bA=\bU\bT\bV$: where $\bU, \bV$ are orthogonal matrices such that the columns of $\bU$ and the rows of $\bV$ constitute an orthonormal basis of the column space and row space of $\bA$ respectively.
\item $\underset{m\times n}{\bA}=\underset{m\times r}{\bB}\gap \underset{r\times n}{\bC}$: where $\bB,\bC$ are full rank matrices that can reduce the memory storage of $\bA$. In practice, a low-rank approximation $\underset{m\times n}{\bA}\approx \underset{m\times k}{\bD}\gap \underset{k\times n}{\bF}$ can be employed where $k<r$ is called the \textit{numerical rank} of the matrix such that  the matrix can be stored much more inexpensively and can be multiplied rapidly with
vectors or other matrices.
An approximation of the form $\bA=\bD\bF$ is useful for storing the matrix $\bA$ more frugally (we can store $\bD$ and $\bF$ using $k(m+n)$ floats, as opposed to $mn$ numbers for storing $\bA$), for efficiently computing a matrix-vector product $\bb = \bA\bx$ (via $\bc = \bF\bx$ and $\bb = \bD\bc$), for data interpretation, and much more.

\item A matrix decomposition, which though is usually expensive to
compute, can be reused to solve new problems involving the
original matrix in different scenarios, e.g., as long as the factorization of $\bA$ is obtained, it can be reused to solve the set of linear systems $\{\bb_1=\bA\bx_1, \bb_2=\bA\bx_2, \ldots, \bb_k=\bA\bx_k\}$.
\item More generally, a matrix decomposition can help to understand the internal meaning of  what happens when multiplied by the matrix such that each constituent has a geometrical transformation (see Section~\ref{section:coordinate-transformation}, p.~\pageref{section:coordinate-transformation}).
\end{itemize}

The matrix decomposition algorithms can fall into many categories. Nonetheless, six categories hold the center and we sketch it here:
\begin{enumerate}
	\item Factorizations arise from Gaussian elimination including the LU decomposition and its positive definite alternative - Cholesky decomposition;
	\item Factorizations obtained when orthogonalizing the columns or the rows of a matrix such that the data can be explained well in an orthonormal basis;
	\item Factorizations where the matrices are skeletoned such that a subset of the columns or the rows can represent the whole data in a small reconstruction error, whilst, the sparsity and nonnegativity of the matrices are kept as they are;
	\item Reduction to Hessenberg, tridiagonal, or bidiagonal form, as a result, the properties of the matrices can be explored in these reduced matrices such as rank, eigenvalues, and so on;
	\item Factorizations result from the computation of the eigenvalues of matrices;
	\item In particular, the rest can be cast as a special kind of decompositions that involve optimization methods, high-level ideas where the category may not be straightforward to determine.
\end{enumerate}

The world pictures for decomposition in Figure~\ref{fig:matrix-decom-world-picture} and \ref{fig:matrix-decom-world-picture2} connect each decomposition method by their internal relations and also separate different methods by the criteria or prerequisites of them. Readers will get more information about the two pictures after reading the text.

\paragraph{Notation and preliminaries} In the rest of this section we will introduce and recap some basic knowledge about linear algebra. For the rest of the important concepts, we define and discuss them as per need for clarity.
The readers with enough background in matrix analysis can skip this section.
In the text, we simplify matters by considering only matrices that are real. Without special consideration, the eigenvalues of the discussed matrices are also real. We also assume throughout that $||\cdot||=||\cdot||_2$.

In all cases, scalars will be denoted in a non-bold font possibly with subscripts (e.g., $a$, $\alpha$, $\alpha_i$). We will use \textbf{boldface} lower case letters possibly with subscripts to denote vectors (e.g., $\bmu$, $\bx$, $\bx_n$, $\bz$) and
\textbf{boldface} upper case letters possibly with subscripts to denote matrices (e.g., $\bA$, $\bL_j$). The $i$-th element of a vector $\bz$ will be denoted by $\bz_i$ in bold font (or $z_i$ in the non-bold font).

The $n$-th element in a sequence is denoted by a superscript in parentheses, e.g., $\bA^{(n)}$ denotes the $n$-th matrix in a sequence, $\ba^{(k)}$ denote the $k$-th vector in a sequence.

Subarrays are formed when a subset of the indices is fixed.
\textit{The $i$-th row and $j$-th column value of matrix $\bA$ (entry ($i,j$) of $\bA$) will be denoted by $\bA_{ij}$ if block submatrices are involved, or by $a_{ij}$ alternatively if block submatrices are not involved}. Furthermore, it will be helpful to utilize the \textbf{Matlab-style notation}, the $i$-th row to the $j$-th row and the $k$-th column to the $m$-th column submatrix of the matrix $\bA$ will be denoted by $\bA_{i:j,k:m}$. A colon is used to indicate all elements of a dimension, e.g., $\bA_{:,k:m}$ denotes the $k$-th column to the $m$-th column of the matrix $\bA$, and $\bA_{:,k}$ denote the $k$-th column of $\bA$. Alternatively, the $k$-th column of $\bA$ may be denoted more compactly by $\ba_k$.

When the index is not continuous, given ordered subindex sets $I$ and $J$, $\bA[I, J]$ denotes the submatrix of $\bA$ obtained by extracting the rows and columns of $\bA$ indexed by $I$ and $J$, respectively; and $\bA[:, J]$ denotes the submatrix of $\bA$ obtained by extracting the columns of $\bA$ indexed by $J$.

\begin{definition}[Matlab Notation]\label{definition:matlabnotation}
Suppose $\bA\in \real^{m\times n}$, and $I=[i_1, i_2, \ldots, i_k]$ and $J=[j_1, j_2, \ldots, j_l]$ are two index vectors, then $\bA[I,J]$ denotes the $k\times l$ submatrix
$$
\bA[I,J]=
\begin{bmatrix}
\bA_{i_1,j_1} & \bA_{i_1,j_2} &\ldots & \bA_{i_1,j_l}\\
\bA_{i_2,j_1} & \bA_{i_2,j_2} &\ldots & \bA_{i_2,j_l}\\
\vdots & \vdots&\ddots & \vdots\\
\bA_{i_k,j_1} & \bA_{i_k,j_2} &\ldots & \bA_{i_k,j_l}\\
\end{bmatrix}.
$$
Whilst, $\bA[I,:]$ denotes the $k\times n$, and $\bA[:,J]$ denotes the $m\times l$ analogously.

We note that it does not matter whether the index vectors $I,J$ are row vectors or column vectors. It matters which axis they index (rows of $\bA$ or columns of $\bA$). We should also notice that range of the index:
$$
\left\{
\begin{aligned}
0&\leq \min(I) \leq \max(I)\leq m;\\
0&\leq \min(J) \leq \max(J)\leq n.
\end{aligned}
\right.
$$
\end{definition}

 And in all cases, vectors are formulated in a column rather than in a row. A row vector will be denoted by a transpose of a column vector such as $\ba^\top$. A specific column vector with values is split by the symbol $``;"$, e.g., $\bx=[1;2;3]$ is a column vector in $\real^3$. Similarly, a specific row vector with values is split by the symbol $``,"$, e.g., $\by=[1,2,3]$ is a row vector with 3 values. Further, a column vector can be denoted by the transpose of a row vector e.g., $\by=[1,2,3]^\top$ is a column vector.

The transpose of a matrix $\bA$ will be denoted by $\bA^\top$ and its inverse will be denoted by $\bA^{-1}$ . We will denote the $p \times p$ identity matrix by $\bI_p$. A vector or matrix of all zeros will be denoted by a \textbf{boldface} zero $\bzero$ whose size should be clear from context, or we denote $\bzero_p$ to be the vector of all zeros with $p$ entries.

\begin{definition}[Eigenvalue]
Given any vector space $E$ and any linear map $A: E \rightarrow E$, a scalar $\lambda \in K$ is called an eigenvalue, or proper value, or characteristic value of $\bA$ if there is some nonzero vector $\bu \in E$ such that
\begin{equation*}
\bA \bu = \lambda \bu.
\end{equation*}
\end{definition}

\begin{definition}[Spectrum and Spectral Radius]\label{definition:spectrum}
The set of all eigenvalues of $\bA$ is called the spectrum of $\bA$ and denoted by $\Lambda(\bA)$. The largest magnitude of the eigenvalues is known as the spectral radius $\rho(\bA)$:
$$
\rho(\bA) = \mathop{\max}_{\lambda\in \Lambda(\bA)}  |\lambda|.
$$
\end{definition}

\begin{definition}[Eigenvector]
A vector $\bu \in E$ is called an eigenvector, or proper vector, or characteristic vector of $\bA$ if $\bu \neq 0$ and if there is some $\lambda \in K$ such that
\begin{equation*}
\bA \bu  = \lambda \bu,
\end{equation*}
where the scalar $\lambda$ is then an eigenvalue. And we say that $\bu$ is an eigenvector associated with $\lambda$.
\end{definition}

Moreover, the tuple $(\lambda, \bu)$ above is said to be an \textbf{eigenpair}. Intuitively, the above definitions mean that multiplying matrix $\bA$ by the vector $\bu$ results in a new vector that is in the same direction as $\bu$, but only scaled by a factor $\lambda$. For any eigenvector $\bu$, we can scale it by a scalar $s$ such that $s\bu$ is still an eigenvector of $\bA$. That's why we call the eigenvector as an eigenvector of $\bA$ associated with eigenvalue $\lambda$. To avoid ambiguity, we usually assume that the eigenvector is normalized to have length $1$ and the first entry is positive (or negative) since both $\bu$ and $-\bu$ are eigenvectors.

In this context, we will highly use the idea about the linear independence of a set of vectors. Two equivalent definitions are given as follows.
\begin{definition}[Linearly Independent]
A set of vectors $\{\ba_1, \ba_2, \ldots, \ba_m\}$ is called linearly independent if there is no combination can get $x_1\ba_1+x_2\ba_2+\ldots+x_m\ba_m=0$ except all $x_i$'s are zero. An equivalent definition is that $\ba_1\neq \bzero$, and for every $k>1$, the vector $\ba_k$ does not belong to the span of $\{\ba_1, \ba_2, \ldots, \ba_{k-1}\}$.
\end{definition}

In the study of linear algebra, every vector space has a basis and every vector is a linear combination of members of the basis. We then define the span and dimension of a subspace via the basis.

\begin{definition}[Span]
	If every vector $\bv$ in subspace $\mathcal{V}$ can be expressed as a linear combination of $\{\ba_1, \ba_2, \ldots,$ $\ba_m\}$, then $\{\ba_1, \ba_2, \ldots, \ba_m\}$ is said to span $\mathcal{V}$.
\end{definition}

\begin{definition}[Subspace]
	A nonempty subset $\mathcal{V}$ of $\real^n$ is called a subspace if $x\ba+y\ba\in \mathcal{V}$ for every $\ba,\bb\in \mathcal{V}$ and every $x,y\in \real$.
\end{definition}

\begin{definition}[Basis and Dimension]
A set of vectors $\{\ba_1, \ba_2, \ldots, \ba_m\}$ is called a basis of $\mathcal{V}$ if they are linearly independent and span $\mathcal{V}$. Every basis of a given subspace has the same number of vectors, and the number of vectors in any basis is called the dimension of the subspace $\mathcal{V}$. By convention, the subspace $\{\bzero\}$ is said to  have dimension zero. Furthermore, every subspace of nonzero dimension has a basis that is orthogonal, i.e., the basis of a subspace can be chosen orthogonal.
\end{definition}

\begin{definition}[Column Space (Range)]
	If $\bA$ is an $m \times n$ real matrix, we define the column space (or range) of $\bA$ to be the set spanned by its columns:
	\begin{equation*}
		\mathcal{C} (\bA) = \{ \by\in \mathbb{R}^m: \exists \bx \in \mathbb{R}^n, \, \by = \bA \bx \}.
	\end{equation*}
	And the row space of $\bA$ is the set spanned by its rows, which is equal to the column space of $\bA^\top$:
	\begin{equation*}
		\mathcal{C} (\bA^\top) = \{ \bx\in \mathbb{R}^n: \exists \by \in \mathbb{R}^m, \, \bx = \bA^\top \by \}.
	\end{equation*}
\end{definition}

\begin{definition}[Null Space (Nullspace, Kernel)]
	If $\bA$ is an $m \times n$ real matrix, we define the null space (or kernel, or nullspace) of $\bA$ to be the set:
	\begin{equation*}
		\nspace (\bA) = \{\by \in \mathbb{R}^n:  \, \bA \by = \bzero \}.
	\end{equation*}
	And the null space of $\bA^\top$ is defined as 	\begin{equation*}
		\nspace (\bA^\top) = \{\bx \in \mathbb{R}^m:  \, \bA^\top \bx = \bzero \}.
	\end{equation*}
\end{definition}

Both the column space of $\bA$ and the null space of $\bA^\top$ are subspaces of $\real^n$. In fact, every vector in $\nspace(\bA^\top)$ is perpendicular to $\cspace(\bA)$ and vice versa.\footnote{Every vector in $\nspace(\bA)$ is also perpendicular to $\cspace(\bA^\top)$ and vice versa.}

\begin{definition}[Rank]
The $rank$ of a matrix $\bA\in \real^{m\times n}$ is the dimension of the column space of $\bA$. That is, the rank of $\bA$ is equal to the maximal number of linearly independent columns of $\bA$, and is also the maximal number of linearly independent rows of $\bA$. The matrix $\bA$ and its transpose $\bA^\top$ have the same rank. We say that $\bA$ has full rank, if its rank is equal to $min\{m,n\}$. In another word, this is true if and only if either all the columns of $\bA$ are linearly independent, or all the rows of $\bA$ are linearly independent. Specifically, given a vector $\bu \in \real^m$ and a vector $\bv \in \real^n$, then the $m\times n$ matrix $\bu\bv^\top$ obtained by the outer product of vectors is of rank 1. In short, the rank of a matrix is equal to:
\begin{itemize}
\item number of linearly independent columns;
\item number of linearly independent rows;
\item and remarkably, these are always the same (see Appendix~\ref{append:row-equal-column}, p.~\pageref{append:row-equal-column}).
\end{itemize}
\end{definition}

\begin{definition}[Orthogonal Complement in General]
The orthogonal complement $\mathcalV^\perp$ of a subspace $\mathcalV$ contains every vector that is perpendicular to $\mathcalV$. That is,
$$
\mathcalV^\perp = \{\bv | \bv^\top\bu=0, \,\,\, \forall \bu\in \mathcalV  \}.
$$
The two subspaces are disjoint that span the entire space. The dimensions of $\mathcalV$ and $\mathcalV^\perp$ add to the dimension of the whole space. Furthermore, $(\mathcalV^\perp)^\perp=\mathcalV$.
\end{definition}

\begin{definition}[Orthogonal Complement of Column Space]
	If $\bA$ is an $m \times n$ real matrix, the orthogonal complement of $\mathcal{C}(\bA)$, $\mathcal{C}^{\bot}(\bA)$ is the subspace defined as:
	\begin{equation*}
		\begin{aligned}
			\mathcal{C}^{\bot}(\bA) &= \{\by\in \mathbb{R}^m: \, \by^\top \bA \bx=\bzero, \, \forall \bx \in \mathbb{R}^n \} \\
			&=\{\by\in \mathbb{R}^m: \, \by^\top \bv = \bzero, \, \forall \bv \in \mathcal{C}(\bA) \}.
		\end{aligned}
	\end{equation*}
\end{definition}
Then we have the four fundamental spaces for any matrix $\bA\in \real^{m\times n}$ with rank $r$:
\begin{description}
\item $\bullet$ $\cspace(\bA)$: Column space of $\bA$, i.e., linear combinations of columns with dimension $r$;

\item $\bullet$ $\nspace(\bA)$: Null space of $\bA$, i.e., all $\bx$ with $\bA\bx=0$ with dimension $n-r$;

\item $\bullet$ $\cspace(\bA^\top)$: Row space of $\bA$, i.e., linear combinations of rows with dimension $r$;

\item $\bullet$ $\nspace(\bA^\top)$: Left null space of $\bA$, i.e., all $\by$ with $\bA^\top \by=0$ with dimension $m-r$,
\end{description}
where $r$ is the rank of the matrix. Furthermore, $\nspace(\bA)$ is the orthogonal complement to $\cspace(\bA^\top)$, and $\cspace(\bA)$ is the orthogonal complement to $\nspace(\bA^\top)$. The proof can be found in Appendix~\ref{appendix:fundamental-rank-nullity}.

\begin{definition}[Orthogonal Matrix]
A real square matrix $\bQ$ is an orthogonal matrix if the inverse of $\bQ$ equals the transpose of $\bQ$, that is $\bQ^{-1}=\bQ^\top$ and $\bQ\bQ^\top = \bQ^\top\bQ = \bI$. In another word, suppose $\bQ=[\bq_1, \bq_2, \ldots, \bq_n]$ where $\bq_i \in \real^n$ for all $i \in \{1, 2, \ldots, n\}$, then $\bq_i^\top \bq_j = \delta(i,j)$ with $\delta(i,j)$ being the Kronecker delta function. If $\bQ$ contains only $\gamma$ of these columns with $\gamma<n$, then $\bQ^\top\bQ = \bI_\gamma$ stills holds with $\bI_\gamma$ being the $\gamma\times \gamma$ identity matrix. But $\bQ\bQ^\top=\bI$ will not be true. For any vector $\bx$, the orthogonal matrix will preserve the length: $||\bQ\bx|| = ||\bx||$.
\end{definition}

\begin{definition}[Permutation Matrix]\label{definition:permutation-matrix}
A permutation matrix $\bP$ is a square binary matrix that has exactly one entry of 1 in each row and each column and 0's elsewhere.
\paragraph{Row Point} That is, the permutation matrix $\bP$ has the rows of the identity $\bI$ in any order and the order decides the sequence of the row permutation. Suppose we want to permute the rows of matrix $\bA$, we just multiply on the left by $\bP\bA$.
\paragraph{Column Point} Or, equivalently, the permutation matrix $\bP$ has the columns of the identity $\bI$ in any order and the order decides the sequence of the column permutation. And now, the column permutation of $\bA$ is to multiply on the right by $\bA\bP$.
\end{definition}

The permutation matrix $\bP$ can be more efficiently represented via a vector $J \in \integer_+^n$ of indices such that $\bP = \bI[:, J]$ where $\bI$ is the $n\times n$ identity matrix and notably, the elements in vector $J$ sum to $1+2+\ldots+n= \frac{n^2+n}{2}$.

\begin{example}[Permutation]
Suppose,
$$\bA=\begin{bmatrix}
1 & 2&3\\
4&5&6\\
7&8&9
\end{bmatrix}
,\qquad \text{and} \qquad
\bP=\begin{bmatrix}
	&1&\\
	&&1\\
	1&&
\end{bmatrix}.
$$
The row permutation is given by
$$
\bP\bA = \begin{bmatrix}
4&5&6\\
7&8&9\\
1 & 2&3\\
\end{bmatrix},
$$
where the order of the rows of $\bA$ appearing in $\bP\bA$ matches the order of the rows of $\bI$ in $\bP$. And the column permutation is given by
$$
\bA\bP = \begin{bmatrix}
3 & 1 & 2 \\
6 & 4 & 5\\
9 & 7 & 8
\end{bmatrix},
$$
where the order of the columns of $\bA$ appearing in $\bA\bP$ matches the order of the columns of $\bI$ in $\bP$. \exampbar
\end{example}

\begin{definition}[Selection Matrix]\label{definition:selection-matrix}
A selection matrix $\bS$ is a square diagonal matrix with diagonals being 1 or 0.  The 1 entries are the rows or columns that will be selected.
\paragraph{Row Point} That is, the selection matrix $\bS$ has the rows of the identity $\bI$ if we want to select the corresponding rows, and otherwise, we mask the rows in the identity $\bI$ by zero.
Suppose we want to select the rows of matrix $\bA$, we just multiply from left by $\bS\bA$.
\paragraph{Column Point} Or, equivalent, the selection matrix $\bS$ has the columns of the identity $\bI$ if we want to select the corresponding columns, or otherwise, we mask the columns in the identity $\bI$ by zero. And now, the column selection of $\bA$ is to multiply from right by $\bA\bS$.
\end{definition}

\begin{example}[Selection and Permutation]
	Suppose,
	$$\bA=\begin{bmatrix}
		1 & 2&3\\
		4&5&6\\
		7&8&9
	\end{bmatrix}
	,\qquad \text{and} \qquad
	\bS=\begin{bmatrix}
		1&&\\
		& 0&\\
		 &&1
	\end{bmatrix}.
	$$
	The row selection is given by
	$$
	\bS\bA = \begin{bmatrix}
		1&2&3\\
		0&0&0\\
		7 & 8&9\\
	\end{bmatrix},
	$$
	where the row of $\bA$ appearing in $\bS\bA$ matches the row entries of $\bS$. And the column selection is given by
	$$
	\bA\bS = \begin{bmatrix}
		1& 0 & 3 \\
		4 & 0 & 6\\
		7 & 0 & 9
	\end{bmatrix},
	$$
	where the columns of $\bA$ appearing in $\bA\bS$ matches column entries of $\bS$. If now, we want to reorder the selected rows or columns in the upper left of the final matrix, we can construct a permutation as follows
	$$
	\bP=\begin{bmatrix}
		1&&\\
		& & 1\\
		&1&
	\end{bmatrix},
	$$
	such that
	$$
	\bP\bS\bA = \begin{bmatrix}
		1&2&3\\
		7 & 8&9\\
		0&0&0\\
	\end{bmatrix},
	$$
	and
	$$
	\bA\bS\bP = \begin{bmatrix}
		1 & 3& 0 \\
		4  & 6& 0\\
		7  & 9& 0
	\end{bmatrix}.
	$$
	The trick is essential to some mathematical proofs, e.g., the properties of positive definite matrices in Lemma~\ref{lemma:pd-more-properties}. \exampbar
\end{example}

From an introductory course on linear algebra, we have the following remark on the equivalent claims on nonsingular matrices.
\begin{remark}[List of Equivalence of Nonsingularity for a Matrix]
For a square matrix $\bA\in \real^{n\times n}$, the following claims are equivalent:
\begin{itemize}
\item $\bA$ is nonsingular;
\item $\bA$ is invertible, i.e., $\bA^{-1}$ exists;
\item $\bA\bx=\bb$ has a unique solution $\bx = \bA^{-1}\bb$;
\item $\bA\bx = \bzero$ has a unique, trivial solution: $\bx=\bzero$;
\item Columns of $\bA$ are linearly independent;
\item Rows of $\bA$ are linearly independent;
\item $\det(\bA) \neq 0$;
\item $\dim(\nspace(\bA))=0$;
\item $\nspace(\bA) = \{\bzero\}$, i.e., the null space is trivial;
\item $\cspace(\bA)=\cspace(\bA^\top) = \real^n$, i.e., the column space or row space span the whole $\real^n$;
\item $\bA$ has full rank $r=n$;
\item The reduced row echelon form is $\bR=\bI$;
\item $\bA^\top\bA$ is symmetric positive definite;
\item $\bA$ has $n$ nonzero (positive) singular values;
\item All eigenvalues are nonzero;
\end{itemize}
\end{remark}
It will be shown important to take the above equivalence into mind, otherwise, we will easily get lost. On the other hand, the following remark also shows the equivalent claims for singular matrices.
\begin{remark}[List of Equivalence of Singularity for a Matrix]
For a square matrix $\bA\in \real^{n\times n}$ with eigenpair $(\lambda, \bu)$, the following claims are equivalent:
\begin{itemize}
	\item $(\bA-\lambda\bI)$ is singular;
	\item $(\bA-\lambda\bI)$ is not invertible;
	\item $(\bA-\lambda\bI)\bx = \bzero$ has nonzero $\bx\neq \bzero$ solutions, and $\bx=\bu$ is one of such solutions;
	\item $(\bA-\lambda\bI)$ has linearly dependent columns;
	\item $\det(\bA-\lambda\bI) = 0$;
	\item $\dim(\nspace(\bA-\lambda\bI))>0$;
	\item Null space of $(\bA-\lambda\bI)$ is nontrivial;
	\item Columns of $\bA$ are linearly dependent;
	\item Rows of $\bA$ are linearly dependent;
	\item $\bA$ has rank $r<n$;
	\item Dimension of column space = dimension of row space = $r<n$;
	\item $\bA^\top\bA$ is symmetric semidefinite;
	\item $\bA$ has $r<n$ nonzero (positive) singular values;
	\item Zero is an eigenvalue of $\bA$
\end{itemize}
\end{remark}

\subsection*{Matrix Decomposition in a Nutshell}
We briefly overview the different decompositional methods that we will cover in this text as follows:
\begin{enumerate}[leftmargin=*]
\item \textit{LU.} $\bA=\bL\bU =
\left(\parbox{8em}{lower triangular $\bL$\\ 1's on the diagonal}\right)
\left(\parbox{9.5em}{upper triangular $\bU$\\pivots on the diagonal}\right)$

\textbf{Requirements:} $\bA$ has nonzero leading principal minors, i.e., no row permutations involve to reduce $\bA$ to upper triangular via the Gaussian elimination (Theorem~\ref{theorem:lu-factorization-without-permutation}, p.~\pageref{theorem:lu-factorization-without-permutation}).

\item \textit{LU.} $\bA=\bL\bD\bU = \left(\parbox{8em}{lower triangular $\bL$\\ 1's on the diagonal}\right)
\left(\parbox{6em}{pivot matrix  \\$\bD$ is diagonal}\right)
\left(\parbox{8.5em}{upper triangular $\bU$\\1's on the diagonal}\right)$

\textbf{Requirements:} $\bA$ has nonzero leading principal minors, i.e., no row permutations involves to reduce $\bA$ to upper triangular via the Gaussian elimination. The $\bD$ contains the pivots such that $\bU$ has 1's on the diagonal. When $\bA$ is symmetric, it follows that $\bA=\bL\bD\bL^\top$ (Corollary~\ref{corollary:ldu-decom}, p.~\pageref{corollary:ldu-decom}).

\item \textit{PLU.} $\bA=\bP\bL\bU =\left(\parbox{7em}{permutation \\matrix  $\bP$ avoids\\ zeros for\\ eliminations}\right) \left(\parbox{8em}{lower triangular $\bL$\\ 1's on the diagonal}\right)\left(\parbox{9.5em}{upper triangular $\bU$\\pivots on the diagonal}\right)$

\textbf{Requirements:} $\bA$ is nonsingular, then $\bP,\bL,\bU$ are nonsingular as well. $\bP^\top$ does the row permutation in advance such that $\bP^\top\bA$ has nonzero leading principle minors (Theorem~\ref{theorem:lu-factorization-with-permutation}, p.~\pageref{theorem:lu-factorization-with-permutation}).

\item \textit{RRLU.} $\bP\bA\bQ = \begin{bmatrix}
	\bL_{11} & \bzero \\
	\bL_{21}^\top & \bI
\end{bmatrix}
\begin{bmatrix}
	\bU_{11} & \bU_{12} \\
	\bzero & \bzero
\end{bmatrix}$

\textbf{Requirements}: Any matrix with rank $r$ such that $\bP,\bQ$ are permutations, $\bL_{11}$ is lower triangular with 1's on the diagonal, $\bU_{11}$ is upper triangular with pivots on the diagonal where both of them reveal the rank of matrix $\bA$ (Section~\ref{section:rank-reveal-lu-short}, p.~\pageref{section:rank-reveal-lu-short}).

\item \textit{Complete Pivoting LU.} $\bP\bA\bQ = \bL\bU=
\left(\parbox{8em}{lower triangular $\bL$\\ 1's on the diagonal}\right)\left(\parbox{9.5em}{upper triangular $\bU$\\pivots on the diagonal}\right)$

\textbf{Requirements}: $\bA$ is nonsingular such that $\bP,\bQ$ are permutations, $\bL$ is unit lower triangular with 1's on the diagonal, $\bU$ is upper triangular with pivots on the diagonal (Section~\ref{section:complete-pivoting}, p.~\pageref{section:complete-pivoting}).

\item \textit{Cholesky.} $\bA=\bR^\top\bR$=(lower triangular)(upper triangular)

\textbf{Requirements:} $\bA$ is positive definite (symmetric) such that the diagonal of $\bR$ contains the diagonals of $\sqrt{\bD}$ which are positive from the LU decomposition. Trivially, $\bR^\top$ can be set as $\bL\sqrt{\bD}$ from the LU decomposition. When $\bA$ is positive semidefinite, $\bR$ is upper triangular with possible zeros on the diagonal (Theorem~\ref{theorem:cholesky-factor-exist}, p.~\pageref{theorem:cholesky-factor-exist}; Theorem~\ref{theorem:semidefinite-factor-exist}, p.~\pageref{theorem:semidefinite-factor-exist}).

\item \textit{Pivoted Cholesky.} $\bP\bA\bP^\top = \bR^\top\bR $=(lower triangular)(upper triangular)

\textbf{Requirements:} $\bA$ is positive definite (symmetric) such that $\bR$ is upper triangular (Section~\ref{section:volum-picot-cholesjy}, p.~\pageref{section:volum-picot-cholesjy}).

\item \textit{Semidefinite Rank-Revealing.} $\bP^\top \bA\bP  = \bR^\top\bR, \qquad \mathrm{with} \qquad
\bR = \begin{bmatrix}
	\bR_{11} & \bR_{12}\\
	\bzero &\bzero
\end{bmatrix} $

\textbf{Requirements:} $\bA$ is positive semidefinite with rank $r$ such that after permuting by $\bP$, $\bR_{11}$ is upper triangular with rank $r$ (Theorem~\ref{theorem:semidefinite-factor-rank-reveal}, p.~\pageref{theorem:semidefinite-factor-rank-reveal}).

\item \textit{CR.} $\bA=\bC\bR$=(first independent columns of $\bA$)(basis for row space of $\bA$)

\textbf{Requirements:} Any matrix $\bA$ with rank $r$, $\bC$ contain first $r$ linearly independent columns of $\bA$, $\bR$ is the reduced row echelon form removing zero rows. $\bR$ contains an $r\times r$ identity submatrix (Theorem~\ref{theorem:cr-decomposition}, p.~\pageref{theorem:cr-decomposition}).

\item \textit{Rank.} $\bA=\bD\bF$=(column basis of $\bA$)(row basis of $\bA$)

\textbf{Requirements:} Any matrix $\bA$ with rank $r$, $\bD$ contains $r$ columns that span the column space of $\bA$, $\bF$ contains $r$ rows that span the row space of $\bA$ (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}).

\item \textit{Skeleton.} $\bA=\bC\bU^{-1}\bR$=($r$ columns of $\bA$)(intersection of $\bC,\bR$)$^{-1}$($r$ rows of $\bA$)

\textbf{Requirements:} Any matrix $\bA$ with rank $r$ such that $\bC, \bR$ come directly from columns and rows of $\bA$. $\bU$ is the mixing matrix that on the intersection of $\bC,\bR$ (Theorem~\ref{theorem:skeleton-decomposition}, p.~\pageref{theorem:skeleton-decomposition}).

\item \textit{Interpolative.} $\bA=\bC\bW$=(independent columns of $\bA$)(columns consisting of identity matrix)

\textbf{Requirements:} Any matrix $\bA$ with rank $r$ such that $\bW$ contains an $r\times r$ identity submatrix (in the sense of permutation). The elements of $\bW$ are no greater than 1 in absolute value (Theorem~\ref{theorem:interpolative-decomposition}, p.~\pageref{theorem:interpolative-decomposition}; Theorem~\ref{theorem:interpolative-decomposition-row}, p.~\pageref{theorem:interpolative-decomposition-row}).

\item \textit{QR.} $\bA = \bQ\bR$=(orthonormal columns in $\bQ$)(upper triangular $\bR$)

\textbf{Requirements:} Rectangular matrix $\bA$ has linearly independent columns otherwise $\bR$ will be singular (diagonals contain at least one zero). In the full QR decomposition, $\bQ$ is orthogonal such that $\bQ^{-1}=\bQ^\top$ (Theorem~\ref{theorem:qr-decomposition}, p.~\pageref{theorem:qr-decomposition}).

\item \textit{CPQR.}
$\bA = \bQ
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix}\bP^\top
=\left(\parbox{5.9em}{orthogonal $\bQ$}\right)
\left(\parbox{8.em}{upper triangular $\bR_{11}$ $|$ full matrix $\bR_{12}$}\right)
\left(\parbox{5.2em}{permutation $\bP^\top$}\right)$

\textbf{Requirements:} Any rectangular matrix $\bA$ with rank $r$ such that $\bR_{11}$ is $r\times r$ upper triangular (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}).

\item \textit{RRQR.}
$\bA = \bQ
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bR_{22}
\end{bmatrix}\bP^\top
=\left(\parbox{5.9em}{orthogonal $\bQ$}\right)
\left(\parbox{8.em}{upper triangular $\bR_{11}$ $|$ full matrix $\bR_{12}$ $|$ $\bR_{22}$ small in norm}\right)
\left(\parbox{5.2em}{permutation $\bP^\top$}\right)$

\textbf{Requirements:} Any rectangular matrix $\bA$ with rank $r$ such that $\bR_{11}$ is $r\times r$ upper triangular and $\bR_{22}$ is small in norm (Section~\ref{section:rank-r-qr}, p.~\pageref{section:rank-r-qr}).

\item \textit{LQ.} $\bA=\bL\bQ$=(lower triangular $\bL$)(orthonormal rows in $\bQ$)

\textbf{Requirements:} Rectangular matrix $\bA$ has linearly independent rows otherwise $\bL$ will be singular (diagonals contain at least one zero). In the full LQ decomposition, $\bQ$ is orthogonal such that $\bQ^{-1}=\bQ^\top$ (Theorem~\ref{theorem:lq-decomposition}, p.~\pageref{theorem:lq-decomposition}).

\item \textit{RPLQ.}
$\bA = \bP^\top
\begin{bmatrix}
		\bL_{11} & \bzero \\
\bL_{21} & \bzero
\end{bmatrix}\bQ
=\left(\text{permutation $\bP^\top$}\right)
\left(\parbox{8.em}{lower triangular $\bL_{11}$ $|$ full matrix $\bL_{21}$}\right)
\left(\parbox{5.9em}{orthogonal $\bQ$}\right)$

\textbf{Requirements:} Any rectangular matrix $\bA$ with rank $r$ such that $\bL_{11}$ is $r\times r$ lower triangular (Section~\ref{section:lq-decomp}, p.~\pageref{section:lq-decomp}).

\item \textit{Two-Sided Orthogonal.} $\bA\bP\bA=\bU\bF\bV^\top=\left(\parbox{5.3em}{orthonormal  \\ basis in $\bU$}\right)
\left(\parbox{4.5em}{Upper-left \\$r\times r$\\ submatrix\\ is nonzero}\right)
\left(\parbox{5.3em}{orthonormal \\ basis in $\bV$}\right)$

\textbf{Requirements:} Square matrix with rank $r$ such that first $r$ columns of $\bU$ span the column space of $\bA$, and the rest columns span the null space of $\bA^\top$. Whilst first $r$ columns of $\bV$ span the row space of $\bA$ and the rest columns span the null space of $\bA$ (Theorem~\ref{theorem:two-sided-orthogonal}, p.~\pageref{theorem:two-sided-orthogonal}).

\item \textit{UTV.} $\bA = \bU\bT\bV$=(orthogonal $\bU$)
$\left(\parbox{8.4em}{upper triangular $\bR$\\lower triangular $\bL$}\right)$
(orthogonal $\bV$)

\textbf{Requirements:} Any matrix $\bA$ with rank $r$ such that $\bT$ is lower or upper triangular with rank $r$ (Theorem~\ref{theorem:ulv-decomposition}, p.~\pageref{theorem:ulv-decomposition}).

\item \textit{Hessenberg.} $\bA=\bQ\bH\bQ^\top$=(orthogonal matrix $\bQ$)(Hessenberg matrix $\bH$)($\bQ^\top=\bQ^{-1}$)

\textbf{Requirements:} Any square matrix $\bA$ such that $\bA,\bH$ are orthogonal similar matrices with the same rank, trace, eigenvalues (Theorem~\ref{theorem:hessenberg-decom}, p.~\pageref{theorem:hessenberg-decom}).

\item \textit{Tridiagonal.} $\bA=\bQ\bT\bQ^\top$=(orthogonal $\bQ$)(Symmetric tridiagonal $\bT$)($\bQ^\top=\bQ^{-1}$)

\textbf{Requirements:} Any symmetric matrix $\bA$ such that $\bA,\bT$ are orthogonal similar matrices with same rank, trace, eigenvalues (Theorem~\ref{definition:tridiagonal-hessenbert}, p.~\pageref{definition:tridiagonal-hessenbert}).

\item \textit{Bidiagonal.} $\bA=\bU\bB\bV^\top$=(orthogonal matrix $\bU$)(Bidiagonal $\bB$)(orthogonal matrix $\bV$)

\textbf{Requirements:} Any rectangular matrix $\bA$ such that $\bB$ is upper bidiagonal. $\bT=\bB^\top\bB$ is tridiagonal if $\bB$ is bidiagonal such that $\bA^\top\bA = \bV\bB^\top\bB\bV^\top$ is the tridiagonal decomposition of $\bA^\top\bA$ (Theorem~\ref{theorem:Golub-Kahan-Bidiagonalization-decom}, p.~\pageref{theorem:Golub-Kahan-Bidiagonalization-decom}).

\item \textit{Eigenvalue.} $\bA = \bX\bLambda\bX^{-1}
=\left(\parbox{6.5em}{eigenvectors in\\ columns of $\bX$}\right)
\left(\parbox{6.1em}{eigenvalues in \\diagonal of $\bLambda$}\right)
\left(\parbox{7em}{left eigenvectors \\in $\bX^{-1}$}\right)$

\textbf{Requirements:} Square matrix $\bA$ has $n$ linearly independent eigenvectors (Theorem~\ref{theorem:eigenvalue-decomposition}, p.~\pageref{theorem:eigenvalue-decomposition}).

\item \textit{Schur.} $\bA = \bQ\bU\bQ^\top
=\left(\parbox{6.6em}{orthogonormal \\columns in $\bQ$}\right)
\left(\parbox{8.5em}{Upper triangular $\bU$}\right)
\left(\parbox{4.8em}{$\bQ^\top=\bQ^{-1}$}\right)$

\textbf{Requirements:} $\bA$ is any real matrix with real eigenvalues (Theorem~\ref{theorem:schur-decomposition}, p.~\pageref{theorem:schur-decomposition}).

\item \textit{Spectral.} $\bA=\bQ\bLambda\bQ^\top
=\left(\parbox{7.6em}{orthogonormal \\eigenvectors in $\bQ$}\right)
\left(\parbox{7.8em}{real eigenvalues in \\diagonal of $\bLambda$}\right)
\left(\parbox{7em}{left eigenvectors \\in $\bQ^{\top}$}\right)$

\textbf{Requirements:} $\bA$ is real and symmetric, which is a special case of the Schur decomposition (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}).

\item \textit{Jordan.} $\bA = \bX\bJ\bX^{-1}
=\left(\parbox{7.3em}{generalized eigenvectors \\in columns of $\bX$}\right)
\left(\parbox{6.4em}{Jordan blocks \\in $\bJ$}\right)
\left(\parbox{8.8em}{left generalized  \\eigenvectors in $\bX^{-1}$}\right)$

\textbf{Requirements:} $\bA$ is any square matrix, $\bJ$ is a Jordan form matrix which has several diagonal blocks that contains identical eigenvalues in each block (Theorem~\ref{theorem:jordan-decomposition}, p.~\pageref{theorem:jordan-decomposition}).

\item \textit{SVD.} $\bA=\bU \bSigma \bV^\top
=\left(\parbox{5.5em}{left singular \\vectors in $\bU$}\right)
\left(\parbox{7.6em}{singular values in \\diagonal of $\bSigma$}\right)
\left(\parbox{5.8em}{right singular \\vectors in $\bV$}\right)$

\textbf{Requirements:} Any matrix $\bA$ such that the diagonals of $\bSigma^2$ contain the eigenvalues of $\bA^\top\bA$ or $\bA\bA^\top$. In the full SVD, both $\bU$ and $\bV$ are orthogonal (Theorem~\ref{theorem:full_svd_rectangular}, p.~\pageref{theorem:full_svd_rectangular}).

\item \textit{Polar.} $\bA=\bQ_l\bS_l
=\left(\parbox{4.6em}{orthogonal matrix $\bQ_l$}\right)
\left(\parbox{6.5em}{positive \\semidefinite $\bS_l$}\right)$

\textbf{Requirements:} $\bA$ is any square matrix. The polar decomposition indicates $\bS_l^2 = \bA^\top\bA$. When $\bA$ is nonsingular, $\bS_l$ is positive definite. Moreover, there exists a right polar decomposition $\bA=\bS_r\bQ_r$ such that $\bS_r^2 = \bA\bA^\top$ (Theorem~\ref{theorem:polar-decomposition}, p.~\pageref{theorem:polar-decomposition}).

\item \textit{ALS.} $\bA \approx \bW\bZ$=$\left(s=rank(\bW)<rank(\bA)=r\right)$
$\left(s=rank(\bZ)<rank(\bA)=r\right)$

\textbf{Requirements:} For any rectangular matrix $\bA$, $\bW\bZ$ approximates $\bA$ in the sense of Frobenius norm (Section~\ref{section:als}, p.~\pageref{section:als}).

\item \textit{NMF.} $\bA \approx \bW\bZ$=$\left(s=rank(\bW)<rank(\bA)=r\right)$
$\left(s=rank(\bZ)<rank(\bA)=r\right)$

\textbf{Requirements:} For any rectangular matrix $\bA$, $\bW\bZ$ approximates $\bA$ in the sense of Frobenius norm with entries of $\bA,\bW,\bZ$ being nonegative (Section~\ref{section:nmf}, p.~\pageref{section:nmf}).

\item \textit{Biconjugate.} $\bA = \bPhi \bOmega^{-1} \bPsi^\top$

\textbf{Requirements:} Any rectangular matrix $\bA$ with rank $r$ such that $\bOmega$ is $r\times r$ diagonal. The columns of $\bPhi, \bPsi$ comes from the Wedderburn sequence (Theorem~\ref{theorem:biconjugate-form1}, p.~\pageref{theorem:biconjugate-form1}; Theorem~\ref{theorem:biconjugate-form2}, p.~\pageref{theorem:biconjugate-form2}).

\item \textit{CP.} $\eX \approx \llbracket \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r=1}^{R} \ba_r^{(1)} \circ \ba_r^{(2)} \circ \ldots \circ \ba_r^{(N)} $

\textbf{Requirements:} Any tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ such that the CP decomposition is a low-rank approximation of the original tensor (Theorem~\ref{theorem:cp-decomp}, p.~\pageref{theorem:cp-decomp}).

\item \textit{Tucker.} $\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}
\ba_r^{(1)} \circ \ba_r^{(2)} \circ \ldots \circ \ba_r^{(N)} $

\textbf{Requirements:} Any tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ such that the Tucker decomposition makes the principal components in high-order dimensions (Theorem~\ref{theorem:tucker-decomp}, p.~\pageref{theorem:tucker-decomp}).

\item \textit{HOSVD.} $\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}
\ba_r^{(1)} \circ \ba_r^{(2)} \circ \ldots \circ \ba_r^{(N)} $

\textbf{Requirements:} Any tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ such that the HOSVD makes the principal components in high-order dimensions where further the slices of $\eG$ are mutually orthogonal and the slices are ordered in a descending manner (Theorem~\ref{theorem:hosvd-decomp}, p.~\pageref{theorem:hosvd-decomp}).

\item \textit{TT.} $\eX \approx \eG^{(1)} \boxtimes \eG^{(2)} \boxtimes \ldots \boxtimes \eG^{(N)}$

\textbf{Requirements:} Any tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ such that the TT decomposition factors the tensor into $N$ third-order tensors (Theorem~\ref{theorem:ttrain-decomp}, p.~\pageref{theorem:ttrain-decomp}).

\end{enumerate}

\part{Triangularization, Orthogonalization and Gram-Schmidt Process}
\section*{Introduction}
Given an $m\times l$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_l]$, with $m\geq l$, the \textit{orthonormalization} admits $\bQ_l=[\bq_1, \bq_2, \ldots, \bq_l]$ such that
$$
span([\bq_1, \bq_2, \ldots, \bq_k]) = span([\bq_1, \ba_2, \ldots, \ba_k]), \qquad \text{for all } k\in \{1,2,\ldots, l\}.
$$
Whilst, columns of $\bQ_l$ are mutually orthonormal:
$$
\bq_i^\top\bq_j =
\left\{
\begin{aligned}
&1, \qquad \text{if $i=j$};\\
&0, \qquad \text{if $i\neq j$}.
\end{aligned}
\right\} \leadto  \bQ_l^\top\bQ_l = \bI_l.
$$
where $\bI_l$ is a $l\times l$ identity matrix. When we complete $\bQ_l$ into $m$ mutually orthonormal columns $\bQ=[\bq_1, \bq_2, \ldots, \bQ_m]$, $\bQ$ is a square matrix and it follows that $\bQ\bQ^\top=\bQ^\top\bQ=\bI_m$. Sometimes, this \textit{column completion} is done by the Gram-Schmidt process that we will introduce in the sequel.

For the rest of this part, we will discuss several factorization methods in the sense of orthogonalization above with different focuses, e.g., in terms of column space, row space, or both. The big picture can be shown in Figure~\ref{fig:orthgonal-world-picture}.

\begin{figure}[htbp]
\centering
\begin{widepage}
\centering
\resizebox{0.6\textwidth}{!}{%
\begin{tikzpicture}[>=latex]

\tikzstyle{state} = [draw, very thick, fill=white, rectangle, minimum height=3em, minimum width=6em, node distance=8em, font={\sffamily\bfseries}]
\tikzstyle{stateEdgePortion} = [black,thick];
\tikzstyle{stateEdge} = [stateEdgePortion,->];
\tikzstyle{stateEdge2} = [stateEdgePortion,<->];
\tikzstyle{edgeLabel} = [pos=0.5, text centered, font={\sffamily\small}];

\node[state, name=qr, node distance=7em, xshift=-9em, yshift=-1em, fill={colorqr}] {QR};
\node[state, name=lq, above of=qr, xshift=-10em, yshift=-4em, fill={colorqr}] {LQ};
\node[state, name=utv, above of=qr, xshift=10em, yshift=-4em, fill={colorqr}] {UTV};
\node[state, name=twosidedortho, draw, above of=qr,xshift=0em, yshift=0em,fill={colorqr}]  {\parbox{5em}{Two-Sided \\Orthogonal}};

\draw (lq.east)
edge[stateEdge] node[edgeLabel, yshift=0em, xshift=-0.7em]{\parbox{3em}{Rank\\Estimation}}
(utv.west) ;

\draw ($(qr.east)$)
edge[stateEdge, bend left=-12.5] node[edgeLabel, xshift=-0.3em,yshift=0.5em]{Rank Estimation}
(utv.south) ;

\draw ($(lq.north) + (0em,0em)$)
edge[stateEdge, bend left=12.5] node[edgeLabel, xshift=1.5em,yshift=-0.5em]{Phase 1}
(twosidedortho.west) ;
\draw (twosidedortho.east)
edge[stateEdge, bend left=+12.5] node[edgeLabel, xshift=-1.5em,yshift=-0.5em]{Phase 2}
($(utv.north) + (0em,0em)$);

\draw ($(qr.west)$)
edge[stateEdge, bend left=+12.5] node[edgeLabel, yshift=0.5em, xshift=1em]{Row Space}
(lq.south) ;

\begin{pgfonlayer}{background}
\draw [join=round,cyan,dotted,fill={colormiddleleft}] ($(qr.south west -| lq.west) + (-0.5em, -0.5em)$) rectangle ($(twosidedortho.north east -| utv.north east) + (0.6em, 0.5em)$);
\end{pgfonlayer}

\end{tikzpicture}
}
\end{widepage}
\caption{Orthogonalization World Map. See also where it's lying in the matrix decomposition world map Figure~\ref{fig:matrix-decom-world-picture}.}
\label{fig:orthgonal-world-picture}
\end{figure}

\newpage
\chapter{QR Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup
\section{QR Decomposition}
In many applications, we are interested in the column space of a matrix $\bA=[\ba_1, \ba_2, ..., \ba_n] \in \real^{m\times n}$. The successive spaces spanned by the columns $\ba_1, \ba_2, \ldots$ of $\bA$ are
$$
\cspace([\ba_1])\,\,\,\, \subseteq\,\,\,\, \cspace([\ba_1, \ba_2]) \,\,\,\,\subseteq\,\,\,\, \cspace([\ba_1, \ba_2, \ba_3])\,\,\,\, \subseteq\,\,\,\, \ldots,
$$
where $\cspace([\ldots])$ is the subspace spanned by the vectors included in the brackets. The idea of QR decomposition is the construction of a sequence of orthonormal vectors $\bq_1, \bq_2, \ldots$ that span the same successive subspaces.
$$
\bigg\{\cspace([\bq_1])=\cspace([\ba_1])\bigg\}\subseteq
\bigg\{\cspace([\bq_1, \bq_2])=\cspace([\ba_1, \ba_2])\bigg\}\subseteq
\bigg\{\cspace([\bq_1, \bq_2, \bq_3])=\cspace([\ba_1, \ba_2, \ba_3])\bigg\}
\subseteq \ldots,
$$
We provide the result of QR decomposition in the following theorem and we delay the discussion of its existence in the next sections.

\begin{theoremHigh}[QR Decomposition]\label{theorem:qr-decomposition}
Every $m\times n$ matrix $\bA=[\ba_1, \ba_2, ..., \ba_n]$ (whether linearly independent or dependent columns) with $m\geq n$ can be factored as
$$
\bA = \bQ\bR,
$$
where

\begin{enumerate}
\item  \textbf{Reduced}: $\bQ$ is $m\times n$ with orthonormal columns and $\bR$ is an $n\times n$ upper triangular matrix which is known as the \textbf{reduced QR decomposition};
\item \textbf{Full}: $\bQ$ is $m\times m$ with orthonormal columns and $\bR$ is an $m\times n$ upper triangular matrix which is known as the \textbf{full QR decomposition}. If further restrict the upper triangular matrix to be a square matrix, the full QR decomposition can be denoted as
$$
\bA = \bQ\begin{bmatrix}
	\bR_0\\
	\bzero
\end{bmatrix},
$$
where $\bR_0$ is an $m\times m$ upper triangular matrix.
\end{enumerate}

Specifically, when $\bA$ has full rank, i.e., has linearly independent columns, $\bR$ also has linearly independent columns, and $\bR$ is nonsingular for the \textit{reduced} case. This implies diagonals of $\bR$ are nonzero. Under this condition, when we further restrict elements on the diagonal of $\bR$ are positive, the \textit{reduced} QR decomposition is \textbf{unique}. The \textit{full} QR decomposition is normally not unique since the right-most $(m-n)$ columns in $\bQ$ can be in any order.
\end{theoremHigh}

\section{Project a Vector Onto Another Vector}\label{section:project-onto-a-vector}
Project a vector $\ba$ to a vector $\bb$ is to find the vector closest to $\ba$ on the line of $\bb$. The projection vector $\widehat{\ba}$ is some multiple of $\bb$. Let $\widehat{\ba} = \widehat{x} \bb$ and $\ba-\widehat{\ba}$ is perpendicular to $\bb$ as shown in Figure~\ref{fig:project-line}. We then get the following result:
\begin{tcolorbox}[title={Project Vector $\ba$ Onto Vector $\bb$}]
$\ba^\perp =\ba-\widehat{\ba}$ is perpendicular to $\bb$, so $(\ba-\widehat{x}\bb)^\top\bb=0$: $\widehat{x}$ = $\frac{\ba^\top\bb}{\bb^\top\bb}$ and $\widehat{\ba} = \frac{\ba^\top\bb}{\bb^\top\bb}\bb = \frac{\bb\bb^\top}{\bb^\top\bb}\ba$.
\end{tcolorbox}

\begin{figure}[h!]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[Project onto a line]{\label{fig:project-line}
	\includegraphics[width=0.47\linewidth]{./imgs/projectline.pdf}}
\quad
\subfigure[Project onto a space]{\label{fig:project-space}
	\includegraphics[width=0.47\linewidth]{./imgs/projectspace.pdf}}
\caption{Project a vector onto a line and a space.}
\label{fig:projection-qr}
\end{figure}

\section{Project a Vector Onto a Plane}\label{section:project-onto-a-plane}
Project a vector $\ba$ to a space spanned by $\bb_1, \bb_2, \ldots, \bb_n$ is to find the vector closest to $\ba$ on the column space of $[\bb_1, \bb_2, \ldots, \bb_n]$. The projection vector $\widehat{\ba}$ is a combination of $\bb_1, \bb_2, \ldots, \bb_n$: $\widehat{\ba} = \widehat{x}_1\bb_1+ \widehat{x}_2\bb_2+\ldots+\widehat{x}_n\bb_n$. This is actually a least squares problem. To find the projection, we just solve the normal equation $\bB^\top\bB\widehat{\bx} = \bB^\top\ba$ where $\bB=[\bb_1, \bb_2, \ldots, \bb_n]$ and $\widehat{\bx}=[\widehat{x}_1, \widehat{x}_2, \ldots, \widehat{x}_n]$. We refer the details of this projection view in the least squares to \citep{strang1993introduction, trefethen1997numerical, yang2000matrix, golub2013matrix, lu2021rigorous} as it is not the main interest of this survey. For each vector $\bb_i$, the projection of $\ba$ in the direction of $\bb_i$ can be analogously obtained by
$$
\widehat{\ba}_i = \frac{\bb_i\bb_i^\top}{\bb_i^\top\bb_i}\ba, \gap \forall i \in \{1,2,\ldots, n\}.
$$
Let $\widehat{\ba}=\sum_{i=1}^{n}\widehat{\ba_i}$, this results in
$$
\ba^\perp = (\ba-\widehat{\ba}) \perp \cspace(\bB),
$$
i.e., $(\ba-\widehat{\ba})$ is perpendicular to the column space of $\bB=[\bb_1, \bb_2, \ldots, \bb_n]$ as shown in Figure~\ref{fig:project-space}.

\section{Existence of the QR Decomposition via the Gram-Schmidt Process}\label{section:gram-schmidt-process}
\subsubsection*{\textbf{First View by Projection Directly}}
For three linearly independent vectors $\{\ba_1, \ba_2, \ba_3\}$ and the space spanned by the three linearly independent vectors $\cspace{([\ba_1, \ba_2, \ba_3])}$, i.e., the column space of the matrix $[\ba_1, \ba_2, \ba_3]$. We intend to construct three orthogonal vectors $\{\bb_1, \bb_2, \bb_3\}$ in which case $\cspace{([\bb_1, \bb_2, \bb_3])}$ = $\cspace{([\ba_1, \ba_2, \ba_3])}$. Then we divide the orthogonal vectors by their length to normalize. This process produces three mutually orthonormal vectors $\bq_1 = \frac{\bb_1}{||\bb_1||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$.

For the first vector, we choose $\bb_1 = \ba_1$ directly. The second vector $\bb_2$ must be perpendicular to the first one. This is actually the vector $\ba_2$ subtracting its projection along $\bb_1$:
\begin{equation}
\begin{aligned}
\bb_2 &= \ba_2- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} \ba_2 = (\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} )\ba_2   \qquad &(\text{Projection view})\\
&= \ba_2-  \underbrace{\frac{ \bb_1^\top \ba_2}{\bb_1^\top\bb_1} \bb_1}_{\widehat{\ba}_2}, \qquad &(\text{Combination view}) \nonumber
\end{aligned}
\end{equation}
where the first equation shows $\bb_2$ is a multiplication of the matrix $(\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} )$ and the vector $\ba_2$, i.e., project $\ba_2$ onto the orthogonal complement space of $\cspace{([\bb_1])}$. The second equality in the above equation shows $\ba_2$ is a combination of $\bb_1$ and $\bb_2$.
Clearly, the space spanned by $\bb_1, \bb_2$ is the same space spanned by $\ba_1, \ba_2$. The situation is shown in Figure~\ref{fig:gram-schmidt1} in which we choose \textbf{the direction of $\bb_1$ as the $x$-axis in the Cartesian coordinate system}. $\widehat{\ba}_2$ is the projection of $\ba_2$ onto line $\bb_1$. It can be clearly shown that the part of $\ba_2$ perpendicular to $\bb_1$ is $\bb_2 = \ba_2 - \widehat{\ba}_2$ from the figure.

For the third vector $\bb_3$, it must be perpendicular to both the $\bb_1$ and $\bb_2$ which is actually the vector $\ba_3$ subtracting its projection along the plane spanned by $\bb_1$ and $\bb_2$
\begin{equation}\label{equation:gram-schdt-eq2}
\begin{aligned}
\bb_3 &= \ba_3- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} \ba_3 - \frac{\bb_2 \bb_2^\top}{\bb_2^\top\bb_2} \ba_3 = (\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1}  - \frac{\bb_2 \bb_2^\top}{\bb_2^\top\bb_2} )\ba_3   \qquad &(\text{Projection view})\\
&= \ba_3- \underbrace{\frac{ \bb_1^\top\ba_3}{\bb_1^\top\bb_1} \bb_1}_{\widehat{\ba}_3} - \underbrace{\frac{ \bb_2^\top\ba_3}{\bb_2^\top\bb_2}  \bb_2}_{\bar{\ba}_3},    \qquad &(\text{Combination view})
\end{aligned}
\end{equation}
where the first equation shows $\bb_3$ is a multiplication of the matrix $(\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1}  - \frac{\bb_2 \bb_2^\top}{\bb_2^\top\bb_2} )$ and the vector $\ba_3$, i.e., project $\ba_3$ onto the orthogonal complement space of $\cspace{([\bb_1, \bb_2])}$. The second equality in the above equation shows $\ba_3$ is a combination of $\bb_1, \bb_2, \bb_3$. We will see this property is essential in the idea of the QR decomposition.
Again, it can be shown that the space spanned by $\bb_1, \bb_2, \bb_3$ is the same space spanned by $\ba_1, \ba_2, \ba_3$. The situation is shown in Figure~\ref{fig:gram-schmidt2}, in which we choose \textbf{the direction of $\bb_2$ as the $y$-axis of the Cartesian coordinate system}. $\widehat{\ba}_3$ is the projection of $\ba_3$ onto line $\bb_1$, $\bar{\ba}_3$ is the projection of $\ba_3$ onto line $\bb_2$. It can be shown that the part of $\ba_3$ perpendicular to both $\bb_1$ and $\bb_2$ is $\bb_3=\ba_3-\widehat{\ba}_3-\bar{\ba}_3$ from the figure.

Finally, we normalize each vector by dividing their length which produces three orthonormal vectors $\bq_1 = \frac{\bb_1}{||\bb_1||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$.

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[Project $\ba_2$ onto the space perpendicular to $\bb_1$.]{\label{fig:gram-schmidt1}
	\includegraphics[width=0.47\linewidth]{./imgs/gram-schmidt1.pdf}}
\quad
\subfigure[Project $\ba_3$ onto the space perpendicular to $\bb_1, \bb_2$.]{\label{fig:gram-schmidt2}
	\includegraphics[width=0.47\linewidth]{./imgs/gram-schmidt2.pdf}}
\caption{The Gram-Schmidt process.}
\label{fig:gram-schmidt-12}
\end{figure}

This idea can be extended to a set of vectors rather than only three. And we call this process as \textit{Gram-Schmidt process}. After this process, matrix $\bA$ will be triangularized. The method is named after Jørgen Pedersen Gram and Erhard Schmidt, but it appeared earlier in the work of Pierre-Simon Laplace in the theory of Lie group decomposition.

As we mentioned previously, the idea of the QR decomposition is the construction of a sequence of orthonormal vectors $\bq_1, \bq_2, \ldots$ that span the same successive subspaces.
$$
\bigg\{\cspace([\bq_1])=\cspace([\ba_1])\bigg\} \subseteq
\bigg\{\cspace([\bq_1, \bq_2])=\cspace([\ba_1, \ba_2])\bigg\} \subseteq
\bigg\{\cspace([\bq_1, \bq_2, \bq_3])=\cspace([\ba_1, \ba_2, \ba_3])\bigg\}
\subseteq \ldots,
$$
This implies any $\ba_k$ is in the space spanned by $\cspace([\bq_1, \bq_2, \ldots, \bq_k])$. \footnote{And also, any $\bq_k$ is in the space spanned by $\cspace([\ba_1, \ba_2, \ldots, \ba_k])$.} As long as we have found these orthonormal vectors, to reconstruct $\ba_i$'s from the orthonormal matrix $\bQ=[\bq_1, \bq_2, \ldots, \bq_n]$, an upper triangular matrix $\bR$ is needed such that $\bA = \bQ\bR$.

The Gram–Schmidt process is not the only algorithm for finding the QR decomposition.
Several other QR decomposition algorithms exist such as Householder reflections and Givens rotations which are more reliable in the presence of round-off errors.  These QR decomposition methods may also change the
order in which the columns of $\bA$ are processed.

\subsubsection*{\textbf{Another View by Inner Product (with Projection Implicitly)}}
In the above direct projection view, we first find orthogonal vectors and then normalize them to unit 1. The projection relies on the finding in Section~\ref{section:project-onto-a-vector} where the projection of the vector $\ba$ onto the vector $\bb$ is given by $\widehat{\ba} = \frac{\ba^\top\bb}{\bb^\top\bb}\bb = \frac{\bb\bb^\top}{\bb^\top\bb}\ba$. Now suppose $\bb$ is of unit length, then
\begin{equation}\label{equation:qr-projection-unit}
\widehat{\ba} = (\ba^\top\bb) \bb.
\end{equation}

Again, for three linearly independent vectors $\{\ba_1, \ba_2, \ba_3\}$ and the space spanned by the three linearly independent vectors $\cspace{([\ba_1, \ba_2, \ba_3])}$. To abuse the notation, we will write $\bb_k$ in the first view as $\ba_k^\perp$ in this view to emphasize that $\ba_k^\perp$ is orthogonal to $\{\bq_1, \ldots, \bq_{k-1}\}$. The process proceeds as follows:
\begin{itemize}
\item Compute $\bq_1$ of unit length so that $\cspace([\bq_1]) = \cspace([\ba_1])$:
\begin{itemize}
\item Compute the length of $\ba_1$: $r_{11} = ||\ba_1||$;
\item Sets $\bq_1$ to a unit vector in the direction of $\ba_1$: $\bq_1=\frac{\ba_1}{||\ba_1||}=\frac{\ba_1}{r_{11}}$;
\end{itemize}
\item Compute $\bq_2$ of unit length so that $\cspace([\bq_1,\bq_2]) = \cspace([\ba_1,\ba_2])$:
\begin{itemize}
\item Compute $r_{12}$ so that $r_{12}\bq_1 = (\ba_2^\top\bq_1) \bq_1$ equals the component of $\ba_2$ in the direction of $\bq_1$ by Equation~\eqref{equation:qr-projection-unit};
\item Compute the component of $\ba_2$ that is orthogonal to $\bq_1$: $\ba_2^\perp = \ba_2 - r_{12}\bq_1$;
\item Compute the length of vector $\ba_2^\perp$: $r_{22}=||\ba_2^\perp||$;
\item Set $\bq_2$ to a unit vector in the direction of $||\ba_2^\perp||$: $\bq_2 = \frac{\ba_2^\perp}{||\ba_2^\perp||}=\frac{\ba_2^\perp}{r_{22}}$;
\item This results in:
$$
\begin{bmatrix}
	\ba_1 & \ba_2
\end{bmatrix}=
\begin{bmatrix}
\bq_1 & \bq_2
\end{bmatrix}
\begin{bmatrix}
r_{11}& r_{12}\\
0 & r_{22}
\end{bmatrix};
$$
\end{itemize}
\item Compute $\bq_3$ of unit length so that $\cspace([\bq_1,\bq_2,\bq_3]) = \cspace([\ba_1,\ba_2,\ba_3])$:
\begin{itemize}
\item Compute $r_{13}$ so that $r_{13}\bq_1 = (\ba_3^\top\bq_1) \bq_1$ equals the component of $\ba_3$ in the direction of $\bq_1$ by Equation~\eqref{equation:qr-projection-unit};
\item Compute $r_{23}$ so that $r_{23}\bq_2 = (\ba_3^\top\bq_2) \bq_2$ equals the component of $\ba_3$ in the direction of $\bq_2$ by Equation~\eqref{equation:qr-projection-unit};
\item Compute the component of $\ba_3$ that is orthogonal to $\bq_1$ and $\bq_2$: $\ba_3^\perp = \ba_3 - r_{13}\bq_1 - r_{23}\bq_2$;
\item Compute the length of vector $\ba_3^\perp$: $r_{33}=||\ba_3^\perp||$;
\item Set $\bq_3$ to a unit vector in the direction of $||\ba_3^\perp||$: $\bq_3 = \frac{\ba_3^\perp}{||\ba_3^\perp||}=\frac{\ba_3^\perp}{r_{33}}$;
\item This results in:
$$
\begin{bmatrix}
\ba_1 & \ba_2 & \ba_3
\end{bmatrix}=
\begin{bmatrix}
\bq_1 & \bq_2 & \bq_3
\end{bmatrix}
\begin{bmatrix}
r_{11}& r_{12} & r_{13}\\
0 & r_{22}  & r_{23} \\
0 & 0 & r_{33}
\end{bmatrix}.
$$
\end{itemize}
\end{itemize}
Again, this idea can be extended to a set of vectors rather than only three. The process above reveals the meaning of $r_{ij}$ in the triangular matrix $\bR$ where $r_{ij}$ represents the component of $\ba_j$ in the direction of $\bq_i$. This matches the matrix multiplication result:
$$
\ba_j = \sum_{i=1}^{j} r_{ij} \bq_j.
$$

\subsubsection*{\textbf{Main Proof}}
Though the existence of the QR decomposition is conceptually intuitive from the two views of the Gram-Schmidt process above, the formal proof is clunky which needs inductive hypothesis. We now prove it rigorously.

\begin{proof}[of Theorem~\ref{theorem:qr-decomposition}]
We will prove by induction that every $m\times n$ matrix $\bA$ with linearly independent columns admits a \textit{reduced} QR decomposition. The \textit{full} QR decomposition can be done by completing the orthonormal columns in $\bQ$. The $1\times 1$ case is trivial by setting $\bQ=\frac{\ba_1}{||\ba_1||}, \bR=||\ba_1||$, thus, $\bA=\ba_1=\bQ\bR$.

Suppose for any $m\times k$ matrix $\bA_k$ with linearly independent columns admits a reduced QR decomposition. If we prove any $m\times (k+1)$ matrix $\bA_{k+1}$ can also be factored as this reduced QR decomposition, then we complete the proof. Suppose $\bA_{k+1}=[\bA_k, \ba_{k+1}]$ where $\bA_k$ admits the reduced QR decomposition by inductive hypothesis
$$
\bA_k = \bQ_k\bR_k,
$$
where $\bQ_k$ contains orthonormal columns $\bQ_k^\top\bQ_k=\bI_k$ and $\bR_k \in \real^{k\times k}$ is upper triangular. Also, by the induction hypothesis,
if the values on the diagonal of $\bR_k$ are chosen to be positive, then the reduced QR decomposition of
$\bA_k = \bQ_k\bR_k$ is \textbf{unique}.

Suppose further $\bA_{k+1}$ can be factored as
\begin{equation}\label{equation:qr-rigorous-proof}
\bA_{k+1} =
\begin{bmatrix}
	\bA_k& \ba_{k+1}
\end{bmatrix}
=
\begin{bmatrix}
	\widetildebQ_k& \bq_{k+1}
\end{bmatrix}
\begin{bmatrix}
	\widetildebR_k&
	\begin{matrix}
		\br_{k}\\
		r_{k+1}
	\end{matrix}
\end{bmatrix}
,
\end{equation}
where apparently $\bA_k = \widetildebQ_k \widetildebR_k$ is a reduced QR decomposition of $\bA_k$ and if restrict the diagonal values of $\widetildebR_k$ are positive, the factorization is \textbf{unique} which indicates $\widetildebQ_k = \bQ_k$ and $\widetildebR_k=\bR_k$. Moreover, Equation~\eqref{equation:qr-rigorous-proof} implies
$$
\begin{aligned}
\ba_{k+1} = \bQ_k\br_k + \bq_{k+1}r_{k+1} \\
\leadto &\bQ_k^\top \ba_{k+1} = \bQ_k^\top ( \bQ_k\br_k + \bq_{k+1}r_{k+1})=\br_k + \bQ_k^\top\bq_{k+1}r_{k+1}.
\end{aligned}
$$
Since we assume columns of $\bQ_k$ are orthonormal to $\bq_{k+1}$, it follows that $\bQ_k^\top\bq_{k+1}=\bzero$ and $\br_k=\bQ_k^\top \ba_{k+1}$. When $\bQ_k$ is fixed, the $\br_k$ is \textbf{uniquely} decided.

Let $\ba_{k+1}^\perp = \ba_{k+1}-\bQ_k\br_k$, then $\ba_{k+1}^\perp$ is orthogonal to the columns of $\bQ_k$. To see this, $\bQ_k^\top \ba_{k+1}^\perp = \bQ_k^\top(\ba_{k+1}-\bQ_k\br_k) =\bzero$ as we construct $\br_k$ by $\br_k=\bQ_k^\top \ba_{k+1}$. Since $\ba_{k+1}$ is linearly independent to columns of $\bA_k$, which is also linearly independent to columns of $\bQ_k$, $\ba_{k+1}^\perp$ is thus nonzero. Therefore, let $r_{k+1}=||\ba_{k+1}^\perp||$ and $\bq_{k+1} = \ba_{k+1}^\perp / r_{k+1}$, we find the \textbf{unique} reduced QR decomposition of $\bA_{k+1}$ with positive diagonals in the upper triangular matrix. This completes the proof.
\end{proof}
\section{Orthogonal vs Orthonormal}\index{Orthogonal}\index{Orthonormal}\label{section:orthogonal-orthonormal-qr}
The vectors $\bq_1, \bq_2, \ldots, \bq_n\in \real^m$ are mutually orthogonal when their dot products $\bq_i^\top\bq_j$ are zero whenever $i \neq j$. When each vector is divided by its length, the vectors become orthogonal unit vectors. Then the vectors $\bq_1, \bq_2, \ldots, \bq_n$ are mutually orthonormal. We put the orthonormal vectors into a matrix $\bQ$.

\begin{itemize}
\item When $m\neq n$: the matrix $\bQ$ is easy to work with because $\bQ^\top\bQ=\bI \in \real^{n\times n}$. Such $\bQ$ with $m\neq n$ is sometimes referred to as a \textbf{semi-orthogonal} matrix.

\item When $m= n$: the matrix $\bQ$ is square, $\bQ^\top\bQ=\bI$ means that $\bQ^\top=\bQ^{-1}$, i.e., the transpose of $\bQ$ is also the inverse of $\bQ$. Then we also have $\bQ\bQ^\top=\bI$, i.e., $\bQ^\top$ is the \textbf{two-sided inverse} of $\bQ$. We call this $\bQ$ an \textbf{orthogonal matrix}. \footnote{Note here we use the term \textit{orthogonal matrix} to mean the matrix $\bQ$ has orthonormal columns. The term \textit{orthonormal matrix} is \textbf{not} used for historical reasons.}
\end{itemize}
To see this, we have
$$
\begin{bmatrix}
\bq_1^\top \\
\bq_2^\top\\
\vdots \\
\bq_n^\top
\end{bmatrix}
\begin{bmatrix}
\bq_1 &\bq_2 & \ldots & \bq_n
\end{bmatrix}
=
\begin{bmatrix}
1 & & & \\
& 1 & & \\
& & \ddots & \\
& & & 1
\end{bmatrix}.
$$
In other words, $\bq_i^\top \bq_j = \delta_{ij}$ where $\delta_{ij}$ is the Kronecker delta. The columns of an orthogonal matrix $\bQ\in \real^{n\times n}$ form an \textbf{orthonormal basis} of $\real^n$. \footnote{Notice that the \textit{orthogonal matrix} $\bQ$ contains \textit{orthonormal basis}, \textbf{not} orthogonal basis.}

Orthogonal matrices can be viewed as matrices that change the basis of other matrices. Hence they preserve the angle (inner product) between the vectors
$$
\text{inner product: } \qquad \bu^\top \bv = (\bQ\bu)^\top(\bQ\bv).
$$
The above invariance of the inner products of angles between the vectors is preserved, which also relies on the invariance of their lengths:
$$
\text{length: } \qquad ||\bQ\bu|| = ||\bu||.
$$
In real cases, multiplied by a orthogonal matrix $\bQ$ will rotate (if $\det(\bQ)=1$) or reflect (if $\det(\bQ)=-1$) the original vector space. Many decomposition algorithms will result in two orthogonal matrices, thus such rotations or reflections will happen twice. See Section~\ref{section:coordinate-transformation} (p.~\pageref{section:coordinate-transformation}) for a discussion on the coordinate transformation in matrix decomposition.

\begin{example}[Rotation and Reflection in Orthogonal Matrices]
To see the rotation and reflection in orthogonal matrices, suppose
$$
\bQ_1 = \begin{bmatrix}
-1 & \\
& -1
\end{bmatrix}
\qquad
\text{and}
\qquad
\bQ_2 = \begin{bmatrix}
1 & \\
& -1
\end{bmatrix},
$$
where $\det(\bQ_1)=1$ and $\det(\bQ_2)=-1$. For vector $\bv=[1,1]^\top$, we have
$$
\bQ_1\bv = \begin{bmatrix}
-1 \\
-1
\end{bmatrix}
\qquad
\text{and}
\qquad
\bQ_2\bv = \begin{bmatrix}
1 \\
-1
\end{bmatrix}.
$$
Thus, $\bQ_1$ rotates $\bv$ along the point $\bzero$, and $\bQ_2$ reflects $\bv$ along the $x$-axis.
The illustration of the rotation and reflection is shown in Figure~\ref{fig:orthogonal-rotate-reflect}.
\exampbar

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[$\bQ_1$ rotates $\bv$ along the $\bzero$ point.]{\label{fig:orthogonal-rotate1}
	\includegraphics[width=0.47\linewidth]{./imgs/orthogonal-rotate.pdf}}
\quad
\subfigure[$\bQ_2$ reflects $\bv$ along the $x$-axis.]{\label{fig:orthogonal-rotate2}
	\includegraphics[width=0.47\linewidth]{./imgs/orthogonal-rotate2.pdf}}
\caption{Rotation and reflection in orthogonal matrices.}
\label{fig:orthogonal-rotate-reflect}
\end{figure}

\end{example}

\section{Properties of the QR Decomposition}
For any matrix, we have the property: $\nspace(\bA^\top)$ is the orthogonal complement of the column space $\cspace(\bA)$ in $\real^m$: $dim(\nspace(\bA^\top))+dim(\cspace(\bA))=m$.
This is known as the rank-nullity theorem. And the proof can be found in Appendix~\ref{appendix:fundamental-rank-nullity} (p.~\pageref{appendix:fundamental-rank-nullity}).
In specific, from QR decomposition, we can find a basis for the corresponding subspaces. In singular value decomposition (SVD), we will also find the orthonormal basis for $\nspace(\bA)$ and $\cspace(\bA^\top)$.

\begin{lemma}[Orthonormal Basis in $\real^m$]\label{lemma:qr-four-orthonormal-Basis}
For full QR decomposition of $\bA\in \real^{m\times n}$ with full rank $n$ and $m\geq n$, we have the following property:
\begin{itemize}
\item $\{\bq_1,\bq_2, \ldots,\bq_n\}$ is an orthonormal basis of $\cspace(\bA)$;
\item $\{\bq_{n+1},\bq_{n+2}, \ldots,\bq_m\}$ is an orthonormal basis of $\nspace(\bA^\top)$.
\end{itemize}
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:qr-four-orthonormal-Basis}]
From the Gram-Schmidt process, it is trivial that $span\{\ba_1, \ba_2, \ldots, \ba_k\}$ is equal to $span\{\bq_1, \bq_2, \ldots, \bq_k\}$ for all $k\in\{1, 2, \ldots, n\}$. Thus $\cspace(\bA) = span\{\ba_1, \ba_2, \ldots, \ba_n\}=span\{\bq_1, \bq_2, \ldots, \bq_n\}$, and $\{\bq_1, \bq_2, \ldots, \bq_n\}$ is an orthonormal basis for the column space of $\bA$. As $ \nspace(\bA^\top) \bot\cspace(\bA)$, $dim(\nspace(\bA^\top))=m-dim(\cspace(\bA))=m-n$. And the space spanned by $\{\bq_{n+1},\bq_{n+2}, \ldots,\bq_m\}$ is also $\bot\cspace(\bA)$ with dimension $m-n$. Thus, $\{\bq_{n+1},\bq_{n+2}, \ldots,\bq_m\}$ is an orthonormal basis for $\nspace(\bA^\top)$.
\end{proof}

\section{Computing the Reduced QR Decomposition via the Gram-Schmidt Process}\label{section:qr-gram-compute}

We write out this form of the reduced QR Decomposition such that $\bA=\bQ\bR$ where $\bQ\in \real^{m\times n}$ and $\bR\in \real^{n\times n}$:
\begin{equation}
\bA=\left[
\begin{matrix}
\ba_1 & \ba_2 & ... & \ba_n
\end{matrix}
\right]
=\left[
\begin{matrix}
\bq_1 & \bq_2 & ... & \bq_n
\end{matrix}
\right]
\begin{bmatrix}
r_{11} & r_{12}& \dots & r_{1n}\\
    & r_{22}& \dots & r_{2n}\\
    &       &    \ddots  & \vdots \\
\multicolumn{2}{c}{\raisebox{1.3ex}[0pt]{\Huge0}} & & r_{nn} \nonumber
\end{bmatrix}.
\end{equation}
The orthonormal matrix $\bQ$ can be easily calculated by the Gram-Schmidt process. To see why we have the upper triangular matrix $\bR$, we write out these equations
\begin{equation*}
\begin{aligned}
\ba_1 & = r_{11}\bq_1 &= \sum_{i=1}^{1} r_{i1}\bq_1, \\
\ba_2 & = r_{12}\bq_1 + r_{22}\bq_2&= \sum_{i=1}^{2} r_{i2}\bq_2, \\
\ba_3 &= r_{13}\bq_1 + r_{23}\bq_2 + r_{33} \bq_3&= \sum_{i=1}^{3}r_{i3} \bq_3, \\
&\vdots& \\
\ba_k &= r_{1k}\bq_1 + r_{2k}\bq_2 + \ldots + r_{kk}\bq_k  &= \sum_{i=1}^{k} r_{ik} \bq_k,\\
&\vdots& \\
\ba_n &= r_{1n}\bq_1 + r_{2n}\bq_2 + \ldots + r_{nn}\bq_n  &= \sum_{i=1}^{n} r_{in} \bq_n ,
\end{aligned}
\end{equation*}
which coincides with the second equation of Equation~\eqref{equation:gram-schdt-eq2} and conforms to the form of an upper triangular matrix $\bR$. And if we extend the idea of Equation~\eqref{equation:gram-schdt-eq2} into the $k$-th term, we will get
$$
\begin{aligned}
\ba_k &= \sum_{i=1}^{k-1}(\bq_i^\top\ba_k)\bq_i + \ba_k^\perp \\
&= \sum_{i=1}^{k-1}(\bq_i^\top\ba_k)\bq_i + ||\ba_k^\perp||\cdot \bq_k,
\end{aligned}
$$
which implies we can gradually orthonormalize $\bA$ to an orthonormal set $\bQ=[\bq_1, \bq_2, \ldots, \bq_n]$ by
\begin{equation}\label{equation:qr-gsp-equation}
\left\{
\begin{aligned}
r_{ik} &= \bq_i^\top\ba_k, \,\,\,\,\forall i \in \{1,2,\ldots, k-1\};\\
\ba_k^\perp&= \ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i;\\
r_{kk} &= ||\ba_k^\perp||;\\
\bq_k &= \ba_k^\perp/r_{kk}.
\end{aligned}
\right.
\end{equation}
The procedure is formulated in Algorithm~\ref{alg:reduced-qr}.
\begin{algorithm}[h]
\caption{Reduced QR Decomposition via Gram-Schmidt Process}
\label{alg:reduced-qr}
\begin{algorithmic}[1]
\Require Matrix $\bA$ has linearly independent columns with size $m\times n $ and $m\geq n$;
\For{$k=1$ to $n$} \Comment{compute $k$-th column of $\bQ,\bR$}
\For{$i=1$ to $k-1$}
\State $r_{ik} =\bq_i^\top\ba_k$; \Comment{entry ($i,k$) of $\bR$, $2m-1$ flops}
\EndFor  \Comment{all $k-1$ iterations: $(k-1)(2m-1)$ flops}
\State $\ba_k^\perp= \ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i$;\Comment{$2m(k-1)$ flops}
\State $r_{kk} = ||\ba_k^\perp||$; \Comment{main diagonal of $\bR$, $2m$ flops}
\State $\bq_k = \ba_k^\perp/r_{kk}$; \Comment{$m$ flops}
\EndFor
\State Output $\bQ=[\bq_1, \ldots, \bQ_n]$ and $\bR$ with entry $(i,k)$ being $r_{ik}$;

\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: Reduced QR via Gram-Schmidt]\label{theorem:qr-reduced}
Algorithm~\ref{alg:reduced-qr} requires $\sim 2mn^2$ flops to compute a reduced QR decomposition of an $m\times n$ matrix with linearly independent columns and $m\geq n$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-reduced}]
In step 3, the computation of $r_{ik}$ is a vector inner product which requires $m$ multiplications, and $m-1$ additions . This makes it $\boxed{(k-1)(2m-1)}$ flops for all the $k-1$ iterations.

In step 5, the computation of $r_{ik}\bq_i$ needs $m$ flops and there are $k-1$ such scalar-vector multiplications, which makes it $m(k-1)$ flops. The vector subtraction and additions require another $m(k-1)$ flops. Thus step $5$ costs $\boxed{2m(k-1)}$ flops.

In step 6, the vector norm involves a vector inner product plus a square root that takes $\boxed{2m}$ flops.

Step 7 costs $\boxed{m}$ for the divisions

Therefore, for computing the $k$-th column of $\bQ,\bR$, it requires $(k-1)(2m-1)+ 2m(k-1)+2m+m=4mk-m-k+1$ flops. Let $f(k) = 4mk-m-k+1$, the total complexity can be obtained by
$$
\mathrm{cost}= f(n) +f(n-1) + \ldots +f(1).
$$
Simple calculations will show the total complexity is $2mn^2+mn-3m-\frac{n^2-n}{2}$ flops, or $2mn^2$ flops if we keep only the leading term.
\end{proof}

\subsubsection*{\textbf{Orthogonal Projection}}
We notice again from Equation~\eqref{equation:qr-gsp-equation}, i.e., step 2 to step 6 in Algorithm~\ref{alg:reduced-qr}, the first two equality imply that
\begin{equation}\label{equation:qr-gsp-equation2}
\left.
\begin{aligned}
r_{ik} &= \bq_i^\top\ba_k, \,\,\,\,\forall i \in \{1,2,\ldots, k-1\}\\
\ba_k^\perp&= \ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i\\
\end{aligned}
\right\}
\rightarrow
\ba_k^\perp= \ba_k- \bQ_{k-1}\bQ_{k-1}^\top \ba_k=(\bI-\bQ_{k-1}\bQ_{k-1}^\top )\ba_k,
\end{equation}
where $\bQ_{k-1}=[\bq_1,\bq_2,\ldots, \bq_{k-1}]$. This implies $\bq_k$ can be obtained by
$$
\bq_k = \frac{\ba_k^\perp}{||\ba_k^\perp||} = \frac{(\bI-\bQ_{k-1}\bQ_{k-1}^\top )\ba_k}{||(\bI-\bQ_{k-1}\bQ_{k-1}^\top )\ba_k||}.
$$
The matrix $(\bI-\bQ_{k-1}\bQ_{k-1}^\top )$ in above equation is known as an \textit{orthogonal projection matrix} \footnote{More details can be referred to Appendix~\ref{section:by-geometry-hat-matrix} (p.~\pageref{section:by-geometry-hat-matrix}).} that will project $\ba_k$ along the column space of $\bQ_{k-1}$, i.e., project a vector so that the vector is perpendicular to the column space of $\bQ_{k-1}$. The net result is that the $\ba_k^\perp$ or $\bq_k$ calculated in this way will be orthogonal to the $\cspace(\bQ_{k-1})$, i.e., in the null space of $\bQ_{k-1}^\top$: $\nspace(\bQ_{k-1}^\top)$ by the fundamental theorem of linear algebra (Theorem~\ref{theorem:fundamental-linear-algebra}, p.~\pageref{theorem:fundamental-linear-algebra}).

Let $\bP_1=(\bI-\bQ_{k-1}\bQ_{k-1}^\top )$ and we claimed above $\bP_1=(\bI-\bQ_{k-1}\bQ_{k-1}^\top )$ is an orthogonal projection matrix such that $\bP_1\bv$ will project the $\bv$ onto the null space of $\bQ_{k-1}$. And actually, let $\bP_2=\bQ_{k-1}\bQ_{k-1}^\top$, then $\bP_2$ is also an orthogonal projection matrix such that $\bP_2\bv$ will project the $\bv$ onto the column space of $\bQ_{k-1}$.

\begin{figure}[h!]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[Orthogonal projection]{\label{fig:project-oblique}
	\includegraphics[width=0.47\linewidth]{./imgs/project-orthogonal.pdf}}
\quad
\subfigure[Oblique projection]{\label{fig:project-orthogonal}
	\includegraphics[width=0.47\linewidth]{./imgs/project-oblique.pdf}}
\caption{Demonstration of the difference between orthogonal projection and oblique projection.}
\label{fig:projection-oblie-and-orthogonal}
\end{figure}

But why do the matrix $\bP_1, \bP_2$ can magically project a vector onto the corresponding subspaces? We will show in Lemma~\ref{lemma:rank-of-ttt} that the column space of $\bQ_{k-1}$ is equal to the column space of $\bQ_{k-1}\bQ_{k-1}^\top$:
$$
\cspace(\bQ_{k-1})=\cspace(\bQ_{k-1}\bQ_{k-1}^\top)=\cspace(\bP_2).
$$
Therefore, the result of $\bP_2\bv$ is a linear combination of the columns of $\bP_2$, which is in the column space of $\bP_2$ or the column space of $\bQ_{k-1}$. The formal definition of a \textit{projection matrix} $\bP$ is that it is idempotent $\bP^2=\bP$ such that projecting twice is equal to projecting once \footnote{See also Definition~\ref{definition:projection-matrix} (p.~\pageref{definition:projection-matrix}).}. What makes the above $\bP_2=\bQ_{k-1}\bQ_{k-1}^\top $ different is that the projection $\widehat{\bv}$ of any vector $\bv$ is perpendicular to $\bv-\widehat{\bv}$:
$$
(\widehat{\bv}=\bP_2\bv) \perp (\bv-\widehat{\bv}).
$$
This goes to the original definition we gave above: the \textit{orthogonal projection matrix} \footnote{See also Definition~\ref{definition:orthogonal-projection-matrix} (p.~\pageref{definition:orthogonal-projection-matrix}).}. To
avoid confusion, one may use the term \textit{oblique projection matrix} in the nonorthogonal
case where the difference is shown in Figure~\ref{fig:projection-oblie-and-orthogonal}. When $\bP_2$ is an orthogonal projection matrix, $\bP_1=\bI-\bP_2$ is also an orthogonal projection matrix that will project any vector onto the space perpendicular to the $\cspace(\bQ_{k-1})$, i.e., $\nspace(\bQ_{k-1}^\top)$. Therefore, we conclude the two orthogonal projections:
$$
\left\{
\begin{aligned}
\bP_1: &\gap \text{project onto $\nspace(\bQ_{k-1}^\top)$;} \\
\bP_2: &\gap \text{project onto $\cspace(\bQ_{k-1})$} .
\end{aligned}
\right.
$$

The further result that is important to notice is when the columns of $\bQ_{k-1}$ are mutually orthonormal, we have the following decomposition:
\begin{equation}\label{equation:qr-orthogonal-equality}
\boxed{\bP_1 = \bI - \bQ_{k-1}\bQ_{k-1}^\top = (\bI-\bq_1\bq_1^\top)(\bI-\bq_2\bq_2^\top)\ldots (\bI-\bq_{k-1}\bq_{k-1}^\top),}
\end{equation}
where $\bQ_{k-1}=[\bq_1,\bq_2,\ldots, \bq_{k-1}]$ and each $(\bI-\bq_i\bq_i^\top)$ is to project a vector into the perpendicular space of $\bq_i$.

\subsubsection*{\textbf{Modified Gram-Schmidt (MGS) Process}}
To emphasize the modified Gram-Schmidt process and make a connection to the equivalent projection in Equation~\eqref{equation:qr-orthogonal-equality}, we first illustrate the lemma how the entries in the upper triangular $\bR$ of the QR decomposition can be obtained in an alternative way.
\begin{lemma}[Modified Gram-Schmidt Process]
Suppose for $[\ba_1, \ba_2, \ldots,\ba_{k-1}, \ba_k$], where the first $k-1$ column are spanned by $k-1$ orthonormal vectors $[\bq_1, \bq_2, \ldots, \bq_{k-1}]$:
$$
\cspace([\ba_1, \ba_2, \ldots,\ba_{i}]) = \cspace([\bq_1, \bq_2, \ldots, \bq_{i}]), \gap \forall i\in \{1,2,\ldots, k-1\}.
$$
Therefore, $r_{ik} = \bq_i^\top\ba_k$ is the projection of $\ba_k$ on the vector $\bq_i$. Then it follows that
$$
\begin{aligned}
\bq_i^\top\ba_k &= \bq_i^\top (\ba_k \underbrace{- r_{1k}\bq_1 - r_{2k}\bq_2 - \ldots - r_{i-1,k}\bq_{i-1}}_{\text{orthogonal to $\bq_i$}}), \gap \forall i\in \{1,2,\ldots, k-1\}\\
&= \bq_i^\top (\ba_k - \sum_{j=1}^{i-1}r_{jk}\bq_j ).
\end{aligned}
$$
This can be easily checked since $\bq_i$ is orthonormal to $\{\bq_1, \bq_2, \ldots, \bq_{i-1}\}$. This observation implies another update on the $k$-th column of $\bR$.
\end{lemma}
The lemma above reveals a second algorithm to compute the reduced QR decomposition of a matrix as shown in Algorithm~\ref{alg:qr-mgs-right} of which the algorithm on the left is exactly the same one as Algorithm~\ref{alg:reduced-qr} (with slight modification) to emphasize the difference.

\noindent
\begin{minipage}[t]{0.495\linewidth}
\begin{algorithm}[H]
\caption{CGS (=Algorithm~\ref{alg:reduced-qr} )}
\label{alg:qr-mgs-left}
\begin{algorithmic}[1]
\Require $\bA\in \real^{m\times n}$ with full column rank;
\For{$k=1$ to $n$}
\State $\ba_k^\perp=\ba_k$;
\For{$i=1$ to $k-1$}
\State $r_{ik} =\bq_i^\top\ba_k$;
\State $\ba_k^\perp= \ba_k^\perp-r_{ik}\bq_i$;
\EndFor
\State $r_{kk} = ||\ba_k^\perp||$;
\State $\bq_k = \ba_k^\perp/r_{kk}$;
\EndFor
\end{algorithmic}
\end{algorithm}
\end{minipage}%
\hfil
\begin{minipage}[t]{0.495\linewidth}
\begin{algorithm}[H]
\caption{MGS}
\label{alg:qr-mgs-right}
\begin{algorithmic}[1]
\Require $\bA\in \real^{m\times n}$ with full column rank;
\For{$k=1$ to $n$}
\State $\ba_k^\perp=\ba_k$;
\For{$i=1$ to $k-1$}
\State $r_{ik} =\bq_i^\top\textcolor{blue}{\ba_k^\perp}$;
\State $\ba_k^\perp= \ba_k^\perp-r_{ik}\bq_i$; (*)
\EndFor
\State $r_{kk} = ||\ba_k^\perp||$;
\State $\bq_k = \ba_k^\perp/r_{kk}$;
\EndFor
\end{algorithmic}
\end{algorithm}
\end{minipage}

The above process is named as the \textit{modified Gram-Schmidt (MGS) process}, whereas the previous one is also known as the \textit{classical Gram-Schmidt (CGS) process}. In theory, all CGS and MGS are equivalent in the
sense that they compute exactly the same QR decompositions when exact arithmetic is employed. In
practice, in the presence of round-off error, the orthonormal columns of $\bQ$ computed by MGS are
often ``more orthonormal" than those computed by CGS.

We notice that the equality (*) in Algorithm~\ref{alg:qr-mgs-right} can be rewritten as (via the step 4 and step 5 of the algorithm)
$$
\begin{aligned}
\ba_k^\perp &:= \ba_k^\perp-r_{ik}\bq_i\\
&= \ba_k^\perp-\bq_i^\top\ba_k^\perp\bq_i\\
&=\ba_k^\perp-\bq_i\bq_i^\top\ba_k^\perp\\
&=(\bI-\bq_i\bq_i^\top)\ba_k^\perp.
\end{aligned}
$$
That is, $\ba_k^\perp$ will be updated by
$$
\left\{(\bI-\bq_{k-1}\bq_{k-1}^\top)\ldots\left[(\bI-\bq_2\bq_2^\top)\left((\bI-\bq_1\bq_1^\top) \ba_k\right)\right]\right\},
$$
where the parentheses indicate the order of the computation, and
which matches the orthogonal projection matrix equality in Equation~\eqref{equation:qr-orthogonal-equality} that
$$
\begin{aligned}
\bP_1 &= \bI-\bQ_{k-1}\bQ_{k-1}^\top\\
&=(\bI-\bq_1\bq_1^\top)(\bI-\bq_2\bq_2^\top)\ldots (\bI-\bq_{k-1}\bq_{k-1}^\top)\\
&=\prod_{i=1}^{k-1}(\bI-\bq_i\bq_i^\top),
\end{aligned}
$$
where $\bQ_{k-1}=[\bq_1,\bq_2,\ldots, \bq_{k-1}]$.

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[CGS, step 1: \textcolor{blue}{blue} vector; step 2: \textcolor{green}{green} vector; step 3: \textcolor{purple}{purple} vector.]{\label{fig:projection-mgs-demons-cgs}
	\includegraphics[width=0.47\linewidth]{./imgs/projectqr-cgs.pdf}}
\quad
\subfigure[MGS, step 1: \textcolor{blue}{blue} vector; step 2: \textcolor{purple}{purple} vector.]{\label{fig:projection-mgs-demons-mgs}
	\includegraphics[width=0.47\linewidth]{./imgs/projectqr-mgs.pdf}}
\caption{CGS vs MGS in 3-dimensional space where $\bq_2^\prime$ is parallel to $\bq_2$ so that projecting on $\bq_2$ is equivalent to projecting on $\bq_2^\prime$.}
\label{fig:projection-mgs-demons-3d}
\end{figure}

\paragraph{What's the difference?}
Taking a three-column matrix $\bA=[\ba_1, \ba_2, \ba_3]$ as an example. Suppose we have computed $\{\bq_1, \bq_2\}$ such that $span\{\bq_1, \bq_2\}=span\{\ba_1, \ba_2\}$. And we want to proceed to compute the $\bq_3$.

In the CGS, the orthogonalization of column $\ba_n$ against column $\{\bq_1, \bq_2\}$ is performed by projecting the original column $\ba_3$ of $\bA$ onto $\{\bq_1, \bq_2\}$ respectively and subtracting at once:
\begin{equation}\label{equation:cgs-3d-exmp}
\left\{
\begin{aligned}
\ba_3^\perp &= \ba_3 - (\bq_1^\top\ba_3)\bq_1 - (\bq_2^\top\ba_3)\bq_2\\
&= \ba_3 - (\bq_1\bq_1^\top)\ba_3 - \boxed{(\bq_2\bq_2^\top)\ba_3}\\
\bq_3 &=  \frac{\ba_3^\perp}{||\ba_3^\perp||},
\end{aligned}
\right.
\end{equation}
as shown in Figure~\ref{fig:projection-mgs-demons-cgs}.

In the MGS, on the other hand, the components along each $\{\bq_1, \bq_2\}$ are immediately subtracted out of the rest of the column $\ba_3$ as soon as the $\{\bq_1, \bq_2\}$ are computed. Therefore the orthogonalization of column $\ba_3$ against $\{\bq_1, \bq_2\}$ is not performed by projecting the original column $\ba_3$ against $\{\bq_1, \bq_2\}$ as it is in CGS, but rather against a vector obtained by subtracting from that column $\ba_3$ of $\bA$ the components in the direction of $\bq_1, \bq_2$ successively. This is important because the error components of $\bq_i$ in $span\{\bq_1, \bq_2\}$ will be smaller (we will further discuss in the next paragraphs).

More precisely, in the MGS the orthogonalization of column $\ba_3$ against $\bq_1$ is performed by subtracting the component of $\bq_1$ from the vector $\ba_3$:
$$
\ba_3^{(1) }=  (\bI-\bq_1\bq_1^\top)\ba_3 = \ba_3 - (\bq_1\bq_1^\top)\ba_3,
$$
where $\ba_3^{(1) }$ is the component of $\ba_3$ lies in a space perpendicular to $\bq_1$. And further step is performed by
\begin{equation}\label{equation:mgs-3d-exmp}
\begin{aligned}
\ba_3^{(2) }=  (\bI-\bq_2\bq_2^\top)\ba_3^{(1) }&=\ba_3^{(1) }-(\bq_2\bq_2^\top)\ba_3^{(1) }\\
&=\ba_3 - (\bq_1\bq_1^\top)\ba_3-\boxed{(\bq_2\bq_2^\top)\textcolor{blue}{\ba_3^{(1) }}}
\end{aligned}
\end{equation}
where $\ba_3^{(2) }$ is the component of $\ba_3^{(1) }$ lies in a space perpendicular to $\bq_2$  and we highlight the difference to the CGS in Equation~\eqref{equation:cgs-3d-exmp} by \textcolor{blue}{blue} text. This net result is that $\ba_3^{(2) }$ is the component of $\ba_3$ lies in the space perpendicular to $\{\bq_1, \bq_2\}$ as shown in Figure~\ref{fig:projection-mgs-demons-mgs}.

\subsubsection*{\textbf{Main difference and catastrophic cancellation}}
The key difference is that the $\ba_3$ can in general have large components in $span\{\bq_1, \bq_2\}$ in which case one starts with large
values and ends up with small values with large relative errors in them. This is known as the problem of \textit{catastrophic cancellation}. Whereas $\ba_3^{(1) }$ is in the direction perpendicular to $\bq_1$ and has only a small ``error" component in the direction of $\bq_1$. Compare the \fbox{boxed} text in Equation~\eqref{equation:cgs-3d-exmp} and \eqref{equation:mgs-3d-exmp}, it is not hard to see $(\bq_2\bq_2^\top)\ba_3^{(1) }$ in Equation~\eqref{equation:mgs-3d-exmp} is more accurate by the above argument. And thus, because of the much smaller error in this projection factor, the MGS introduces less orthogonalization error at each subtraction step than that is in the CGS. In fact, it can be shown that the final $\bQ$ obtained in the CGS satisfies
$$
||\bI-\bQ\bQ^\top|| \leq O(\epsilon \kappa^2(\bA)),
$$
where $\kappa(\bA)$ is a value larger than 1 determined by $\bA$.
Whereas, in MGS, the error satisfies
$$
||\bI-\bQ\bQ^\top|| \leq O(\epsilon \kappa(\bA)).
$$
That is, the $\bQ$ obtained in the MGS is more orthogonal.
Therefore we summarize the difference between the CGS and MGS processes for obtaining $\bq_k$ via the $k$-th column $\ba_k$ of $\bA$ and the orthonormalized vectors $\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$:
$$
\begin{aligned}
\text{(CGS)}: &\, \text{obtain $\bq_k$ by normalizing $\ba_k^\perp=(\bI-\bQ_{k-1}\bQ_{k-1}^\top)\ba_k$;} \\
\text{(MGS)}: &\, \text{obtain $\bq_k$ by normalizing  $\ba_k^\perp=\left\{(\bI-\bq_{k-1}\bq_{k-1}^\top)\ldots\left[(\bI-\bq_2\bq_2^\top)\left((\bI-\bq_1\bq_1^\top) \ba_k\right)\right]\right\}$.}
\end{aligned}
$$

\subsubsection*{\textbf{Triangular Orthogonalization in CGS and MGS}}
We here illustrate that in the CGS or MGS, the orthogonal matrix $\bQ$ is obtained via a set of triangular matrices. For simplicity, we only discuss the situation in the CGS and follow up the three-column example where $\bA=[\ba_1, \ba_2, \ba_3]\in \real^{3\times 3}$. The above discussion shows that the mutually orthonormal vectors $\{\bq_1, \bq_2, \bq_3\}$ can be obtained as follows:
$$
\left\{
\begin{aligned}
\bq_1 &= \frac{\ba_1}{r_{11}};\\
\bq_2 &= \frac{\ba_2 - r_{12}\bq_1}{r_{22}};\\
\bq_3 &= \frac{\ba_3-r_{13}\bq_1-r_{23}\bq_2}{r_{33}}.
\end{aligned}
\right.
$$
Whilst, the three mutually orthonormal vectors can be equivalently obtained by
$$
\bQ\bR_3\bR_2\bR_1= \bA \leadto \bQ=\bA\bR_1^{-1}\bR_2^{-1}\bR_3^{-1},
$$
where
$$
\bR_3=
\begin{bmatrix}
1 & 0 & r_{13} \\
0 & 1 & r_{23}\\
0 & 0 & r_{33}
\end{bmatrix},
\gap
\bR_2=
\begin{bmatrix}
1 & r_{12} & 0 \\
0 & r_{22} &0\\
0 & 0 & 1
\end{bmatrix},
\gap
\bR_1=
\begin{bmatrix}
r_{11} & 0 & 0 \\
0 & 1 &0\\
0 & 0 & 1
\end{bmatrix},
$$
such that
$$
\bR_3\bR_2\bR_1 = \bR=
\begin{bmatrix}
r_{11}& r_{12} & r_{13}\\
0 & r_{22} & r_{13}\\
0 & 0 & r_{33}
\end{bmatrix}.
$$
The above procedure $\bA\bR_1^{-1}\bR_2^{-1}\bR_3^{-1}$ will obtain the $\{\bq_1, \bq_2, \bq_3\}$ in a successive manner where $\bA\bR_1^{-1}$ will get $\bq_1$ into the first column of $\bQ$; $(\bA\bR_1^{-1})\bR_2^{-1}$ will get $\bq_2$ into the second column of $\bQ$; and $(\bA\bR_1^{-1}\bR_2^{-1})\bR_3^{-1}$ will get $\bq_3$ into the third column of $\bQ$. This is called the \textit{triangular orthogonalization} in the Gram-Schmidt process. The triangular orthogonalization is problematic in the sense that the condition number of a triangular matrix ($\bR_1^{-1}, \bR_2^{-1}, \bR_3^{-1}$ in the above three-column example) can be anything. And the Gram-Schmidt process have a series of them where the condition number can grow very large so that the orthogonalization is not numerical stable.

\subsubsection*{\textbf{More to go, preliminaries for Householder and Givens methods}}
Although, we claimed here that the MGS usually works better than the CGS in practice. An example will be given in the sequel. The MGS can still fall victim to the \textit{catastrophic cancellation} problem. Suppose in iteration $k$ of the MGS Algorithm~\ref{alg:qr-mgs-right}, $\ba_k$ is almost in the span of $\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$. This will result in that $\ba_k^\perp$ has only a small component that is perpendicular to $span\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$, whereas the ``error" component in the $span\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$ will be amplified and the net result is $\bQ$ will be less orthonormal. As discussed above, the main disadvantage in both the CGS and MGS can be described by that the algorithms find the orthogonal matrix $\bQ$ via the upper triangular $\bR$, i.e., if $\bA\in \real^{m\times n}$, one obtains $\bQ$ by
$$
\bQ=\bA\underbrace{\bR_1^{-1}\bR_2^{-1}\ldots \bR_n^{-1}}_{\bR^{-1}}.
$$

In this case, if we can find a successive set of orthogonal matrices $\{\bQ_1, \bQ_2, \ldots, \bQ_l\}$ such that $\bQ_l\ldots\bQ_2\bQ_1\bA$ is triangularized, then $\bQ=(\bQ_l\ldots\bQ_2\bQ_1)^\top$ will be ``more" orthogonal than the CGS or the MGS since the condition numbers for the orthogonal matrices are all 1. We will discuss this method in Section~\ref{section:qr-via-householder} and \ref{section:qr-givens} via the Householder reflectors and the Givens rotations.

\subsubsection*{\textbf{Example for MGS vs CGS}}
For better understanding, we will show by a $4\times 3$ \textit{Lauchli matrix} where
the Lauchli matrix is an $(n+1) \times n$ rectangular matrix that has ones on the top row and the parameter $\epsilon=\sqrt{\epsilon_{mach}}$ on the diagonal starting at entry $(2,1)$ i.e., on the lower subdiagonal.\index{Lauchli matrix}
\begin{example}[MGS vs CGS]
Let $\epsilon = \sqrt{\epsilon_{mach}}$ and consider the QR decomposition of the following matrix by CGS and MGS:
$$
\bA =
\begin{bmatrix}
1& 1 & 1 \\
\epsilon & 0  & 0 \\
0 & \epsilon  & 0 \\
0 & 0 & \epsilon
\end{bmatrix}
=
\begin{bmatrix}
\ba_1 & \ba_2 & \ba_3
\end{bmatrix}.
$$
Note that we will round  $1 +\epsilon_{mach}$ to 1
whenever encountered in the calculation.
The CGS proceeds as follows:
\begin{itemize}
\item Compute $\bq_1$ of unit length so that $\cspace([\bq_1]) = \cspace([\ba_1])$:
\begin{itemize}
\item Compute $r_{11}$: $r_{11} = ||\ba_1||=\sqrt{1+\epsilon_{mach}}\approx1$;
\item Compute $\bq_1$: $\bq_1=\frac{\ba_1}{||\ba_1||}=\ba_1$;
\end{itemize}
\item Compute $\bq_2$ of unit length so that $\cspace([\bq_1,\bq_2]) = \cspace([\ba_1,\ba_2])$:
\begin{itemize}
\item Compute $r_{12}$: $r_{12} = \ba_2^\top\bq_1=1$;
\item Compute $\ba_2^\perp$: $\ba_2^\perp = \ba_2 - r_{12}\bq_1=\ba_2-\ba_1 = [0,-\epsilon, \epsilon, 0]^\top$;
\item Compute $r_{22}$: $r_{22}=||\ba_2^\perp||=\sqrt{2\epsilon_{mach}}=\sqrt{2}\epsilon$;
\item Compute $\bq_2$: $\bq_2 = \frac{\ba_2^\perp}{||\ba_2^\perp||}=\frac{\ba_2^\perp}{r_{22}}=[0,-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}, 0]^\top$;
\end{itemize}
\item Compute $\bq_3$ of unit length so that $\cspace([\bq_1,\bq_2,\bq_3]) = \cspace([\ba_1,\ba_2,\ba_3])$:
\begin{itemize}
\item Compute $r_{13}$: $r_{13}= \ba_3^\top\bq_1=\ba_3^\top\ba_1=1$;
\item Compute $r_{23}$: $r_{23}= \ba_3^\top\bq_2=0$;
\item Compute $\ba_3^\perp$: $\ba_3^\perp = \ba_3 - r_{13}\bq_1 - r_{23}\bq_2=[0,-\epsilon, 0, \epsilon]^\top$;
\item Compute $r_{33}$: $r_{33}=||\ba_3^\perp||=\sqrt{2\epsilon_{mach}}=\sqrt{2}\epsilon$;
\item Compute $\bq_3$: $\bq_3 = \frac{\ba_3^\perp}{||\ba_3^\perp||}=\frac{\ba_3^\perp}{r_{33}}=[0,-\frac{1}{\sqrt{2}},0, \frac{1}{\sqrt{2}}]^\top$;
\end{itemize}
\item This results in
$$
\bA=
\begin{bmatrix}
1& 0& 0 \\
\epsilon & -\frac{1}{\sqrt{2}}  & -\frac{1}{\sqrt{2}} \\
0 & \frac{1}{\sqrt{2}}  & 0 \\
0 & 0 & \frac{1}{\sqrt{2}}
\end{bmatrix}
\begin{bmatrix}
1& 1 & 1 \\
0 & \sqrt{2}\epsilon  & 0 \\
0 & 0  & \sqrt{2}\epsilon
\end{bmatrix}=
\bQ_1\bR_1
$$
\end{itemize}

Whilst, the MGS proceeds as follows:
\begin{itemize}
\item Compute $\bq_1$ of unit length so that $\cspace([\bq_1]) = \cspace([\ba_1])$:
\begin{itemize}
\item Compute $r_{11}$: $r_{11} = ||\ba_1||=\sqrt{1+\epsilon_{mach}}\approx1$;
\item Compute $\bq_1$: $\bq_1=\frac{\ba_1}{||\ba_1||}=\ba_1$;
\end{itemize}
\item Compute $\bq_2$ of unit length so that $\cspace([\bq_1,\bq_2]) = \cspace([\ba_1,\ba_2])$:
\begin{itemize}
\item Compute $r_{12}$: $r_{12} = \ba_2^\top\bq_1=1$;
\item Compute $\ba_2^\perp$: $\ba_2^\perp = \ba_2 - r_{12}\bq_1=\ba_2-\ba_1 = [0,-\epsilon, \epsilon, 0]^\top$;
\item Compute $r_{22}$: $r_{22}=||\ba_2^\perp||=\sqrt{2\epsilon_{mach}}=\sqrt{2}\epsilon$;
\item Compute $\bq_2$: $\bq_2 = \frac{\ba_2^\perp}{||\ba_2^\perp||}=\frac{\ba_2^\perp}{r_{22}}=[0,-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}, 0]^\top$;
\item \textcolor{blue}{Till now, we are still the same as the CGS};
\end{itemize}
\item Compute $\bq_3$ of unit length so that $\cspace([\bq_1,\bq_2,\bq_3]) = \cspace([\ba_1,\ba_2,\ba_3])$:
\begin{itemize}
\item Compute $r_{13}$: $r_{13}= \ba_3^\top\bq_1=\ba_3^\top\ba_1=1$;
\item Compute temporary $\ba_3^\perp=\ba_3 - r_{13}\bq_1=[0,-\epsilon, 0, \epsilon]^\top$;
\item Compute $r_{23}$: $r_{23}= \bq_2^\top \textcolor{blue}{\ba_3^\perp}=\textcolor{blue}{\frac{\epsilon}{\sqrt{2}}}$;
\item Compute final $\ba_3^\perp$: $\ba_3^\perp = \underbrace{\ba_3 - r_{13}\bq_1}_{\text{the old $\ba_3^\perp$}} - r_{23}\bq_2=\textcolor{blue}{[0,-\epsilon/2, -\epsilon/2, \epsilon]^\top}$;
\item Compute $r_{33}$: $r_{33}=||\ba_3^\perp||=\sqrt{2\epsilon_{mach}}=\textcolor{blue}{\frac{\sqrt{6}}{2}\epsilon}$;
\item Compute $\bq_3$: $\bq_3 = \frac{\ba_3^\perp}{||\ba_3^\perp||}=\frac{\ba_3^\perp}{r_{33}}=\textcolor{blue}{[0,-\frac{1}{\sqrt{6}}, -\frac{1}{\sqrt{6}}, \frac{2}{\sqrt{6}}]^\top}$;
\end{itemize}
\item This results in
$$
\bA=
\begin{bmatrix}
1& 0& 0 \\
\epsilon & -\frac{1}{\sqrt{2}}  & \textcolor{blue}{-\frac{1}{\sqrt{6}}}\\
0 & \frac{1}{\sqrt{2}}  &  \textcolor{blue}{-\frac{1}{\sqrt{6}}}\\
0 & 0 &  \textcolor{blue}{\frac{2}{\sqrt{6}}}\\
\end{bmatrix}
\begin{bmatrix}
1& 1 & 1 \\
0 & \sqrt{2}\epsilon  & \textcolor{blue}{\frac{\epsilon}{2}} \\
0 & 0  & \frac{\sqrt{6}}{2}\epsilon
\end{bmatrix}=
\bQ_2\bR_2.
$$

\end{itemize}
We notice that
$$
\bQ_1^\top\bQ_1=
\begin{bmatrix}
1+\epsilon_{mach} & -\frac{1}{\sqrt{2}}\epsilon&-\frac{1}{\sqrt{2}}\epsilon\\
-\frac{1}{\sqrt{2}}\epsilon & 1 & \frac{1}{2}\\
-\frac{1}{\sqrt{2}}\epsilon & \frac{1}{2} & 1
\end{bmatrix}
\gap \text{and}\gap
\bQ_2^\top\bQ_2=
\begin{bmatrix}
1+\epsilon_{mach} & -\frac{1}{\sqrt{2}}\epsilon & -\frac{1}{\sqrt{6}}\epsilon\\
-\frac{1}{\sqrt{2}}\epsilon & 1 & 0\\
-\frac{1}{\sqrt{6}}\epsilon & 0 & 1
\end{bmatrix},
$$
which shows that $\bQ_2$ is better in the sense of orthogonality.
\exampbar
\end{example}

\subsubsection*{\textbf{Row-Wise MGS, Recursive Algorithm}}
The algorithms introduced above in Algorithm~\ref{alg:qr-mgs-left} and \ref{alg:qr-mgs-right} are to compute the entries in the upper triangular matrix $\bR$ element-wise and column-by-column. Suppose $\bA$ has column partition $\bA=[\ba_1, \bA_2]$ where $\bA_2=[\ba_2, \ba_3, \ldots, \ba_n]\in \real^{m\times (n-1)}$. Notice in the CGS Algorithm~\ref{alg:qr-mgs-left}, the first row of $\bR$ can be obtained by
$$
\left.
\begin{aligned}
r_{11} &= ||\ba_1||\\
r_{1k} & = \bq_1^\top\ba_k, \,\,\, \forall k\in\{2,3,\ldots, n\}.
\end{aligned}\right\}
\leadto
\left\{
\begin{aligned}
r_{11} &= ||\ba_1||\\
\br_{12}^\top & = \bq_1^\top\bA_2, \,\,\, \br_{12}=[r_{12}, r_{13}, \ldots, r_{1n}].
\end{aligned}\right.
$$
Therefore, the QR decomposition of $\bA$ is given by
$$
\bA =
\begin{bmatrix}
\ba_1 & \bA_2
\end{bmatrix}=
\begin{bmatrix}
\bq_1 & \bQ_2
\end{bmatrix}
\begin{bmatrix}
r_{11} & \br_{12}^\top\\
\bzero & \bR_{22}
\end{bmatrix}
=
\begin{bmatrix}
r_{11}\bq_1 & \bq_1\br_{12}^\top +\bQ_2\bR_{22}
\end{bmatrix},
$$
where columns of $\bQ_2\in \real^{m\times (n-1)}$ are mutually orthonormal and $\bR_{22}\in \real^{(n-1)\times (n-1)}$ is upper triangular. And this implies $\bQ_2\bR_{22} $ is the reduced QR decomposition of $\bA_2-\bq_1\br_{12}^\top$ which reveals a recursive algorithm for the reduced QR decomposition of $\bA$. This is actually the same as the MGS that subtracts each component in the span of $\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$ when computing column $k$ of $\bQ$ (i.e., equality (*) in Algorithm~\ref{alg:qr-mgs-right}). The process is formulated in Algorithm~\ref{alg:qr-mgs-fulll-rowwise-recursive}.
\begin{algorithm}[H]
\caption{MGS (\textcolor{blue}{Row-Wise and Recursively})=Algorithm~\ref{alg:qr-mgs-right} }
\label{alg:qr-mgs-fulll-rowwise-recursive}
\begin{algorithmic}[1]
	\Require $\bA\in \real^{m\times n}$ with full column rank;
	\For{$k=1$ to $n$}  \Comment{i.e., compute $k$-th column of $\bQ$ and $k$-th row of $\bR$}
	\State $\ba_1=\bA[:,1]$; \Comment{$1$-st column of $\bA\in \real^{m\times (n-k+1)}$}
	\State $r_{kk}=||\ba_1||$;\Comment{$\ba_1\in \real^{m\times 1}$}
	\State $\bq_k = \ba_1/r_{kk}$;
	\State $\br_{k2}^\top=\bq_k^\top\bA_2$; \Comment{$\bA_2=\bA[:,2:n]\in \real^{m\times (n-k)}$, $\br_{k2}^\top\in \real^{1\times (n-k)}$}
	\State $\bA=\bA_2-\bq_k\br_{k2}^\top$; \Comment{$\bA \in \real^{m\times (n-k)}$}
	\EndFor
	\State Output $\bQ=[\bq_1, \ldots, \bQ_n]$ and $\bR$ with entry $(i,k)$ being $r_{ik}$;
\end{algorithmic}
\end{algorithm}

\section{Computing the Full QR Decomposition via the Gram-Schmidt Process}
A full QR decomposition of an $m\times n$ matrix with linearly independent columns goes further by appending additional $m-n$ orthonormal columns to $\bQ$ so that it becomes an $m\times m$ orthogonal matrix. In addition, rows of zeros are appended to $\bR$ so that it becomes an $m\times n$ upper triangular matrix. We call the additional columns in $\bQ$ \textbf{silent columns} and additional rows in $\bR$ \textbf{silent rows}. The comparison between the reduced QR decomposition and the full QR decomposition is shown in Figure~\ref{fig:qr-comparison} where silent columns in $\bQ$ are denoted in \textcolor{gray}{gray}, blank entries are zero and \textcolor{blue}{blue} entries are elements that are not necessarily zero.

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[Reduced QR decomposition]{\label{fig:gphalf}
	\includegraphics[width=0.47\linewidth]{./imgs/qrreduced.pdf}}
\quad
\subfigure[Full QR decomposition]{\label{fig:gpall}
	\includegraphics[width=0.47\linewidth]{./imgs/qrfull.pdf}}
\caption{Comparison between the reduced and full QR decompositions.}
\label{fig:qr-comparison}
\end{figure}

\section{Dependent Columns}\label{section:dependent-gram-schmidt-process}
Previously, we assumed matrix $\bA$ has linearly independent columns. However, this is not always necessary. Suppose in step $k$ of Algorithm~\ref{alg:reduced-qr}, $\ba_k$ is in the plane spanned by $\bq_1, \bq_2, \ldots, \bq_{k-1}$ which is equivalent to the space spanned by $\ba_1, \ba_2, \ldots, \ba_{k-1}$, i.e., vectors $\ba_1, \ba_2, \ldots, \ba_k$ are dependent. Then $r_{kk}$ will be zero and $\bq_k$ does not exist because of the zero division. At this moment, we simply pick $\bq_k$ arbitrarily to be any normalized vector that is orthogonal to $\cspace([\bq_1, \bq_2, \ldots, \bq_{k-1}])$ and continue the Gram-Schmidt process. Again, for matrix $\bA$ with dependent columns, we have both reduced and full QR decomposition algorithms. We reformulate the step $k$ in the algorithm as follows:
$$
\bq_k=\left\{
\begin{aligned}
&(\ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i)/r_{kk}, \qquad r_{ik}=\bq_i^\top\ba_k, r_{kk}=||\ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i||, &\mathrm{if\,} r_{kk}\neq0, \\
&\mathrm{pick \, one\, in\,}\cspace^{\bot}([\bq_1, \bq_2, \ldots, \bq_{k-1}]),\qquad &\mathrm{if\,} r_{kk}=0.
\end{aligned}
\right.
$$

This idea can be further extended that when $\bq_k$ does not exist, we just skip the current steps. And add the silent columns in the end. In this sense, QR decomposition for a matrix with dependent columns is not unique. However, as long as you stick to a systematic process, QR decomposition for any matrix is unique.

This finding can also help to decide whether a set of vectors are linearly independent or not. Whenever $r_{kk}$ in Algorithm~\ref{alg:reduced-qr} is zero, we report the vectors $\ba_1, \ba_2, \ldots, \ba_k$ are dependent and stop the algorithm for ``independent checking".

\section{QR with Column Pivoting: Column-Pivoted QR (CPQR)}\label{section:cpqr}

Suppose $\bA$ has dependent columns, a column-pivoted QR (CPQR) decomposition can be found as follows.
\begin{theoremHigh}[Column-Pivoted QR Decomposition\index{Column-pivoted QR (CPQR)}]\label{theorem:rank-revealing-qr-general}
Every $m\times n$ matrix $\bA=[\ba_1, \ba_2, ..., \ba_n]$ with $m\geq n$ and rank $r$ can be factored as
$$
\bA\bP = \bQ
\begin{bmatrix}
\bR_{11} & \bR_{12} \\
\bzero   & \bzero
\end{bmatrix},
$$
where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ\in \real^{m\times m}$ is an orthogonal matrix, and $\bP$ is a permutation matrix. This is also known as the \textbf{full} CPQR decomposition. Similarly, the \textbf{reduced} version is given by
$$
\bA\bP = \bQ_r
\begin{bmatrix}
\bR_{11} & \bR_{12} \\
\end{bmatrix},
$$
where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ_r\in \real^{m\times r}$ contains orthonormal columns, and $\bP$ is a permutation matrix.
\end{theoremHigh}
\subsection{A Simple CPQR via CGS}
\paragraph{A Simple CPQR via CGS}
The classical Gram-Schmidt process can compute this CPQR decomposition. Following from the QR decomposition for dependent columns that when $r_{kk}=0$, the column $k$ of $\bA$ is dependent on the previous $k-1$ columns. Whenever this happens, we permute this column into the last column and continue the Gram-Schmidt process. We notice that $\bP$ is the permutation matrix that interchanges the dependent columns into the last $n-r$ columns. Suppose the first $r$ columns of $\bA\bP$ are $[\widehat{\ba}_1, \widehat{\ba}_2, \ldots, \widehat{\ba}_r]$, and the span of them is just the same as the span of $\bQ_r$ (in the reduced version), or the span of $\bQ_{:,:r}$ (in the full version)
$$
\cspace([\widehat{\ba}_1, \widehat{\ba}_2, \ldots, \widehat{\ba}_r]) = \cspace(\bQ_r) = \cspace(\bQ_{:,:r}).
$$
And $\bR_{12}$ is a matrix that recovers the dependent $n-r$ columns from the column space of $\bQ_r$ or column space of $\bQ_{:,:r}$. The comparison of reduced and full CPQR decomposition is shown in Figure~\ref{fig:qr-comparison-rank-reveal} where silent columns in $\bQ$ are denoted in \textcolor{gray}{grey}, blank entries are zero and \textcolor{blue}{blue}/\textcolor{orange}{orange} entries are elements that are not necessarily zero.

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[Reduced CPQR decomposition]{\label{fig:gphalf-rank-reveal}
	\includegraphics[width=0.475\linewidth]{./imgs/qrreduced-revealing.pdf}}
\quad
\subfigure[Full CPQR decomposition]{\label{fig:gpall-rank-reveal}
	\includegraphics[width=0.475\linewidth]{./imgs/qrfull-revealing.pdf}}
\caption{Comparison between the reduced and full CPQR decompositions.}
\label{fig:qr-comparison-rank-reveal}
\end{figure}

\begin{algorithm}[h]
\caption{\textcolor{blue}{Simple} Reduced CPQR Decomposition via CGS}
\label{alg:reduced-qr-rank-revealing}
\begin{algorithmic}[1]
	\Require
	Matrix $\bA$ with size $m\times n$ and $m\geq n$;
	\State $cnt = 0$; \Comment{i.e., the count for the permutations}
	\State $\bq_1   = \ba_1/r_{11},  r_{11}=||\ba_1||$; \Comment{i.e., the first column of $\bR_{11}$}
	\For{$k=2$ to $n$}   \Comment{i.e., compute column $k$ of $\bR_{11}$}
	\State Set the initial value $r_{kk}=0$;
	\While{$r_{kk} ==0$} \Comment{the column is dependent if $r_{kk}$ is equal to 0}
	\State $r_{ik}=\bq_i^\top\ba_k$, $\forall i \in \{1, 2, \ldots, k-1\}$;\Comment{first $k-1$ elements in column $k$ of $\bR_{11}$}
	\State $r_{kk}=||\ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i||$;\Comment{$k$-th elements in column $k$ of $\bR_{11}$}
	\State $\bq_k = (\ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i)/r_{kk}$;
	\If{$r_{kk}==0$}
	\State $cnt = cnt+1$;
	\State Permute the column $k$ to last column;
	\State i.e., $[\ba_k, \ba_{k+1}, \ldots, \ba_n] \leftarrow [\ba_{k+1}, \ba_{k+2}, \ldots, \ba_n, \ba_k]$;
	\EndIf
	\EndWhile
	\If{$k+cnt == n$}
	\State rank $r=k$;
	\For{$k=r+1$ to $n$} \Comment{i.e., compute column $k$ of $\bR_{12}$}
	\State $r_{ik}=\bq_i^\top\ba_k$, $\forall i \in \{1, 2, \ldots, \textcolor{blue}{r}\}$;

	\EndFor
	\EndIf
	\State Output rank $r$, output $\bR_{11}, \bR_{12}$, $\bQ_r=[\bq_1, \bq_2, \ldots, \bq_r]$. And exit the loop;
	\EndFor
\end{algorithmic}
\end{algorithm}

The reduced algorithm is formulated in Algorithm~\ref{alg:reduced-qr-rank-revealing}. And it is trivial to find the last $n-r$ columns of $\bQ$ in the orthogonal complement of $\cspace(\bQ_r)$.
Note step 6 of Algorithm~\ref{alg:reduced-qr-rank-revealing} can be rewritten as $\bQ_{k-1}^\top \ba_k$, where $\bQ_{k-1}=[\bq_1, \ldots, \bq_{k-1}]$.

\begin{theorem}[Algorithm Complexity: Reduced CPQR]\label{theorem:qr-reduced-rank-revealing}
	Algorithm~\ref{alg:reduced-qr-rank-revealing} requires $\sim 6mnr-4mr^2$ flops to compute a CPQR decomposition of an $m\times n$ matrix with $m\geq n$ and rank $r$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-reduced-rank-revealing}]
From Theorem~\ref{theorem:qr-reduced}, to compute $\bR_{11}$, we just need to replace $n$ in Theorem~\ref{theorem:qr-reduced} by $r$ such that the complexity of computing $\bR_{11}$ is \fbox{$2mr^2$} flops if keep only the leading term.

To compute $\bR_{12}$, there are $r\times (n-r)$ values, each taking $2m-1$ flops ($m$ multiplications and $m-1$ additions) from step 18. That is \fbox{$r(n-r)(2m-1)$} flops.

The upper bound on steps 6, 7, 8 is set $k-1=r$ in these steps. This makes \fbox{$(2m-1)r$} flops for step 6 ($mr$ multiplications and $(m-1)r$ additions), \fbox{$2m(r+1)$} flops for step 7 ($mr$ multiplications, $(r-1)m$ additions, $m$ subtractions, $2m$ for the norm), and \fbox{$m$} flops for step 8. The total complexity for steps 6, 7, 8 is thus \fbox{$4mr+3m-r$} flops. And there are $n-r$ such iterations, which imply \fbox{$(4mr+3m-r)(n-r)$} flops needed to find the dependent columns.

Therefore, the final complexity is
$$
2mr^2 + r(n-r)(2m-1)+ (4mr+3m-r)(n-r).
$$
And it is $6mnr-4mr^2$ flops if we keep only the leading term.
\end{proof}
We notice that when $r=n$, the complexity for Algorithm~\ref{alg:reduced-qr-rank-revealing} is $2mn^2$ which agrees with the complexity of Algorithm~\ref{alg:reduced-qr}.

\subsection{A Practical CPQR via CGS}\label{section:practical-cpqr-cgs}
\paragraph{A Practical CPQR via CGS}
We notice that the simple CPQR algorithm pivot the first $r$ independent columns into the first $r$ columns of $\bA\bP$. Let $\bA_1$ be the first $r$ columns of $\bA\bP$, and $\bA_2$ be the rest. Then, from the full CPQR, we have
$$
[\bA_1, \bA_2] =
\bQ \begin{bmatrix}
\bR_{11} & \bR_{12} \\
\bzero   & \bzero
\end{bmatrix}
=\left[
\bQ \begin{bmatrix}
\bR_{11}  \\
\bzero
\end{bmatrix}
,
\bQ \begin{bmatrix}
\bR_{12}  \\
\bzero
\end{bmatrix}
\right]
.
$$
It is not easy to see that
$$
||\bA_2|| = \left\Vert\bQ \begin{bmatrix}
\bR_{12}  \\
\bzero
\end{bmatrix}\right\Vert
=
\left\Vert\begin{bmatrix}
\bR_{12}  \\
\bzero
\end{bmatrix}\right\Vert
=
\left\Vert \bR_{12}  \right\Vert,
$$
where the penultimate equality comes from the orthogonal equivalence under the matrix norm (Lemma~\ref{lemma:frobenius-orthogonal-equi}, p.~\pageref{lemma:frobenius-orthogonal-equi}). Therefore, the norm of $\bR_{12}$ is decided by the norm of $\bA_2$. When favoring well-conditioned CPQR, $\bR_{12}$ should be small in norm. And a practical CPQR decomposition is to permute columns of the matrix $\bA$ firstly such that the columns are ordered decreasingly in vector norm:
$$
\widetildebA = \bA\bP_0 = [\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_n}],
$$
where $\{j_1, j_2, \ldots, j_n\}$ is a permuted index set of $\{1,2,\ldots, n\}$ and
$$
||\ba_{j_1}||\geq  ||\ba_{j_2}||\geq  \ldots\geq  ||\ba_{j_n}||.
$$
Then apply the ``simple" reduced CPQR decomposition on $\widetildebA$ such that $\widetildebA \bP_1= \bQ_r[\bR_{11}, \bR_{12}]$. The ``practical" reduced CPQR of $\bA$ is then recovered as
$$
\bA\underbrace{\bP_0\bP_1}_{\bP} =\bQ_r[\bR_{11}, \bR_{12}].
$$
When the $l_2$ vector norm (i.e., inner product, Appendix~\ref{section:vector-norm}, p.~\pageref{section:vector-norm}) is applied, extra $n(2m-1)$ flops are required to compute the $n$ norms of the column vectors and $\frac{n(n-1)}{2}$ comparisons needed to determine the order of the norms.

\subsection{A Practical CPQR via MGS}
\paragraph{A Practical CPQR via MGS}
Now, based on the recursive MGS in Algorithm~\ref{alg:qr-mgs-fulll-rowwise-recursive}, we can also develop a practical CPQR. The algorithm is formulated in Algorithm~\ref{alg:cpqr-partial-fact-cpqr} where the only difference to Algorithm~\ref{alg:qr-mgs-fulll-rowwise-recursive} is highlighted in the \textcolor{blue}{blue} text that we permute the column with the largest norm into the first column.

\begin{algorithm}[H]
\caption{\textcolor{blue}{Practical} CPQR via MGS (\textcolor{blue}{Row-Wise and Recursively}) }
\label{alg:cpqr-partial-fact-cpqr}
\begin{algorithmic}[1]
	\Require $\bA\in \real^{m\times n}$ with \textcolor{blue}{exact} rank $r$;
	\For{$k=1$ to $n$}  \Comment{i.e., compute $k$-th column of $\bQ$ and $k$-th row of $\bR$}
	\State \textcolor{blue}{Find the column with largest norm in $\bA$, and permute to first column;}
	\State $\ba_1=\bA[:,1]$; \Comment{$1$-st column of $\bA\in \real^{m\times (n-k+1)}$}
	\State $r_{kk}=||\ba_1||$;\Comment{$\ba_1\in \real^{m\times 1}$}
	\State $\bq_k = \ba_1/r_{kk}$;
	\State $\br_{k2}^\top=\bq_k^\top\bA_2$; \Comment{$\bA_2=\bA[:,2:n]\in \real^{m\times (n-k)}$, $\br_{k2}^\top\in \real^{1\times (n-k)}$}
	\State $\bA=\bA_2-\bq_k\br_{k2}^\top$; \Comment{$\bA \in \real^{m\times (n-k)}$}
	\State \textcolor{blue}{Exit when $r_{kk}=0$;}
	\EndFor
	\State Output $\bQ=[\bq_1, \ldots, \bQ_n]$ and $\bR$ with entry $(i,k)$ being $r_{ik}$;
\end{algorithmic}
\end{algorithm}

The difference is in that, in each iteration, we need to all the norms of the columns of $\bA$ rather than compute the norm of it at once as that is in CGS. Suppose in iteration $k$, we need to compute the reduced QR decomposition of a matrix of size $m\times (n-k+1)$ if the original matrix $\bA$ is of size $m\times n$. That is, extra $(n-k+1)(2m-1)$ flops required flops needed to do the CPQR via MGS. Let $f(k)=(n-k+1)(2m-1)$, simple calculation can show that additional complexity for CPQR via MGS is:
\begin{equation}\label{equation:mgs-cpqr-extra1}
\text{extra cost = }f(1)+f(2)+\ldots +f(n)\sim mn^2 \text{  flops},
\end{equation}
if only keep the leading term. This costs more than $n(2m-1)$ flops in the ``practical" CPQR via CGS.

But do not worry, this extra cost in CPQR via MGS can be partially solved. Suppose the column partition of $\bA\in \real^{m\times n}$ is $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$, and each squared norm of the columns are given in the vector
$$
\bl =
\begin{bmatrix}
l_1 \\
l_2 \\
\vdots \\
l_n
\end{bmatrix}=
\begin{bmatrix}
||\ba_1||^2\\
||\ba_2||^2\\
\vdots \\
||\ba_n||^2\\
\end{bmatrix}.
$$
Suppose further $\bq\in \real^m$ is a unit-length vector such that $\bq^\top\bq=1$ and $\br\in \real^n$ is a trivial vector given by
$$
\br = \bA^\top\bq
=
\begin{bmatrix}
r_1 \\
r_2 \\
\vdots \\
r_n
\end{bmatrix}
. \gap \text{(similar to step 6 of above Algorithm~\ref{alg:cpqr-partial-fact-cpqr})}
$$
Let further $\bB=\bA-\bq\br^\top=[\bb_1, \bb_2, \ldots, \bb_n]$ (similar to step 7 of above Algorithm~\ref{alg:cpqr-partial-fact-cpqr}). Then the squared length vector of $\bB$ is given by
$$
\bl =
\begin{bmatrix}
s_1 \\
s_2 \\
\vdots \\
s_n
\end{bmatrix}=
\begin{bmatrix}
||\bb_1||^2\\
||\bb_2||^2\\
\vdots \\
||\bb_n||^2\\
\end{bmatrix}
=
\begin{bmatrix}
l_1 -r_1^2\\
l_2 -r_2^2\\
\vdots \\
l_n-r_n^2
\end{bmatrix}.
$$
This can be easily checked since $\bb_i = \ba_i -r_{i}\bq =\ba_i-(\ba_i^\top\bq) \bq$ such that
$$
||\bb_i||^2 = || \ba_i -r_{i}\bq||^2=( \ba_i -r_{i}\bq)^\top( \ba_i -r_{i}\bq)=l_i-r_i^2.
$$
Come back to step 2 of Algorithm~\ref{alg:cpqr-partial-fact-cpqr}. Suppose we have computed squared norm of the original matrix $\bA\in \real^{m\times n}$ (which takes $n(2m-1)$, the same as that in ``practical" CPQR via CGS). The squared norm of $\bA_2-\bq_1\br_{12}^\top$ (suppose $k=1$ in step 7 of Algorithm~\ref{alg:cpqr-partial-fact-cpqr}) can be obtained by extra $2(n-1)$ flops. And for all the $n$ iterations, the cost is $2(n-1)+2(n-2)+\ldots +2(1)=n^2-n$ flops. This is much less than $\sim mn^2$ in Equation~\eqref{equation:mgs-cpqr-extra1}.

\subsection{Partial Factorization for CPQR: Extra Bonus of CPQR via MGS}\label{section:partial-cpqr-mgs}
\paragraph{Partial Factorization for CPQR}
The extra bonus for the CPQR via MGS is that we can do partial factorization at some point. We notice that at step of Algorithm~\ref{alg:cpqr-partial-fact-cpqr}, we permute the column with the largest norm into the first column, and step 4 of Algorithm~\ref{alg:cpqr-partial-fact-cpqr} is to compute the norm into the main diagonal of the upper triangular $\bR$. When $\bA$ has full rank $n$, $r_{kk}$ in all iterations will be positive. When $\bA$ has an ``exact" rank $r$, the algorithm will stop after iteration $r$. However, when $\bA$ has an ``effective" rank $r$ with rank deficiency
\footnote{\textit{Effective rank}, or also known as the \textit{numerical rank}. Assume the $i$-th largest singular value of $\bA$ is denoted as $\sigma_i(\bA)$. Then if $\sigma_r(\bA)\gg \sigma_{r+1}(\bA)\approx 0$, $r$ is known as the numerical rank of $\bA$. The singular value of matrix $\bA$ will be introduced in the SVD section (Section~\ref{section:SVD}, p.~\pageref{section:SVD}). Whereas, when $\sigma_i(\bA)>\sigma_{r+1}(\bA)=0$, it is known as having \textit{exact rank} $r$ as we have used in most of our discussions.}, the algorithm can proceed well in the first $r$ iterations since $\{r_{11}, r_{22}, \ldots, r_{rr}\}$ are relatively large in value that are far from 0. When it comes to iteration $k=r+1, \ldots$, the $r_{kk}$ is small which means the column $k$ of $\bA\bP$ has a small component in the direction of $\bq_k$ and is ``almost" dependent on the previous $\{\bq_1, \bq_2, \ldots, \bq_k\}$. The situation is the same for the rest $\{k+1, k+2, \ldots\}$ columns since $r_{kk}\geq r_{k+1, k+1}\geq \ldots r_{nn}$ in iteration $k$. The partial factorization CPQR via MGS is formulated in Algorithm~\ref{alg:cpqr-partial-fact-cpqr-partial}. This is related to the \textit{rank-revealing QR decomposition (RRQR)} that we will introduce in the next sections. The algorithm will result in the factorization
\begin{equation}
\bA\bP =
\bQ\bR=
\bQ
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero & \bR_{22}
\end{bmatrix},
\end{equation}
where $\bR_{22}$ is small in norm.

However, in the ``practical" CPQR with CGS (Algorithm~\ref{alg:reduced-qr-rank-revealing} with $\bA$ permuted at once at the beginning of the procedure), it is wasteful when $r \ll \min(m, n)$. Since we only permute the column with the largest norm into the beginning of $\bA\bP$. A value $r_{kk}$ close to 0 only means column $k$ of $\bA\bP$ is almost dependent on $\{\bq_1, \bq_2, \ldots, \bq_k\}$, and it does not mean the rest $\{k+1, k+2, \ldots\}$ columns of $\bA\bP$ are also almost dependent on $\{\bq_1, \bq_2, \ldots, \bq_k\}$.

\begin{algorithm}[H]
\caption{\textcolor{blue}{Practical} and \textcolor{blue}{Partial} CPQR via MGS (\textcolor{blue}{Row-Wise and Recursively}) }
\label{alg:cpqr-partial-fact-cpqr-partial}
\begin{algorithmic}[1]
	\Require $\bA\in \real^{m\times n}$ with rank deficiency;
	\State Select a stopping criteria $\delta$;
	\For{$k=1$ to $n$}  \Comment{i.e., compute $k$-th column of $\bQ$ and $k$-th row of $\bR$}
	\State \textcolor{blue}{Find the column with largest norm in $\bA$, and permute to first column;}
	\State $\ba_1=\bA[:,1]$; \Comment{$1$-st column of $\bA\in \real^{m\times (n-k+1)}$}
	\State $r_{kk}=||\ba_1||$;\Comment{$\ba_1\in \real^{m\times 1}$}
	\State $\bq_k = \ba_1/r_{kk}$;
	\State $\br_{k2}^\top=\bq_k^\top\bA_2$; \Comment{$\bA_2=\bA[:,2:n]\in \real^{m\times (n-k)}$, $\br_{k2}^\top\in \real^{1\times (n-k)}$}
	\State $\bA=\bA_2-\bq_k\br_{k2}^\top$; \Comment{$\bA \in \real^{m\times (n-k)}$}
	\State \textcolor{blue}{Exit when $r_{kk}<\delta$, set effective rank $r=k$;}
	\EndFor
	\State Output $\bQ=[\bq_1, \ldots, \bQ_n]$, $\bR$ with entry $(i,k)$ being $r_{ik}$, and \textcolor{blue}{effective rank $r$};
\end{algorithmic}
\end{algorithm}
\section{QR with Column Pivoting: Revealing Rank One Deficiency}\label{section:rank-one-qr-revealing}\index{Rank-revealing}
We notice that Algorithm~\ref{alg:reduced-qr-rank-revealing} is just one method to find the column permutation where $\bA$ is rank deficient and we interchange the first linearly independent $r$ columns of $\bA$ into the first $r$ columns of the $\bA\bP$. If $\bA$ is nearly rank-one deficient and we would like to find a column permutation of $\bA$ such that the resulting pivotal element $r_{nn}$ of the QR decomposition is small. This is known as the \textit{revealing rank-one deficiency} problem.
\begin{theoremHigh}[Revealing Rank One Deficiency, \citep{chan1987rank}]\label{theorem:finding-good-qr-ordering}
If $\bA\in \real^{m\times n}$ and $\bv\in \real^n$ is a unit 2-norm vector (i.e., $||\bv||=1$), then there exists a permutation $\bP$ such that the reduced QR decomposition
$$
\bA\bP = \bQ\bR
$$
satisfies $r_{nn} \leq \sqrt{n} \epsilon$ where $\epsilon = ||\bA\bv||$ and $r_{nn}$ is the $n$-th diagonal of $\bR$. Note that $\bQ\in \real^{m\times n}$ and $\bR\in \real^{n\times n}$ in the reduced QR decomposition.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:finding-good-qr-ordering}]
Suppose $\bP\in \real^{n\times n}$ is a permutation matrix such that if $\bw=\bP^\top\bv$ where
$$
|w_n| = \max |v_i|,  \,\,\,\, \forall i \in \{1,2,\ldots,n\},
$$
i.e., interchange the largest magnitude into the last entry
such that the last component of $\bw$ is equal to the maximal component of $\bv$ in absolute value. Then we have $|w_n| \geq 1/\sqrt{n}$. Suppose the QR decomposition of $\bA\bP$ is $\bA\bP = \bQ\bR$, then
$$
\epsilon = ||\bA\bv|| = ||(\bQ^\top\bA\bP) (\bP^\top\bv)|| = ||\bR\bw|| =
\begin{bmatrix}
\vdots \\
r_{nn} w_n
\end{bmatrix}
\geq |r_{nn} w_n| \geq |r_{nn}|/\sqrt{n},
$$
where the second equality above is from the length preservation under orthogonal transformation and $\bP$ is orthogonal such that $\bP\bP^\top=\bI$. This completes the proof.
\end{proof}
The following discussion is based on the existence of the singular value decomposition (SVD) which will be introduced in Section~\ref{section:SVD} (p.~\pageref{section:SVD}). Feel free to skip at a first reading. Suppose the SVD of $\bA$ is given by $\bA = \sum_{i=1}^{n} \sigma_i \bu_i\bv_i^\top$, where $\sigma_i$'s are the singular values with $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_n$, i.e., $\sigma_n$ is the smallest singular value, and $\bu_i$'s, $\bv_i$'s are left and right singular vectors respectively. Then, if we let $\bv = \bv_n$ such that $\bA\bv_n = \sigma_n \bu_n$, \footnote{We will prove that the right singular vector of $\bA$ is equal to the right singular vector of $\bR$ if the $\bA$ has QR decomposition $\bA=\bQ\bR$ in Lemma~\ref{lemma:svd-for-qr} (p.~\pageref{lemma:svd-for-qr}). The claim can also be applied to the singular values. So $\bv_n$ here is also the right singular vector of $\bR$.} we have
$$
||\bA\bv|| = \sigma_n.
$$
By constructing a permutation matrix $\bP$ such that
$$
|\bP^\top \bv|_n = \max |\bv_i|,  \,\,\,\, \forall i \in \{1,2,\ldots,n\},
$$
we will find a QR decomposition of $\bA=\bQ\bR$ with a pivot $r_{nn}$ smaller than $\sqrt{n}\sigma_n$. If $\bA$ is rank-one deficient, then $\sigma_n$ will be close to 0 and $r_{nn}$ is thus bounded to a small value in magnitude which is close to 0.

\section{QR with Column Pivoting: Revealing Rank r Deficiency*}\label{section:rank-r-qr}\index{Rank-revealing QR}
Following from the last section, suppose now we want to compute the reduced QR decomposition where $\bA\in \real^{m\times n}$ is nearly rank $r$ deficient \footnote{Note that rank $r$ here does not mean the matrix has rank $r$, but rather it has rank $(\min\{m,n\}-r)$.} with $r>1$. Our goal now is to find a permutation $\bP$ such that
\begin{equation}\label{equation:rankr-reval-qr}
\bA\bP =
\bQ\bR=
\bQ
\begin{bmatrix}
\bL & \bM \\
\bzero & \bN
\end{bmatrix},
\end{equation}\footnote{To abuse the notation, we use the notation $\bL,\bM,\bN$ for clarity on the derivation. It is better to replace $\bL,\bM,\bN$ by $\bR_{11}, \bR_{12}, \bR_{22}$ to match other contexts.}
where $\bN \in \real^{r\times r}$ and $||\bN||$ is small in some norm (and $\bL\in \real^{(n-r)\times (n-r)}, \bM\in \real^{(n-r)\times r}$ that can be inferred from context).

A recursive algorithm can be applied to do so. Suppose we have already isolated a small $k\times k$ block $\bN_k$, based on which, if we can isolate a small $(k+1)\times (k+1)$ block $\bN_{k+1}$, then we can find the permutation matrix recursively. To repeat, suppose we have the permutation $\bP_k$ such that the $\bN_k \in \real^{k\times k}$ has a small norm,
$$
\bA\bP_k = \bQ_k \bR_k=
\bQ_k
\begin{bmatrix}
\bL_k & \bM_k \\
\bzero & \bN_k
\end{bmatrix}.
$$
We want to find a permutation $\bP_{k+1}$, such that $\bN_{k+1} \in \real^{(k+1)\times (k+1)}$ also has a small norm,
$$
\boxed{
\bA\bP_{k+1} = \bQ_{k+1} \bR_{k+1}=
\bQ_{k+1}
\begin{bmatrix}
\bL_{k+1} & \bM_{k+1} \\
\bzero & \bN_{k+1}
\end{bmatrix}}.
$$
From the algorithm introduced in the last section, there is an $(n-k)\times (n-k)$ permutation matrix $\widetilde{\bP}_{k+1}$ such that $\bL_k \in \real^{(n-k)\times (n-k)}$ has the QR decomposition $\bL_k \widetilde{\bP}_{k+1} = \widetilde{\bQ}_{k+1}\widetilde{\bL}_k$ such that the entry $(n-k, n-k)$ of $\widetilde{\bL}_k$ is small. By constructing
$$
\bP_{k+1} = \bP_k
\begin{bmatrix}
\widetilde{\bP}_{k+1} & \bzero \\
\bzero & \bI
\end{bmatrix},
\qquad
\bQ_{k+1} = \bQ_k
\begin{bmatrix}
\widetilde{\bQ}_{k+1} & \bzero \\
\bzero & \bI
\end{bmatrix},
$$
we have
$$
\boxed{
\bA \bP_{k+1} = \bQ_{k+1}
\begin{bmatrix}
\widetilde{\bL}_k & \widetilde{\bQ}_{k+1}^\top  \bM_k \\
\bzero & \bN_k
\end{bmatrix}}.
$$
We know that entry $(n-k, n-k)$ of $\widetilde{\bL}_k$ is small, if we can prove the last row of $\widetilde{\bQ}_{k+1}^\top  \bM_k$ is small in norm, then we find the QR decomposition revealing rank $k+1$ deficiency (see \citep{chan1987rank} for a proof). And the procedure is formulated in Algorithm~\ref{alg:qr-reveal-rank-r}.

\begin{algorithm}[h]
\caption{Reveal Rank $r$ Deficiency}
\label{alg:qr-reveal-rank-r}
\begin{algorithmic}[1]
\Require
Matrix $\bA$ with size $m\times n$ and $m\geq n$, and $rank(\bA)=n-r$;
\State Initialize $\bW \in \real^{n\times r}$ to zero; \Comment{store the singular vectors}
\State Initial QR decomposition by $\bA = \bQ\bR$;
\For{$i=n$ to $n-r+1$}
\State $\bL \leftarrow $ leading $i\times i$ block of $\bR$;
\State Compute the singular vector $\bv\in \real^i$ corresponding to the min singular value of $\bL$;
\State Compute a permutation $\widetilde{\Pi} \in \real^{i\times i}$ such that $|\widetilde{\Pi}^\top \bv|_i = \max |\bv_j|, \forall j \in \{1,2,\ldots, i\}$;
\State Assign $\begin{bmatrix}
	\bv \\
	\bzero
\end{bmatrix}$ to the $i$-th column of $\bW$;
\State Compute $\bW \leftarrow \Pi^\top \bW$, where $\Pi = \begin{bmatrix}
	\widetilde{\Pi} & \bzero \\
	\bzero & \bI
\end{bmatrix}$;
\State Compute the QR decomposition: $\bL\widetilde{\Pi} = \widetilde{\bQ} \widetilde{\bL}$;
\State $\bP \leftarrow \bP \Pi$;
\State $\bQ \leftarrow \bQ \begin{bmatrix}
	\widetilde{\bQ} & \bzero  \\
	\bzero & \bI
\end{bmatrix}$;
\State $\bR \leftarrow  \begin{bmatrix}
	\widetilde{\bL} & \widetilde{\bQ}^\top \bB \\
	\bzero & \bC
\end{bmatrix}$, where $\bB =\bR_{1:n-i, n-i+1:n}$, and $\bC = \bR_{n-i+1:n, n-i+1:n}$;
\EndFor

\end{algorithmic}
\end{algorithm}

\section{Existence of the QR Decomposition via the Householder Reflector}\label{section:qr-via-householder}
\subsubsection*{\textbf{Householder Reflectors}}
We first give the formal definition of a Householder reflector and we will take a look at its properties.
\begin{definition}[Householder Reflector\index{Householder reflector}]\label{definition:householder-reflector}
Let $\bu \in \real^n$ be a vector of unit length (i.e., $||\bu||=1$). Then $\bH = \bI - 2\bu\bu^\top$ is said to be a \textit{Householder reflector}, a.k.a., a \textit{Householder transformation}. We call this $\bH$ the Householder reflector associated with the unit vector $\bu$ where the unit vector $\bu$ is also known as \textit{Householder vector}. If a vector $\bx$ is multiplied by $\bH$, then it is reflected in the hyperplane $span\{\bu\}^\perp$.

\item Note that if $||\bu|| \neq 1$, we can define $\bH = \bI - 2  \frac{\bu\bu^\top}{\bu^\top\bu} $ as the Householder reflector.
\end{definition}
From the definition of the Householder reflector, then we have the following corollary that a special kind of vectors will remain unchanged under the Householder reflector.
\begin{corollary}[Unreflected by Householder]
Suppose $||\bu||=1$, and define the Householder reflector $\bH=\bI-2\bu\bu^\top$. Then any vector $\bv$ that is perpendicular to $\bu$ is left unchanged by the Householder transformation, that is, $\bH\bv=\bv$ if $\bu^\top\bv=0$.
\end{corollary}
The proof is trivial that $(\bI - 2\bu\bu^\top)\bv = \bv - 2\bu\bu^\top\bv=\bv$.

Suppose $\bu$ is a unit vector with $||\bu||=1$, and a vector $\bv$ is perpendicular to $\bu$. Then any vector $\bx$ on the plane can be decomposed into two parts
$$
\bx = \bx_{\bv} + \bx_{\bu},
$$
where the first one $\bx_{\bu}$ is parallel to $\bu$ and the second one $\bx_{\bv}$ is perpendicular to $\bu$ (i.e., parallel to $\bv$). From Section~\ref{section:project-onto-a-vector} on the projection of a vector onto another one, $\bx_{\bu}$ can be computed by $\bx_{\bu} = \frac{\bu\bu^\top}{\bu^\top\bu} \bx = \bu\bu^\top\bx$, i.e., the projection of $\bx$ onto the vector $\bu$. We then transform this $\bx$ by the Householder reflector associated with $\bu$,
$$\bH\bx = (\bI - 2\bu\bu^\top)(\bx_{\bv} + \bx_{\bu}) = \bx_{\bv} -\bu\bu^\top \bx = \bx_{\bv} - \bx_{\bu},
$$
i.e., the Householder reflector transforms $\bx_{\bv} + \bx_{\bu}$ into $\bx_{\bv} - \bx_{\bu}$.
That is, the space perpendicular to $\bu$ acts as a mirror and any vector $\bx$ is reflected by the Householder reflector associated with $\bu$ (i.e., reflected in the hyperplane $span\{\bu\}^\perp$). The situation is shown in Figure~\ref{fig:householder}.

\begin{SCfigure
\centering
\includegraphics[width=0.5\textwidth]{imgs/householder.pdf}
\caption{Demonstration of the Householder reflector. The Householder reflector obtained by $\bH=\bI-2\bu\bu^\top$ where $||\bu||=1$ will reflect vector $\bx$ along the plane perpendicular to $\bu$: $\bx=\bx_{\bv} + \bx_{\bu} \rightarrow \bx_{\bv} - \bx_{\bu}$.}
\label{fig:householder}
\end{SCfigure}

The above discussion tells us how to find the reflected vector given the Householder reflector. The further question can be posed that if we know two vectors are reflected to each other up front, the next corollary tells us how to find the corresponding Householder reflector. The property is important for computing the QR decomposition if we want to reflect a column into a specific form.
\begin{corollary}[Finding the Householder Reflector]\label{corollary:householder-reflect-finding}
Suppose $\bx$ is reflected to $\by$ by a Householder reflector with $||\bx|| = ||\by||$, then the Householder reflector is obtained by
$$
\bH = \bI - 2 \bu\bu^\top, \text{ where } \bu = \frac{\bx-\by}{||\bx-\by||}.
$$
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:householder-reflect-finding}]
Write out the equation, we have
$$
\begin{aligned}
\bH\bx &= \bx - 2 \bu\bu^\top\bx =\bx - 2\frac{(\bx-\by)(\bx^\top-\by^\top)}{(\bx-\by)^\top(\bx-\by)} \bx\\
&= \bx - (\bx-\by) = \by.
\end{aligned}
$$
Note that the condition $||\bx|| = ||\by||$ is required to prove the result.
\end{proof}
The Householder reflectors are useful to set a block of components of a given vector to zero. Particularly, we usually would like to set the vector $\ba\in \real^n$ to be zero except the $i$-th element. Then the Householder vector can be chosen to be
$$
\bu = \frac{\ba - r\be_i}{||\ba - r\be_i||}, \qquad \text{where } r = \pm||\ba||
$$
which is a reasonable Householder vector since $||\ba|| = ||r\be_i|| = |r|$. We carefully notice that when $r=||\ba||$, $\ba$ is reflected to $||\ba||\be_i$ via the Householder reflector $\bH = \bI - 2 \bu\bu^\top$; otherwise when $r=-||\ba||$, $||\ba||$ is reflected to $-||\ba||\be_i$ via the Householder reflector.

Recall that in Section~\ref{section:qr-gram-compute} (p.~\pageref{section:qr-gram-compute}), we claimed the Householder or Givens method is to employ a set of orthogonal matrices to triangularize the matrix such that the QR decomposition is obtained and the orthogonal matrix is ``more" orthogonal in this sense. The Householder reflector is such orthogonal matrix for this purpose. We not provide some more properties of the Householder reflector in the following remark.
\begin{remark}[Householder Properties]\label{remark:householder-propes}
If $\bH$ is a Householder reflector, then it has the following properties:

$\bullet$ $\bH\bH = \bI$;

$\bullet$ $\bH = \bH^\top$;

$\bullet$ $\bH^\top\bH = \bH\bH^\top = \bI$ such that Householder reflector is an orthogonal matrix;

$\bullet$ $\bH\bu = -\bu$, if $\bH = \bI - 2 \bu\bu^\top$.
\end{remark}

\subsubsection*{\textbf{Orthogonal Triangularization}}
To repeat, we see in the Gram-Schmidt section that QR decomposition is to use a triangular matrix to orthogonalize a matrix $\bA$. The further idea is that, if we have a set of orthogonal matrices that can make $\bA$ to be triangular step by step, then we can also recover the QR decomposition.
Specifically, if we have an orthogonal matrix $\bQ_1$ that can introduce zeros to the $1$-st column of $\bA$ except the entry (1,1); and an orthogonal matrix $\bQ_2$ that can introduce zeros to the $2$-nd column except the entries (2,1), (2,2); $\ldots$. Then, we can also find the QR decomposition.
For the way to introduce zeros, we could reflect the columns of the matrix to a basis vector $\be_1$ whose entries are all zero except the first entry.

Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n]\in \real^{m\times n}$ be the column partition of $\bA$, and let further
\begin{equation}\label{equation:qr-householder-to-chooose-r-numeraically}
r_1 = ||\ba_1||,\qquad \bu_1 = \frac{\ba_1 - r_1 \be_1}{||\ba_1 - r_1 \be_1||}, \qquad  \text{and}\qquad \bH_1 = \bI - 2\bu_1\bu_1^\top,
\end{equation}
where $\be_1$ here is the first basis for $\real^m$, i.e., $\be_1 = [1;0;0;\ldots;0]\in \real^m$.
Then
\begin{equation}\label{equation:householder-qr-projection-step1}
\bH_1\bA = [\bH_1\ba_1, \bH_1\ba_2, \ldots, \bH_1\ba_n] =
\begin{bmatrix}
r_1 & \bR_{1,2:n} \\
\bzero&  \bB_2
\end{bmatrix},
\end{equation}
which reflects $\ba_1$ to $r_1\be_1$ and introduces zeros below the diagonal in the $1$-st column. We observe that the entries below $r_1$ are all zero now under this specific reflection. Notice that we reflect $\ba_1$ to $||\ba_1||\be_1$ the two of which have same length, rather than reflect $\ba_1$ to $\be_1$ directly. This is for the purpose of \textbf{numerical stability} and matches the requirement in Corollary~\ref{corollary:householder-reflect-finding}.

\textbf{Choice of $r_1$:} moreover, the choice of $r_1$ is \textbf{not unique}. For \textbf{numerical stability}, it is also desirable to choose $r_1 =-\text{sign}(a_{11}) ||\ba_1||$, where $a_{11}$ is the first component of $\ba_{1}$. Or even, $r_1 =\text{sign}(a_{11}) ||\ba_1||$ is also possible as long as $||\ba_1||$ is equal to $||r_1\be_1||$. However, we will not cover this topic here.

We can then apply this process to $\bB_2$ in Equation~\eqref{equation:householder-qr-projection-step1} to make the entries  below the entry (2,2) to be all zeros. Note that, we do not apply this process to the entire $\bH_1\bA$ but rather the submatrix $\bB_2$ in it because we have already introduced zeros in the first column, and reflecting again will introduce nonzero values back and destroy what have accomplished.

Suppose $\bB_2 = [\bb_2, \bb_3, \ldots, \bb_n]$ is the column partition of $\bB_2$, and let
$$
r_2 = ||\bb_2||,\qquad \bu_2 = \frac{\bb_2 - r_2 \be_1}{||\bb_2 - r_2 \be_1||}, \qquad  \qquad \widetilde{\bH}_2 = \bI - 2\bu_2\bu_2^\top, \qquad \text{and}\qquad  \bH_2 =
\begin{bmatrix}
1 & \bzero \\
\bzero & \widetilde{\bH}_2
\end{bmatrix},
$$
where now $\be_1$ here is the first basis for $\real^{m-1}$ and $\bH_2$ is also an orthogonal matrix since $\widetilde{\bH}_2$ is an orthogonal matrix. Then it follows that
$$
\widetilde{\bH_2}\bB_2 = [\bH_2\bb_2, \bH_2\bb_3, \ldots, \bH_2\bb_n]=
\begin{bmatrix}
r_2 & \bR_{2,3:n} \\
\bzero &\bC_3
\end{bmatrix},
$$
and
$$
\bH_2\bH_1\bA = [\bH_2\bH_1\ba_1, \bH_2\bH_1\ba_2, \ldots, \bH_2\bH_1\ba_n] =
\begin{bmatrix}
r_1 & \bR_{12} & \bR_{1,3:n} \\
0 & r_2 & \bR_{2,3:n} \\
\bzero &  \bzero &\bC_3
\end{bmatrix}.
$$

Same process can go on, and if $\bA\in \real^{m\times n}$, after $n$ stages we will finally triangularize $\bA = (\bH_n \bH_{n-1}\ldots\bH_1)^{-1} \bR = \bQ\bR$. Since the $\bH_i$'s are symmetric and orthogonal (Remark~\ref{remark:householder-propes}), we have orthogonal $\bQ=(\bH_n \bH_{n-1}\ldots\bH_1)^{-1} = \bH_1 \bH_2\ldots\bH_n$.

An example of a $5\times 4$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\bH_1}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bH_2}{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
&\stackrel{\bH_3}{\rightarrow}
\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes \\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} \\
0 & 0 & \bm{0} & \bm{\boxtimes} \\
0 & 0 & \bm{0} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bH_4}{\rightarrow}
\begin{sbmatrix}{\bH_4\bH_3\bH_2\bH_1\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes \\
0 & 0 & \boxtimes & \boxtimes \\
0 & 0 & 0 & \bm{\boxtimes} \\
0 & 0 & 0 & \bm{0}
\end{sbmatrix}
\end{aligned}
$$

\paragraph{A closer look at the QR factorization} The Householder algorithm is a process that makes a matrix triangular by a sequence of orthogonal matrix operations. In the Gram-Schmidt process (both CGS and MGS), we use a triangular matrix to orthogonalize the matrix. However, in the Householder algorithm, we use orthogonal matrices to triangularize. The difference between the two approaches is then summarized as follows:
\begin{itemize}
\item Gram-Schmidt: triangular orthogonalization;
\item Householder: orthogonal triangularization.
\end{itemize}

We further notice that, in the Householder algorithm or the Givens algorithm that we will shortly see, a set of orthogonal matrices are applied so that the QR decomposition obtained is a \textit{full} QR decomposition. Whereas, the direct QR decomposition obtained by CGS or MGS is a \textit{reduced} one (although the silent columns or rows can be further added to find the full one).

\section{Computing the Full QR Decomposition via the Householder Reflector}
Since $\bA$ has $n$ columns, and for every step $i\in \{1, 2, \ldots, n\}$ to introduce zeros in the $i$-th column below the diagonal, we operate on a submatrix of size $(m-i+1)\times(n-i+1)$.
To compute the upper triangular matrix $\bR = \bH_n\bH_{n-1}\ldots\bH_1 \bA$, we notice that
$$
\begin{aligned}
\bR
&= (\bH_n\ldots(\bH_3(\bH_2(\bH_1 \bA))))\\
&=
\begin{bmatrix}
\bI_{n-1}& \bzero \\
\bzero& \bI - 2\bu_n \bu_n^\top
\end{bmatrix}
\ldots
\begin{bmatrix}
\bI_2& \bzero \\
\bzero& \bI - 2\bu_3\bu_3^\top
\end{bmatrix}
\begin{bmatrix}
\bI_1& \bzero \\
\bzero& \bI - 2\bu_2\bu_2^\top
\end{bmatrix}
\begin{bmatrix}
\bI - 2\bu_1\bu_1^\top
\end{bmatrix}\bA,
\end{aligned}
$$
where the parentheses indicate the order of the computation, the upper-left of $\bH_2$ is a $1\times 1$ identity matrix, and it will not change the \textbf{first row} and \textbf{first column} of $\bH_1\bA$ from the ``triangular property"; and the upper-left of $\bH_3$ is a $2\times 2$ identity matrix which will not change the \textbf{first 2 rows} and \textbf{first 2 columns} of $\bH_2\bH_2\bA$; $\ldots$. This property yields the step 8 in Algorithm~\ref{alg:qr-decomposition-householder} which operates only on the $i:m$ rows and $i+1:n$ column of $\bR$ in step $i$ (since the $i$-th column takes 1 flop explicitly in step 7 though this hardly reduces the complexity).
After the Householder transformation, we output the final triangular matrix $\bR$, and the process is shown in Algorithm~\ref{alg:qr-decomposition-householder}.

Furthermore, in Algorithm~\ref{alg:qr-decomposition-householder}, to get the final orthogonal matrix $\bQ=\bH_1 \bH_2\ldots\bH_n$, we notice that
$$
\begin{aligned}
\bQ
&=(((\bH_1 \bH_2)\bH_3)\ldots\bH_n) \\
&= \begin{bmatrix}
	\bI - 2\bu_1\bu_1^\top
\end{bmatrix}
\begin{bmatrix}
	\bI_1& \bzero \\
	\bzero& \bI - 2\bu_2\bu_2^\top
\end{bmatrix}
\begin{bmatrix}
	\bI_2& \bzero \\
	\bzero& \bI - 2\bu_3\bu_3^\top
\end{bmatrix}
\ldots
\begin{bmatrix}
	\bI_{n-1}& \bzero \\
	\bzero& \bI - 2\bu_n\bu_n^\top
\end{bmatrix},
\end{aligned}
$$
where the parentheses indicate the order of the computation, the upper-left of $\bH_2$ is a $1\times 1$ identity matrix, and it will not change the \textbf{first column} of $\bH_1$; and the upper-left of $\bH_3$ is a $2\times 2$ identity matrix which will not change the \textbf{first 2 columns} of $\bH_1\bH_2$; $\ldots$. This property yields the step 14 in the algorithm.
\begin{algorithm}[H]
\caption{Full QR Decomposition via the Householder Reflector}
\label{alg:qr-decomposition-householder}
\begin{algorithmic}[1]
	\Require matrix $\bA$ with size $m\times n $ and $m\geq n$;
	\State Initially set $\bR = \bA$;
	\For{$i=1$ to $n$}
	\State $\ba = \bR_{i:m,i}$, i.e., first column of $\bR_{i:m,i:n} \in \real^{(m-i+1)\times(n-i+1)}$;
	\State $r = ||\ba||$; \Comment{$2(m-i+1)$ flops;}
	\State $\bu_i = \ba-r\be_1 \in \real^{m-i+1}$; \Comment{ $1$ flop;}
	\State $\bu_i = \bu_i / ||\bu_i||$ ; \Comment{$3(m-i+1)$ flops;}
	\State $\bR_{i,i} = r$, $\bR_{i+1:m,i}=\bzero$; \Comment{0 flops, update first column of $\bR_{i:m,i:n}$}
	\State $\bR_{i:m,i+1:n} = \bR_{i:m,i+1:n} - 2\bu_i (\bu_i^\top \bR_{i:m,i+1:n})$; \Comment{update $i+1:n$ columns of $\bR_{i:m,i:n}$}
	\EndFor
	\State Output $\bR$ as the triangular matrix;

	\State Get $\bQ=\bH_1 \bH_2\ldots\bH_n$;
	\State Initially set $\bQ = \bH_1$;
	\For{$i=1$ to $n-1$}
	\State $\bQ_{1:m,i+1:m} = \bQ_{1:m,i+1:m}(\bI - 2\bu_{i+1}\bu_{i+1}^\top)=\bQ_{1:m,i+1:m}-\bQ_{1:m,i+1:m}2\bu_{i+1}\bu_{i+1}^\top$;
	\EndFor
	\State Output $\bQ$ as the orthogonal matrix;
\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: QR via Householder]\label{theorem:qr-full-householder}
Algorithm~\ref{alg:qr-decomposition-householder} requires $\sim 2mn^2-\frac{2}{3}n^3$ flops to compute a full QR decomposition of an $m\times n$ matrix with linearly independent columns and $m\geq n$. Further, if $\bQ$ is needed explicitly, additional $\sim 4m^2n-2mn^2$ flops are required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-full-householder}]
For loop $i$, $\bA_{i:m,i:n}$ is of size $(m-i+1)\times(n-i+1)$. Thus $\ba_1$ is in $\real^{m-i+1}$.

In step 4, to compute $r = ||\ba||$ involves $m-i+1$ multiplications, $m-i$ additions, and 1 square root operation which is \fbox{$2(m-i+1)$} flops.

In step 5, $\bu_i = \ba-r\be_1$ involves 1 subtraction which is \fbox{$1$} flop as the special structure of $\be_1$;

In step 6, same as that in step 4, it requires $2(m-i+1)$ flops ($m-i+1$ multiplications, $m-i$ additions, and 1 square root) to compute the norm $||\bu_i||$ and $m-i+1$ additional divisions which is \fbox{$3(m-i+1)$} flops totally.

In step 8, suppose loop $i=1$, $\bu_1^\top \bR_{1:m,2:n}$ requires $n-1$ times ($m$ multiplications and $m-1$ additions) which is \fbox{$(n-1)(2m-1)$} flops. $2\bu_1$ requires \fbox{$m$} multiplications. Further, $2\bu_1 (\bu_1^\top \bR_{1:m,2:n})$ requires \fbox{$m(n-1)$} multiplications to make an $m\times (n-1)$ matrix. The final matrix subtraction needs \fbox{$m(n-1)$} subtractions. Thus the total complexity for step 8 if loop $i=1$ is \fbox{$4m(n-1)+m-(n-1)$} flops. This analysis can be applied to any loop $i$, and the complexity of step 8 for loop $i$ can be obtained by \fbox{$4(m-i+1)(n-i)+m-n+i$} flops.

So for loop $i$, the total complexity from step 3 to step 8 can be defined as $f(i)$ flops.
To compute $\bR$, the final complexity is
$$
\mathrm{cost}=f(1)+f(2)+\ldots +f(n).
$$
Simple calculations will show the sum of $n$ loops is \fbox{$2mn^2-\frac{2}{3}n^3$} flops if we keep only the leading terms.

To get the final orthogonal matrix $\bQ$, since we have already computed $2\bu_{i+1}$ in step 8, this will not have additional costs. The computation of $\bQ_{1:m,i+1:m}2\bu_{i+1}$ involves $m$ times ($m-i$ multiplications and $m-i-1$ additions) which is \fbox{$m(2(m-i)-1)$} flops. Multiplied to $\bu_{i+1}^\top$ takes \fbox{$m(m-i)$} multiplications. Further, the final matrix subtraction requires \fbox{$m(m-i)$} subtractions. So in loop $i$, the complexity of step 14 is $g(i)=4m(m-i)-m = 4m^2-4mi-m$ flops. To compute $\bQ$, the final complexity is
$$
\mathrm{cost}=g(1)+g(2)+\ldots +g(n-1).
$$
Simple calculation will show the sum of $n-1$ loops is $4m^2n-2mn^2-4m^2+mn+m$ flops, or $4m^2n-2mn^2$ flops if we keep only the leading terms.
\end{proof}

After computing the full QR decomposition via the Householder algorithm, it is trivial to recover the reduced QR decomposition by just removing the silent columns in $\bQ$ and silent rows in $\bR$. However, there is no direct way to compute the reduced QR decomposition without the full decomposition form.

In \citep{golub2013matrix}, a Householder method for rank-revealing QR decomposition is discussed, the complexity is $4mnr-2r^2(m+n)+4r^3/3$ flops for rank $r$ matrix $\bA$. If $r=n$, the complexity agrees with the result in Theorem~\ref{theorem:qr-full-householder}.

\section{Existence of the QR Decomposition via the Givens Rotation}\label{section:qr-givens}
We have seen the Givens rotation can be utilized to find the rank-one update/downdate of the Cholesky decomposition in Section~\ref{section:cholesky-rank-one-update} (p.~\pageref{section:cholesky-rank-one-update}). Now let's take a look at what does the Givens rotation accomplish by specific examples. Consider the following $2\times 2$ orthogonal matrices
$$
\bF =
\begin{bmatrix}
-c & s\\
s & c
\end{bmatrix},
\qquad
\bJ=
\begin{bmatrix}
c & -s \\
s & c
\end{bmatrix},
\qquad
\bG=
\begin{bmatrix}
c & s \\
-s & c
\end{bmatrix},
$$
where $s = \sin \theta$ and $c=\cos \theta$ for some $\theta$. The first matrix has $\det(\bF)=-1$ and is a special case of a Householder reflector in dimension 2 such that $\bF=\bI-2\bu\bu^\top$ where $\bu=\begin{bmatrix}
\sqrt{\frac{1+c}{2}}, &\sqrt{\frac{1-c}{2}}
\end{bmatrix}^\top$ or $\bu=\begin{bmatrix}
-\sqrt{\frac{1+c}{2}}, &-\sqrt{\frac{1-c}{2}}
\end{bmatrix}^\top$. The latter two matrices have $\det(\bJ)=\det(\bG)=1$ and effects a rotation instead of a reflection. Such a matrix is called a \textbf{\textit{Givens rotation}}.

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[$\by = \bJ\bx$, counter-clockwise rotation.]{\label{fig:rotation1}
	\includegraphics[width=0.47\linewidth]{imgs/rotation.pdf}}
\quad
\subfigure[$\by = \bG\bx$, clockwise rotation.]{\label{fig:rotation2}
	\includegraphics[width=0.47\linewidth]{imgs/rotation2.pdf}}
\caption{Demonstration of two Givens rotations.}
\label{fig:rotation}
\end{figure}

Figure~\ref{fig:rotation} demonstrate a rotation of $\bx$ under $\bJ$, where $\by = \bJ\bx$ such that
$$
\left\{
\begin{aligned}
&y_1 = c\cdot x_1 - s\cdot x_2,   \\
&y_2 = s \cdot x_1 + c\cdot x_2.
\end{aligned}
\right.
$$
We want to verify the angle between $\bx$ and $\by $ is actually $\theta$ (and counter-clockwise rotation) after the Givens rotation $\bJ$ as shown in Figure~\ref{fig:rotation1}. Firstly, we have
$$
\left\{
\begin{aligned}
&\cos(\alpha) =\frac{x_1}{\sqrt{x_1^2+x_2^2}},   \\
&\sin (\alpha) =\frac{x_2}{\sqrt{x_1^2+x_2^2}}.
\end{aligned}
\right.
\qquad
\text{and }\qquad
\left\{
\begin{aligned}
&\cos(\theta) =c,   \\
&\sin (\theta) =s.
\end{aligned}
\right.
$$
This implies $\cos(\theta+\alpha) = \cos(\theta)\cos(\alpha)-\sin(\theta)\sin(\alpha)$.
If we can show $\cos(\theta+\alpha) = \cos(\theta)\cos(\alpha)-\sin(\theta)\sin(\alpha)$ is equal to $\frac{y_1}{\sqrt{y_1^2+y_2^2}}$, then we complete the proof.

For the former one, $\cos(\theta+\alpha) = \cos(\theta)\cos(\alpha)-\sin(\theta)\sin(\alpha)=\frac{c\cdot x_1 - s\cdot x_2}{\sqrt{x_1^2+x_2^2}}$. For the latter one, it can be verified that $\sqrt{y_1^2+y_2^2}=\sqrt{x_1^2+x_2^2}$, and $\frac{y_1}{\sqrt{y_1^2+y_2^2}} = \frac{c\cdot x_1 - s\cdot x_2}{\sqrt{x_1^2+x_2^2}}$. This completes the proof. Similarly, we can also show that the angle between $\by=\bG\bx$ and $\bx$ is also $\theta$ in Figure~\ref{fig:rotation2} and the rotation is clockwise.

More generally, we define the $n$-th order Givens rotation as follows.

\begin{definition}[$n$-th Order Givens Rotation\index{Givens rotation}]\label{definition:givens-rotation-in-qr}
An $n\times n$ Givens rotation is represented by a matrix of the following form
$$
\begin{blockarray}{cccccccccccc}
\begin{block}{c[cccccccccc]c}
	&1 &          &   &  &   &   & &  & & &\\
	&& \ddots  &  &  &  & && & & &\\
	&&      & 1 &  & & &  && & & \\
	&&      &  & c &  &  &  & s & & &k\\
	&&& &   & 1 & & && & &\\
	\bG_{kl}=&&& &   &   &\ddots &  && & &\\
	&&& &  &   &  & 1&& & &\\
	&&& & -s &  &  & &c& & & l\\
	&&& & &  &  & & &1 & &\\
	&&& & &  &  & & & &\ddots &\\
\end{block}
&&      &  & k &  &  &  & l & & &\\
\end{blockarray},
$$
where the $(k,k), (k,l), (l,k), (l,l)$ entries are $c, s, -s, c$ respectively, and $s = \cos \theta$ and $c=\cos \theta$ for some $\theta$.

Let $\bdelta_k \in \real^n$ be the zero vector where the $k$-th entry is 1. Then mathematically, the Givens rotation defined above can be denoted by
$$
\bG_{kl}= \bI + (c-1)(\bdelta_k\bdelta_k^\top + \bdelta_l\bdelta_l^\top) + s(\bdelta_k\bdelta_l^\top -\bdelta_l\bdelta_k^\top ),
$$
where the subscripts $k,l$ indicate the \textbf{rotation is in plane $k$ and $l$}.

Specifically, one can also define the $n$-th order Givens rotation where $(k,k),$ $(k,l),$ $(l,k),$ $(l,l)$ entries are $c, \textcolor{blue}{-s, s}, c$ respectively (note the difference in the sign of $s$). The ideas are the same.
\end{definition}

It can be easily verified that the $n$-th order Givens rotation is an orthogonal matrix and its determinant is 1. For any vector $\bx =[x_1, x_2, \ldots, x_n]^\top \in \real^n$, we have $\by = \bG_{kl}\bx$, where
$$
\left\{
\begin{aligned}
&y_k = c \cdot x_k + s\cdot x_l,   \\
&y_l = -s\cdot x_k +c\cdot x_l,  \\
&y_j = x_j , &  (j\neq k,l)
\end{aligned}
\right.
$$
That is, a Givens rotation applied to $\bx$ rotates two components of $\bx$ by some angle $\theta$ and leaves all other components the same.
When $\sqrt{x_k^2 + x_l^2} \neq 0$,
let $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$. Then,
$$
\left\{
\begin{aligned}
&y_k = \sqrt{x_k^2 + x_l^2},   \\
&y_l = 0,  \\
&y_j = x_j . &  (j\neq k,l)
\end{aligned}
\right.
$$
This finding above is essential for the QR decomposition via the Givens rotation.

\begin{corollary}[Basis From Givens Rotations Forwards]\label{corollary:basis-from-givens}
For any vector $\bx \in \real^n$, there exists a set of Givens rotations $\{\bG_{12}, \bG_{13}, \ldots, \bG_{1n}\}$ such that $\bG_{1n}\ldots \bG_{13}\bG_{12}\bx = ||\bx||\be_1$ where $\be_1\in \real^n$ is the first unit basis in $\real^n$.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:basis-from-givens}]
From the finding above, we can find a $\bG_{12}, \bG_{13}, \bG_{14}$ such that
$$
\bG_{12}\bx = \left[\sqrt{x_1^2 + x_2^2}, 0, x_3, \ldots, x_n \right]^\top,
$$
$$
\bG_{13}\bG_{12}\bx = \left[\sqrt{x_1^2 + x_2^2+x_3^2}, 0, 0, x_4, \ldots, x_n \right]^\top,
$$
and
$$
\bG_{14}\bG_{13}\bG_{12}\bx = \left[\sqrt{x_1^2 + x_2^2+x_3^2+x_4^2},0, 0, 0, x_5, \ldots, x_n \right]^\top.
$$
Continue this process, we will obtain $\bG_{1n}\ldots \bG_{13}\bG_{12} = ||\bx||\be_1$.
\end{proof}

\begin{remark}[Basis From Givens Rotations Backwards]\label{remark:basis-from-givens2}
In Corollary~\ref{corollary:basis-from-givens}, we find the Givens rotation that introduces zeros from the $2$-nd entry to th $n$-th entry (i.e., forward). Sometimes we want the reverse order, i.e., introduce zeros from the $n$-th entry to the $2$-nd entry such that $\bG_{12}\bG_{13}\ldots  \bG_{1n}\bx = ||\bx||\be_1$ where $\be_1\in \real^n$ is the first unit basis in $\real^n$.

The procedure is similar, we can find a $\bG_{1n},\bG_{1,(n-1)}, \bG_{1,(n-2)}$ such that
$$
\bG_{1n}\bx = \left[\sqrt{x_1^2 + x_n^2}, x_2, x_3, \ldots, x_{n-1}, 0 \right]^\top,
$$
$$
\bG_{1,(n-1)}\bG_{1n}\bx = \left[\sqrt{x_1^2 + x_{n-1}^2+x_n^2}, x_2, x_3, \ldots, x_{n-2}, 0, 0  \right]^\top,
$$
and
$$
\bG_{1,(n-2)}\bG_{1,(n-1)}\bG_{1n}\bx = \left[\sqrt{x_1^2 + x_{n-2}^2+x_{n-1}^2+x_n^2}, x_2, x_3,  \ldots,x_{n-3},0, 0, 0  \right]^\top.
$$
Continue this process, we will obtain $\bG_{12}\bG_{13}\ldots  \bG_{1n}\bx = ||\bx||\be_1$.

\paragraph{An alternative form} Alternatively, there are rotations $\{\bG_{12}, \bG_{23}, \ldots,\bG_{(n-1),n}\}$ such that $\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}\bx = ||\bx||\be_1$ where $\be_1\in \real^n$ is the first unit basis in $\real^n$ and where
$$
\bG_{(n-1),n}\bx = \left[x_1, x_2, \ldots, x_{n-2},\sqrt{x_{n-1}^2 + x_n^2}, 0 \right]^\top,
$$
$$
\bG_{(n-2),(n-1)}\bG_{(n-1),n}\bx = \left[x_1, x_2, \ldots,x_{n-3}, \sqrt{x_{n-2}^2+x_{n-1}^2 + x_n^2}, 0, 0  \right]^\top,
$$
and
$$
\begin{aligned}
\bG_{(n-3),(n-2)}\bG_{(n-2),(n-1)}\bG_{(n-1),n}\bx &= \\
&\,\left[x_1,  x_2, \ldots ,x_{n-4}, \sqrt{x_{n-3}^2+x_{n-2}^2+x_{n-1}^2 + x_n^2},0, 0, 0  \right]^\top.
\end{aligned}
$$
Continue this process, we will obtain $\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}\bx = ||\bx||\be_1$.

The above backward Givens rotation basis update will be proved useful in the rank-one changes of the QR decomposition (Section~\ref{section:qr-rank-one-changes}, p.~\pageref{section:qr-rank-one-changes}).
\end{remark}

From the Corollary~\ref{corollary:basis-from-givens} above, for the way to introduce zeros, we could \textbf{rotate} the columns of the matrix to a basis vector $\be_1$ whose entries are all zero except the first entry.
Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n] \in \real^{m\times n}$ be the column partition of $\bA$, and let
\begin{equation}\label{equation:qr-rotate-to-chooose-r-numeraically}
\bG_1 = \bG_{1m}\ldots \bG_{13}\bG_{12},
\end{equation}
where $\be_1$ here is the first basis for $\real^m$, i.e., $\be_1 = [1;0;0;\ldots;0]\in \real^m$.
Then
\begin{equation}\label{equation:rotate-qr-projection-step1}
\begin{aligned}
\bG_1\bA &= [\bG_1\ba_1, \bG_1\ba_2, \ldots, \bG_1\ba_n] \\
&=
\begin{bmatrix}
||\ba_1|| & \bR_{1,2:n} \\
\bzero&  \bB_2
\end{bmatrix},
\end{aligned}
\end{equation}
which rotates $\ba_1$ to $||\ba_1||\be_1$ and introduces zeros below the diagonal in the $1$-st column of $\bA$. And bear in mind that the $\bG_1$ above will sequentially change the ($1$-st, $2$-nd), ($1$-st, $3$-rd), ($1$-st, $4$-th), \ldots, ($1$-st, $m$-th) elements in pair for any vector $\bv\in \real^m$.

We can then apply this process to $\bB_2$ in Equation~\eqref{equation:rotate-qr-projection-step1} to make the entries  below the (2,2)-th entry to be all zeros.
Suppose $\bB_2 = [\bb_2, \bb_3, \ldots, \bb_n]$ is the column partition of $\bB_2$, and let
$$
\bG_2 = \bG_{2m}\ldots\bG_{24}\bG_{23},
$$
where $\bG_{2n}, \ldots, \bG_{24}, \bG_{23}$ can be implied from context.
Then
$$
\begin{aligned}
\bG_2
\begin{bmatrix}
\bR_{1,2:n} \\
\bB_2
\end{bmatrix}
&=
\begin{bmatrix}
\bR_{12} & \bR_{1,3:n} \\
||\bb_2|| & \bR_{2,3:n} \\
\bzero &\bC_3
\end{bmatrix},
\end{aligned}
$$
and
$$
\begin{aligned}
\bG_2\bG_1\bA &= [\bG_2\bG_1\ba_1, \bG_2\bG_1\ba_2, \ldots, \bG_2\bG_1\ba_n]=
\begin{bmatrix}
||\ba_1|| & \bR_{12} & \bR_{1,3:n} \\
0 & ||\bb_2|| & \bR_{2,3:n} \\
\bzero &  \bzero &\bC_3
\end{bmatrix}.
\end{aligned}
$$

Same process can go on, and we will finally triangularize $\bA = (\bG_n \bG_{n-1}\ldots\bG_1)^{-1} \bR = \bQ\bR$. And since $\bG_i$'s are orthogonal, the orthogonal $\bQ$ can also be obtained by $\bQ=(\bG_n \bG_{n-1}\ldots\bG_1)^{-1} = \bG_1^\top \bG_2^\top\ldots\bG_n^\top $, and
\begin{equation}\label{equation:givens-q}
\begin{aligned}
\bG_1^\top \bG_2^\top\ldots\bG_n^\top &=(\bG_n \ldots \bG_2 \bG_1)^\top \\
&=\left\{(\bG_{nm} \ldots \bG_{n,(n+1)}) \ldots (\bG_{2m}\ldots \bG_{23})  ( \bG_{1m} \ldots \bG_{12} )\right\}^\top .
\end{aligned}
\end{equation}

\paragraph{When will the Givens work better?} In practice, compared to the Householder algorithms, the Givens rotation algorithm works better when $\bA$ already has a lot of zeros below the main diagonal. Therefore, the Givens rotations can be applied to the rank-one changes of the QR decomposition as rank-one change will only introduce a small amount of nonzero values (Section~\ref{section:qr-rank-one-changes}, p.~\pageref{section:qr-rank-one-changes}).

An example of a $5\times 4$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\paragraph{Givens rotations in $\bG_1$} For a $5\times 4$ example, we realize that $\bG_1 = \bG_{15}\bG_{14}\bG_{13}\bG_{12}$. And the process is shown as follows:
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\bG_{12}}{\rightarrow}
\begin{sbmatrix}{\bG_{12}\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bG_{13}}{\rightarrow}
\begin{sbmatrix}{\bG_{13}\bG_{12}\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes &\boxtimes \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}\\
&\stackrel{\bG_{14}}{\rightarrow}
\begin{sbmatrix}{\bG_{14}\bG_{13}\bG_{12}\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
0 & \boxtimes & \boxtimes &\boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bG_{15}}{\rightarrow}
\begin{sbmatrix}{\bG_{15}\bG_{14}\bG_{13}\bG_{12}\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
0 & \boxtimes & \boxtimes &\boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\end{sbmatrix}
\end{aligned}
$$

\paragraph{Givens rotation as a big picture} Take $\bG_1, \bG_2, \bG_3, \bG_4$ as a single matrix, we have
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\bG_1}{\rightarrow}
\begin{sbmatrix}{\bG_1\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bG_2}{\rightarrow}
\begin{sbmatrix}{\bG_2\bG_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
&\stackrel{\bG_3}{\rightarrow}
\begin{sbmatrix}{\bG_3\bG_2\bG_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & 0 & \bm{0} & \bm{\boxtimes} \\
	0 & 0 & \bm{0} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bG_4}{\rightarrow}
\begin{sbmatrix}{\bG_4\bG_3\bG_2\bG_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes & \boxtimes \\
	0 & 0 & 0 & \bm{\boxtimes} \\
	0 & 0 & 0 & \bm{0}
\end{sbmatrix}
\end{aligned}
$$

\paragraph{Orders to introduce the zeros} With the Givens rotations for the QR decomposition, it is flexible to choose different orders to introduce the zeros of $\bR$. In our case, we introduce zeros column by column. It is also possible to introduce zeros row by row.

\section{Computing the Full QR Decomposition via the Givens Rotation}
The algorithm to compute the full QR decomposition via the Givens rotation is straightforward from the example shown above and is illustrated in Algorithm~\ref{alg:qr-decomposition-givens}.
\begin{algorithm}[H]
\caption{Full QR Decomposition via the Givens Rotation}
\label{alg:qr-decomposition-givens}
\begin{algorithmic}[1]
\Require matrix $\bA$ with size $m\times n $ and $m\geq n$;
\State Initially set $\bR = \bA$, $\bQ=\bI$;
\For{$i=1$ to $n$}
\For{$j=i+1$ to $m$}
\State Get Givens rotation $\bG_{i,j}$ with the following parameters $c, s$:
\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k = \bR_{i,i}$, $x_l = \bR_{j,i}$;
\State Calculate $\bR = \bG_{i,j}\bR $ in following two steps:
\State $i$-th row: $\bR_{i,:} = c\cdot \bR_{i,:} + s \bR_{j,:} $;
\State $j$-th row: $\bR_{j,:} = -s\cdot \bR_{i,:} + c \bR_{j,:} $;
\EndFor
\EndFor
\State Output $\bR$ as the triangular matrix;

\For{$i=1$ to $n$}
\For{$j=i+1$ to $m$}
\State Calculate $\bQ = \bG_{i,j}\bQ $ in following two steps:
\State $i$-th row: $\bQ_{i,:} = c\cdot \bQ_{i,:} + s \bQ_{j,:} $;
\State $j$-th row: $\bQ_{j,:} = -s\cdot \bQ_{i,:} + c \bQ_{j,:} $;
\EndFor
\EndFor
\State Output $\bQ=\bQ^\top$ from Equation~\eqref{equation:givens-q};
\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: QR via Givens]\label{theorem:qr-full-givens}
	Algorithm~\ref{alg:qr-decomposition-givens}  requires $\sim 3mn^2-n^3$ flops to compute a full QR decomposition of an $m\times n$ matrix with linearly independent columns and $m\geq n$. Further, if $\bQ$ is needed explicitly, additional $\sim 3mn^2-n^3$ flops required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-full-givens}]
For step 5, each iteration $i,j$ requires $6$ flops (which are 2 square operations, 1 addition, 1 square root, and 2 divisions). And there are $m-i$ iterations for each $i$ which means $(m-1)+(m-2)+\ldots+(m-n)=(mn-\frac{n^2+n}{2})$ iterations. Therefore, the complexity for all the step 5's is \fbox{$6(mn-\frac{n^2+n}{2})$} flops.

For each iteration $i$, step 7 and step 8 operate on two length-$(n-i+1)$ vectors. The two steps take $6(n-i+1)$ flops for each iteration $i$ (which are $4(n-i+1)$ multiplications and $2(n-i+1)$ additions). And for each $i$, there are $(m-i)$ such operations which takes $(m-i)\times 6(n-i+1)$ flops for each $i$. Let $f(i) = (m-i)\times 6(n-i+1) = m(n+1)-(m+n+1)i +i^2$. The total complexity for the two steps is equal to
$$
\mathrm{cost} =f(1)+ f(2) +\ldots +f(n),
$$
or \fbox{$3mn^2-n^3$} flops if keep only the leading terms.

Similarly, we can get the complexity of step 15 and 16 with \fbox{$3mn^2-n^3$} flops if keep only the leading terms.
\end{proof}

Same as the Householder algorithm, after computing the full QR decomposition via the Givens algorithm, it is trivial to recover the reduced QR decomposition by just removing the silent columns in $\bQ$ and silent rows in $\bR$.

\section{Uniqueness of the QR Decomposition}\label{section:nonunique-qr}
The results of the QR decomposition from the Gram-Schmidt process , the Householder algorithm, and the Givens algorithms are different. Even in the Householder algorithm, we have different methods to choose the sign of $r_1$ in Equation~\eqref{equation:qr-householder-to-chooose-r-numeraically}. Thus, from this point, QR decomposition is not unique.

\begin{example}[Non-Uniqueness of the QR Decomposition\index{Uniqueness}]
Suppose the matrix $\bA$ is given by
$$
\bA =
\begin{bmatrix}
4 & 1 \\
3 & 2
\end{bmatrix}.
$$
The QR decomposition of $\bA$ can be obtained by
$$
\begin{aligned}
\bA &= \bQ_1\bR_1=
\begin{bmatrix}
0.8 & 0.6 \\
0.6 & -0.8
\end{bmatrix}
\begin{bmatrix}
5 & 2 \\
0 & -1
\end{bmatrix}\\
&= \bQ_2\bR_2=
\begin{bmatrix}
0.8 & -0.6 \\
0.6 & 0.8
\end{bmatrix}
\begin{bmatrix}
5 & 2 \\
0 & 1
\end{bmatrix}\\
&=\bQ_3\bR_3=
\begin{bmatrix}
-0.8 & -0.6 \\
-0.6 & 0.8
\end{bmatrix}
\begin{bmatrix}
-5 & -2 \\
0 & 1
\end{bmatrix}\\
&= \bQ_4\bR_4=
\begin{bmatrix}
-0.8 & 0.6 \\
-0.6 & -0.8
\end{bmatrix}
\begin{bmatrix}
-5 & -2 \\
0 & -1
\end{bmatrix}.
\end{aligned}
$$
Thus the QR decomposition of $\bA$ is not unique.
\exampbar
\end{example}

However, if we use just the procedure described in the Gram-Schmidt process, or systematically choose the sign in the Householder algorithm, then the decomposition is unique.
The uniqueness of the \textit{reduced} QR decomposition for full column rank matrix $\bA$ is assured when $\bR$ has positive diagonals as shown in the ``main proof" of Section~\ref{section:gram-schmidt-process} by inductive analysis.
We here provide another proof for the uniqueness of the \textit{reduced} QR decomposition for matrices if the diagonal values of $\bR$ are positive which will shed light on the implicit Q theorem in Hessenberg decomposition (Section~\ref{section:hessenberg-decomposition}, p.~\pageref{section:hessenberg-decomposition}) or tridiagonal decomposition (Theorem~\ref{section:tridiagonal-decomposition}, p.~\pageref{section:tridiagonal-decomposition}).
\begin{corollary}[Uniqueness of the reduced QR Decomposition]\label{corollary:unique-qr}
	Suppose matrix $\bA$ is an $m\times n$ matrix with full column rank $n$ and $m\geq n$. Then, the \textit{reduced} QR decomposition is unique if the main diagonal values of $\bR$ are positive.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:unique-qr}]
Suppose the \textit{reduced} QR decomposition is not unique, we can complete it into a \textit{full} QR decomposition, then we can find two such full decompositions so that $\bA=\bQ_1\bR_1 = \bQ_2\bR_2$ which implies $\bR_1 = \bQ_1^{-1}\bQ_2\bR_2 = \bV \bR_2$ where $\bV= \bQ_1^{-1}\bQ_2$ is an orthogonal matrix. Write out the equation, we have
$$
\begin{aligned}
	\bR_1 &=
	\begin{bmatrix}
		r_{11} & r_{12}& \dots & r_{1n}\\
		& r_{22}& \dots & r_{2n}\\
		&       &    \ddots  & \vdots \\
		\multicolumn{2}{c}{\raisebox{1.3ex}[0pt]{\Huge0}} & & r_{nn} \\
		\bzero & \bzero &\ldots & \bzero
	\end{bmatrix}=
	\begin{bmatrix}
		v_{11}& v_{12} & \ldots & v_{1m}\\
		v_{21} & v_{22} & \ldots & v_{2m}\\
		\vdots & \vdots & \ddots & \vdots\\
		v_{m1} & v_{m2} & \ldots & v_{\textcolor{black}{mm}}
	\end{bmatrix}
	\begin{bmatrix}
		s_{11} & s_{12}& \dots & s_{1n}\\
		& s_{22}& \dots & s_{2n}\\
		&       &    \ddots  & \vdots \\
		\multicolumn{2}{c}{\raisebox{1.3ex}[0pt]{\Huge0}} & & s_{nn} \\
		\bzero & \bzero &\ldots & \bzero
	\end{bmatrix}= \bV\bR_2,
\end{aligned}
$$

This implies
$$
r_{11} = v_{11} s_{11}, \qquad v_{21}=v_{31}=v_{41}=\ldots=v_{m1}=0.
$$
Since $\bV$ contains mutually orthonormal columns and the first column of $\bV$ is of norm 1. Thus, $v_{11} = \pm 1$.
We notice that $r_{ii}> 0$ and $s_{ii}> 0$ for $i\in \{1,2,\ldots,n\}$ by assumption such that $r_{11}> 0$ and $s_{11}> 0$ and $v_{11}$ can only be positive 1. Since $\bV$ is an orthogonal matrix, we also have
$$
v_{12}=v_{13}=v_{14}=\ldots=v_{1m}=0.
$$
Applying this process to the submatrices of $\bR_1, \bV, \bR_2$, we will find the upper-left submatrix of $\bV$ is an identity: $\bV[1:n,1:n]=\bI_n$ such that $\bR_1=\bR_2$. This implies $\bQ_1[:,1:n]=\bQ_2[:,1:n]$ and leads to a contradiction such that the reduced QR decomposition is unique.
\end{proof}
We notice that the uniqueness of the reduced QR decomposition shown above is from the diagonals of $\bR$ are positive. Therefore, if we restrict the QR decomposition that the diagonal values of $\bR$ are positive in Householder or Givens algorithms, then the decomposition will be unique. And actually, the Gram-Schmidt process in Algorithm~\ref{alg:reduced-qr} is such a decomposition that the diagonal values of $\bR$ will be positive if $\bA$ has full column rank. If $\bA$ has dependent columns, the diagonal entries of $\bR$ can only be nonnegative, and the factorization may not be unique.

\section{LQ Decomposition}\label{section:lq-decomp}
We previously proved the existence of the QR decomposition via the Gram-Schmidt process in which case we are interested in the column space of a matrix $\bA=[\ba_1, \ba_2, ..., \ba_n] \in \real^{m\times n}$. The successive spaces spanned by the columns $\ba_1, \ba_2, \ldots$ of $\bA$ are
$$
\cspace([\ba_1])\,\,\,\, \subseteq\,\,\,\, \cspace([\ba_1, \ba_2]) \,\,\,\,\subseteq\,\,\,\, \cspace([\ba_1, \ba_2, \ba_3])\,\,\,\, \subseteq\,\,\,\, \ldots,
$$
The idea of QR decomposition is the construction of a sequence of orthonormal vectors $\bq_1, \bq_2, \ldots$ that span the same successive subspaces:
$$
\left\{\cspace([\bq_1])=\cspace([\ba_1]) \right\}\,\,\,\, \subseteq\,\,\,\, \{\cspace([\bq_1, \bq_2])=\cspace([\ba_1, \ba_2])\} \,\,\,\, \subseteq\,\,\,\, \ldots,
$$
However, in many applications (see \citep{schilders2009solution}), we are also interested in the row space of a matrix $\bB=[\bb_1^\top; \bb_2^\top; ...;\bb_m^\top] \in \real^{m\times n}$, where $\bb_i$ is the $i$-th row of $\bB$. The successive spaces spanned by the rows $\bb_1, \bb_2, \ldots$ of $\bB$ are
$$
\cspace([\bb_1])\,\,\,\, \subseteq\,\,\,\, \cspace([\bb_1, \bb_2]) \,\,\,\,\subseteq\,\,\,\, \cspace([\bb_1, \bb_2, \bb_3])\,\,\,\, \subseteq\,\,\,\, \ldots.
$$
The QR decomposition thus has its sibling which finds the orthogonal row space.
By applying QR decomposition on $\bB^\top = \bQ_0\bR$, we recover the LQ decomposition of the matrix $\bB = \bL \bQ$ where $\bQ = \bQ_0^\top$ and $\bL = \bR^\top$.

\begin{theoremHigh}[LQ Decomposition]\label{theorem:lq-decomposition}
	Every $m\times n$ matrix $\bB$ (whether linearly independent or dependent rows) with $n\geq m$ can be factored as
	$$
	\bB = \bL\bQ,
	$$
	where

	1. \textbf{Reduced}: $\bL$ is an $m\times m$ lower triangular matrix and $\bQ$ is $m\times n$ with orthonormal rows which is known as the \textbf{reduced LQ decomposition};

	2. \textbf{Full}: $\bL$ is an $m\times n$ lower triangular matrix and $\bQ$ is $n\times n$ with orthonormal rows which is known as the \textbf{full LQ decomposition}. If further restrict the lower triangular matrix to be a square matrix, the full LQ decomposition can be denoted as
	$$
	\bB = \begin{bmatrix}
		\bL_0 & \bzero
	\end{bmatrix}\bQ,
	$$
	where $\bL_0$ is an $m\times m$ square lower triangular matrix.
\end{theoremHigh}
Similarly, a comparison between the reduced and full LQ decomposition is shown in Figure~\ref{fig:lq-comparison}.

\begin{figure}[H]
\centering
\vspace{-0.35cm}
\subfigtopskip=2pt
\subfigbottomskip=2pt
\subfigcapskip=-5pt
\subfigure[Reduced LQ decomposition]{\label{fig:lqhalf}
	\includegraphics[width=0.47\linewidth]{./imgs/qrreduced-LR.pdf}}
\quad
\subfigure[Full LQ decomposition]{\label{fig:lqall}
	\includegraphics[width=0.47\linewidth]{./imgs/qrfull-LR.pdf}}
\caption{Comparison between the reduced and full LQ decomposition.}
\label{fig:lq-comparison}
\end{figure}

\paragraph{Row-pivoted LQ (RPLQ)\index{Row-pivoted}\index{RPLQ}} Similar to the column-pivoted QR in Section~\ref{section:cpqr}, there exists a row-pivoted LQ decomposition:
$$
\left\{
\begin{aligned}
\text{Reduced RPLQ: }&\qquad
\bP\bB &=&
\underbrace{\begin{bmatrix}
	\bL_{11} \\
	\bL_{21}
\end{bmatrix}}_{m\times r}
\underbrace{\bQ_r }_{r\times n};\\
\text{Full RPLQ: }&\qquad
\bP\bB &=&
\underbrace{\begin{bmatrix}
	\bL_{11} & \bzero \\
	\bL_{21} & \bzero
\end{bmatrix}}_{m\times m}
\underbrace{\bQ }_{m\times n},\\
\end{aligned}
\right.
$$
where $\bL_{11}\in \real^{r\times r}$ is lower triangular, $\bQ_r$ or $\bQ_{1:r,:}$ spans the same row space as $\bB$, and $\bP$ is a permutation matrix that interchange independent rows into the upper-most rows.

\section{Two-Sided Orthogonal Decomposition}
\begin{theoremHigh}[Two-Sided Orthogonal Decomposition]\label{theorem:two-sided-orthogonal}
When square matrix $\bA\in \real^{n\times n}$ with rank $r$, the full CPQR, RPLQ of $\bA$ are given by
$$\bA\bP_1=\bQ_1
\begin{bmatrix}
\bR_{11} & \bR_{12}\\
\bzero & \bzero
\end{bmatrix},\qquad
\bP_2\bA=
\begin{bmatrix}
\bL_{11} & \bzero \\
\bL_{21} & \bzero
\end{bmatrix}
\bQ_2
$$
respectively. Then we would find out
$$
\bA\bP\bA = \bQ_1
\underbrace{\begin{bmatrix}
	\bR_{11}\bL_{11}+\bR_{12}\bL_{21} & \bzero \\
	\bzero & \bzero
\end{bmatrix}}_{\text{rank $r$}}
\bQ_2,
$$
where the first $r$ columns of $\bQ_1$ span the same column space of $\bA$, first $r$ rows of $\bQ_2$ span the same row space of $\bA$, and $\bP$ is a permutation matrix. We name this decomposition as \textbf{two-sided orthogonal decomposition}.
\end{theoremHigh}

This decomposition is very similar to the property of SVD: $\bA=\bU\bSigma\bV^\top$ that the first $r$ columns of $\bU$ span the column space of $\bA$ and the first $r$ columns of $\bV$ span the row space of $\bA$ (we shall see in Lemma~\ref{lemma:svd-four-orthonormal-Basis}, p.~\pageref{lemma:svd-four-orthonormal-Basis}). Therefore, the two-sided orthogonal decomposition can be regarded as an inexpensive alternative in this sense.

\begin{lemma}[Four Orthonormal Basis]
Given the two-sided orthogonal decomposition of matrix $\bA\in \real^{n\times n}$ with rank $r$: $\bA\bP\bA = \bU \bF\bV^\top$, where $\bU=[\bu_1, \bu_2, \ldots,\bu_n]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU$ and $\bV$. Then, we have the following property:

$\bullet$ $\{\bv_1, \bv_2, \ldots, \bv_r\} $ is an orthonormal basis of $\cspace(\bA^\top)$;

$\bullet$ $\{\bv_{r+1},\bv_{r+2}, \ldots, \bv_n\}$ is an orthonormal basis of $\nspace(\bA)$;

$\bullet$ $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is an orthonormal basis of $\cspace(\bA)$;

$\bullet$ $\{\bu_{r+1}, \bu_{r+2},\ldots,\bu_n\}$ is an orthonormal basis of $\nspace(\bA^\top)$.
\end{lemma}
\section{Applications}
\subsection{Application: Least Squares via the Full QR Decomposition}\label{section:application-ls-qr}
Let's consider the overdetermined system $\bA\bx = \bb$, where $\bA\in \real^{m\times n}$ is the data matrix, $\bb\in \real^m$ with $m>n$ is the observation matrix. Normally $\bA$ will have full column rank since the data from real work has a large chance to be unrelated. And the least squares (LS) solution is given by $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ for minimizing $||\bA\bx-\bb||^2$, where $\bA^\top\bA$ is invertible since $\bA$ has full column rank and $rank(\bA^T\bA)=rank(\bA)$.\index{Least squares}

However, the inverse of a matrix is not easy to compute, we can then use QR decomposition to find the least squares solution as illustrated in the following theorem.
\begin{theorem}[LS via QR for Full Column Rank Matrix]\label{theorem:qr-for-ls}
Let $\bA\in \real^{m\times n}$ and $\bA=\bQ\bR$ is its full QR decomposition with $\bQ\in\real^{m\times m}$ being an orthogonal matrix, $\bR\in \real^{m\times n}$ being an upper triangular matrix appended by additional $m-n$ zero rows, and $\bA$ has full column rank with $m\geq n$. Suppose $\bR = \begin{bmatrix}
	\bR_1 \\
	\bzero
\end{bmatrix}$, where $\bR_1 \in \real^{n\times n}$ is the square upper triangular in $\bR$, $\bb\in \real^m$, then the LS solution to $\bA\bx=\bb$ is given by
$$
\bx_{LS} = \bR_1^{-1}\bc,
$$
where $\bc$ is the first $n$ components of $\bQ^\top\bb$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-for-ls}]
Since $\bA=\bQ\bR$ is the full QR decomposition of $\bA$ and $m\geq n$, the last $m-n$ rows of $\bR$ are zero as shown in Figure~\ref{fig:qr-comparison}. Then $\bR_1 \in \real^{n\times n}$ is the square upper triangular in $\bR$ and
$
\bQ^\top \bA = \bR = \begin{bmatrix}
	\bR_1 \\
	\bzero
\end{bmatrix}.
$
Thus,
$$
\begin{aligned}
	||\bA\bx-\bb||^2 &= (\bA\bx-\bb)^\top(\bA\bx-\bb)\\
	&=(\bA\bx-\bb)^\top\bQ\bQ^\top (\bA\bx-\bb) \qquad &(\text{Since $\bQ$ is an orthogonal matrix})\\
	&=||\bQ^\top \bA \bx-\bQ^\top\bb||^2 \qquad &(\text{Invariant under orthogonal})\\
	&=\left\Vert\begin{bmatrix}
		\bR_1 \\
		\bzero
	\end{bmatrix} \bx-\bQ^\top\bb\right\Vert^2\\
	&=||\bR_1\bx - \bc||^2+||\bd||^2,
\end{aligned}
$$
where $\bc$ is the first $n$ components of $\bQ^\top\bb$ and $\bd$ is the last $m-n$ components of $\bQ^\top\bb$. And the LS solution can be calculated by back substitution of the upper triangular system $\bR_1\bx = \bc$, i.e., $\bx_{LS} = \bR_1^{-1}\bc$.
\end{proof}

To verify Theorem~\ref{theorem:qr-for-ls}, for the full QR decomposition of $\bA = \bQ\bR$ where $\bQ\in \real^{m\times m}$ and $\bR\in \real^{m\times n}$. Together with the LS solution $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$, we obtain
\begin{equation}\label{equation:qr-for-ls-1}
\begin{aligned}
	\bx_{LS} &= (\bA^\top\bA)^{-1}\bA^\top\bb \\
	&= (\bR^\top\bQ^\top\bQ\bR)^{-1} \bR^\top\bQ^\top \bb\\
	&= (\bR^\top\bR)^{-1} \bR^\top\bQ^\top \bb \\
	&= (\bR_1^\top\bR_1)^{-1} \bR^\top\bQ^\top \bb \\
	&=\bR_1^{-1} \bR_1^{-\top} \bR^\top\bQ^\top \bb\\
	&= \bR_1^{-1} \bR_1^{-\top} \bR_1^\top\bQ_1^\top \bb \\
	&=\bR_1^{-1} \bQ_1^\top \bb,
\end{aligned}
\end{equation}
where $\bR = \begin{bmatrix}
\bR_1 \\
\bzero
\end{bmatrix}$ and $\bR_1\in \real^{n\times n}$  is an upper triangular matrix, and $\bQ_1 =\bQ_{1:m,1:n}\in \real^{m\times n}$ is the first $n$ columns of $\bQ$ (i.e., $\bQ_1\bR_1$ is the reduced QR decomposition of $\bA$). Then the result of Equation~\eqref{equation:qr-for-ls-1} agrees with Theorem~\ref{theorem:qr-for-ls}.

To conclude, using the QR decomposition, we first derive directly the least squares result which results in the argument in Theorem~\ref{theorem:qr-for-ls}. Moreover, we verify the result of LS from calculus indirectly by QR decomposition as well. The two results coincide with each other. For those who are interested in LS in linear algebra, a pictorial view of least squares for full column rank $\bA$ in the fundamental theorem of linear algebra is provided in Appendix~\ref{appendix:ls-fundation-theorem} or a detailed discussion in \citep{lu2021revisit}.

\subsection{Application: Rank-One Changes}\label{section:qr-rank-one-changes}
We previously discussed the rank-one update/downdate of the Cholesky decomposition in Section~\ref{section:cholesky-rank-one-update} (p.~\pageref{section:cholesky-rank-one-update}). The rank-one change $\bA^\prime$ of matrix $\bA$ in the QR decomposition is defined in a similar form:\index{Rank-one update}
$$
\begin{aligned}
	\bA^\prime &= \bA + \bu\bv^\top, \\
	\downarrow &\gap  \downarrow\\
	\bQ^\prime\bR^\prime &=\bQ\bR + \bu\bv^\top,
\end{aligned}
$$
where if we set $\bA^\prime = \bA - (-\bu)\bv^\top$, we recover the downdate form such that the update or downdate in the QR decomposition are the same. Let $\bw = \bQ^\top\bu$, we have
$$
\bA^\prime = \bQ(\bR + \bw\bv^\top).
$$
From the second form in Remark~\ref{remark:basis-from-givens2} (p.~\pageref{remark:basis-from-givens2}) on introducing zeros backwards, there exists a set of Givens rotations $\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}$ such that
$$
\bG_{12} \bG_{23} \ldots \bG_{(n-1),n} \bw = \pm ||\bw|| \be_1,
$$
where $\bG_{(k-1),k}$ is the Givens rotation in plane $k-1$ and $k$ that introduces zero in the $k$-th entry of $\bw$. Apply this rotation to $\bR$, we have
$$
\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}\bR = \bH_0 ,
$$
where the Givens rotations in this \textit{reverse order} are useful to transform the upper triangular $\bR$ into a ``simple" upper Hessenberg which is close to upper triangular matrices (see Definition~\ref{definition:upper-hessenbert}, p.~\pageref{definition:upper-hessenbert} that we will introduce in the Hessenberg decomposition). If the rotations are transforming $\bw$ into $\pm ||\bw||\be_1$ from \textit{forward order} as in Corollary~\ref{corollary:basis-from-givens} (p.~\pageref{corollary:basis-from-givens}), we will not have this upper Hessenberg $\bH_0$. To see this, suppose $\bR\in \real^{5\times 5}$, an example is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed. The backwards rotations result in the upper Hessenberg $\bH_0$ which is relatively simple to handle:
$$
\begin{aligned}
	\text{\parbox{7em}{Backwards\\(Right Way)}: }
	\begin{sbmatrix}{\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes \\
		0 & 0 & 0 & 0& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{45}}{\rightarrow}
	\begin{sbmatrix}{\bG_{45}\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & 0 & \bm{\boxtimes}& \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\bG_{34}}{\rightarrow}
	\begin{sbmatrix}{\bG_{34}\bG_{45}\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & 0 & \boxtimes & \boxtimes
	\end{sbmatrix}\\
	&\stackrel{\bG_{23}}{\rightarrow}
	\begin{sbmatrix}{\bG_{23}\bG_{34}\bG_{45}\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	\stackrel{\bG_{12}}{\rightarrow}
	\begin{sbmatrix}{\bG_{12}\bG_{23}\bG_{34}\bG_{45}\bR}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}.
\end{aligned}
$$
And the forward rotations result in a full matrix:
$$
\begin{aligned}
	\text{\parbox{7em}{Forwards\\(Wrong Way)}: }
	\begin{sbmatrix}{\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes \\
		0 & 0 & 0 & 0& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}}{\rightarrow}
	\begin{sbmatrix}{\bG_{12}\bR}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes \\
		0 & 0 & 0 & 0& \boxtimes
	\end{sbmatrix}
	\stackrel{\bG_{23}}{\rightarrow}
	\begin{sbmatrix}{\bG_{23}\bG_{12}\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & 0 & \boxtimes& \boxtimes \\
		0 & 0 & 0 & 0& \boxtimes
	\end{sbmatrix}\\
	&\stackrel{\bG_{34}}{\rightarrow}
	\begin{sbmatrix}{\bG_{34}\bG_{23}\bG_{12}\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & 0 & 0& \boxtimes
	\end{sbmatrix}
	\stackrel{\bG_{45}}{\rightarrow}
	\begin{sbmatrix}{\bG_{45}\bG_{34}\bG_{23}\bG_{12}\bR}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
	\end{sbmatrix}.
\end{aligned}
$$
I.e., the backward rotation will keep a lot of the zeros as they are, whereas the forward rotation will destroy these zeros.
Generally, the backward rotation results in,
$$
\bG_{12} \bG_{23} \ldots \bG_{(n-1),n} (\bR+\bw\bv^\top) = \bH_0  \pm ||\bw|| \be_1 \bv^\top = \bH,
$$
which is also upper Hessenberg. Similar to triangularization via the Givens rotation in Section~\ref{section:qr-givens} (p.~\pageref{section:qr-givens}), there exists a set of rotations $\bJ_{12}, \bJ_{23}, \ldots, \bJ_{(n-1),n}$ such that
$$
\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}\bH = \bR^\prime,
$$
is upper triangular. Following from the $5\times 5$ example, the triangularization is shown as follows
$$
\begin{aligned}
	\underbrace{\bH_0  \pm ||\bw|| \be_1 \bv^\top}_{\bH} =
	\begin{sbmatrix}{\bH}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bJ_{12}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{12}\bH}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	\stackrel{\bJ_{23}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{23}\bJ_{12}\bH}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}\\
	&\stackrel{\bJ_{34}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{34}\bJ_{23}\bJ_{12}\bH}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & 0 & 0 & \boxtimes& \boxtimes
	\end{sbmatrix}
	\stackrel{\bJ_{45}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{45}\bJ_{34}\bJ_{23}\bJ_{12}\bH}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & 0 & 0 & \bm{0} & \bm{\boxtimes} \\
	\end{sbmatrix}.
\end{aligned}
$$
And the QR decomposition of $\bA^\prime$ is thus given by
$$
\bA^\prime = \bQ^\prime \bR^\prime,
$$
where
\begin{equation}\label{equation:qr-rank-one-update}
\left\{
\begin{aligned}
\bR^\prime &=(\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}) (\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}) (\bR+\bw\bv^\top);\\
\bQ^\prime &= \bQ\left\{(\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}) (\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}) \right\}^\top; \\
\text{(or) }\bQ^{\prime\top}&= \left\{(\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}) (\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}) \right\}\bQ^\top .
\end{aligned}
\right.
\end{equation}
The procedure is then formulated in Algorithm~\ref{alg:qr-rankoneChange}.

\begin{algorithm}[h]
	\caption{QR Rank-One Changes}
	\label{alg:qr-rankoneChange}
	\begin{algorithmic}[1]
\Require Matrix $\bA \in \real^{n\times n}$ with QR decomposition $\bA=\bQ\bR$, and $\bA^\prime = \bA+\bu\bv^\top$;
\State Calculate $\bw\leftarrow\bQ^\top\bu$; \Comment{$2n^2-n$ flops}
\State Calculate $\bH \leftarrow \bR$;
\For{$i=n-1$ to $1$}
\State Get Givens rotation $\bG_{i,i+1}$ with the following parameters $c, s$:
\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k = \bw_i$, $x_l = \bw_{i-1}$; \Comment{6 flops}
\State Calculate $\bH = \bG_{i,i+1}\bH $ in following two steps:
\State $i$-th row: $\bH_{i,:} = c\cdot \bH_{i,:} + s \bH_{j,:} $, where $j=i+1$; \Comment{$3(n-i+1)$ flops}
\State $(i+1)$-th row: $\bH_{i+1,:} = -s\cdot \bH_{i,:} + c \bH_{j,:} $, where $j=i+1$; \Comment{$3(n-i+1)$ flops}
\EndFor
\State Set $\bR^\prime =\bH \pm ||\bw|| \be_1 \bv^\top$; \Comment{$\bH, \bR^\prime$ are both upper Hessenberg}
\For{$i=1$ to $n-1$}
\State Get Givens rotation $\bJ_{i,i+1}$ with the following parameters $c, s$:
\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k = \bH_{i,i}$, $x_l = \bH_{i+1,i}$;
\State Calculate $\bR^\prime = \bJ_{i,i+1}\bR^\prime $ in following two steps:
\State $i$-th row: $\bR^\prime_{i,:} = c\cdot \bR^\prime_{i,:} + s \bR^\prime_{j,:} $, where $j=i+1$;
\State $(i+1)$-th row: $\bR^\prime_{i+1,:} = -s\cdot \bR^\prime_{i,:} + c \bR^\prime_{j,:} $, where $j=i+1$;
\EndFor

		\State Output $\bR^\prime$;
\State Set $\bQ^{\prime\top} = \bQ^\top$;
\For{$i=n-1$ to $1$}    \Comment{The following $c,s$ are from step 5}
\State $i$-th row: $\bQ^{\prime\top}_{i,:} = c\cdot \bQ^{\prime\top}_{i,:} + s \bQ^{\prime\top}_{j,:} $, where $j=i+1$;  \Comment{$6n$ flops}
\State $(i+1)$-th row: $\bQ^{\prime\top}_{i+1,:} = -s\cdot \bQ^{\prime\top}_{i,:} + c \bQ^{\prime\top}_{j,:} $, where $j=i+1$;\Comment{$6n$ flops}
\EndFor
\For{$i=1$ to $n-1$} \Comment{The following $c,s$ are from step 13}
\State $i$-th row: $\bQ^{\prime\top}_{i,:} = c\cdot \bQ^{\prime\top}_{i,:} + s \bQ^{\prime\top}_{j,:} $, where $j=i+1$; \Comment{$6n$ flops}
\State $(i+1)$-th row: $\bQ^{\prime\top}_{i+1,:} = -s\cdot \bQ^{\prime\top}_{i,:} + c \bQ^{\prime\top}_{j,:} $, where $j=i+1$;\Comment{$6n$ flops}
\EndFor
\State Output $\bQ^\prime$;

	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: QR Rank-One Change]\label{theorem:qr-full-givens-rank1}
	Algorithm~\ref{alg:qr-rankoneChange} requires $\sim 8n^2$ flops to compute a full QR decomposition of an $\bA^\prime \in \real^{n\times n}$ matrix with rank-one change to $\bA$ and the full QR decomposition of $\bA$ is known. Further, if $\bQ^\prime$ is needed explicitly, additional $\sim 12n^2$ flops are required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-full-givens-rank1}]
	It is trivial that step 1 needs \fbox{(*). $n(2n-1)=2n^2-n$} flops to calculate $\bw=\bQ^\top\bu$.

	For step 5, each iteration $i$ requires $6$ flops (which are 2 square operations, 1 addition, 1 square root, and 2 divisions). And there are $n-1$ such iterations so that the complexity for all the step 5's is \fbox{$6(n-1)$} flops.

	For each iteration $i$, step 7 and step 8 operate on two length-$(n-i+1)$ vectors. The two steps take $6(n-i+1)$ flops for each iteration $i$ (which are $4(n-i+1)$ multiplications and $2(n-i+1)$ additions).

	Let $f(i) =  6(n-i+1)$, the total complexity for the two steps is equal to
	$$
	\mathrm{cost} =f(1)+ f(2) +\ldots +f(n-1) = \boxed{3n^2-3n} \,\, \mathrm{flops}.
	$$
	Therefore, the complexity for step 3 to step 9 is \fbox{(*). $6(n-1) + 3n^2-3n = 3n^2+3n-6$} flops. Similarly, the complexity for step 11 to step 16 is again \fbox{(*). $3n^2+3n-6$} flops.

	However, for each iteration $i$, step 21 and step 22 operate on two length-$n$ vectors. The two steps take $6n$ flops for each iteration $i$, and the total complexity for step 20 to step 23 is \fbox{(*). $6n(n-1) =6n^2-6n$} flops. Again, step 24 to step 27 take another \fbox{(*). $6n^2-6n$} flops.

	Therefore, we can calculate the final complexity by summing up the equations marked as $(*)$ with \fbox{$20n^2$} flops if keep only the leading terms ($8n^2$ flops for calculating $\bR^\prime$, and $12n^2$ flops for calculating $\bQ^\prime$).

	Note that for each iteration $i$, step 7 and step 8 operate on two length-$(n-i+1)$ vectors since $\bR$ is upper triangular. If we do not favor such a structure, the final complexity is $26n^2$ flops as stated in \citep{golub2013matrix}.
\end{proof}
The algorithm can be easily applied to a rectangular matrix $\bA\in \real^{m\times n}$, or $\bA+\bU\bV^\top$ where $\bU\in \real^{m\times k}$ and $\bV\in \real^{n\times k}$.

\subsection{Application: Appending or Deleting a Column}\label{section:append-column-qr}
\paragraph{Deleting a column}
Suppose the QR decomposition of $\bA\in \real^{m\times n}$ is given by $\bA=\bQ\bR$ where the column partition of $\bA$ is $\bA=[\ba_1,\ba_2,\ldots,\ba_n]$. Now, if we delete the $k$-th column of $\bA$ such that $\bA^\prime = [\ba_1,\ldots,\ba_{k-1},\ba_{k+1},\ldots,\ba_n] \in \real^{m\times (n-1)}$. We want to find the QR decomposition of $\bA^\prime$ efficiently. Suppose further $\bR$ has the following form
$$
\begin{aligned}
	\begin{blockarray}{ccccc}
		\begin{block}{c[ccc]c}
			&	\bR_{11} & \ba & \bR_{12} & k-1  \\
			\bR = 	&  \bzero    &	r_{kk} & \bb^\top  & 1 \\
			&	\bzero  &\bzero& \bR_{22} & m-k  \\
		\end{block}
		& k-1 & 1 &  n-k & \\
	\end{blockarray}.\\
\end{aligned}
$$
Apparently,
$$
\bQ^\top \bA^\prime =
\begin{bmatrix}
	\bR_{11} &\bR_{12} \\
	\bzero & \bb^\top \\
	\bzero & \bR_{22}
\end{bmatrix} = \bH
$$
is upper Hessenberg. A $6\times 5$ example is shown as follows where $k=3$:
$$
\begin{aligned}
	\begin{sbmatrix}{\bR = \bQ^\top\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & 0 & \boxtimes& \boxtimes \\
		0 & 0 & 0 & 0& \boxtimes \\
		0 & 0 & 0 & 0& 0
	\end{sbmatrix}
	&\longrightarrow
	\begin{sbmatrix}{\bH = \bQ^\top\bA^\prime}
		\boxtimes & \boxtimes  & \boxtimes & \boxtimes \\
		0 & \boxtimes  & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes& \boxtimes \\
		0 & 0  & \boxtimes& \boxtimes \\
		0 & 0  & 0& \boxtimes \\
		0 & 0  & 0& 0
	\end{sbmatrix}.
\end{aligned}
$$

Again, for columns $k$ to $n-1$ of $\bH$, there exists a set of rotations $\bG_{k,k+1}$, $\bG_{k+1,k+2}$, $\ldots$, $\bG_{n-1,n}$ that could introduce zeros for the elements $h_{k+1,k}$, $h_{k+2,k+1}$, $\ldots$, $h_{n,n-1}$ of $\bH$. The the triangular matrix $\bR^\prime$ is given by
$$
\bR^\prime = \bG_{n-1,n}\ldots \bG_{k+1,k+2}\bG_{k,k+1}\bQ^\top \bA^\prime.
$$
And the orthogonal matrix
\begin{equation}\label{equation:qr-delete-column-finalq}
\bQ^\prime = (\bG_{n-1,n}\ldots \bG_{k+1,k+2}\bG_{k,k+1}\bQ^\top )^\top = \bQ \bG_{k,k+1}^\top  \bG_{k+1,k+2}^\top \ldots \bG_{n-1,n}^\top,
\end{equation}
such that $\bA^\prime = \bQ^\prime\bR^\prime$. The procedure is formulated in Algorithm~\ref{alg:qr-delete-a-column}. And the $6\times 5$ example is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed:
$$
\begin{aligned}
\begin{sbmatrix}{\bR = \bQ^\top\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
0 & 0 & 0 & \boxtimes& \boxtimes \\
0 & 0 & 0 & 0& \boxtimes \\
0 & 0 & 0 & 0& 0
\end{sbmatrix}
&\stackrel{k=3}{\rightarrow}
\begin{sbmatrix}{\bH = \bQ^\top\bA^\prime}
\boxtimes & \boxtimes  & \boxtimes & \boxtimes \\
0 & \boxtimes  & \boxtimes & \boxtimes \\
0 & 0 & \boxtimes& \boxtimes \\
0 & 0  & \boxtimes& \boxtimes \\
0 & 0  & 0& \boxtimes \\
0 & 0  & 0& 0
\end{sbmatrix}
\stackrel{\bG_{34}}{\rightarrow}
\begin{sbmatrix}{\bG_{34}\bH }
	\boxtimes & \boxtimes  & \boxtimes & \boxtimes \\
	0 & \boxtimes  & \boxtimes & \boxtimes \\
	0 & 0 & \bm{\boxtimes}& \bm{\boxtimes} \\
	0 & 0  & \bm{0}& \bm{\boxtimes} \\
	0 & 0  & 0& \boxtimes \\
	0 & 0  & 0& 0
\end{sbmatrix}
\stackrel{\bG_{45}}{\rightarrow}
\begin{sbmatrix}{\bG_{45}\bG_{34}\bH}
	\boxtimes & \boxtimes  & \boxtimes & \boxtimes \\
	0 & \boxtimes  & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes & \boxtimes \\
	0 & 0  &0 & \bm{\boxtimes} \\
	0 & 0  & 0& \bm{0} \\
	0 & 0  & 0& 0
\end{sbmatrix}.
\end{aligned}
$$

\begin{algorithm}[h]
	\caption{QR Deleting a Column}
	\label{alg:qr-delete-a-column}
	\begin{algorithmic}[1]
		\Require Matrix $\bA \in \real^{m\times n}$ with full QR decomposition $\bA=\bQ\bR$, and $\bA^\prime \in \real^{m\times (n-1)}$ by deleting column $k$ of $\bA$;
		\State Obtain $\bH$ by deleting $k$ of $\bR$, that is, $\bH=\bQ^\top\bA^\prime$;
		\For{$i=k$ to $n-1$}
		\State Get Givens rotation $\bG_{i,i+1}$ with the following parameters $c, s$:
		\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k = h_{ii}$, $x_l = h_{i+1,i}$;
		\State Calculate $\bH = \bG_{i,i+1}\bH $ in following two steps:
		\State $i$-th row: $\bH_{i,:} = c\cdot \bH_{i,:} + s \bH_{j,:} $, where $j=i+1$;
		\State $(i+1)$-th row: $\bH_{i+1,:} = -s\cdot \bH_{i,:} + c \bH_{j,:} $, where $j=i+1$;
		\EndFor
		\State Set $\bR^\prime \leftarrow \bH$ and output $\bR^\prime$;
		\State Set $\bQ^\prime \leftarrow \bQ^\top $;
		\For{$i=k$ to $n-1$}
		\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k$, $x_l$ are from step 4;
		\State Calculate $\bQ^\prime = \bG_{i,i+1}\bQ^\prime $ in following two steps:
		\State $i$-th row: $\bQ^\prime_{i,:} = c\cdot \bQ^\prime_{i,:} + s \bQ^\prime_{j,:} $, where $j=i+1$;
		\State $(i+1)$-th row: $\bQ^\prime_{i+1,:} = -s\cdot \bQ^\prime_{i,:} + c \bQ^\prime_{j,:} $, where $j=i+1$;
		\EndFor
		\State Output $\bQ^\prime \leftarrow \bQ^{\prime\top}$ from Equation~\eqref{equation:qr-delete-column-finalq};
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: QR Deleting Column]\label{theorem:qr-full-givens-delete-column}
	Algorithm~\ref{alg:qr-delete-a-column}  requires $\sim 3n^2-6nk+3k^2$ flops to compute a full QR decomposition of an $\bA^\prime \in \real^{m\times (n-1)}$ matrix where we delete the column $k$ of $\bA\in \real^{m\times n}$ and the full QR decomposition of $\bA$ is known. Further, if $\bQ^\prime$ is needed explicitly, additional $\sim 6m(n-k)$ flops required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-full-givens-delete-column}]
	For step 4, each iteration $i$ requires $6$ flops (which are 2 square operations, 1 addition, 1 square root, and 2 divisions). And there are $n-k$ such iterations so that the complexity for all the step 5's is \fbox{$6(n-k)$} flops.

	For each iteration $i$, step 6 and step 7 operate on two length-$(n-i)$ vectors since $\bH$ is upper Hessenberg. The two steps take $6(n-i)$ flops for each iteration $i$ (which are $4(n-i)$ multiplications and $2(n-i)$ additions).
	Let $f(i) =  6(n-i)$, the total complexity for the two steps is equal to
	$$
	\mathrm{cost} =f(k)+ f(k+1) +\ldots + f(n-1) = \boxed{3n^2-6nk+3k^2+3n-3k} \,\, \mathrm{flops}.
	$$
	Therefore, the complexity for step 2 to step 8 is \fbox{(*). $3n^2-6nk+3k^2$} flops if we keep only the leading terms.

	However, for each iteration $i$, step 14 and step 15 operate on two length-$m$ vectors. The two steps take $6m$ flops for each $i$, and  there are $n-k$ such iterations so that the total complexity for step 11 to step 16 is \fbox{(*). $6m(n-k)$} flops.
\end{proof}
Note that the number of column $k$ takes a role in the complexity, when $k=n$, the complexity is 0; and when $k=1$, the complexity gets its maximal value.

\paragraph{Appending a column}
Similarly, suppose $\widetilde{\bA} = [\ba_1,\ba_k,\bw,\ba_{k+1},\ldots,\ba_n]$ where we append $\bw$ into the $(k+1)$-th column of $\bA$. We can obtain
$$
\bQ^\top \widetilde{\bA} = [\bQ^\top\ba_1,\ldots, \bQ^\top\ba_k, \bQ^\top\bw, \bQ^\top\ba_{k+1}, \ldots,\bQ^\top\ba_n] = \widetilde{\bH}.
$$
A set of Givens rotations $\bJ_{m-1,m}, \bJ_{m-2,m-1}, \ldots, \bJ_{k+1,k+2}$ can introduce zeros for the $\widetilde{h}_{m,k+1}$, $\widetilde{h}_{m-1,k+1}$, $\ldots$, $\widetilde{h}_{k+2,k+1}$ elements of $\widetilde{\bH}$ such that
$$
\widetilde{\bR} =\bJ_{k+1,k+2}\ldots \bJ_{m-2,m-1}\bJ_{m-1,m} \bQ^\top \widetilde{\bA},
$$
is upper triangular.
Suppose $\widetilde{\bH}$ is of size $6\times 5$ and $k=2$, an example is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
	\begin{sbmatrix}{\widetilde{\bH}}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & 0& \boxtimes \\
		0 & 0 & \boxtimes & 0& 0 \\
		0 & 0 & \boxtimes & 0& 0
	\end{sbmatrix}
	&\stackrel{\bJ_{56}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{56}\widetilde{\bH} \rightarrow \widetilde{h}_{63}=0}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \boxtimes & 0& \boxtimes \\
		0 & 0 & \bm{\boxtimes} & 0& 0 \\
		0 & 0 & \bm{0} & 0& 0
	\end{sbmatrix}
	\stackrel{\bJ_{45}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{45}\bJ_{56}\widetilde{\bH}\rightarrow \widetilde{h}_{53}=0}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
		0 & 0 & \bm{\boxtimes} & 0& \bm{\boxtimes} \\
		0 & 0 & \bm{0} & 0& \bm{\boxtimes}\\
		0 & 0 & 0 & 0& 0
	\end{sbmatrix}\\
	&\stackrel{\bJ_{34}}{\rightarrow}
	\begin{sbmatrix}{\bJ_{34}\bJ_{45}\bJ_{56}\widetilde{\bH}\rightarrow \widetilde{h}_{43}=0}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \bm{\boxtimes}  & \bm{\boxtimes} & \bm{\boxtimes}  \\
		0 & 0 & \bm{0}  & \bm{\boxtimes} & \bm{\boxtimes}  \\
		0 & 0 & 0 & 0& \boxtimes\\
		0 & 0 & 0 & 0& 0
	\end{sbmatrix} = \widetilde{\bR}.
\end{aligned}
$$
And finally, the orthogonal matrix
\begin{equation}\label{equation:qr-add-column-finalq}
	\widetilde{\bQ} = (\bJ_{k+1,k+2}\ldots \bJ_{m-2,m-1}\bJ_{m-1,m} \bQ^\top )^\top = \bQ \bJ_{m-1,m}^\top  \bJ_{m-2,m-1}^\top \ldots \bJ_{k+1,k+2}^\top,
\end{equation}
such that $\widetilde{\bA} = \widetilde{\bQ}\widetilde{\bR}$. The procedure is again formulated in Algorithm~\ref{alg:qr-adding-a-column}.

\begin{algorithm}[h]
	\caption{QR Adding a Column}
	\label{alg:qr-adding-a-column}
	\begin{algorithmic}[1]
		\Require Matrix $\bA \in \real^{m\times n}$ with full QR decomposition $\bA=\bQ\bR$, and $\widetilde{\bA}\in \real^{m\times (n+1)}$ by adding column $\bw$ into $(k+1)$-th column of $\bA$;
		\State Calculate $\bQ^\top\bw$;
		\State Obtain $\widetilde{\bH}$ by inserting $\bQ^\top \bw$ into $(k+1)$-th column of $\bR$;
		\For{$i=m-1$ to $k+1$}
		\State Get Givens rotation $\bJ_{i,i+1}$ with the following parameters $c, s$:
		\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k = \widetilde{h}_{i,k+1}$, $x_l = \widetilde{h}_{i+1,k+1}$;
		\State Calculate $\widetilde{\bH} = \bJ_{i,i+1}\bH $ in following two steps:
		\State $i$-th row: $\widetilde{\bH}_{i,:} = c\cdot \widetilde{\bH}_{i,:} + s \widetilde{\bH}_{j,:} $, where $j=i+1$;
		\State $(i+1)$-th row: $\widetilde{\bH}_{i+1,:} = -s\cdot \widetilde{\bH}_{i,:} + c \widetilde{\bH}_{j,:} $, where $j=i+1$;
		\EndFor
		\State Set $\widetilde{\bR} \leftarrow \widetilde{\bH}$ and output $\widetilde{\bR}$;
		\State Set $\widetilde{\bQ} \leftarrow \bQ^\top $;
		\For{$i=m-1$ to $k+1$}
		\State $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$ where $x_k$, $x_l$ are from step 5;
		\State Calculate $\widetilde{\bQ} = \bJ_{i,i+1}\widetilde{\bQ} $ in following two steps:
		\State $i$-th row: $\widetilde{\bQ}_{i,:} = c\cdot \widetilde{\bQ}_{i,:} + s \widetilde{\bQ}_{j,:} $, where $j=i+1$;
		\State $(i+1)$-th row: $\bQ_{i+1,:} = -s\cdot \widetilde{\bQ}_{i,:} + c \widetilde{\bQ}_{j,:} $, where $j=i+1$;
		\EndFor
		\State Output $\widetilde{\bQ} \leftarrow \widetilde{\bQ}^{\top}$ from Equation~\eqref{equation:qr-add-column-finalq};
	\end{algorithmic}
\end{algorithm}
\begin{theorem}[Algorithm Complexity: QR Adding Column]\label{theorem:qr-full-givens-add-column}
	Algorithm~\ref{alg:qr-adding-a-column} requires $\sim 2m^2+6(mn+k^2-nk-mk)$ flops to compute a full QR decomposition of an $\widetilde{\bA} \in \real^{m\times (n+1)}$ matrix where we add a column into the $(k+1)$-th column of $\bA\in \real^{m\times n}$ and the full QR decomposition of $\bA$ is known. Further, if $\widetilde{\bQ}$ is needed explicitly, additional $\sim 6m(m-k)$ flops required.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-full-givens-add-column}]
	It is trivial that step 1 requires \fbox{$m(2m-1)$} flops to calculate $\bQ^\top\bw$.

	For step 5, each iteration $i$ requires $6$ flops (which are 2 square operations, 1 addition, 1 square root, and 2 divisions). And there are $m-k-1$ such iterations so that the complexity for all the step 5's is \fbox{$6(m-k-1)$} flops.

	For each iteration $i$, step 7 and step 8 operate on two length-$(n-k+1)$ vectors since $\widetilde{\bH}$ is upper Hessenberg. The two steps take $6(n-k+1)$ flops for each iteration $i$ (which are $4(n-k+1)$ multiplications and $2(n-k+1)$ additions).
	And there are $m-k-1$ such iterations so that the complexity for all the step 7 and step 8's is \fbox{$6(n-k+1)(m-k-1)$} flops.

	Therefore, the complexity for step 1 to step 9 is
	$$
	m(2m-1)+6(m-k-1)+6(n-k+1)(m-k-1)=\boxed{2m^2+6(n-k+2)(m-k-1)},
	$$
	or \fbox{$2m^2+6(mn+k^2-nk-mk)$} flops if we keep only the leading terms.

	However, for each iteration $i$, step 15 and step 16 operate on two length-$m$ vectors. The two steps take $6m$ flops for each $i$, and  there are $m-k-1$ such iterations so that the total complexity for step 12 to step 17 is \fbox{$6m(m-k-1)$} flops or \fbox{$6m(m-k)$} flops if we keep only the leading terms.
\end{proof}

\paragraph{Real world application} The method introduced above is useful for the efficient variable selection in the least squares problem via the QR decomposition. At each time we delete a column of the data matrix $\bA$, and apply an $F$-test to see if the variable is significant or not. If not, we will delete the variable and favor a simpler model. A short review is given as follows and more details can be referred to \citep{lu2021rigorous}.

Following the setup in Section~\ref{section:application-ls-qr},
let's consider the overdetermined system $\bA\bx = \bb$, where $\bA\in \real^{m\times n}$ is the data matrix, $\bb\in \real^m$ with $m>n$ is the observation matrix. The LS solution is given by $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ for minimizing $||\bA\bx-\bb||^2$, where $\bA^\top\bA$ is invertible since $\bA$ has full column rank and $rank(\bA^T\bA)=rank(\bA)$.

Suppose we delete a column of $\bA$ into $\widehat{\bA}$, the LS solution is reduced from $\bx_{LS}$ to $\widehat{\bx}_{LS}$.
Define
$$
\begin{aligned}
	RSS(\widehat{\bx}_{LS}) &= ||\bb - \widehat{\bb}_{LS}||^2, \qquad \text{where } \widehat{\bb}_{LS} = \widehat{\bA}\widehat{\bx}_{LS},   \\
	RSS(\bx_{LS})&= ||\bb - \bb_{LS}||^2, \qquad \text{where } \bb_{LS} = \bA\bx_{LS},\\
	\bH & = \bA(\bA^\top\bA)^{-1}\bA^\top, \\
	\widehat{\bH} &= \widehat{\bA}(\widehat{\bA}^\top \widehat{\bA})^{-1}\widehat{\bA}^\top. \\
\end{aligned}
$$
Suppose the \textit{reduced} QR decomposition of $\bA,\widehat{\bA}$ are given by $\bA=\bQ\bR$, $\widehat{\bA}=\widehat{\bQ}\widehat{\bR}$.
Thus $RSS(\bx_{LS}) = \bb^\top (\bI-\bH)\bb = \bb^\top\bb - (\bb^\top\bQ)(\bQ^\top\bb)$
and $RSS(\widehat{\bx}_{LS}) - RSS(\bx_{LS}) = ||\bb_{LS} - \widehat{\bb}_{LS}||^2 = \bb^\top(\bH-\widehat{\bH})\bb=(\bb^\top\bQ)(\bQ^\top\bb)-(\bb^\top\widehat{\bQ})(\widehat{\bQ}^\top\bb)$, which are the differences of two inner products.
It can be shown that $RSS(\bx_{LS})\sim \sigma^2 \chi^2_{(m-n)}$ which is a Chi-square distribution and $\sigma$ is the noise level. Under the hypothesis that the deleted column is not significant, we could conclude that
$$
T=\frac{\frac{1}{n-q}\left(RSS(\widehat{\bx}_{LS}) - RSS(\bx_{LS})\right)  }{\frac{1}{m-n}RSS(\bx_{LS})} \sim F_{n-q,m-n},
$$
which is the \textbf{test statistic for $F$-test} with $q=n-1$.
Suppose we have the data set $(\ba_1, b_1)$, $(\ba_2, b_2)$, $\ldots$, $(\ba_n, b_m)$, and we observe $T=t$ for this specific data set. Then
$$
p=P[T((\ba_1, b_1), (\ba_2, b_2), \ldots, (\ba_n, b_m)) \geq t] = P[F_{n-q,m-n} \geq t].
$$
We reject the hypothesis if $p<\alpha$, for some small $\alpha$, say 0.05. This is called the \textit{p-value}.

\subsection{Application: Appending or Deleting a Row}

\paragraph{Appending a row}
Suppose the full QR decomposition of $\bA\in \real^{m\times n}$ is given by $\bA= \begin{bmatrix}
	\bA_1 \\
	\bA_2
\end{bmatrix}=\bQ\bR$ where $\bA_1\in \real^{k\times n}$ and $\bA_2 \in \real^{(m-k)\times n}$. Now, if we add a row such that $\bA^\prime = \begin{bmatrix}
	\bA_1 \\
	\bw^\top\\
	\bA_2
\end{bmatrix} \in \real^{(m+1)\times n}$. We want to find the full QR decomposition of $\bA^\prime$ efficiently. Construct a permutation matrix
$$
\bP=
\begin{bmatrix}
	\bzero & 1 & \bzero  \\
	\bI_k & \bzero & \bzero \\
	\bzero & \bzero & \bI_{m-k}
\end{bmatrix}
\longrightarrow
\bP
\begin{bmatrix}
	\bA_1 \\
	\bw^\top\\
	\bA_2
\end{bmatrix}
=
\begin{bmatrix}
	\bw^\top\\
	\bA_1 \\
	\bA_2
\end{bmatrix}
.
$$
Then,
$$
\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bQ^\top
\end{bmatrix}
\bP
\bA^\prime
=
\begin{bmatrix}
	\bw^\top \\
	\bR
\end{bmatrix}=\bH
$$
is upper Hessenberg. Similarly, a set of rotations $\bG_{12}, \bG_{23}, \ldots, \bG_{n,n+1}$ can be applied to introduce zeros for the elements $h_{21}$, $h_{32}$, $\ldots$, $h_{n+1,n}$ of $\bH$. The triangular matrix $\bR^\prime$ is given by
$$
\bR^\prime = \bG_{n,n+1}\ldots \bG_{23}\bG_{12}\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bQ^\top
\end{bmatrix}
\bP   \bA^\prime.
$$
And the orthogonal matrix
$$
\bQ^\prime = \left(\bG_{n,n+1}\ldots \bG_{23}\bG_{12}\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bQ^\top
\end{bmatrix}
\bP \right)^\top
=
\bP^\top
\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bQ
\end{bmatrix}
\bG_{12}^\top  \bG_{23}^\top \ldots \bG_{n,n+1}^\top,
$$
such that $\bA^\prime = \bQ^\prime\bR^\prime$.

	\paragraph{Deleting a row} Suppose $\bA = \begin{bmatrix}
	\bA_1 \\
	\bw^\top\\
	\bA_2
\end{bmatrix} \in \real^{m\times n}$ where $\bA_1\in\real^{k\times n}$, $\bA_2 \in \real^{(m-k-1)\times n}$ with the full QR decomposition given by $\bA=\bQ\bR$ where $\bQ\in \real^{m\times m}, \bR\in \real^{m\times n}$. We want to compute the full QR decomposition of $\widetilde{\bA} =  \begin{bmatrix}
	\bA_1 \\
	\bA_2
\end{bmatrix}$ efficiently (assume $m-1\geq n$). Analogously, we can construct a permutation matrix
$$
\bP =
\begin{bmatrix}
	\bzero & 1 & \bzero  \\
	\bI_k & \bzero & \bzero \\
	\bzero & \bzero & \bI_{m-k-1}
\end{bmatrix}
$$
such that
$$
\bP\bA =
\begin{bmatrix}
	\bzero & 1 & \bzero  \\
	\bI_k & \bzero & \bzero \\
	\bzero & \bzero & \bI_{m-k-1}
\end{bmatrix}
\begin{bmatrix}
	\bA_1 \\
	\bw^\top\\
	\bA_2
\end{bmatrix}=
\begin{bmatrix}
	\bw^\top \\
	\bA_1\\
	\bA_2
\end{bmatrix} = \bP\bQ\bR =\bM\bR ,
$$
where $\bM = \bP\bQ$ is an orthogonal matrix. Let $\bmm^\top$ be the first row of $\bM$, and a set of givens rotations $\bG_{m-1,m}, \bG_{m-2,m-1}, \ldots, \bG_{1,2}$ introducing zeros for elements $m_m, m_{m-1}, \ldots, m_2$ of $\bmm$ respectively such that $\bG_{1,2}\ldots \bG_{m-2,m-1}\bG_{m-1,m}\bmm = \alpha \be_1$ where $\alpha = \pm 1$. Therefore, we have
$$
\bG_{1,2}\ldots \bG_{m-2,m-1}\bG_{m-1,m} \bR =
\begin{blockarray}{cc}
	\begin{block}{[c]c}
		\bv^\top & 1 \\
		\bR_1& m-1  \\
	\end{block}
\end{blockarray},
$$
which is upper Hessenberg with $\bR_1\in \real^{(m-1)\times n}$ being upper triangular. And
$$
\bM   \bG_{m-1,m}^\top \bG_{m-2,m-1}^\top \ldots  \bG_{1,2}^\top  =
\begin{bmatrix}
	\alpha & \bzero \\
	\bzero & \bQ_1
\end{bmatrix},
$$
where $\bQ_1\in \real^{(m-1)\times (m-1)}$ is an orthogonal matrix. The bottom-left block of the above matrix is a zero vector since $\alpha=\pm 1$ and $\bM$ is orthogonal. To see this, let $\bG=\bG_{m-1,m}^\top \bG_{m-2,m-1}^\top $ $\ldots  \bG_{1,2}^\top $ with the first column being $\bg$ and $\bM = [\bmm^\top; \bmm_2^\top; \bmm_3^\top; \ldots, \bmm_{m}^\top]$ being the row partition of $\bM$. We have
$$
\begin{aligned}
	\bmm^\top\bg &= \pm 1 \qquad  \rightarrow \qquad  \bg = \pm \bmm, \\
	\bmm_i^\top \bmm &=0,  \qquad \forall i \in \{2,3,\ldots,m\}.
\end{aligned}
$$
This results in
$$
\begin{aligned}
	\bP\bA&=\bM\bR\\
	&=(\bM   \bG_{m-1,m}^\top \bG_{m-2,m-1}^\top \ldots  \bG_{1,2}\top ) (\bG_{1,2}\ldots \bG_{m-2,m-1}\bG_{m-1,m} \bR ) \\
	&=
	\begin{bmatrix}
		\alpha & \bzero \\
		\bzero & \bQ_1
	\end{bmatrix}
	\begin{bmatrix}
		\bv^\top \\
		\bR_1
	\end{bmatrix} =
	\begin{bmatrix}
		\alpha \bv^\top \\
		\bQ_1\bR_1
	\end{bmatrix}
	=
	\begin{bmatrix}
		\bw^\top \\
		\widetilde{\bA}
	\end{bmatrix}
	.
\end{aligned}
$$
This implies $\bQ_1\bR_1$ is the full QR decomposition of $\widetilde{\bA}=\begin{bmatrix}
	\bA_1\\
	\bA_2
\end{bmatrix}$.

\subsection{Application: Reducing the Ill-Condition via the QR decomposition}

\paragraph{Well-determined linear system}
Consider the well-determined linear equation $\bA\bx = \bb$, where $\bA\in \real^{n\times n}$ is nonsingular. The solution $\bx = \bA^{-1}\bb$ exists and is unique. Now suppose the vector $\bb$ is perturbed by $\delta\bb$, the solution is now given by
$$
\bx + \delta\bx = \bA^{-1}(\bb+\delta\bb)=\bA^{-1}\bb+\bA^{-1}\delta\bb = \bx + \bA^{-1}\delta\bb.
$$
That is $\delta\bx = \bA^{-1}\delta\bb $. By properties of matrix-vector inequality of matrix 2-norm (see Appendix~\ref{appendix:matrix-norm-sect2}), we have
$$
||\delta\bx|| = ||\bA^{-1}\delta\bb ||_2 \leq ||\bA^{-1} ||_2  ||\delta\bb||.
$$
This is known as the \textbf{absolute error bound} of $||\delta\bx||$. And if $||\bA^{-1} ||_2$ is small, then small changes in $\bb$ (i.e., $||\delta\bb||$ is small) will result in small changes in $\bx$. However, if $||\bA^{-1} ||_2$ is large, the changes in $\bx$ may be large as well.

Now we divide the above equation by $||\bx||$
$$
\frac{||\delta\bx|| }{||\bx||} \leq ||\bA^{-1} ||_2  \frac{||\delta\bb||}{||\bx||}.
$$
By $ ||\bb|| = ||\bA\bx|| \rightarrow ||\bx|| \geq \frac{||\bb||}{||\bA||_2}$, we have
$$
\frac{||\delta\bx|| }{||\bx||}
\leq ||\bA ||_2 ||\bA^{-1} ||_2  \frac{||\delta\bb||}{||\bb||}.
$$
This is known as the \textbf{relative error bound} of $\frac{||\delta\bx|| }{||\bx||}$. The product $||\bA ||_2 ||\bA^{-1} ||_2 $ is called the \textbf{condition number} of $\bA$, and is denoted as $\kappa(\bA)$:
$$
\kappa(\bA) = ||\bA ||_2 ||\bA^{-1} ||_2 .
$$
Similarly, by the inequality of Frobenius norm $||\bb|| = ||\bA\bx|| \rightarrow ||\bb|| \leq ||\bA||_F ||\bx||$, we can also define the condition number to be
$$
\kappa(\bA)= ||\bA ||_F ||\bA^{-1} ||_F.
$$
We will only consider the first definition of the condition in this context. If the relative error in $\bx$ is not much larger than the relative error in $\bb$, the matrix is said to be \textbf{well-conditioned}. That is, if the condition number $\kappa(\bA)$ is small, the matrix $\bA$ is well-conditioned. Otherwise, the matrix is called \textbf{ill-conditioned}.

\paragraph{Over-determined linear system} Now, we further consider the over-determined linear equation $\bA\bx = \bb$, where $\bA\in \real^{m\times n}$ with $m>n$. Suppose $\bA$ has full column rank such that $\bA^\top\bA$ is invertible. Then the unique least squares solution is given by
$$
\bx = (\bA^\top\bA)^{-1}\bA^\top\bb,
$$
which results from the normal equation
$$
(\bA^\top\bA)\bx = \bA^\top\bb.
$$
It can be shown that the condition number is given by
$$
\kappa(\bA^\top\bA) = \kappa(\bA)^2.
$$

\begin{example}[Reducing ill-condition]
	Consider the matrix
	$$
	\bA =
	\begin{bmatrix}
		1+\delta & 1\\
		1 & 1+\delta
	\end{bmatrix},
	$$
	where $\delta$ is small. The condition number of $\bA$ is of order $\delta^{-1}$. If we use the QR decomposition $\bA = \bQ\bR$ to solve the linear equation, then
	$$
	\kappa(\bQ)=1, \qquad
	\kappa(\bA) \rightarrow \kappa(\bQ^\top\bA) = \kappa(\bR).
	$$
	If one believes the condition number of $\bR$ is smaller than $\bA$, then we would overcome the ill-condition problem. Specifically, two QR decomposition results of $\bA$ are given by
	$$
	\begin{aligned}
		\bA &= \bQ_1\bR_1\\
		&=
		\begin{bmatrix}
			q_{11} & q_{12} \\
			q_{21} & q_{22}
		\end{bmatrix}
		\begin{bmatrix}
			r_{11} & r_{12} \\
			0 & r_{22}
		\end{bmatrix}\\
		&=
		\begin{bmatrix}
			\frac{1+\delta}{\sqrt{\delta^2+2\delta+2}} & \frac{1}{\sqrt{\delta^2+2\delta+2}}\\
			\frac{1}{\sqrt{\delta^2+2\delta+2}} & -\frac{1+\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
		\begin{bmatrix}
			\sqrt{\delta^2+2\delta+2} & \frac{\delta^2+\delta}{(1+\delta)\sqrt{\delta^2+2\delta+2}} + \frac{\sqrt{\delta^2+2\delta+2}}{1+\delta}\\
			0 & - \frac{\delta^2+2\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
	\end{aligned}
	$$
	or
	$$
	\begin{aligned}
		\bA &= \bQ_2\bR_2\\
		&=
		\begin{bmatrix}
			q_{11} & \textcolor{blue}{-}q_{12} \\
			q_{21} & \textcolor{blue}{-}q_{22}
		\end{bmatrix}
		\begin{bmatrix}
			r_{11} & r_{12} \\
			0 & \textcolor{blue}{-}r_{22}
		\end{bmatrix}\\
		&=
		\begin{bmatrix}
			\frac{1+\delta}{\sqrt{\delta^2+2\delta+2}} & -\frac{1}{\sqrt{\delta^2+2\delta+2}}\\
			\frac{1}{\sqrt{\delta^2+2\delta+2}} & \frac{1+\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
		\begin{bmatrix}
			\sqrt{\delta^2+2\delta+2} & \frac{\delta^2+\delta}{(1+\delta)\sqrt{\delta^2+2\delta+2}} + \frac{\sqrt{\delta^2+2\delta+2}}{1+\delta}\\
			0 & \frac{\delta^2+2\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
	\end{aligned}
	$$
	or
	$$
	\begin{aligned}
		\bA &= \bQ_3\bR_3\\
		&=
		\begin{bmatrix}
			\textcolor{blue}{-}q_{11} & q_{12} \\
			\textcolor{blue}{-}q_{21} & q_{22}
		\end{bmatrix}
		\begin{bmatrix}
			\textcolor{blue}{-}r_{11} & \textcolor{blue}{-}r_{12} \\
			0 & r_{22}
		\end{bmatrix}\\
		&=
		\begin{bmatrix}
			-\frac{1+\delta}{\sqrt{\delta^2+2\delta+2}} & \frac{1}{\sqrt{\delta^2+2\delta+2}}\\
			-\frac{1}{\sqrt{\delta^2+2\delta+2}} & -\frac{1+\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
		\begin{bmatrix}
			-\sqrt{\delta^2+2\delta+2} & -\frac{\delta^2+\delta}{(1+\delta)\sqrt{\delta^2+2\delta+2}} - \frac{\sqrt{\delta^2+2\delta+2}}{1+\delta}\\
			0 & - \frac{\delta^2+2\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
	\end{aligned}
	$$
	or
	$$
	\begin{aligned}
		\bA &= \bQ_4\bR_4\\
		&=
		\begin{bmatrix}
			\textcolor{blue}{-}q_{11} & \textcolor{blue}{-}q_{12} \\
			\textcolor{blue}{-}q_{21} & \textcolor{blue}{-}q_{22}
		\end{bmatrix}
		\begin{bmatrix}
			\textcolor{blue}{-}r_{11} & \textcolor{blue}{-}r_{12} \\
			0 & \textcolor{blue}{-}r_{22}
		\end{bmatrix}\\
		&=
		\begin{bmatrix}
			-\frac{1+\delta}{\sqrt{\delta^2+2\delta+2}} & -\frac{1}{\sqrt{\delta^2+2\delta+2}}\\
			-\frac{1}{\sqrt{\delta^2+2\delta+2}} & \frac{1+\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
		\begin{bmatrix}
			-\sqrt{\delta^2+2\delta+2} & -\frac{\delta^2+\delta}{(1+\delta)\sqrt{\delta^2+2\delta+2}} - \frac{\sqrt{\delta^2+2\delta+2}}{1+\delta}\\
			0 & \frac{\delta^2+2\delta}{\sqrt{\delta^2+2\delta+2}}
		\end{bmatrix}
	\end{aligned}
	$$

	Suppose $\delta=0.01$, the condition number of $\bA$ is 200, and the condition number of $\bR_1$ or $\bR_2$ is 1.
	\exampbar
\end{example}

The example shown above solving linear equation via the QR decomposition can reduce the ill-condition problem, more details on the numerical stability topics can be referred to \citep{higham2002accuracy, zhang2017matrix, golub2013matrix, boyd2018introduction}.

\chapter{Skeleton/CUR Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Skeleton Decomposition}
\begin{theoremHigh}[Skeleton Decomposition]\label{theorem:skeleton-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA }=
\underset{m\times r}{\bC} \gap \underset{r\times r}{\bU^{-1} }\gap \underset{r\times n}{\bR},
$$
where $\bC$ is some $r$ linearly independent columns of $\bA$, $\bR$ is some $r$ linearly independent rows of $\bA$ and $\bU$ is the nonsingular submatrix on the intersection.
\begin{itemize}
	\item The storage for the decomposition is then reduced or potentially increased from $mn$ floats to $r(m+n)+r^2$ floats.
	\item Or further, if we only record the position of the indices, it requires $mr$, $nr$ floats for storing $\bC, \bR$ respectively and extra $2r$ integers to remember the position of each column of $\bC$ in that of $\bA$ and each row of $\bR$ in that of $\bA$ (i.e., construct $\bU$ from $\bC,\bR$).
\end{itemize}
\end{theoremHigh}

Skeleton decomposition is also known as the \textit{CUR decomposition} follows from the notation in the decomposition. The illustration of skeleton decomposition is shown in Figure~\ref{fig:skeleton} where the \textcolor{canaryyellow}{yellow} vectors denote the linearly independent columns of $\bA$ and \textcolor{green}{green} vectors denote the linearly independent rows of $\bA$. Specifically, if $I,J$ index vectors both with size $r$ that contain the indices of rows and columns selected from $\bA$ into $\bR$ and $\bC$ respectively, $\bU$ can be denoted as $\bU=\bA[I,J]$ (see Definition~\ref{definition:matlabnotation}, p.~\pageref{definition:matlabnotation}).
\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{imgs/skeleton.pdf}
\caption{Demonstration of skeleton decomposition of a matrix.}
\label{fig:skeleton}
\end{figure}

\section{Existence of the Skeleton Decomposition}
In Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}), we proved the row rank and the column rank of a matrix are equal. In another word, we can also claim that the dimension of the column space and the dimension of the row space are equal. This property is essential for the existence of the skeleton decomposition.

We are then ready to prove the existence of the skeleton decomposition. The proof is rather elementary.
\begin{proof}[of Theorem~\ref{theorem:skeleton-decomposition}]
The proof relies on the existence of such nonsingular matrix $\bU$ which is central to this decomposition method.

\paragraph{Existence of such nonsingular matrix $\bU$} Since matrix $\bA$ is rank-$r$, we can pick $r$ columns from $\bA$ so that they are linearly independent. Suppose we put the specific $r$ independent columns $\ba_{i1}, \ba_{i2}, \ldots, \ba_{ir}$ into the columns of an $m\times r$ matrix $\bN=[\ba_{i1}, \ba_{i2}, \ldots, \ba_{ir}] \in \real^{m\times r}$. The dimension of the column space of $\bN$ is $r$ so that the dimension of the row space of $\bN$ is also $r$ by Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}). Again, we can pick $r$ linearly independent rows $\bn_{j1}^\top,\bn_{j2}^\top, \ldots, \bn_{jr}^\top $ from $\bN$ and put the specific $r$ rows into rows of an $r\times r$ matrix $\bU = [\bn_{j1}^\top; \bn_{j2}^\top; \ldots; \bn_{jr}^\top]\in \real^{r\times r}$. Using Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}) again, the dimension of the column space of $\bU$ is also $r$ which means there are the $r$ linearly independent columns from $\bU$. So $\bU$ is such a nonsingular matrix with size $r\times r$.

\paragraph{Main proof}
As long as we find the nonsingular $r\times r$ matrix $\bU$ inside $\bA$, we can find the existence of the skeleton decomposition as follows.

Suppose $\bU=\bA[I,J]$ where $I,J$ are index vectors of size $r$. Since $\bU$ is a nonsingular matrix, the columns of $\bU$ are linearly independent. Thus the columns of matrix $\bC$ based on the columns of $\bU$ are also linearly independent (i.e., select the $r$ columns of $\bA$ with the same entries of the matrix $\bU$. Here $\bC$ is equal to the $\bN$ we construct above and $\bC=\bA[:,J]$).

As the rank of the matrix $\bA$ is $r$, if we take any other column $\ba_i$ of $\bA$, $\ba_i$ can be represented as a linear combination of the columns of $\bC$, i.e., there exists a vector $\bx$ such that $\ba_i = \bC \bx$, for all $ i\in \{1, 2, \ldots, n\}$. Let $r$ rows of $\ba_i$ corresponding to the row entries of $\bU$ be $\br_i \in \real^r$ for all $i\in \{1, 2, \ldots, n\}$ (i.e., $\br_i$ contains $r$ entries of $\ba_i$). That is, select the $r$ entries of $\ba_i$'s corresponding to the entries of $\bU$ as follows:
$$
\bA = [\ba_1,\ba_2, \ldots, \ba_n]\in \real^{m\times n} \qquad \longrightarrow \qquad
\bA[I,:]=[\br_1, \br_2, \ldots, \br_n] \in \real^{r\times n}.
$$
Since $\ba_i = \bC\bx$, $\bU$ is a submatrix inside $\bC$, and $\br_i$ is a subvector inside $\ba_i$, we have $\br_i = \bU \bx$ which is equivalent to $\bx = \bU^{-1} \br_i$. Thus for every $i$, we have $\ba_i = \bC \bU^{-1} \br_i$. Combine the $n$ columns of such $\br_i$ into $\bR=[\br_1, \br_2, \ldots, \br_n]$, we obtain
$$
\bA = [\ba_1, \ba_2, \ldots, \ba_n] = \bC \bU^{-1} \bR,
$$
from which the result follows.

In short, we first find $r$ linearly independent columns of $\bA$ into $\bC\in \real^{m\times r}$. From $\bC$, we find an $r\times r$ nonsingular submatrix $\bU$. The $r$ rows of $\bA$ corresponding to entries of $\bU$ can help to reconstruct the columns of $\bA$. Again, the situation is shown in Figure~\ref{fig:skeleton}.
\end{proof}

In case $\bA$ is square and invertible, we have skeleton decomposition $\bA=\bC\bU^{-1} \bR$ where $\bC=\bR=\bU=\bA$ such that the decomposition reduces to $\bA = \bA\bA^{-1}\bA$.

\paragraph{CR decomposition vs skeleton decomposition} We note that CR decomposition and skeleton decomposition share a similar form. Even for the symbols used $\bA=\bC\bR$ for the CR decomposition and $\bA=\bC\bU^{-1}\bR$ for the skeleton decomposition.

Both in the CR decomposition and the skeleton decomposition, we \textbf{can}\footnote{Here, we highlight that we can, but not necessarily, as we will see in the randomized algorithm for the skeleton decomposition.} select the first $r$ independent columns to obtain the matrix $\bC$ (the symbol for both the CR decomposition and the skeleton decomposition). So $\bC$'s in the CR decomposition and the skeleton decomposition are exactly the same. On the contrary, $\bR$ in the CR decomposition is the reduced row echelon form without the zero rows, whereas $\bR$ in the skeleton decomposition is exactly some rows from $\bA$ so that $\bR$'s have different meanings in the two decompositional methods. We will formally show that $\bU^{-1}\bR$ in skeleton decomposition is the row reduced echelon form without zero rows in Theorem~\ref{theorem:cur-row-reduced-form}.

\paragraph{A word on the uniqueness of CR decomposition and skeleton decomposition} As mentioned above, both in the CR decomposition and the skeleton decomposition, we select the first $r$ linearly independent columns to obtain the matrix $\bC$. In this sense, the CR and skeleton decompositions have a unique form.
However, if we select the last $r$ linearly independent columns, we will get a different CR decomposition or skeleton decomposition. We will not discuss this situation here as it is not the main interest of this text.

To repeat, in the above proof for the existence of the skeleton decomposition, we first find the $r$ linearly independent columns of $\bA$ into the matrix $\bC$. From $\bC$, we find an $r\times r$ nonsingular submatrix $\bU$. From the submatrix $\bU$, we finally find the final row submatrix $\bR\in \real^{r\times n}$. A further question can be posed that if matrix $\bA$ has rank $r$, matrix $\bC$ contains $r$ linearly independent columns, and matrix $\bR$ contains $r$ linearly independent rows, then whether the $r\times r$ ``intersection" of $\bC$ and $\bR$ is invertible or not \footnote{We thank Gilbert Strang for raising this interesting question.}.

\begin{corollary}[Nonsingular Intersection]\label{corollary:invertible-intersection}
If matrix $\bA \in \real^{m\times n}$ has rank $r$, matrix $\bC$ contains $r$ linearly independent columns, and matrix $\bR$ contains $r$ linearly independent rows, then the $r\times r$ ``intersection" matrix $\bU$ of $\bC$ and $\bR$ is invertible.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:invertible-intersection}]
If $I,J$ are the indices of rows and columns selected from $\bA$ into $\bR$ and $\bC$ respectively, then, $\bR$ can be denoted as $\bR=\bA[I, :]$, $\bC$ can be represented as $\bC = \bA[:,J]$, and $\bU$ can be denoted as $\bU=\bA[I,J]$.

Since $\bC$ contains $r$ linearly independent columns of $\bA$, any column $\ba_i$ of $\bA$ can be represented as $\ba_i = \bC\bx_i = \bA[:,J]\bx_i$ for all $i \in \{1,2,\ldots, n\}$. This implies the $r$ entries of $\ba_i$ corresponding to the $I$ indices can be represented by the columns of $\bU$ such that $\ba_i[I] = \bU\bx_i \in \real^{r1}$ for all $i \in \{1,2,\ldots, n\}$, i.e.,
$$
\ba_i = \bC\bx_i = \bA[:,J]\bx_i \in \real^{m} \qquad \longrightarrow  \qquad
\ba_i[I] =\bA[I,J]\bx_i= \bU\bx_i \in \real^{r}.
$$
Since $\bR$ contains $r$ linearly independent rows of $\bA$, the row rank and column rank of $\bR$ are equal to $r$. Combining the facts above, the $r$ columns of $\bR$ corresponding to indices $J$ (i.e., the $r$ columns of $\bU$) are linearly independent.

Again, by applying Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}), the dimension of the row space of $\bU$ is also equal to $r$ which means there are the $r$ linearly independent rows from $\bU$, and $\bU$ is invertible.
\end{proof}

\section{Computing the Skeleton Decomposition via the Gram-Schmidt Process}

In Section~\ref{section:dependent-gram-schmidt-process}, we have discussed how to tackle dependent columns when orthogonalizing the matrix. We can thus use the Gram-Schmidt process to select $r$ linearly independent columns from $\bA$ resulting in $\bC$. And use the Gram-Schmidt process to select $r$ linearly independent columns from $\bC^\top$ which results in $\bU$ and $\bR$. The process is shown in Algorithm~\ref{alg:skeleton-decomposition}.

\begin{algorithm}[h]
	\caption{Skeleton Decomposition via Gram-Schmidt Process}
	\label{alg:skeleton-decomposition}
	\begin{algorithmic}[1]
		\Require
		Rank-$r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ with size $m\times n $;
		\State Initially set column count $ck=0$ and row count $rk=0$;
		\State Set $\bq_1   = \ba_1/r_{11},  r_{11}=||\ba_1||$; \Comment{Suppose the first column is nonzero and $r_{11}\neq 0$}
		\For{$k=2$ to $n$}
		\State $\bq_k = (\ba_k-\sum_{i=1}^{k-1}r_{in}\bq_i)/r_{kk}, r_{ik}=\bq_i^\top\ba_k, r_{kk}=||\ba_n-\sum_{i=1}^{k-1}r_{ik}\bq_i||$, $\forall i \in \{1,\ldots, k-1\}$;
		\If{$r_{kk}\neq 0$}
		\State 	Select the $k$-th column of $\bA$ into $ck$-th column of $\bC$;
		\State  $ck=ck+1$;
		\EndIf
		\EndFor
		\State $\bC=[\bc_1^\top; \bc_2^\top; \ldots; \bc_m^\top]$, where $\bc_i^\top$ is the $i$-th row of $\bC$;
		\State Set $\bq_1   = \bc_1/r_{11},  r_{11}=||\bc_1||$;\Comment{Suppose the first row is nonzero and $r_{11}\neq 0$}
		\For{$k=2$ to $m$}
		\State $\bq_k = (\bc_k-\sum_{i=1}^{k-1}r_{ik}\bq_i)/r_{kk}, r_{ik}=\bq_i^\top\bc_k, r_{kk}=||\ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i|| $, $\forall i \in \{1,\ldots, k-1\}$;
		\If{$r_{kk}\neq 0$}
		\State 	Select the $k$-th row of $\bC$ into $rk$-th row of $\bU$;
		\State  $rk=rk+1$;
		\EndIf
		\EndFor
		\State Select the rows from $\bA$ corresponding to the rows of $\bU$ into $\bR$.
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: Skeleton via Gram-Schmidt]\label{theorem:comp-skeleton-decom1}
Algorithm~\ref{alg:skeleton-decomposition} requires $\sim 2(mn^2+rm^2+r^3)$ flops to compute a skeleton decomposition of an $m\times n$ matrix. We will show this algorithm is not the best way to get the skeleton decomposition.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:comp-skeleton-decom1}]
	This is just applying Theorem~\ref{theorem:qr-reduced} twice to get matrix $\bU$ (to get $\bC$, it costs $2mn^2$ flops. And to get $\bU$, it costs $2rm^2$ flops.) and applying Theorem~\ref{theorem:inverse-by-lu2} to compute the inverse of $\bU$ ($2r^3$ flops).
\end{proof}

Note that, choosing the first linearly independent columns from $\bA$ will make the inverse of $\bU$ not stable. A maxvol procedure is used in \citep{goreinov2010find}. We will introduce a pseudoskeleton decomposition to overcome this problem at the end of the section.

\section{Computing the Skeleton Decomposition via Modified Gram-Schmidt Process}
In the Gram-Schmidt process, we want to find orthonormal vectors. However, this orthogonal property is not useful for skeleton decomposition. We just need to decide whether the vectors are linearly independent or not.
We can thus use the Gram-Schmidt process with slight modification to select $r$ linearly independent columns from $\bA$ to get $\bC$. And use the Gram-Schmidt process (again with slight modification) to select $r$ linearly independent columns from $\bC^\top$ to get $\bU$ and $\bR$.

For a set of $n$ vectors $\ba_1, \ba_2, \ldots, \ba_n$ from the columns of $\bA$, if we want to project a vector $\ba_k$ onto the space perpendicular to the space spanned by $\ba_1, \ba_2, \ba_{k-1}$ (suppose the $k-1$ vectors are linearly independent), we can write out this projection from Equation~\eqref{equation:gram-schdt-eq2}
$$
\begin{aligned}
\bb_k &= \ba_k -  \left(\frac{ \bb_1^\top\ba_k}{\bb_1^\top\bb_1} \bb_1 + \frac{ \bb_2^\top\ba_k}{\bb_2^\top\bb_2} \bb_2+\ldots + \frac{ \bb_{k-1}^\top\ba_k}{\bb_{k-1}^\top\bb_{k-1}} \bb_{k-1} \right)\\
&=\ba_k -  \sum_{i=1}^{k-1}\frac{ \bb_i^\top\ba_k}{\bb_i^\top\bb_i} \bb_i,
\end{aligned}
$$
where $\bb_k$ is the projection of $\ba_k$ onto the spaced perpendicular to the space spanned by $\{\ba_1, \ba_2, \ba_{k-1}\}$, $\bb_{k-1}$ is the projection of $\ba_{k-1}$ onto the spaced perpendicular to the space spanned by $\{\ba_1, \ba_2, \ba_{k-2}\}$, and so on.

If $\bb_k$ is nonzero, $\ba_k$ is linearly independent of $\ba_1, \ba_2, \ldots, \ba_{k-1}$ which will be chosen into the column of $\bC$.
Similarly, we can choose linearly independent rows from $\bC$ to produce $\bU$. The process is shown in Algorithm~\ref{alg:skeleton-decomposition-modified}.

\begin{algorithm}[h]
	\caption{Skeleton Decomposition via \textbf{Modified Gram-Schmidt Process}}
	\label{alg:skeleton-decomposition-modified}
	\begin{algorithmic}[1]
		\Require
		Rank-$r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ with size $m\times n $;
		\State Initially set column count $ck=0$ and row count $rk=0$;
		\State Set $\bb_1= \ba_1$; \Comment{Suppose the first column is nonzero, $0$ flops}
		\For{$k=2$ to $n$}
		\State $\bb_k=\ba_k -  \sum_{i=1}^{k-1}\frac{ \bb_i^\top\ba_k}{\bb_i^\top\bb_i} \bb_i$, (Skip the zero terms $\bb_i$);
		\If{$\bb_k\neq \bzero$}
		\State 	Select the $k$-th column of $\bA$ into the $ck$-th column of $\bC$;
		\State  $ck=ck+1$;
		\EndIf
		\EndFor
		\State $\bC=[\bc_1^\top; \bc_2^\top; \ldots; \bc_m^\top]$, where $\bc_i^\top$ is the $i$-th row of $\bC$;
		\State Set $\bb_1   = \bc_1$, (Suppose the first row is nonzero);
		\For{$k=2$ to $m$}
		\State $\bb_k = \bc_k -  \sum_{i=1}^{k-1}\frac{ \bb_i^\top\bc_k}{\bb_i^\top\bb_i} \bb_i$, (Skip the zero terms $\bb_i$);
		\If{$\bb_k\neq \bzero$}
		\State 	Select the $k$-th row of $\bC$ into the $rk$-th row of $\bU$;
		\State  $rk=rk+1$;
		\EndIf
		\EndFor
		\State Select the rows from $\bA$ corresponding to the rows of $\bU$ into $\bR$.
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: Skeleton via Modified Gram-Schmidt]\label{theorem:comp-skeleton-decom2}
	Algorithm~\ref{alg:skeleton-decomposition-modified} requires $\sim 2(mn^2+rm^2+ r^3)$ flops to compute a skeleton decomposition of an $m\times n$ matrix.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:comp-skeleton-decom2}]
For the last loop $n$ in step 4,  the $\frac{ \bb_i^\top\ba_n}{\bb_i^\top\bb_i} \bb_i$ involves:

a. $m$ multiplications and $m-1$ additions for the calculation of $\bb_i^\top\ba_n$;

b. $m$ multiplications for the calculation of $\bb_i^\top\ba_n *\bb_i $;

c. $m$ multiplications and $m-1$ additions for the calculation of $\bb_i^\top\bb_i$;

d. 1 division;

There are $n-1$ such terms to compute procedure $a, b, d$ above. However, for procedure $c$, we did the calculation of $\bb_i^\top\bb_i$ for $i \in \{1, 2, \ldots, n-2\}$ in the previous loops. As a result, we need \fbox{$3m(n-1)+2m-1$} to compute $\frac{ \bb_i^\top\ba_n}{\bb_i^\top\bb_i} \bb_i$ for $i\in \{1, 2, \ldots, n-1\}$ where the last $2m-1$ flops is to compute $\bb_{n-1}^\top\bb_{n-1}$.
The final $\ba_k -  \sum_{i=1}^{k-1}\frac{ \bb_i^\top\ba_k}{\bb_i^\top\bb_i} \bb_i$ then takes $(n-2)m$ additions and $m$ subtractions which cost \fbox{$(n-1)m$} flops. So the total cost of step 4 in the last loop $n$ is $4m(n-1)+2m-1$ flops. Let $f(i)=4m(i-1)+2m-1$, the total cost for step 3 to step 9 can be obtained by
$$
\mathrm{cost}=f(2)+f(3)+\ldots f(n).
$$
Simple calculations can show this loop takes $4m \frac{n^2-n}{2} + (2m-1)(n-1)$ flops, or \fbox{$2mn^2$} flops if keep only the leading term.

Similarly, for step 13, the total cost for the second loop is $4r \frac{m^2-m}{2} + (2r-1)(m-1)$ flops, or \fbox{$2rm^2$} flops if keep only the leading term.

As a result, the total cost is $2(mn^2+rm^2+r^3)$ flops if keep only the leading term, where $2r^3$ results from the computation of the inverse of $\bU$ by Theorem~\ref{theorem:inverse-by-lu2} (p.~\pageref{theorem:inverse-by-lu2}).
\end{proof}

The complexity of Algorithm~\ref{alg:skeleton-decomposition-modified} is the same as that of Algorithm~\ref{alg:skeleton-decomposition} as they are mathematically equivalent. However, the meaning from Algorithm~\ref{alg:skeleton-decomposition-modified} is much more clear.

\section{Computing the Skeleton Decomposition via the Gaussian Elimination}
In Algorithm~\ref{alg:cr-decomposition-alg}, we discussed the method to get the row reduced echelon form by Gaussian elimination. And the columns containing the pivots are the ones that are linearly independent. We thus can use this row reduced echelon form to find the skeleton decomposition.

\begin{algorithm}[H]
	\caption{Skeleton Decomposition via Gaussian Elimination}
	\label{alg:skeleton-by-row-reduced-echelon}
	\begin{algorithmic}[1]
		\Require
		Rank-$r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ with size $m\times n $;
		\State Use Algorithm~\ref{alg:cr-decomposition-alg} to get the row reduced echelon form of $\bA$ and choose the columns containing pivots from $\bA$ into $\bC$;
		\State Use Algorithm~\ref{alg:cr-decomposition-alg} to get the row reduced echelon form of $\bC^\top$ and choose the columns containing pivots from $\bC^\top$ into the columns of $\bU^\top$;
		\State Select the rows from $\bA$ corresponding to the rows of $\bU$ into $\bR$.
	\end{algorithmic}
\end{algorithm}

\begin{theorem}[Algorithm Complexity: Skeleton via Guassian Elimination]\label{theorem:comp-skeleton-decom-row-reduced}
Algorithm~\ref{alg:skeleton-by-row-reduced-echelon} requires
$$
\text{cost}=\left\{
\begin{aligned}
	&(2m^2n-m^3) + (2r^2m-r^3) + 2r^3=2m^2n-m^3 + 2r^2m+r^3 \text{}, \qquad  &\text{if }m\leq n; \\
	&(mn^2)+ (2r^2m-r^3)+2r^3 = mn^2+ 2r^2m+r^3\text{},\qquad &\text{if }m>n.
\end{aligned}
\right.
$$
flops to compute a skeleton decomposition of an $m\times n$ matrix. The cost of $2r^3$ above is again from the calculation of the inverse of $\bU$.
\end{theorem}

The proof of the above theorem is trivial since $r\leq m$ by Theorem~\ref{theorem:cr-decomposition-alg}.

\section{Recover Reduced Row Echelon Form from Skeleton Decomposition}
In this section, we formally claim that $\bU^{-1}\bR$ is the row reduced echelon form without zero rows.
\begin{theorem}[Recover RREF from Skeleton Decomposition]\label{theorem:cur-row-reduced-form}
Suppose we have skeleton decomposition via Algorithm~\ref{alg:skeleton-decomposition}, Algorithm~\ref{alg:skeleton-decomposition-modified}, or Algorithm~\ref{alg:skeleton-by-row-reduced-echelon}
that $\bA = \bC\bU^{-1}\bR$, then $\bU^{-1}\bR$ is the row reduced echelon form without zero rows.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:cur-row-reduced-form}]
Without loss of generality, we assume matrix $\bA\in \real^{4\times 5}$ and the linearly independent columns of matrix $\bA$ are the columns $1, 3, 5$, which means column 2 is a multiple $m_1$ of column 1, column 4 is a linear combination of column 1 and 3: column 4 = $n_1$ column 1 + $n_2$ column 2.

Then we use $\bU$ to recover the column 1, we will have $\bx_1 = \bU^{-1}\br_1 = [1;0;0]$;

When we use $\bU$ to recover the column 2, we will have $\bx_2 = \bU^{-1}\br_2 = [m_1;0;0]$;

When we use $\bU$ to recover column 3, we will have $\bx_3 = \bU^{-1}\br_3 = [0;1;0]$;

When we use $\bU$ to recover column 4, we will have $\bx_4 = \bU^{-1}\br_4 = [n_1;n_2;0]$;

When we use $\bU$ to recover column 5, we will have $\bx_5 = \bU^{-1}\br_5 = [0;0;1]$.

This clearly gives this row reduced echelon form without zero rows:
$$
\bX = \begin{bmatrix}
	1 & m_1 & 0 & n_1 & 0\\
	0 & 0 & 1 & n_2 & 0\\
	0 & 0 & 0 & 0 & 1
\end{bmatrix},
$$
where we find the pivots are all 1's, the entries above the pivots are all 0's.
Now we go back to matrix $\bA\in \real^{m\times n}$, Algorithm~\ref{alg:skeleton-decomposition}, Algorithm~\ref{alg:skeleton-decomposition-modified}, or Algorithm~\ref{alg:skeleton-by-row-reduced-echelon} find the first linearly independent columns from left to right as columns $k, l, m, n, \ldots$ where $k\leq l \leq m\leq n\leq \ldots$. We will get similar results. Pivots are in 1: row 1, column $k$; 2: row 2, column $l$; 3: row 3, column $m$; 4: row 4, column $n$, $\ldots$.

The entries above the pivots are 0's for sure.

For columns $k+1, k+2, \ldots, l-1$, they are combinations of columns $k$, so the entries below row 1, columns $k+1, k+2, \ldots, l-1$ are zero.

For columns $l+1, l+2, \ldots, m-1$, they are combinnations of columns $k, l$, so the entries below row 2, columns $l+1, l+2, \ldots, m-1$ are zero.

The process can go on and it produces the reduced row echelon form without zero rows.
\end{proof}

\section{Randomized Algorithms}
The computing methods for skeleton decomposition described here are choosing the first linearly independent columns from $\bA$ into $\bC$ same as that in the CR decomposition. However, this is not necessarily. Randomized algorithms are discussed in  \citep{mahoney2009cur, boutsidis2009improved, drineas2012fast, kishore2017literature} which do not choose the linearly independent columns from left to right.

In \citep{goreinov1997pseudo, goreinov2001maximal}, an approximation of skeleton decomposition has been developed which is called \textit{pseudoskeleton approximation}. The $k$ columns in $\bC$ and $k$ rows in $\bR$ with $k<r$ were chosen such that their intersection $\bU_{k\times k}$ has maximum volume (i.e., the maximum determinant among all $k\times k$ submatrices of $\bA$).

\section{Pseudoskeleton Decomposition via the SVD}\label{section:pseudoskeleton}
We will discuss singular value decomposition (SVD) extensively in Section~\ref{section:SVD}. But now, let's just assume we have the background of SVD, and we will show how to use the SVD to approximate the skeleton decomposition. Feel free to skip this section for a first reading.

For matrix $\bA\in\real^{m\times n}$, we want to construct an approximation of $\bA$ with rank $\gamma \leq \min(m,n)$ in the form of skeleton decomposition. That is $\bA$ is approximated by $\bA\approx \bC\bG\bR$ where $\bC$ and $\bR$ contain $\gamma$ selected columns and rows respectively and $\bG = \bU^{-1}$ with $\bU$ being the intersection of $\bC$ and $\bR$. Specifically, if $I, J$ are the indices of the selected rows and columns respectively, $\bU=\bA[I,J]$. Note here the $\gamma$ is not necessarily equal to the rank $r$ of $\bA$.

Instead of choosing the $r$ linearly independent columns from $\bA$ (as shown in skeleton decomposition), we choose $k$ random columns into matrix $\bC$ with $k>r$ or even $k=\min(m,n)$ given by the indices $J$ from the matrix $\bA$. That is $\bC = \bA[:,J]\in \real^{m\times k}$. Now we select $k$ rows from $\bA$ by the indices $I$ from the matrix $\bA$ into matrix $\bR=\bA[I,:]$ such that the volume of the intersection matrix $\bU=\bA[I,J]$ is maximized, i.e., $\det(\bU)$ is maximized given the $\bC$ chosen randomly. Note here $\bC$ is randomly chosen, but $\bR$ is not randomly chosen. Now the decomposition becomes
$$
\bA = \bC_{m\times k}\bU_{k\times k}^{-1}\bR_{k\times n}.
$$
Again, the inverse of $\bU_{k\times k}$ is not stable due to the random choice. To overcome this, we decompose $\bU_{k\times k}$ by full SVD (refer to Section~\ref{section:SVD}, p.~\pageref{section:SVD} for the difference between the reduced SVD and full SVD):
$$
\bU_{k\times k} = \bU_k\bSigma_k\bV_k^\top,
$$
where $\bU_k, \bV_k\in \real^{k\times k}$ are orthogonal matrices, $\bSigma_k$ is a diagonal matrix containing $k$ singular values $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_k$ with zeros allowed. Now we choose $\gamma$ singular values that are greater than some value $\epsilon$ and truncate the $\bU_k, \bV_k, \bSigma_k$ according to the $\gamma$ selected singular values such that $\bU_{k\times k}$ is approximated by a rank-$\gamma$ matrix $\bU_{k\times k} \approx \bU_\gamma\bSigma_\gamma\bV_\gamma^\top$, where $\bU_\gamma,\bV_\gamma\in \real^{k\times \gamma}$, and $\bSigma_\gamma \in \real^{\gamma\times \gamma}$. Therefore, the pseudoinverse of $\bU_{k\times k}$ is \footnote{See Section~\ref{section:application-ls-svd} (p.~\pageref{section:application-ls-svd}) and Appendix~\ref{appendix:pseudo-inverse} (p.\pageref{appendix:pseudo-inverse}) for the detailed discussion of pseudoinverse.}
$$
\bU^+ = (\bU_\gamma\bSigma_\gamma\bV_\gamma^\top)^{-1} =\bV_\gamma \bSigma_\gamma^{-1}\bU_\gamma^\top.
$$
As a result, the matrix $\bA$ is approximated by a rank-$\gamma$ matrix
\begin{equation}\label{equation:skeleton-low-rank}
\begin{aligned}
	\bA &\approx \bC\bV_\gamma \bSigma_\gamma^{-1}\bU_\gamma^\top \bR\\
	&=\bC_2 \bR_2,  \qquad (\text{Let $\bC_2=\bC\bV_\gamma \bSigma_\gamma^{-1/2}$ and $\bR_2=\bSigma_\gamma^{-1/2}\bU_\gamma^\top \bR$})
\end{aligned}
\end{equation}
where $\bC_2$ and $\bR_2$ are rank-$\gamma$ matrices. Please refer to \citep{goreinov1997pseudo, kishore2017literature} for further details about the choice of how to choose such $\epsilon$ systematically. In the method described above, we only choose $\bC$ randomly. Algorithms introduced in \citep{zhu2011randomised} choose both $\bC$ and $\bR$ randomly and more stable results are obtained.

\chapter{Spectral Decomposition (Theorem)}\label{section:spectral-decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Spectral Decomposition}
	\begin{theoremHigh}[Spectral Decomposition]\label{theorem:spectral_theorem}
		A real matrix $\bA \in \real^{n\times n}$ is symmetric if and only if there exists an orthogonal matrix $\bQ$ and a diagonal matrix $\bLambda$ such that
		\begin{equation*}
			\bA = \bQ \bLambda \bQ^\top,
		\end{equation*}
		where the columns of $\bQ = [\bq_1, \bq_2, \ldots, \bq_n]$ are eigenvectors of $\bA$ and are mutually orthonormal, and the entries of $\bLambda=diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$ are the corresponding eigenvalues of $\bA$, which are real. And the rank of $\bA$ is the number of nonzero eigenvalues. This is known as the \textbf{spectral decomposition} or \textbf{spectral theorem} of real symmetric matrix $\bA$. Specifically, we have the following properties:

		1. A symmetric matrix has only \textbf{real eigenvalues};

		2. The eigenvectors are orthogonal such that they can be chosen \textbf{orthonormal} by normalization;

		3. The rank of $\bA$ is the number of nonzero eigenvalues;

		4. If the eigenvalues are distinct, the eigenvectors are unique as well.
	\end{theoremHigh}

The above decomposition is called the spectral decomposition for real symmetric matrices and is often known as the \textit{spectral theorem}.
\paragraph{Spectral theorem vs eigenvalue decomposition} In the eigenvalue decomposition, we require the matrix $\bA$ to be square and the eigenvectors to be linearly independent. Whereas in the spectral theorem, any symmetric matrix can be diagonalized, and the eigenvectors are chosen to be orthogonormal.

\paragraph{A word on the spectral decomposition} In Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}), we proved that the eigenvalues of similar matrices are the same. From the spectral decomposition, we notice that $\bA$ and $\bLambda$ are similar matrices such that their eigenvalues are the same. For any diagonal matrices, the eigenvalues are the diagonal components. \footnote{Actually, we have shown in the last section that the diagonal values for triangular matrices are the eigenvalues of it.} To see this, we realize that
$$
\bLambda \be_i = \lambda_i \be_i,
$$
where $\be_i$ is the $i$-th basis vector. Therefore, the matrix $\Lambda$ contains the eigenvalues of $\bA$.

\section{Existence of the Spectral Decomposition}\label{section:existence-of-spectral}

We prove the theorem in several steps.

\begin{tcolorbox}[title={Symmetric Matrix Property 1 of 4}]
\begin{lemma}[Real Eigenvalues]\label{lemma:real-eigenvalues-spectral}
The eigenvalues of any symmetric matrix are all real.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:real-eigenvalues-spectral}]
	Suppose eigenvalue $\lambda$ is a complex number $\lambda=a+ib$ where $a,b$ are real. Its complex conjugate is $\bar{\lambda}=a-ib$. Same for complex eigenvector $\bx = \bc+i\bd$ and its complex conjugate $\bar{\bx}=\bc-i\bd$ where $\bc, \bd$ are real vectors. We then have the following property
	$$
	\bA \bx = \lambda \bx\qquad   \underrightarrow{\text{ leads to }}\qquad  \bA \bar{\bx} = \bar{\lambda} \bar{\bx}\qquad   \underrightarrow{\text{ transpose to }}\qquad  \bar{\bx}^\top \bA =\bar{\lambda} \bar{\bx}^\top.
	$$
	We take the dot product of the first equation with $\bar{\bx}$ and the last equation with $\bx$:
	$$
	\bar{\bx}^\top \bA \bx = \lambda \bar{\bx}^\top \bx, \qquad \text{and } \qquad \bar{\bx}^\top \bA \bx = \bar{\lambda}\bar{\bx}^\top \bx.
	$$
	Then we have the equality $\lambda\bar{\bx}^\top \bx = \bar{\lambda} \bar{\bx}^\top\bx$. Since $\bar{\bx}^\top\bx = (\bc-i\bd)^\top(\bc+i\bd) = \bc^\top\bc+\bd^\top\bd$ is a real number. Therefore the imaginary part of $\lambda$ is zero and $\lambda$ is real.
\end{proof}

\begin{tcolorbox}[title={Symmetric Matrix Property 2 of 4}]
\begin{lemma}[Orthogonal Eigenvectors]\label{lemma:orthogonal-eigenvectors}
The eigenvectors  corresponding to distinct eigenvalues of any symmetric matrix are orthogonal so that we can normalize eigenvectors to make them orthonormal since $\bA\bx = \lambda \bx \underrightarrow{\text{ leads to } } \bA\frac{\bx}{||\bx||} = \lambda \frac{\bx}{||\bx||}$ which corresponds to the same eigenvalue.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:orthogonal-eigenvectors}]
	Suppose eigenvalues $\lambda_1, \lambda_2$ correspond to eigenvectors $\bx_1, \bx_2$ so that $\bA\bx_1=\lambda \bx_1$ and $\bA\bx_2 = \lambda_2\bx_2$. We have the following equality:
	$$
	\bA\bx_1=\lambda_1 \bx_1 \leadto \bx_1^\top \bA =\lambda_1 \bx_1^\top \leadto \bx_1^\top \bA \bx_2 =\lambda_1 \bx_1^\top\bx_2,
	$$
	and
	$$
	\bA\bx_2 = \lambda_2\bx_2 \leadto  \bx_1^\top\bA\bx_2 = \lambda_2\bx_1^\top\bx_2,
	$$
	which implies $\lambda_1 \bx_1^\top\bx_2=\lambda_2\bx_1^\top\bx_2$. Since eigenvalues $\lambda_1\neq \lambda_2$, the eigenvectors are orthogonal.
\end{proof}

In the above Lemma~\ref{lemma:orthogonal-eigenvectors}, we prove that the eigenvectors corresponding to distinct eigenvalues  of symmetric matrices are orthogonal. More generally, we prove the important theorem that eigenvectors corresponding to distinct eigenvalues of any matrix are linearly independent.
\begin{theorem}[Independent Eigenvector Theorem]\label{theorem:independent-eigenvector-theorem}
If a matrix $\bA\in \real^{n\times n}$ has $k$ distinct eigenvalues, then any set of $k$ corresponding eigenvectors are linearly independent.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:independent-eigenvector-theorem}]
We will prove by induction. Firstly, we will prove that any two eigenvectors corresponding to distinct eigenvalues are linearly independent. Suppose $\bv_1,\bv_2$ correspond to distinct eigenvalues $\lambda_1$ and $\lambda_2$ respectively. Suppose further there exists a nonzero vector $\bx=[x_1,x_2] \neq \bzero $ that
\begin{equation}\label{equation:independent-eigenvector-eq1}
x_1\bv_1+x_2\bv_2=\bzero.
\end{equation}
That is, $\bv_1,\bv_2$ are linearly independent.
Multiply Equation~\eqref{equation:independent-eigenvector-eq1} on the left by $\bA$, we get
\begin{equation}\label{equation:independent-eigenvector-eq2}
x_1 \lambda_1\bv_1 + x_2\lambda_2\bv_2 = \bzero.
\end{equation}
Multiply Equation~\eqref{equation:independent-eigenvector-eq1} on the left by $\lambda_2$, we get
\begin{equation}\label{equation:independent-eigenvector-eq3}
x_1\lambda_2\bv_1 + x_2\lambda_2\bv_2 = \bzero.
\end{equation}
Subtract Equation~\eqref{equation:independent-eigenvector-eq2} from Equation~\eqref{equation:independent-eigenvector-eq3} to find
$$
x_1(\lambda_2-\lambda_1)\bv_1 = \bzero.
$$
Since $\lambda_2\neq \lambda_1$, $\bv_1\neq \bzero$, we must have $x_1=0$. From Equation~\eqref{equation:independent-eigenvector-eq1}, $\bv_2\neq \bzero$, we must also have $x_2=0$ which arrives at a contradiction. Thus $\bv_1,\bv_2$ are linearly independent.

Now, suppose any $j<k$ eigenvectors are linearly independent, if we could prove that any $j+1$ eigenvectors are also linearly independent, we finish the proof. Suppose $\bv_1, \bv_2, \ldots, \bv_j$ are linearly independent and $\bv_{j+1}$ is dependent on the first $j$ eigenvectors. That is, there exists a nonzero vector $\bx=[x_1,x_2,\ldots, x_{j}]\neq \bzero$ that
\begin{equation}\label{equation:independent-eigenvector-zero}
\bv_{j+1}=	x_1\bv_1+x_2\bv_2+\ldots+x_j\bv_j .
\end{equation}
Suppose the $j+1$ eigenvectors correspond to distinct eigenvalues $\lambda_1,\lambda_2,\ldots,\lambda_j,\lambda_{j+1}$.
Multiply Equation~\eqref{equation:independent-eigenvector-zero} on the left by $\bA$, we get
\begin{equation}\label{equation:independent-eigenvector-zero2}
\lambda_{j+1} \bv_{j+1} = x_1\lambda_1\bv_1+x_2\lambda_2\bv_2+\ldots+x_j \lambda_j\bv_j .
\end{equation}
Multiply Equation~\eqref{equation:independent-eigenvector-zero} on the left by $\lambda_{j+1}$, we get
\begin{equation}\label{equation:independent-eigenvector-zero3}
	\lambda_{j+1} \bv_{j+1} = x_1\lambda_{j+1}\bv_1+x_2\lambda_{j+1}\bv_2+\ldots+x_j \lambda_{j+1}\bv_j .
\end{equation}
Subtract Equation~\eqref{equation:independent-eigenvector-zero3} from Equation~\eqref{equation:independent-eigenvector-zero2}, we find
$$
x_1(\lambda_{j+1}-\lambda_1)\bv_1+x_2(\lambda_{j+1}-\lambda_2)\bv_2+\ldots+x_j (\lambda_{j+1}-\lambda_j)\bv_j = \bzero.
$$
From assumption, $\lambda_{j+1} \neq \lambda_i$ for all $i\in \{1,2,\ldots,j\}$, and $\bv_i\neq \bzero$ for all $i\in \{1,2,\ldots,j\}$. We must have $x_1=x_2=\ldots=x_j=0$ which leads to a contradiction. Then $\bv_1,\bv_2,\ldots,\bv_j,\bv_{j+1}$ are linearly independent. This completes the proof.
\end{proof}

A direct consequence of above theorem is as follows:
\begin{corollary}[Independent Eigenvector Theorem, CNT.]\label{theorem:independent-eigenvector-theorem-basis}
If a matrix $\bA\in \real^{n\times n}$ has $n$ distinct eigenvalues, then any set of $n$ corresponding eigenvectors form a basis for $\real^n$.
\end{corollary}

\begin{tcolorbox}[title={Symmetric Matrix Property 3 of 4}]
\begin{lemma}[Orthonormal Eigenvectors for Duplicate Eigenvalue]\label{lemma:eigen-multiplicity}
If $\bA$ has a duplicate eigenvalue $\lambda_i$ with multiplicity $k\geq 2$, then there exist $k$ orthonormal eigenvectors corresponding to $\lambda_i$.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:eigen-multiplicity}]
We note that there is at least one eigenvector $\bx_{i1}$ corresponding to $\lambda_i$. And for such eigenvector $\bx_{i1}$, we can always find additional $n-1$ orthonormal vectors $\by_2, \by_3, \ldots, \by_n$ so that $\{\bx_{i1}, \by_2, \by_3, \ldots, \by_n\}$ forms an orthonormal basis in $\real^n$. Put the $\by_2, \by_3, \ldots, \by_n$ into matrix $\bY_1$ and $\{\bx_{i1}, \by_2, \by_3, \ldots, \by_n\}$ into matrix $\bP_1$
$$
\bY_1=[\by_2, \by_3, \ldots, \by_n] \qquad \text{and} \qquad \bP_1=[\bx_{i1}, \bY_1].
$$
We then have
$$
\bP_1^\top\bA\bP_1 = \begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bY_1^\top \bA\bY_1
\end{bmatrix}.
$$
As a result, $\bA$ and $\bP_1^\top\bA\bP_1$ are similar matrices such that they have the same eigenvalues since $\bP_1$ is nonsingular (even orthogonal here, see Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). We obtain
$$
\det(\bP_1^\top\bA\bP_1 - \lambda\bI_n) =
\footnote{By the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bA & \bB \\
		\bC & \bD
	\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$.
}
(\lambda_i - \lambda )\det(\bY_1^\top \bA\bY_1 - \lambda\bI_{n-1}).
$$
If $\lambda_i$ has multiplicity $k\geq 2$, then the term $(\lambda_i-\lambda)$ occurs $k$ times in the polynomial from the determinant $\det(\bP_1^\top\bA\bP_1 - \lambda\bI_n)$, i.e., the term occurs $k-1$ times in the polynomial from $\det(\bY_1^\top \bA\bY_1 - \lambda\bI_{n-1})$. In another word, $\det(\bY_1^\top \bA\bY_1 - \lambda_i\bI_{n-1})=0$ and $\lambda_i$ is an eigenvalue of $\bY_1^\top \bA\bY_1$.

Let $\bB=\bY_1^\top \bA\bY_1$. Since $\det(\bB-\lambda_i\bI_{n-1})=0$, the null space of $\bB-\lambda_i\bI_{n-1}$ is not none. Suppose $(\bB-\lambda_i\bI_{n-1})\bn = \bzero$, i.e., $\bB\bn=\lambda_i\bn$ and $\bn$ is an eigenvector of $\bB$.

From $
\bP_1^\top\bA\bP_1 = \begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bB
\end{bmatrix},
$
we have $
	\bA\bP_1
	\begin{bmatrix}
		z \\
		\bn
	\end{bmatrix}
	=
	\bP_1
	\begin{bmatrix}
		\lambda_i &\bzero \\
		\bzero & \bB
	\end{bmatrix}
	\begin{bmatrix}
		z \\
		\bn
	\end{bmatrix}$, where $z$ is any scalar. From the left side of this equation, we have
\begin{equation}\label{equation:spectral-pro4-right}
	\begin{aligned}
		\bA\bP_1
		\begin{bmatrix}
			z \\
			\bn
		\end{bmatrix}
		&=
		\begin{bmatrix}
			\lambda_i\bx_{i1}, \bA\bY_1
		\end{bmatrix}
		\begin{bmatrix}
			z \\
			\bn
		\end{bmatrix} \\
		&=\lambda_iz\bx_{i1} + \bA\bY_1\bn.
	\end{aligned}
\end{equation}
And from the right side of the equation, we have
\begin{equation}\label{equation:spectral-pro4-left}
\begin{aligned}
\bP_1
\begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bB
\end{bmatrix}
\begin{bmatrix}
	z \\
	\bn
\end{bmatrix}
&=
\begin{bmatrix}
	\bx_{i1} & \bY_1
\end{bmatrix}
\begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bB
\end{bmatrix}
\begin{bmatrix}
	z \\
	\bn
\end{bmatrix}\\
&=
\begin{bmatrix}
\lambda_i\bx_{i1} & \bY_1\bB
\end{bmatrix}
\begin{bmatrix}
	z \\
	\bn
\end{bmatrix}\\
&= \lambda_i z \bx_{i1} + \bY_1\bB \bn \\
&=\lambda_i z \bx_{i1} + \lambda_i \bY_1 \bn.  \qquad (\text{Since $\bB \bn=\lambda_i\bn$})\\
\end{aligned}
\end{equation}
Combine Equation~\eqref{equation:spectral-pro4-left} and Equation~\eqref{equation:spectral-pro4-right}, we obtain
$$
\bA\bY_1\bn = \lambda_i\bY_1 \bn,
$$
which means $\bY_1\bn$ is an eigenvector of $\bA$ corresponding to the eigenvalue $\lambda_i$ (same eigenvalue corresponding to $\bx_{i1}$). Since $\bY_1\bn$ is a combination of $\by_2, \by_3, \ldots, \by_n$ which are orthonormal to $\bx_{i1}$, the $\bY_1\bn$ can be chosen to be orthonormal to $\bx_{i1}$.

To conclude, if we have one eigenvector $\bx_{i1}$ corresponding to $\lambda_i$ whose multiplicity is $k\geq 2$, we could construct the second eigenvector by choosing one vector from the null space of $(\bB-\lambda_i\bI_{n-1})$ constructed above. Suppose now, we have constructed the second eigenvector $\bx_{i2}$ which is orthonormal to $\bx_{i1}$.
For such eigenvectors $\bx_{i1}, \bx_{i2}$, we can always find additional $n-2$ orthonormal vectors $\by_3, \by_4, \ldots, \by_n$ so that $\{\bx_{i1},\bx_{i2}, \by_3, \by_4, \ldots, \by_n\}$ forms an orthonormal basis in $\real^n$. Put the $\by_3, \by_4, \ldots, \by_n$ into matrix $\bY_2$ and $\{\bx_{i1},\bx_{i2},  \by_3, \by_4, \ldots, \by_n\}$ into matrix $\bP_2$:
$$
\bY_2=[\by_3, \by_4, \ldots, \by_n] \qquad \text{and} \qquad \bP_2=[\bx_{i1}, \bx_{i2},\bY_1].
$$
We then have
$$
\bP_2^\top\bA\bP_2 =
\begin{bmatrix}
	\lambda_i & 0 &\bzero \\
	 0& \lambda_i &\bzero \\
	\bzero & \bzero & \bY_2^\top \bA\bY_2
\end{bmatrix}
=
\begin{bmatrix}
	\lambda_i & 0 &\bzero \\
	0& \lambda_i &\bzero \\
	\bzero & \bzero & \bC
\end{bmatrix},
$$
where $\bC=\bY_2^\top \bA\bY_2$ such that $\det(\bP_2^\top\bA\bP_2 - \lambda\bI_n) = (\lambda_i-\lambda)^2 \det(\bC - \lambda\bI_{n-2})$. If the multiplicity of $\lambda_i$ is $k\geq 3$, $\det(\bC - \lambda_i\bI_{n-2})=0$ and the null space of $\bC - \lambda_i\bI_{n-2}$ is not none so that we can still find a vector from null space of $\bC - \lambda_i\bI_{n-2}$ and $\bC\bn = \lambda_i \bn$. Now we can construct a vector $\begin{bmatrix}
	z_1 \\
	z_2\\
	\bn
\end{bmatrix}\in \real^n $, where $z_1, z_2$ are any scalar values, such that
$$
\bA\bP_2\begin{bmatrix}
	z_1 \\
	z_2\\
	\bn
\end{bmatrix} = \bP_2
\begin{bmatrix}
	\lambda_i & 0 &\bzero \\
	0& \lambda_i &\bzero \\
	\bzero & \bzero & \bC
\end{bmatrix}
\begin{bmatrix}
	z_1 \\
	z_2\\
	\bn
\end{bmatrix}.
$$
Similarly, from the left side of the above equation, we will get $\lambda_iz_1\bx_{i1} +\lambda_iz_2\bx_{i2}+\bA\bY_2\bn$. From the right side of the above equation, we will get $\lambda_iz_1\bx_{i1} +\lambda_i z_2\bx_{i2}+\lambda_i\bY_2\bn$. As a result,
$$
\bA\bY_2\bn = \lambda_i\bY_2\bn,
$$
where $\bY_2\bn$ is an eigenvector of $\bA$ and orthogonal to $\bx_{i1}, \bx_{i2}$. And it is easy to construct the eigenvector to be orthonormal to the first two.

The process can go on, and finally, we will find $k$ orthonormal eigenvectors corresponding to $\lambda_i$.

Actually, the dimension of the null space of $\bP_1^\top\bA\bP_1 -\lambda_i\bI_n$ is equal to the multiplicity $k$. It also follows that if the multiplicity of $\lambda_i$ is $k$, there cannot be more than $k$ orthogonal eigenvectors corresponding to $\lambda_i$. Otherwise, it will come to the conclusion that we could find more than $n$ orthogonal eigenvectors which leads to a contradiction.
\end{proof}

The proof of the existence of the spectral decomposition is trivial from the lemmas above. Also, we can use Schur decomposition to prove the existence of it.
\begin{proof}[\textbf{of Theorem~\ref{theorem:spectral_theorem}: Existence of Spectral Decomposition}]
From the Schur decomposition in Theorem~\ref{theorem:schur-decomposition} (p.~\pageref{theorem:schur-decomposition}), symmetric matrix $\bA=\bA^\top$ leads to $\bQ\bU\bQ^\top = \bQ\bU^\top\bQ^\top$. Then $\bU$ is a diagonal matrix. And this diagonal matrix actually contains eigenvalues of $\bA$. All the columns of $\bQ$ are eigenvectors of $\bA$. We conclude that all symmetric matrices are diagonalizable even with repeated eigenvalues.
\end{proof}

For any matrix multiplication, we have the rank of the multiplication result no larger than the rank of the inputs. However, the symmetric matrix $\bA^\top \bA$ is rather special in that the rank of $\bA^\top \bA$ is equal to that of $\bA$ which will be used in the proof of singular value decomposition in the next section.
	\begin{lemma}[Rank of $\bA\bB$]\label{lemma:rankAB}
		For any matrix $\bA\in \real^{m\times n}$, $\bB\in \real^{n\times k}$, then the matrix multiplication $\bA\bB\in \real^{m\times k}$ has $rank$($\bA\bB$)$\leq$min($rank$($\bA$), $rank$($\bB$)).
	\end{lemma}

\begin{proof}[of Lemma~\ref{lemma:rankAB}]
	For matrix multiplication $\bA\bB$, we have

	$\bullet$ All rows of $\bA\bB$ are combinations of rows of $\bB$, the row space of $\bA\bB$ is a subset of the row space of $\bB$. Thus $rank$($\bA\bB$)$\leq$$rank$($\bB$).

	$\bullet$ All columns of $\bA\bB$ are combinations of columns of $\bA$, the column space of $\bA\bB$ is a subset of the column space of $\bA$. Thus $rank$($\bA\bB$)$\leq$$rank$($\bA$).

	Therefore, $rank$($\bA\bB$)$\leq$min($rank$($\bA$), $rank$($\bB$)).
\end{proof}
\begin{tcolorbox}[title={Symmetric Matrix Property 4 of 4}]
	\begin{lemma}[Rank of Symmetric Matrices]\label{lemma:rank-of-symmetric}
		If $\bA$ is an $n\times n$ real symmetric matrix, then rank($\bA$) =
		the total number of nonzero eigenvalues of $\bA$.
		In particular, $\bA$ has full rank if and only if $\bA$ is nonsingular. Further, $\cspace(\bA)$ is the linear space spanned by the eigenvectors of $\bA$ that correspond to nonzero eigenvalues.
	\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:rank-of-symmetric}]
	For any symmetric matrix $\bA$, we have $\bA$, in spectral form, as $\bA = \bQ \bLambda\bQ^\top$ and also $\bLambda = \bQ^\top\bA\bQ$. Since we have shown in Lemma~\ref{lemma:rankAB} that the rank of the multiplication $rank$($\bA\bB$)$\leq$min($rank$($\bA$), $rank$($\bB$)).

	$\bullet$ From $\bA = \bQ \bLambda\bQ^\top$, we have $rank(\bA) \leq rank(\bQ \bLambda) \leq rank(\bLambda)$;

	$\bullet$ From $\bLambda = \bQ^\top\bA\bQ$, we have $rank(\bLambda) \leq rank(\bQ^\top\bA) \leq rank(\bA)$,

	The inequalities above give us a contradiction. And thus $rank(\bA) = rank(\bLambda)$ which is the total number of nonzero eigenvalues.

	Since $\bA$ is nonsingular if and only if all of its eigenvalues are nonzero, $\bA$ has full rank if and only if $\bA$ is nonsingular.
\end{proof}

Similar to the eigenvalue decomposition, we can compute the $m$-th power of matrix $\bA$ via the spectral decomposition more efficiently.
	\begin{remark}[$m$-th Power]\label{remark:power-spectral}
		The $m$-th power of $\bA$ is $\bA^m = \bQ\bLambda^m\bQ^\top$ if the matrix $\bA$ can be factored as $\bA=\bQ\bLambda\bQ^\top$.
	\end{remark}

\section{Uniqueness of Spectral Decomposition}\label{section:uniqueness-spectral-decomposition}
Clearly, the spectral decomposition is not unique essentially because of the multiplicity of eigenvalues. One can imagine that eigenvalue $\lambda_i$ and $\lambda_j$ are the same for some $1\leq i,j\leq n$, and interchange the corresponding eigenvectors in $\bQ$ will have the same results but the decompositions are different.
But the \textit{eigenspaces} (i.e., the null space $\nspace(\bA - \lambda_i\bI)$ for eigenvalue $\lambda_i$) corresponding to each eigenvalue are fixed. So there is a unique decomposition in terms
of eigenspaces and then any orthonormal basis of these eigenspaces can be chosen.

\section{Other Forms, Connecting Eigenvalue Decomposition*}\label{section:otherform-spectral}
In this section, we discuss other forms of the spectral decomposition under different conditions.
\begin{definition}[Characteristic Polynomial\index{Characteristic polynomial}]
For any square matrix $\bA \in \real^{n\times n}$, the \textbf{characteristic polynomial} $\det(\bA - \lambda \bI)$ is given by
$$
\begin{aligned}
\det(\lambda\bI-\bA ) &=\lambda^n - \gamma_{n-1} \lambda^{n-1} + \ldots + \gamma_1 \lambda  + \gamma_0\\
&=(\lambda-\lambda_1)^{k_1} (\lambda-\lambda_2)^{k_2} \ldots (\lambda-\lambda_m)^{k_m},
\end{aligned}
$$
where $\lambda_1, \lambda_2, \ldots, \lambda_m$ are the distinct roots of $\det( \lambda\bI-\bA)$ and also the eigenvalues of $\bA$, and $k_1+k_2+\ldots +k_m=n$, i.e., $\det(\lambda\bI-\bA)$ is a polynomial of degree $n$ for any matrix $\bA\in \real^{n\times n}$ (see proof of Lemma~\ref{lemma:eigen-multiplicity}, p.~\pageref{lemma:eigen-multiplicity}).
\end{definition}

An important multiplicity arises from the characteristic polynomial of a matrix is then defined as follows:
\begin{definition}[Algebraic Multiplicity and Geometric Multiplicity\index{Algebraic multiplicity}\index{Geometric multiplicity}]
Given the characteristic polynomial of matrix $\bA\in \real^{n\times n}$:
$$
\begin{aligned}
\det(\lambda\bI-\bA ) =(\lambda-\lambda_1)^{k_1} (\lambda-\lambda_2)^{k_2} \ldots (\lambda-\lambda_m)^{k_m}.
\end{aligned}
$$
The integer $k_i$ is called the \textbf{algebraic multiplicity} of the eigenvalue $\lambda_i$, i.e., the algebraic multiplicity of eigenvalue $\lambda_i$ is equal to the multiplicity of the corresponding root of the characteristic polynomial.

The \textbf{eigenspace associated to eigenvalue $\lambda_i$} is defined by the null space of $(\bA - \lambda_i\bI)$, i.e., $\nspace(\bA - \lambda_i\bI)$.

And the dimension of the eigenspace associated to $\lambda_i$, $\nspace(\bA - \lambda_i\bI)$, is called the \textbf{geometric multiplicity} of $\lambda_i$.

In short, we denote the algebraic multiplicity of $\lambda_i$ by $alg(\lambda_i)$, and its geometric multiplicity by $geo(\lambda_i)$.
\end{definition}

\begin{remark}[Geometric Multiplicity]\label{remark:geometric-mul-meaning}
Note that for matrix $\bA$ and the eigenspace $\nspace(\bA-\lambda_i\bI)$, the dimension of the eigenspace is also the number of linearly independent eigenvectors of $\bA$ associated to $\lambda_i$, namely a basis for the  eigenspace. This implies that
while there are an infinite number of eigenvectors associated with each eigenvalue $\lambda_i$, the fact that they form a subspace (provided the zero vector is added) means that they can be described by a finite number of vectors.
\end{remark}

By definition, the sum of the algebraic multiplicities is equal to $n$, but the sum of the geometric multiplicities can be strictly smaller.

\begin{corollary}[Multiplicity in Similar Matrices\index{Similar matrices}]\label{corollary:multipli-similar-matrix}
Similar matrices have same algebraic multiplicities and geometric multiplicities.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:multipli-similar-matrix}]
In Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}), we proved that the eigenvalues of similar matrices are the same, therefore, the algebraic multiplicities of similar matrices are the same as well.

Suppose $\bA$ and $\bB= \bP\bA\bP^{-1}$ are similar matrices where $\bP$ is nonsingular. And the geometric multiplicity of an eigenvalue of $\bA$, say $\lambda$, is $k$. Then there exists a set of orthogonal vectors $\bv_1, \bv_2, \ldots, \bv_k$ that are the basis for the eigenspace $\nspace(\bA-\lambda\bI)$ such that $\bA\bv_i = \lambda \bv_i$ for all $i\in \{1, 2, \ldots, k\}$. Then, $\bw_i = \bP\bv_i$'s are the eigenvectors of $\bB$ associated with eigenvalue $\lambda$. Further, $\bw_i$'s are linearly independent since $\bP$ is nonsingular. Thus, the dimension of the eigenspace $\nspace(\bB-\lambda_i\bI)$ is at least $k$, that is, $dim(\nspace(\bA-\lambda\bI)) \leq dim(\nspace(\bB-\lambda\bI)) $.

Similarly, there exists a set of orthogonal vectors $\bw_1, \bw_2, \ldots, \bw_k$ that are the bases for the eigenspace $\nspace(\bB-\lambda\bI)$, then $\bv_i = \bP^{-1}\bw_i$ for all $i \in \{1, 2, \ldots, k\}$ are the eigenvectors of $\bA$ associated to $\lambda$. This will result in $dim(\nspace(\bB-\lambda\bI)) \leq dim(\nspace(\bA-\lambda\bI)) $.

Therefore, by ``sandwiching", we get $dim(\nspace(\bA-\lambda\bI)) = dim(\nspace(\bB-\lambda\bI)) $, which is the equality of the geometric multiplicities, and the claim follows.
\end{proof}

\begin{lemma}[Bounded Geometric Multiplicity]\label{lemma:bounded-geometri}
For any matrix $\bA\in \real^{n\times n}$, its geometric multiplicity is bounded by algebraic multiplicity for any eigenvalue $\lambda_i$:
$$
geo(\lambda_i) \leq alg(\lambda_i).
$$
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:bounded-geometri}]
If we can find a similar matrix $\bB$ of $\bA$ that has a specific form of the characteristic polynomial, then we complete the proof.

Suppose $\bP_1 = [\bv_1, \bv_2, \ldots, \bv_k]$ contains the eigenvectors of $\bA$ associated with $\lambda_i$ which are linearly independent. That is, the $k$ vectors are bases for the eigenspace $\nspace(\bA-\lambda\bI)$ and the geometric multiplicity associated with $\lambda_i$ is $k$. We can expand it to $n$ linearly independent vectors such that
$$
\bP = [\bP_1, \bP_2] = [\bv_1, \bv_2, \ldots, \bv_k, \bv_{k+1}, \ldots, \bv_n],
$$
where $\bP$ is nonsingular. Then $\bA\bP = [\lambda_i\bP_1, \bA\bP_2]$.

Construct a matrix $\bB = \begin{bmatrix}
\lambda_i \bI_k & \bC \\
\bzero  & \bD
\end{bmatrix}$ where $\bA\bP_2 = \bP_1\bC + \bP_2\bD$, then $\bP^{-1}\bA\bP = \bB$ such that $\bA$ and $\bB$ are similar matrices. We can always find such $\bC,\bD$ that satisfy the above condition, since $\bv_i$'s are linearly independent with spanning the whole space $\real^n$, and any column of $\bA\bP_2$ is in the column space of $\bP=[\bP_1,\bP_2]$.
Therefore,
$$
\begin{aligned}
\det(\bA-\lambda\bI) &= \det(\bP^{-1})\det(\bA-\lambda\bI)\det(\bP)  \qquad &(\text{$\det(\bP^{-1}) = 1/\det(\bP)$})\\
&= \det(\bP^{-1}(\bA-\lambda\bI)\bP)  \qquad &(\det(\bA)\det(\bB) = \det(\bA\bB))\\
&=  \det(\bB-\lambda\bI) \\
&= \det(\begin{bmatrix}
	(\lambda_i-\lambda) \bI_k  & \bC \\
	\bzero  & \bD - \lambda \bI
\end{bmatrix})\\
&= (\lambda_i-\lambda)^k \det(\bD-\lambda\bI),
\end{aligned}
$$
where the last equality is from the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
	\bA & \bB \\
	\bC & \bD
\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$.
This implies
$$
geo(\lambda_i) \leq alg(\lambda_i).
$$
And we complete the proof.
\end{proof}

Following from the proof of Lemma~\ref{lemma:eigen-multiplicity}, we notice that the algebraic multiplicity and geometric multiplicity are the same for symmetric matrices. We call these matrices simple matrices.

\begin{definition}[Simple Matrix]
When the algebraic multiplicity and geometric multiplicity are the same for a matrix, we call it a simple matrix.
\end{definition}

\begin{definition}[Diagonalizable]
A matrix $\bA$ is diagonalizable if there exists a nonsingular matrix $\bP$ and a diagonal matrix $\bD$ such that $\bA = \bP\bD\bP^{-1}$.
\end{definition}
Eigenvalue decomposition in Theorem~\ref{theorem:eigenvalue-decomposition} and spectral decomposition in Theorem~\ref{theorem:spectral_theorem} are such kinds of matrices that are diagonalizable.

\begin{lemma}[Simple Matrices are Diagonalizable]\label{lemma:simple-diagonalizable}
A matrix is a simple matrix if and only if it is diagonalizable.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:simple-diagonalizable}]
We will show by forward implication and backward implication separately as follows.
\paragraph{Forward implication} Suppose that $\bA\in \real^{n\times n}$ is a simple matrix, such that the algebraic and geometric multiplicities for each eigenvalue are equal. For a specific eigenvalue $\lambda_i$, let $\{\bv_1^i, \bv_2^i, \ldots, \bv_{k_i}^i\}$ be a basis for the eigenspace $\nspace(\bA - \lambda_i\bI)$, that is, $\{\bv_1^i, \bv_2^i, \ldots, \bv_{k_i}^i\}$ is a set of linearly independent eigenvectors of $\bA$ associated to $\lambda_i$, where ${k_i}$ is the algebraic or geometric multiplicity associated to $\lambda_i$: $alg(\lambda_i)=geo(\lambda_i)=k_i$. Suppose there are $m$ distinct eigenvalues, since $k_1+k_2+\ldots +k_m = n$, the set of eigenvectors consists of the union of $n$ vectors. Suppose there is a set of $x_j$'s such that
\begin{equation}\label{equation:proof-simple-diagonalize}
	\bz = \sum_{j=1}^{k_1} x_j^1 \bv_j^1+ \sum_{j=1}^{k_2} x_j^2 \bv_j^2 + \ldots \sum_{j=1}^{k_m} x_j^m \bv_j^m = \bzero.
\end{equation}
Let $\bw^i = \sum_{j=1}^{k_i} x_j^i \bv_j^i$. Then $\bw^i$ is either an eigenvector associated to $\lambda_i$, or it is a zero vector. That is $\bz = \sum_{i=1}^{m} \bw^i$ is a sum of either zero vector or an eigenvector associated with different eigenvalues of $\bA$. Since eigenvectors associated with different eigenvalues are linearly independent. We must have $\bw^i =0$ for all $i\in \{1, 2, \ldots, m\}$. That is
$$
\bw^i = \sum_{j=1}^{k_i} x_j^i \bv_j^i = \bzero, \qquad \text{for all $i\in \{1, 2, \ldots, m\}$}.
$$
Since we assume the eigenvectors $\bv_j^i$'s associated to $\lambda_i$ are linearly independent, we must have $x_j^i=0$ for all $i \in \{1,2,\ldots, m\}, j\in \{1,2,\ldots,k_i\}$. Thus, the $n$ vectors are linearly independent:
$$
\{\bv_1^1, \bv_2^1, \ldots, \bv_{k_i}^1\},\{\bv_1^2, \bv_2^2, \ldots, \bv_{k_i}^2\},\ldots,\{\bv_1^m, \bv_2^m, \ldots, \bv_{k_i}^m\}.
$$
By eigenvalue decomposition in Theorem~\ref{theorem:eigenvalue-decomposition}, matrix $\bA$ can be diagonalizable.

\paragraph{Backward implication} Suppose $\bA$ is diagonalizable. That is, there exists a nonsingular matrix $\bP$ and a diagonal matrix $\bD$ such that $\bA =\bP\bD\bP^{-1} $. $\bA$ and $\bD$ are similar matrices such that they have the same eigenvalues (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}), same algebraic multiplicities, and geometric multiplicities (Corollary~\ref{corollary:multipli-similar-matrix}, p.~\pageref{corollary:multipli-similar-matrix}). It can be easily verified that a diagonal matrix has equal algebraic multiplicity and geometric multiplicity such that $\bA$ is a simple matrix.
\end{proof}

	\begin{remark}[Equivalence on Diagonalization]
From Theorem~\ref{theorem:independent-eigenvector-theorem} that any eigenvectors corresponding to different eigenvalues are linearly independent, and Remark~\ref{remark:geometric-mul-meaning} that the geometric multiplicity is the dimension of the eigenspace. We realize, if the geometric multiplicity is equal to the algebraic multiplicity, the eigenspace can span the whole space $\real^n$ if matrix $\bA\in \real^{n\times n}$. So the above Lemma is equivalent to claim that if the eigenspace can span the whole space $\real^n$, then $\bA$ can be diagonalizable.
	\end{remark}

\begin{corollary}
A square matrix $\bA$ with linearly independent eigenvectors is a simple matrix. Or if $\bA$ is symmetric, it is also a simple matrix.
\end{corollary}
From the eigenvalue decomposition in Theorem~\ref{theorem:eigenvalue-decomposition} (p.~\pageref{theorem:eigenvalue-decomposition}) and the spectral decomposition in Theorem~\ref{theorem:spectral_theorem} (p.~\pageref{theorem:spectral_theorem}), the proof is trivial for the corollary.

Now we are ready to show the second form of the spectral decomposition.
\begin{theoremHigh}[Spectral Decomposition: The Second Form]\label{theorem:spectral_theorem_secondForm}
A \textbf{simple matrix} $\bA \in \real^{n\times n}$ can be factored as a sum of a set of idempotent matrices
$$
\bA = \sum_{i=1}^{n} \lambda_i \bA_i,
$$
where $\lambda_i$ for all $i\in \{1,2,\ldots, n\}$ are eigenvalues of $\bA$ (duplicate possible), and also known as the \textbf{spectral values} of $\bA$. Specifically, we have the following properties:

1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, n\}$;

2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

3. Additivity: $\sum_{i=1}^{n} \bA_i = \bI_n$;

4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_n) = n$.

\end{theoremHigh}

\begin{proof}[of Theorem~\ref{theorem:spectral_theorem_secondForm}]
Since $\bA$ is a simple matrix, from Lemma~\ref{lemma:simple-diagonalizable}, there exists a nonsingular matrix $\bP$ and a diagonal matrix $\bLambda$ such that $\bA=\bP\bLambda\bP^{-1}$ where $\bLambda=\diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$, and $\lambda_i$'s are eigenvalues of $\bA$ and columns of $\bP$ are eigenvectors of $\bA$. Suppose
$$
\bP = \begin{bmatrix}
\bv_1 & \bv_2&\ldots & \bv_n
\end{bmatrix}
\qquad
\text{and }
\qquad
\bP^{-1} =
\begin{bmatrix}
\bw_1^\top \\
\bw_2^\top \\
\vdots \\
\bw_n^\top
\end{bmatrix}
$$
are the column and row partitions of $\bP$ and $\bP^{-1}$ respectively. Then, we have
$$
\bA= \bP\bLambda\bP^{-1} =
\begin{bmatrix}
	\bv_1 & \bv_2&\ldots & \bv_n
\end{bmatrix}
\bLambda
\begin{bmatrix}
	\bw_1^\top \\
	\bw_2^\top \\
	\vdots \\
	\bw_n^\top
\end{bmatrix}=
\sum_{i=1}^{n}\lambda_i \bv_i\bw_i^\top.
$$
Let $\bA_i = \bv_i\bw_i^\top$, we have $\bA = \sum_{i=1}^{n} \lambda_i \bA_i$.
We realize that $\bP^{-1}\bP = \bI$ such that
$$
\left\{
\begin{aligned}
	&\bw_i^\top\bv_j = 1 ,& \mathrm{\,\,if\,\,} i = j.  \\
	&\bw_i^\top\bv_j = 0 ,& \mathrm{\,\,if\,\,} i \neq j.
\end{aligned}
\right.
$$
Therefore,
$$
\bA_i\bA_j =\bv_i\bw_i^\top\bv_j\bw_j^\top = \left\{
\begin{aligned}
	&\bv_i\bw_i^\top = \bA_i ,& \mathrm{\,\,if\,\,} i = j.  \\
	& \bzero ,& \mathrm{\,\,if\,\,} i \neq j.
\end{aligned}
\right.
$$
This implies the idempotency and orthogonality of $\bA_i$'s. We also notice that $\sum_{i=1}^{n}\bA_i = \bP\bP^{-1}=\bI$, that is the additivity of $\bA_i$'s. The rank-additivity of the $\bA_i$'s is trivial since $rank(\bA_i)=1$ for all $i\in \{1,2,\ldots, n\}$.
\end{proof}
The decomposition is highly related to the Cochran's theorem in Appendix~\ref{appendix:cochran-theorem} and its application in the distribution theory of linear models \citep{lu2021rigorous}.

\begin{theoremHigh}[Spectral Decomposition: The Third Form]\label{Corollary:spectral_theorem_3Form}
A \textbf{simple matrix} $\bA \in \real^{n\times n}$ \textcolor{blue}{with $k$ distinct eigenvalues} can be factored as a sum of a set of idempotent matrices
$$
\bA = \sum_{i=1}^{\textcolor{blue}{k}} \lambda_i \bA_i,
$$
where $\lambda_i$ for all $i\in \{1,2,\ldots, \textcolor{blue}{k}\}$ are the distinct eigenvalues of $\bA$, and also known as the \textbf{spectral values} of $\bA$. Specifically, we have the following properties:

1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, \textcolor{blue}{k}\}$;

2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

3. Additivity: $\sum_{i=1}^{\textcolor{blue}{k}} \bA_i = \bI_n$;

4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{\textcolor{blue}{k}}) = n$.

\end{theoremHigh}
\begin{proof}[of Theorem~\ref{Corollary:spectral_theorem_3Form}]
From Theorem~\ref{theorem:spectral_theorem_secondForm}, we can decompose $\bA$ by $\bA =\sum_{j=1}^{n} \beta_i \bB_j$. Without loss of generality,
the eigenvalues $\beta_i$'s are ordered such that $\beta_1 \leq \beta_2 \leq \ldots \beta_n$ where duplicate is possible. Let $\lambda_i$'s be the distinct eigenvalues, and $\bA_i$ be the sum of the $\bB_j$'s associated with $\lambda_i$.
Suppose the multiplicity of $\lambda_i$ is $m_i$, and the $\bB_j$'s associated to $\lambda_i$ can be denoted as $\{\bB_{1}^i, \bB_{2}^i, \ldots, \bB_{m_i}^i\}$. Then $\bA_i$ can be denoted as $\bA_i = \sum_{j=1}^{m_i} \bB_{j}^i$. Apparently $\bA = \sum_{i=1}^{k} \lambda_i \bA_i$.

\paragraph{Idempotency} $\bA_i^2 = (\bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i)(\bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i)= \bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i = \bA_i$ from the idempotency and orthogonality of $\bB_j^i$'s.

\paragraph{Ortogonality} $\bA_i\bA_j = (\bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i)(\bB_1^j + \bB_2^j+\ldots \bB_{m_j}^j)=\bzero$ from the orthogonality of the $\bB_j^i$'s.

\paragraph{Additivity} It is trivial that  $\sum_{i=1}^{k} \bA_i = \bI_n$.

\paragraph{Rank-Additivity} $rank(\bA_i ) = rank(\sum_{j=1}^{m_i} \bB_{j}^i) = m_i$ such that $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{k}) = m_1+m_2+\ldots+m_k=n$.
\end{proof}

\begin{theoremHigh}[Spectral Decomposition: Backward Implication]\label{Corollary:spectral_theorem_4Form}
If a matrix $\bA \in \real^{n\times n}$ with $k$ distinct eigenvalues can be factored as a sum of a set of idempotent matrices
$$
\bA = \sum_{i=1}^{k} \lambda_i \bA_i,
$$
where $\lambda_i$ for all $i\in \{1,2,\ldots, k\}$ are the distinct eigenvalues of $\bA$, and

1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, k\}$;

2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

3. Additivity: $\sum_{i=1}^{k} \bA_i = \bI_n$;

4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{k}) = n$.

\item Then, the matrix $\bA$ is a simple matrix.
\end{theoremHigh}
\begin{proof}[of Corollary~\ref{Corollary:spectral_theorem_4Form}]
Suppose $rank(\bA_i) = r_i$ for all $i \in \{1,2,\ldots, k\}$. By ULV decomposition in Theorem~\ref{theorem:ulv-decomposition}, $\bA_i$ can be factored as
$$
\bA_i = \bU_i \begin{bmatrix}
	\bL_i & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV_i,
$$
where $\bL_i \in \real^{r_i \times r_i}$, $\bU_i \in \real^{n \times n}$ and $\bV_i\in \real^{n \times n}$ are orthogonal matrices. Let
$$
\bX_i =
\bU_i \begin{bmatrix}
	\bL_i  \\
	\bzero
\end{bmatrix}
\qquad
\text{and}
\qquad
\bV_i =
\begin{bmatrix}
\bY_i \\
\bZ_i
\end{bmatrix},
$$
where $\bX_i \in \real^{n\times r_i}$, and $\bY_i \in \real^{r_i \times n}$ is the first $r_i$ rows of $\bV_i$. Then, we have
$$
\bA_i = \bX_i \bY_i.
$$
This can be seen as a \textbf{reduced} ULV decomposition of $\bA_i$. Appending the $\bX_i$'s and $\bY_i$'s into $\bX$ and $\bY$,
$$
\bX = [\bX_1, \bX_2, \ldots, \bX_k],
\qquad
\bY =
\begin{bmatrix}
\bY_1\\
\bY_2\\
\vdots \\
\bY_k
\end{bmatrix},
$$
where $\bX\in \real^{n\times n}$ and $\bY\in \real^{n\times n}$ (from rank-additivity). By block matrix multiplication and the additivity of $\bA_i$'s, we have
$$
\bX\bY = \sum_{i=1}^{k} \bX_i\bY_i = \sum_{i=1}^{k} \bA_i = \bI.
$$
Therefore $\bY$ is the inverse of $\bX$, and
$$
\bY\bX =
\begin{bmatrix}
	\bY_1\\
	\bY_2\\
	\vdots \\
	\bY_k
\end{bmatrix}
[\bX_1, \bX_2, \ldots, \bX_k]
=
\begin{bmatrix}
\bY_1\bX_1 & \bY_1\bX_2 & \ldots & \bY_1\bX_k\\
\bY_2\bX_1 & \bY_2\bX_2 & \ldots & \bY_2\bX_k\\
\vdots & \vdots & \ddots & \vdots\\
\bY_k\bX_1 & \bY_k\bX_2 & \ldots & \bY_k\bX_k\\
\end{bmatrix}
=\bI,
$$
such that
$$
\bY_i\bX_j = \left\{
\begin{aligned}
	&\bI_{r_i}  ,& \mathrm{\,\,if\,\,} i = j;  \\
	& \bzero ,& \mathrm{\,\,if\,\,} i \neq j.
\end{aligned}
\right.
$$
This implies
$$
\bA_i\bX_j = \left\{
\begin{aligned}
	&\bX_i  ,& \mathrm{\,\,if\,\,} i = j;  \\
	& \bzero ,& \mathrm{\,\,if\,\,} i \neq j,
\end{aligned}
\right.
\qquad
\text{and}
\qquad
\bA \bX_i = \lambda_i\bX_i.
$$
Finally, we have
$$
\begin{aligned}
\bA\bX &= \bA[\bX_1, \bX_2, \ldots, \bX_k] = [\lambda_1\bX_1, \lambda_2\bX_2, \ldots, \lambda_k\bX_k] = \bX\bLambda,
\end{aligned}
$$
where
$$\bLambda =
\begin{bmatrix}
\lambda_1 \bI_{r_1} & \bzero & \ldots & \bzero \\
\bzero &  \lambda_2 \bI_{r_2} & \ldots & \bzero \\
\vdots &  \vdots & \ddots & \vdots \\
\bzero & \bzero & \ldots &  \lambda_k \bI_{r_k} \\
\end{bmatrix}
$$
is a diagonal matrix. This implies $\bA$ can be diagonalized and from Lemma~\ref{lemma:simple-diagonalizable}, $\bA$ is a simple matrix.
\end{proof}

	\begin{corollary}[Forward and Backward Spectral]
		Combine Theorem~\ref{Corollary:spectral_theorem_3Form} and Theorem~\ref{Corollary:spectral_theorem_4Form}, we can claim that matrix $\bA \in \real^{n\times n}$ is a simple matrix with $k$ distinct eigenvalues if and only if it can be factored as a sum of a set of idempotent matrices
		$$
		\bA = \sum_{i=1}^{k} \lambda_i \bA_i,
		$$
		where $\lambda_i$ for all $i\in \{1,2,\ldots, k\}$ are the distinct eigenvalues of $\bA$, and

		1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, k\}$;

		2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

		3. Additivity: $\sum_{i=1}^{k} \bA_i = \bI_n$;

		4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{k}) = n$.
	\end{corollary}

\section{Skew-Symmetric Matrices and its Properties*}
We have introduced the spectral decomposition for symmetric matrices. A special kind of matrices that's related to symmetric is called the skew-symmetric matrices.
\begin{definition}[Skew-Symmetric Matrix\index{Skew-symmetric matrix}]
If matrix $\bA\in \real^{n\times n}$ have the following property, then it is known as a \textbf{skew-symmetric matrix}:
$$
\bA^\top = -\bA.
$$
Note that under this definition, for the diagonal values $a_{ii}$ for all $i \in \{1,2,\ldots, n\}$, we have $a_{ii} = -a_{ii}$ which implies all the diagonal components are 0.
\end{definition}

We have proved in Lemma~\ref{lemma:real-eigenvalues-spectral} that all the eigenvalues of symmetric matrices are real. Similarly, we could show that all the eigenvalues of skew-symmetric matrices are imaginary.

	\begin{lemma}[Imaginary Eigenvalues]\label{lemma:real-eigenvalues-spectral-skew}
		The eigenvalues of any skew-symmetric matrix are all imaginary or zero.
	\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:real-eigenvalues-spectral-skew}]
	Suppose eigenvalue $\lambda$ is a complex number $\lambda=a+ib$ where $a,b$ are real. Its complex conjugate is $\bar{\lambda}=a-ib$. Same for complex eigenvector $\bx = \bc+i\bd$ and its complex conjugate $\bar{\bx}=\bc-i\bd$ where $\bc, \bd$ are real vectors. We then have the following property
	$$
	\bA \bx = \lambda \bx\qquad   \underrightarrow{\text{ leads to }}\qquad  \bA \bar{\bx} = \bar{\lambda} \bar{\bx}\qquad   \underrightarrow{\text{ transpose to }}\qquad  \bar{\bx}^\top \bA^\top =\bar{\lambda} \bar{\bx}^\top.
	$$
	We take the dot product of the first equation with $\bar{\bx}$ and the last equation with $\bx$:
	$$
	\bar{\bx}^\top \bA \bx = \lambda \bar{\bx}^\top \bx, \qquad \text{and } \qquad \bar{\bx}^\top \bA^\top \bx = \bar{\lambda}\bar{\bx}^\top \bx.
	$$
	Then we have the equality $-\lambda\bar{\bx}^\top \bx = \bar{\lambda} \bar{\bx}^\top\bx$ (since $\bA^\top=-\bA$). Since $\bar{\bx}^\top\bx = (\bc-i\bd)^\top(\bc+i\bd) = \bc^\top\bc+\bd^\top\bd$ is a real number. Therefore the real part of $\lambda$ is zero and $\lambda$ is either imaginary or zero.
\end{proof}

	\begin{lemma}[Odd Skew-Symmetric Determinant]\label{lemma:skew-symmetric-determinant}
For skew-symmetric matrix $\bA\in \real^{n\times n}$, if $n$ is odd, then $\det(\bA)=0$.
	\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:skew-symmetric-determinant}]
When $n$ is odd, we have
$$
\det(\bA) = \det(\bA^\top) = \det(-\bA) = (-1)^n \det(\bA) = -\det(\bA).
$$
This implies $\det(\bA)=0$.
\end{proof}

\begin{theoremHigh}[Block-Diagonalization of Skew-Symmetric Matrices]\label{theorem:skew-block-diagonalization_theorem}
A real skew-symmetric matrix $\bA \in \real^{n\times n}$ can be factored as
\begin{equation*}
\bA = \bZ \bD \bZ^\top,
\end{equation*}
where $\bZ$ is an $n\times n$ nonsingular matrix, and $\bD$ is a block-diagonal matrix with the following form
$$
\bD =
\diag\left(\begin{bmatrix}
	0 & 1 \\
	-1 & 0
\end{bmatrix},
\ldots,
\begin{bmatrix}
	0 & 1 \\
	-1 & 0
\end{bmatrix},
0, \ldots, 0\right).
$$
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:skew-block-diagonalization_theorem}]
We will prove by recursive calculation.
As usual, we will denote the entry ($i,j$) of matrix $\bA$ by $\bA_{ij}$.

\paragraph{Case 1).} Suppose the first row of $\bA$ is nonzero, we notice that $\bE\bA\bE^\top$ is skew-symmetric if $\bA$ is skew-symmetric for any matrix $\bE$. This will make both the diagonals of $\bA$ and $\bE\bA\bE^\top$ are zeros, and the upper-left $2\times 2$ submatrix of $\bE\bA\bE^\top$ has the following form
$$
(\bE\bA\bE^\top)_{1:2,1:2}=
\begin{bmatrix}
0 & x \\
-x & 0
\end{bmatrix}.
$$
Since we suppose the first row of $\bA$ is nonzero, there exists a permutation matrix $\bP$ (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}), such that we will exchange the nonzero value, say $a$, in the first row to the second column of $\bP\bA\bP^\top$. And as discussed above, the upper-left $2\times 2$ submatrix of $\bP\bA\bP^\top$ has the following form
$$
(\bP\bA\bP^\top)_{1:2,1:2}=
\begin{bmatrix}
	0 & a \\
	-a & 0
\end{bmatrix}.
$$
Construct a nonsingular matrix $\bM = \begin{bmatrix}
1/a & \bzero \\
\bzero   & \bI_{n-1}
\end{bmatrix}$ such that the upper left $2\times 2$ submatrix of $\bM\bP\bA\bP^\top\bM^\top$ has the following form
$$
(\bM\bP\bA\bP^\top\bM^\top)_{1:2,1:2}=
\begin{bmatrix}
0 & 1 \\
-1 & 0
\end{bmatrix}.
$$
Now we finish diagonalizing the upper-left $2\times 2$ block. Suppose now $(\bM\bP\bA\bP^\top\bM^\top)$ above has a nonzero value, say $b$, in the first row with entry $(1,j)$ for some $j>2$, we can construct a nonsingular matrix $\bL = \bI - b\cdot\bE_{j2}$ where $\bE_{2j}$ is an all-zero matrix except the entry ($2,j$) is 1, such that $(\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top)$ will introduce 0 for the entry with value $b$.

\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={A Trivial Example}]
For example, suppose $\bM\bP\bA\bP^\top\bM^\top$ is a $3\times 3$ matrix with the following value
$$
\bM\bP\bA\bP^\top\bM^\top =
\begin{bmatrix}
0 & 1 & b \\
-1 & 0 & \times \\
\times & \times & 0
\end{bmatrix},
\qquad \text{and}\qquad
\bL =\bI - b\cdot\bE_{j2}=
\begin{bmatrix}
	1 & 0 & 0 \\
	0 & 1 & 0 \\
	0 &-b & 1
\end{bmatrix},
$$
where $j=3$ for this specific example. This results in
$$
\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top =
\begin{bmatrix}
	1 & 0 & 0 \\
	0 & 1 & 0 \\
	0 &-b & 1
\end{bmatrix}
\begin{bmatrix}
	0 & 1 & \textcolor{blue}{b} \\
	-1 & 0 & \times \\
	\times & \times & 0
\end{bmatrix}
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & -b \\
0 & 0 & 1
\end{bmatrix}
=
\begin{bmatrix}
	0 & 1 & \textcolor{blue}{0} \\
	-1 & 0 & \times \\
	\times & \times & 0
\end{bmatrix}.
$$
\end{mdframed}
Similarly, if the second row of $\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top$ contains a nonzero value, say $c$, we could construct a nonsingular matrix $\bK = \bI+c\cdot \bE_{j1}$ such that $\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top$ will introduce 0 for the entry with value $c$.
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={A Trivial Example}]
	For example, suppose $\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top$ is a $3\times 3$ matrix with the following value
	$$
	\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top =
	\begin{bmatrix}
		0 & 1 & 0 \\
		-1 & 0 & c \\
		\times & \times & 0
	\end{bmatrix},
	\qquad \text{and}\qquad
	\bK =\bI + c\cdot\bE_{j1}=
	\begin{bmatrix}
		1 & 0 & 0 \\
		0 & 1 & 0 \\
		c &0 & 1
	\end{bmatrix},
	$$
	where $j=3$ for this specific example. This results in
	$$
	\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top =
	\begin{bmatrix}
		1 & 0 & 0 \\
		0 & 1 & 0 \\
		c & 0 & 1
	\end{bmatrix}
	\begin{bmatrix}
		0 & 1 & 0 \\
		-1 & 0 & \textcolor{blue}{c} \\
		\times & \times & 0
	\end{bmatrix}
	\begin{bmatrix}
		1 & 0 & c \\
		0 & 1 & 0 \\
		0 & 0 & 1
	\end{bmatrix}
	=
	\begin{bmatrix}
		0 & 1 & 0 \\
		-1 & 0 & \textcolor{blue}{0} \\
		\times & \times & 0
	\end{bmatrix}.
	$$
Since we have shown that $\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top$ is also skew-symmetric, then, it is actually
$$
\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top=
\begin{bmatrix}
	0 & 1 & 0 \\
	-1 & 0 & \textcolor{blue}{0} \\
	\textcolor{red}{0} & \textcolor{red}{0} & 0
\end{bmatrix},
$$
so that we do not need to tackle the first 2 columns of the above equation.
\end{mdframed}
Apply this process for the bottom-right $(n-2)\times(n-2)$ submatrix, we will complete the proof.

\paragraph{Case 2).} Suppose the first row of $\bA$ is zero, a permutation matrix to put the first row into the last row and apply the process in case 1 to finish the proof.
\end{proof}

From the block-diagonalization of skew-symmetric matrices above, we could easily find that the rank of a skew-symmetric matrix is even.  And we could prove the determinant of skew-symmetric with even order is nonnegative as follows.
	\begin{lemma}[Even Skew-Symmetric Determinant]\label{lemma:skew-symmetric-determinant-even}
		For skew-symmetric matrix $\bA\in \real^{n\times n}$, if $n$ is even, then $\det(\bA)\geq 0$.
	\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:skew-symmetric-determinant-even}]
By Theorem~\ref{theorem:skew-block-diagonalization_theorem}, we could block-diagonalize $\bA = \bZ\bD\bZ^\top$ such that
$$
\det(\bA) = \det(\bZ\bD\bZ^\top) = \det(\bZ)^2 \det(\bD) \geq 0.
$$
This completes the proof.
\end{proof}

\section{Applications}
\subsection{Application: Eigenvalue of Projection Matrix}

In Section~\ref{section:application-ls-qr} (p.~\pageref{section:application-ls-qr}), we introduced the QR decomposition can be applied to solve the least squares problem, where we consider the overdetermined system $\bA\bx = \bb$ with $\bA\in \real^{m\times n}$ being the data matrix, $\bb\in \real^m$ with $m>n$ being the observation matrix. Normally $\bA$ will have full column rank since the data from real work has a large chance to be unrelated. And the least squares solution is given by $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ for minimizing $||\bA\bx-\bb||^2$, where $\bA^\top\bA$ is invertible since $\bA$ has full column rank and $rank(\bA^T\bA)=rank(\bA)$. The recovered observation matrix is then $\hat{\bb} = \bA\bx_{LS} = \bA(\bA^\top\bA)^{-1}\bA^\top\bb$. $\bb$ may not be in the column space of $\bA$, but the recovered $\hat{\bb}$ is in this column space. We then define such matrix $\bH=\bA(\bA^\top\bA)^{-1}\bA^\top$ to be a projection matrix \footnote{A detailed ananlysis of orthogonal projection is provided in Appendix~\ref{section:by-geometry-hat-matrix} (p.~\pageref{section:by-geometry-hat-matrix}).}, i.e., projecting $\bb$ onto the column space of $\bA$. Or, it is also known as hat matrix, since we put a hat on $\bb$. It can be easily verified the projection matrix is symmetric and idempotent (i.e., $\bH^2=\bH$). \index{Projection matrix}
\begin{remark}[Column Space of Projection Matrix]
	We notice that the hat matrix $\bH = \bA(\bA^\top\bA)^{-1}\bA^\top$ is to project any vector in $\real^m$ into the column space of $\bA$. That is, $\bH\by \in \cspace(\bA)$. Notice again $\bH\by$ is the nothing but a combination of the columns of $\bH$, thus $\cspace(\bH) = \cspace(\bA)$.

	In general, for any projection matrix $\bH$ to project vector onto subspace $\mathcalV$, then $\cspace(\bH) = \mathcalV$. More formally, in a mathematical language, this property can be proved by SVD.
\end{remark}

We now show that for any projection matrix, it has specific eigenvalues. See Appendix~\ref{appendix:orthogonal} for a detailed discussion on the orthogonal projection.
\begin{proposition}[Eigenvalue of Projection Matrix]\label{proposition:eigen-of-projection-matrix}
The only possible eigenvalues of a projection matrix are 0 and 1.
\end{proposition}
\begin{proof}[of Proposition~\ref{proposition:eigen-of-projection-matrix}]
Since $\bH$ is symmetric, we have spectral decomposition $\bH =\bQ\bLambda\bQ^\top$. From the idempotent property, we have
$$
\begin{aligned}
(\bQ\bLambda\bQ^\top)^2 &= \bQ\bLambda\bQ^\top \\
\bQ\bLambda^2\bQ^\top &= \bQ\bLambda\bQ^\top \\
\bLambda^2 &=\bLambda \\
\lambda_i^2 &=\lambda_i,
\end{aligned}
$$
Therefore, the only possible eigenvalues for $\bH$ are 0 and 1.
\end{proof}

This property of the projection matrix is important for the analysis of distribution theory for linear models. See \citep{lu2021rigorous} for more details.
Following from the eigenvalue of the projection matrix, it can also give rise to the perpendicular projection $\bI-\bH$.
\begin{proposition}[Project onto $\mathcalV^\perp$]\label{proposition:orthogonal-projection_tmp}
	Let $\mathcalV$ be a subspace and $\bH$ be a projection onto $\mathcalV$. Then $\bI-\bH$ is the projection matrix onto $\mathcalV^\perp$.
\end{proposition}

\begin{proof}[of Proposition~\ref{proposition:orthogonal-projection_tmp}]
	First, $(\bI-\bH)$ is symmetric, $(\bI-\bH)^\top = \bI - \bH^\top = \bI-\bH$ since $\bH$ is symmatrix. And
	$$
	(\bI-\bH)^2 = \bI^2 -\bI\bH -\bH\bI +\bH^2 = \bI-\bH.
	$$
	Thus $\bI-\bH$ is a projection matrix. By spectral theorem again, let $\bH =\bQ\bLambda\bQ^\top$. Then $\bI-\bH = \bQ\bQ^\top - \bQ\bLambda\bQ^\top = \bQ(\bI-\bLambda)\bQ^\top$. Hence the column space of $\bI-\bH$ is spanned by the eigenvectors of $\bH$ corresponding to the zero eigenvalues of $\bH$ (by Proposition~\ref{proposition:eigen-of-projection-matrix}, p.~\pageref{proposition:eigen-of-projection-matrix}), which coincides with $\mathcalV^\perp$.
\end{proof}

Again, for a detailed analysis of the origin of the projection matrix and results behind the projection matrix, we highly recommend the readers refer to Appendix~\ref{appendix:orthogonal} although it is not the main interest of matrix decomposition results.

\subsection{Application: An Alternative Definition on PD and PSD of Matrices}\label{section:equivalent-pd-psd}
In Definition~\ref{definition:psd-pd-defini} (p.~\pageref{definition:psd-pd-defini}), we defined the positive definite matrices and positive semidefinite matrices by the quadratic form of the matrices. We here prove that a symmetric matrix is positive definite if and only if all eigenvalues are positive.

\begin{lemma}[Eigenvalues of PD and PSD Matrices\index{Positive definite}\index{Positive semidefinite}]\label{lemma:eigens-of-PD-psd}
A matrix $\bA\in \real^{n\times n}$ is positive definite (PD) if and only if $\bA$ has only positive eigenvalues.
And a matrix $\bA\in \real^{n\times n}$ is positive semidefinite (PSD) if and only if $\bA$ has only nonnegative eigenvalues.
\end{lemma}

\begin{proof}[of Lemma~\ref{lemma:eigens-of-PD-psd}]
We will prove by forward implication and reverse implication separately as follows.
\paragraph{Forward implication:}
	Suppose $\bA$ is PD, then for any eigenvalue $\lambda$ and its corresponding eigenvector $\bv$ of $\bA$, we have $\bA\bv = \lambda\bv$. Thus
$$
\bv^\top \bA\bv = \lambda||\bv||^2 > 0.
$$
This implies $\lambda>0$.
\paragraph{Reverse implication:}
Conversely, suppose the eigenvalues are positive. By spectral decomposition of $\bA =\bQ\bLambda \bQ^\top$. If $\bx$ is a nonzero vector, let $\by=\bQ^\top\bx$, we have
$$
\bx^\top \bA \bx = \bx^\top (\bQ\bLambda \bQ^\top) \bx = (\bx^\top \bQ) \bLambda (\bQ^\top\bx) = \by^\top\bLambda\by = \sum_{i=1}^{n} \lambda_i y_i^2>0.
$$
That is, $\bA$ is PD.

Analogously, we can prove the second part of the claim.
\end{proof}

\begin{theoremHigh}[Nonsingular Factor of PSD and PD Matrices]\label{lemma:nonsingular-factor-of-PD}
A real symmetric matrix $\bA$ is PSD if and only if $\bA$ can be factored as $\bA=\bP^\top\bP$, and is PD if and only if $\bP$ is nonsingular.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{lemma:nonsingular-factor-of-PD}]
For the first part, we will prove by forward implication and reverse implication separately as follows.
\paragraph{Forward implication: }Suppose $\bA$ is PSD, its spectral decomposition is given by $\bA = \bQ\bLambda\bQ^\top$. Since eigenvalues of PSD matrices are nonnegative, we can decompose $\bLambda=\bLambda^{1/2}\bLambda^{1/2}$. Let $\bP = \bLambda^{1/2}\bQ^\top$, we can decompose $\bA$ by $\bA=\bP^\top\bP$.

\paragraph{Reverse implication: } If $\bA$ can be factored as $\bA=\bP^\top\bP$, then all eigenvalues of $\bA$ are nonnegative since for any eigenvalues $\lambda$ and its corresponding eigenvector $\bv$ of $\bA$, we have
$$
\lambda = \frac{\bv^\top\bA\bv}{\bv^\top\bv} = \frac{\bv^\top\bP^\top\bP\bv}{\bv^\top\bv}=\frac{||\bP\bv||^2}{||\bv||^2} \geq 0.
$$
This implies $\bA$ is PSD by Lemma~\ref{lemma:eigens-of-PD-psd}.

Similarly, we can prove the second part for PD matrices where the positive definiteness will result in the nonsingular $\bP$ and the nonsingular $\bP$ will result in the positiveness of the eigenvalues. \footnote{See also wiki page: https://en.wikipedia.org/wiki/Sylvester's\_criterion.}
\end{proof}

\subsection{Proof for Semidefinite Rank-Revealing Decomposition}\label{section:semi-rank-reveal-proof}

In this section, we provide a proof for Theorem~\ref{theorem:semidefinite-factor-rank-reveal} (p.~\pageref{theorem:semidefinite-factor-rank-reveal}), the existence of the rank-revealing decomposition for positive semidefinite matrix.\index{Rank-revealing}\index{Semidefinite rank-revealing}
\begin{proof}[of Theorem~\ref{theorem:semidefinite-factor-rank-reveal}]
The proof is a consequence of the nonsingular factor of PSD matrices (Theorem~\ref{lemma:nonsingular-factor-of-PD}, p.~\pageref{lemma:nonsingular-factor-of-PD}) and the existence of column-pivoted QR decomposition (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}).

By Theorem~\ref{lemma:nonsingular-factor-of-PD}, the nonsingular factor of PSD matrix $\bA$ is given by $\bA = \bZ^\top\bZ$, where $\bZ=\bLambda^{1/2}\bQ^\top$ and $\bA=\bQ\bLambda\bQ^\top$ is the spectral decomposition of $\bA$.

By Lemma~\ref{lemma:rank-of-symmetric}, the rank of matrix $\bA$ is the number of nonzero eigenvalues (here the number of positive eigenvalues since $\bA$ is PSD). Therefore only $r$ components in $\bLambda^{1/2}$ are nonzero, and $\bZ=\bLambda^{1/2}\bQ^\top$ contains only $r$ independent columns, i.e., $\bZ$ is of rank $r$. By column-pivoted QR decomposition, we have
$$
\bZ\bP = \bQ
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix},
$$
where $\bP$ is a permutation matrix, $\bR_{11}\in \real^{r\times r}$ is upper triangular with positive diagonals, and $\bR_{12}\in \real^{r\times (n-r)}$. Therefore
$$
\bP^\top\bA\bP  =
\bP^\top\bZ^\top\bZ\bP =
\begin{bmatrix}
\bR_{11}^\top & \bzero \\
\bR_{12}^\top & \bzero
\end{bmatrix}
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix}.
$$
Let
$$
\bR = \begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix},
$$
we find the rank-revealing decomposition for semidefinite matrix $\bP^\top\bA\bP = \bR^\top\bR$.
\end{proof}

This decomposition is produced by using complete pivoting, which at each stage permutes the largest diagonal element
in the active submatrix into the pivot position. The procedure is similar to the partial pivoting discussed in Section~\ref{section:partial-pivot-lu} (p.~\pageref{section:partial-pivot-lu}).

\subsection{Application: Cholesky Decomposition via the QR Decomposition and the Spectral Decomposition}\label{section:cholesky-by-qr-spectral}
In this section, we provide another proof for the existence of the Cholesky decomposition.
\begin{theoremHigh}[Cholesky Decomposition: A Simpler Version of Theorem~\ref{theorem:cholesky-factor-exist}]\label{lemma:cholesky-former-one}
Every positive definite matrix $\bA\in \real^{n\times n}$ can be factored as
$$
\bA = \bR^\top\bR,
$$
where $\bR$ is an upper triangular matrix with positive diagonals.

\end{theoremHigh}
\begin{proof}[of Theorem~\ref{lemma:cholesky-former-one}]
From Theorem~\ref{lemma:nonsingular-factor-of-PD}, the PD matrix $\bA$ can be factored as $\bA=\bP^\top\bP$ where $\bP$ is a nonsingular matrix. Then, the QR decomposition of $\bP$ is given by $\bP = \bQ\bR$. This implies
$$
\bA = \bP^\top\bP = \bR^\top\bQ^\top\bQ\bR = \bR^\top\bR,
$$
where we notice that the form is very similar to the Cholesky decomposition except that we do not claim the $\bR$ has only positive diagonal values. From Algorithm~\ref{alg:reduced-qr}, the existence of QR decomposition via the Gram-Schmidt process, we realize that the diagonals of $\bR$ are nonnegative, and if $\bP$ is nonsingular, the diagonals of $\bR$ are also positive.
\end{proof}
The proof for the above theorem is a consequence of the existence of both the QR decomposition and the spectral decomposition. Thus, the existence of Cholesky decomposition can be proved via the QR decomposition and the spectral decomposition in this sense.

\subsection{Application: Unique Power Decomposition of Positive Definite Matrices}\label{section:unique-posere-pd}
\begin{theoremHigh}[Unique Power Decomposition of PD Matrices]\label{theorem:unique-factor-pd}
Any $n\times n$ positive matrix $\bA$ can be \textbf{uniquely} factored as a product of a positive definite matrix $\bB$ such that $\bA =\bB^2$.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:unique-factor-pd}]
We first prove that there exists such positive definite matrix $\bB$ so that $\bA = \bB^2$.
\paragraph{Existence} Since $\bA$ is PD which is also symmetric, the spectral decomposition of $\bA$ is given by $\bA = \bQ\bLambda\bQ^\top$. Since eigenvalues of PD matrices are positive by Lemma~\ref{lemma:eigens-of-PD-psd}, the square root of $\bLambda$ exists. We can define $\bB = \bQ\bLambda^{1/2}\bQ^\top$ such that $\bA = \bB^2$ where $\bB$ is apparently PD.

\paragraph{Uniqueness} Suppose such factorization is not unique, then there exist two of this decomposition such that
$$
\bA = \bB_1^2 = \bB_2^2,
$$
where $\bB_1$ and $\bB_2$ are both PD. The spectral decompositions of them are given by
$$
\bB_1 = \bQ_1 \bLambda_1\bQ_1^\top, \qquad \text{and} \qquad \bB_2 = \bQ_2 \bLambda_2\bQ_2^\top.
$$
We notice that $\bLambda_1^2$ and $\bLambda_2^2$ contains the eigenvalues of $\bA$, and both eigenvalues of $\bB_1$ and $\bB_2$ contained in $\bLambda_1$ and $\bLambda_2$ are positive (since $\bB_1$ and $\bB_2$ are both PD). Without loss of generality, we suppose $\bLambda_1=\bLambda_2=\bLambda^{1/2}$, and $\bLambda=\diag(\lambda_1,\lambda_2, \ldots, \lambda_n)$ such that $\lambda_1\geq \lambda_2 \geq \ldots \geq \lambda_n$. By $\bB_1^2 = \bB_2^2$, we have
$$
\bQ_1 \bLambda \bQ_1^\top = \bQ_2 \bLambda \bQ_2^\top  \leadto \bQ_2^\top\bQ_1 \bLambda = \bLambda \bQ_2^\top\bQ_1.
$$
Let $\bZ = \bQ_2^\top\bQ_1 $, this implies $\bLambda$ and $\bZ$ commute, and $\bZ$ must be a block diagonal matrix whose partitioning conforms to the block structure of $\bLambda$. This results in $\bLambda^{1/2} = \bZ\bLambda^{1/2}\bZ^\top$ and
$$
\bB_2 = \bQ_2 \bLambda^{1/2}\bQ_2^\top = \bQ_2 \bQ_2^\top\bQ_1\bLambda^{1/2} \bQ_1^\top\bQ_2 \bQ_2^\top=\bB_1.
$$
This completes the proof.
\end{proof}
Similarly, we could prove the unique decomposition of PSD matrix $\bA =  \bB^2$ where $\bB$ is PSD. A more detailed discussion on this topic can be referred to \citep{koeber2006unique}.

\paragraph{Decomposition for PD matrices} To conclude, for PD matrix $\bA$, we can factor it into $\bA=\bR^\top\bR$ where $\bR$ is an upper triangular matrix with positive diagonals as shown in Theorem~\ref{theorem:cholesky-factor-exist} by Cholesky decomposition, $\bA = \bP^\top\bP$ where $\bP$ is nonsingular in Theorem~\ref{lemma:nonsingular-factor-of-PD}, and $\bA = \bB^2$ where $\bB$ is PD in Theorem~\ref{theorem:unique-factor-pd}.

\chapter{Singular Value Decomposition (SVD)}\label{section:SVD}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Singular Value Decomposition}
In eigenvalue decomposition, we factor the matrix into a diagonal matrix. However, this is not always true. If $\bA$ does not have linearly independent eigenvectors, such diagonalization does not exist. The singular value decomposition (SVD) fills this gap. Instead of factoring the matrix into an eigenvector matrix, SVD gives rise to two orthogonal matrices. We provide the result of SVD in the following theorem and we will discuss the existence of SVD in the next sections.

\begin{theoremHigh}[Reduced SVD for Rectangular Matrices]\label{theorem:reduced_svd_rectangular}
For every real $m\times n$ matrix $\bA$ with rank $r$, then matrix $\bA$ can be factored as
$$
\bA = \bU \bSigma \bV^\top,
$$
where $\bSigma\in \real^{r\times r}$ is a diagonal matrix $\bSigma=diag(\sigma_1, \sigma_2 \ldots, \sigma_r)$ with $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$ and
\begin{itemize}
\item $\sigma_i$'s are the nonzero \textbf{singular values} of $\bA$, in the meantime, they are the (positive) square roots of the nonzero \textbf{eigenvalues} of $\trans{\bA} \bA$ and $ \bA \trans{\bA}$.

\item Columns of $\bU\in \real^{m\times r}$ contain the $r$ eigenvectors of $\bA\bA^\top$ corresponding to the $r$ nonzero eigenvalues of $\bA\bA^\top$.

\item Columns of $\bV\in \real^{n\times r}$ contain the $r$ eigenvectors of $\bA^\top\bA$ corresponding to the $r$ nonzero eigenvalues of $\bA^\top\bA$.

\item Moreover, the columns of $\bU$ and $\bV$ are called the \textbf{left and right singular vectors} of $\bA$, respectively.

\item Further, the columns of $\bU$ and $\bV$ are orthonormal (by Spectral Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}).
\end{itemize}

In particular, we can write out the matrix decomposition by the sum of outer products of vectors $\bA = \bU \bSigma \bV^\top = \sum_{i=1}^r \sigma_i \bu_i \bv_i^\top$, which is a sum of $r$ rank-one matrices.

\end{theoremHigh}

If we append additional $m-r$ silent columns that are orthonormal to the $r$ eigenvectors of $\bA\bA^\top$, just like the silent columns in the QR decomposition, we will have an orthogonal matrix $\bU\in \real^{m\times m}$. Similar situation for the columns of $\bV$.
We then illustrate the full SVD for matrices in the following theorem. We formulate the difference between reduced and full SVD in the \textcolor{blue}{blue} text.
\begin{theoremHigh}[Full SVD for Rectangular Matrices]\label{theorem:full_svd_rectangular}
For every real $m\times n$ matrix $\bA$ with rank $r$, then matrix $\bA$ can be factored as
$$
\bA = \bU \bSigma \bV^\top,
$$
where the left-upper side of $\bSigma\in $\textcolor{blue}{$\real^{m\times n}$} is a diagonal matrix, that is $\bSigma=\begin{bmatrix}
	\bSigma_1 & \bzero \\
	\bzero & \bzero
\end{bmatrix}$ where $\bSigma_1=diag(\sigma_1, \sigma_2 \ldots, \sigma_r)\in \real^{r\times r}$ with $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$ and
\begin{itemize}
\item $\sigma_i$'s are the nonzero \textbf{singular values} of matrix $\bA$, in the meantime, they are the (positive) square roots of the nonzero \textbf{eigenvalues} of $\trans{\bA} \bA$ and $ \bA \trans{\bA}$.

\item $\bU\in \textcolor{blue}{\real^{m\times m}}$ contains the $r$ eigenvectors of $\bA\bA^\top$ corresponding to the $r$ nonzero eigenvalues of $\bA\bA^\top$ \textcolor{blue}{and $m-r$ extra orthonormal vectors from $\nspace(\bA^\top)$}.

\item $\bV\in \textcolor{blue}{\real^{n\times n}}$ contains the $r$ eigenvectors of $\bA^\top\bA$ corresponding to the $r$ nonzero eigenvalues of $\bA^\top\bA$ \textcolor{blue}{and $n-r$ extra orthonormal vectors from $\nspace(\bA)$}.

\item Moreover, the columns of $\bU$ and $\bV$ are called the \textbf{left and right singular vectors} of $\bA$, respectively.

\item Further, the columns of $\bU$ and $\bV$ are orthonormal (by Spectral Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}), and \textcolor{blue}{$\bU$ and $\bV$ are orthogonal matrices}.
\end{itemize}

In particular, we can write the matrix decomposition by the sum of outer products of vectors $ \bA = \bU \bSigma \bV^\top = \sum_{i=1}^r \sigma_i \bu_i \bv_i^\top$, which is a sum of $r$ rank-one matrices.
\end{theoremHigh}

The comparison between the reduced and the full SVD is shown in Figure~\ref{fig:svd-comparison} where white entries are zero and blue entries are not necessarily zero.
\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Reduced SVD decomposition]{\label{fig:svdhalf}
		\includegraphics[width=0.47\linewidth]{./imgs/svdreduced.pdf}}
	\quad
	\subfigure[Full SVD decomposition]{\label{fig:svdall}
		\includegraphics[width=0.47\linewidth]{./imgs/svdfull.pdf}}
	\caption{Comparison between the reduced and full SVD.}
	\label{fig:svd-comparison}
\end{figure}

\section{Existence of the SVD}
To prove the existence of the SVD, we need to use the following lemmas. We mentioned that the singular values are the square roots of the eigenvalues of $\bA^\top\bA$. While, negative values do not have square roots such that the eigenvalues must be nonnegative.
\begin{lemma}[Nonnegative Eigenvalues of $\bA^\top \bA$]\label{lemma:nonneg-eigen-ata}
For any matrix $\bA\in \real^{m\times n}$, $\bA^\top \bA$ has nonnegative eigenvalues.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:nonneg-eigen-ata}]
For eigenvalue and its corresponding eigenvector $\lambda, \bx$ of $\bA^\top \bA$, we have
$$
\bA^\top \bA \bx = \lambda \bx \leadto \bx^\top \bA^\top \bA \bx = \lambda \bx^\top\bx.
$$
Since $\bx^\top \bA^\top \bA \bx  = ||\bA \bx||^2 \geq 0$ and $\bx^\top\bx \geq 0$. We then have $\lambda \geq 0$.
\end{proof}

Since $\bA^\top\bA$ has nonnegative eigenvalues, we then can define the singular value $\sigma\geq 0$ of $\bA$ such that $\sigma^2$ is the eigenvalue of $\bA^\top\bA$, i.e., \fbox{$\bA^\top\bA \bv = \sigma^2 \bv$}. This is essential to the existence of the SVD.

We have shown in Lemma~\ref{lemma:rankAB} (p.~\pageref{lemma:rankAB}) that $rank$($\bA\bB$)$\leq$min$\{rank$($\bA$), $rank$($\bB$)\}.
However, the symmetric matrix $\bA^\top \bA$ is rather special in that the rank of $\bA^\top \bA$ is equal to $rank(\bA)$. And we now prove it.
\begin{lemma}[Rank of $\bA^\top \bA$]\label{lemma:rank-of-ata}
$\bA^\top \bA$ and $\bA$ have same rank.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-of-ata}]
Let $\bx\in \nspace(\bA)$, we have
$$
	\bA\bx  = \bzero \leadto \bA^\top\bA \bx =\bzero,
$$
i.e., $\bx\in \nspace(\bA) \leadtosmall \bx \in \nspace(\bA^\top \bA)$, therefore $\nspace(\bA) \subseteq \nspace(\bA^\top\bA)$.

Further, let $\bx \in \nspace(\bA^\top\bA)$, we have
$$
	\bA^\top \bA\bx = \bzero\leadtosmall \bx^\top \bA^\top \bA\bx = 0\leadtosmall ||\bA\bx||^2 = 0 \leadtosmall \bA\bx=\bzero,
$$
i.e., $\bx\in \nspace(\bA^\top \bA) \leadtosmall \bx\in \nspace(\bA)$, therefore $\nspace(\bA^\top\bA) \subseteq\nspace(\bA) $.

As a result, by ``sandwiching", it follows that
$$\nspace(\bA) = \nspace(\bA^\top\bA) \qquad
\text{and} \qquad
dim(\nspace(\bA)) = dim(\nspace(\bA^\top\bA)).
$$
By the fundamental theorem of linear algebra in Appendix~\ref{appendix:fundamental-rank-nullity} (p.~\pageref{appendix:fundamental-rank-nullity}), $\bA^\top \bA$ and $\bA$ have the same rank.
\end{proof}
Apply the observation to $\bA^\top$, we can also prove that $\bA\bA^\top$ and $\bA$ have the same rank:
$$
rank(\bA) = rank(\bA^\top \bA) = rank(\bA\bA^\top).
$$

In the form of the SVD, we claimed the matrix $\bA$ is a sum of $r$ rank-one matrices where $r$ is the number of nonzero singular values. And the number of nonzero singular values is actually the rank of the matrix.

\begin{lemma}[The Number of Nonzero Singular Values Equals the Rank]\label{lemma:rank-equal-singular}
The number of nonzero singular values of matrix $\bA$ equals the rank of $\bA$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-equal-singular}]
The rank of any symmetric matrix (here $\bA^\top\bA$) equals the number of nonzero eigenvalues (with repetitions) by Lemma~\ref{lemma:rank-of-symmetric} (p.~\pageref{lemma:rank-of-symmetric}). So the number of nonzero singular values equals the rank of $\bA^\top \bA$. By Lemma~\ref{lemma:rank-of-ata}, the number of nonzero singular values equals the rank of $\bA$.
\end{proof}

We are now ready to prove the existence of the SVD.
\begin{proof}[\textbf{of Theorem~\ref{theorem:reduced_svd_rectangular}: Existence of the SVD}]
Since $\bA^\top \bA$ is a symmetric matrix, by Spectral Theorem~\ref{theorem:spectral_theorem} (p.~\pageref{theorem:spectral_theorem}) and Lemma~\ref{lemma:nonneg-eigen-ata}, there exists an orthogonal matrix $\bV$ such that

$$
\boxed{\bA^\top \bA = \bV \bSigma^2 \bV^\top},
$$
where $\bSigma$ is a diagonal matrix containing the singular values of $\bA$, i.e., $\bSigma^2$ contains the eigenvalues of $\bA^\top \bA$.
Specifically, $\bSigma=diag(\sigma_1, \sigma_2, \ldots, \sigma_r)$ and $\{\sigma_1^2, \sigma_2^2, \ldots, \sigma_r^2\}$ are the nonzero eigenvalues of $\bA^\top \bA$ with $r$ being the rank of $\bA$. I.e., $\{\sigma_1, \ldots, \sigma_r\}$ are the singular values of $\bA$. In this case, $\bV\in \real^{n\times r}$.
Now we are into the central part.
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
Start from \fbox{$\bA^\top\bA \bv_i = \sigma_i^2 \bv_i$}, $\forall i \in \{1, 2, \ldots, r\}$, i.e., the eigenvector $\bv_i$ of $\bA^\top\bA$ corresponding to $\sigma_i^2$:

1. Multiply both sides by $\bv_i^\top$:
$$
\bv_i^\top\bA^\top\bA \bv_i = \sigma_i^2 \bv_i^\top \bv_i \leadto ||\bA\bv_i||^2 = \sigma_i^2 \leadto ||\bA\bv_i||=\sigma_i
$$

2. Multiply both sides by $\bA$:
$$
\bA\bA^\top\bA \bv_i = \sigma_i^2 \bA \bv_i \leadto \bA\bA^\top \frac{\bA \bv_i }{\sigma_i}= \sigma_i^2 \frac{\bA \bv_i }{\sigma_i} \leadto \bA\bA^\top \bu_i = \sigma_i^2 \bu_i
$$
where we notice that this form can find the eigenvector of $\bA\bA^\top$ corresponding to $\sigma_i^2$ which is $\bA \bv_i$. Since the length of $\bA \bv_i$ is $\sigma_i$, we then define $\bu_i = \frac{\bA \bv_i }{\sigma_i}$ with norm 1.
\end{mdframed}
These $\bu_i$'s are orthogonal because $(\bA\bv_i)^\top(\bA\bv_j)=\bv_i^\top\bA^\top\bA\bv_j=\sigma_j^2 \bv_i^\top\bv_j=0$. That is
$$
\boxed{\bA \bA^\top = \bU \bSigma^2 \bU^\top}.
$$
Since \fbox{$\bA\bv_i = \sigma_i\bu_i$}, we have
$$
[\bA\bv_1, \bA\bv_2, \ldots, \bA\bv_r] = [ \sigma_1\bu_1,  \sigma_2\bu_2, \ldots,  \sigma_r\bu_r]\leadto
\bA\bV = \bU\bSigma,
$$
which completes the proof.
\end{proof}
By appending silent columns in $\bU$ and $\bV$, we can easily find the full SVD. A byproduct of the above proof is that the spectral decomposition of $\bA^\top\bA = \bV \bSigma^2 \bV^\top$ will result in the spectral decomposition of $\bA \bA^\top = \bU \bSigma^2 \bU^\top$ with the same eigenvalues.

\begin{corollary}[Eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$]
The nonzero eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$ are the same.
\end{corollary}
We have shown in Lemma~\ref{lemma:nonneg-eigen-ata} that the eigenvalues of $\bA^\top \bA$ are nonnegative, such that the eigenvalues of $\bA\bA^\top$ are nonnegative as well.
	\begin{corollary}[Nonnegative Eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$]
		The eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$ are nonnegative.
	\end{corollary}

The existence of the SVD is important for defining the effective rank of a matrix.
\begin{definition}[Effective Rank vs Exact Rank]\label{definition:effective-rank-in-svd}
	\textit{Effective rank}, or also known as the \textit{numerical rank}.
	Following from Lemma~\ref{lemma:rank-equal-singular}, the number of nonzero singular values is equal to the rank of a matrix.
	Assume the $i$-th largest singular value of $\bA$ is denoted as $\sigma_i(\bA)$. Then if $\sigma_r(\bA)\gg \sigma_{r+1}(\bA)\approx 0$, $r$ is known as the numerical rank of $\bA$. Whereas, when $\sigma_i(\bA)>\sigma_{r+1}(\bA)=0$, it is known as having \textit{exact rank} $r$ as we have used in most of our discussions.
\end{definition}
\section{Properties of the SVD}\label{section:property-svd}
\subsection{Four Subspaces in SVD}\label{section:four-space-svd}
For any matrix $\bA\in  \real^{m\times n}$, we have the following property:

$\bullet$ $\nspace(\bA)$ is the orthogonal complement of the row space $\cspace(\bA^\top)$ in $\real^n$: $dim(\nspace(\bA))+dim(\cspace(\bA^\top))=n$;

$\bullet$ $\nspace(\bA^\top)$ is the orthogonal complement of the column space $\cspace(\bA)$ in $\real^m$: $dim(\nspace(\bA^\top))+dim(\cspace(\bA))=m$;

This is called the fundamental theorem of linear algebra and is also known as the rank-nullity theorem. And the proof can be found in Appendix~\ref{appendix:fundamental-rank-nullity}. Furthermore, we find the basis for the four subspaces via the CR decomposition in Appendix~\ref{appendix:cr-decomposition-four-basis}.
In specific, from the SVD, we can find an orthonormal basis for each subspace.

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.98\textwidth]{imgs/lafundamental3-SVD.pdf}
	\caption{Orthonormal bases that diagonalize $\bA$ from SVD.}
	\label{fig:lafundamental3-SVD}
\end{figure}

\begin{lemma}[Four Orthonormal Basis]\label{lemma:svd-four-orthonormal-Basis}
Given the full SVD of matrix $\bA = \bU \bSigma \bV^\top$, where $\bU=[\bu_1, \bu_2, \ldots,\bu_m]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU$ and $\bV$. Then, we have the following property:

$\bullet$ $\{\bv_1, \bv_2, \ldots, \bv_r\} $ is an orthonormal basis of $\cspace(\bA^\top)$;

$\bullet$ $\{\bv_{r+1},\bv_{r+2}, \ldots, \bv_n\}$ is an orthonormal basis of $\nspace(\bA)$;

$\bullet$ $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is an orthonormal basis of $\cspace(\bA)$;

$\bullet$ $\{\bu_{r+1}, \bu_{r+2},\ldots,\bu_m\}$ is an orthonormal basis of $\nspace(\bA^\top)$.

The relationship of the four subspaces is demonstrated in Figure~\ref{fig:lafundamental3-SVD} where $\bA$ transfer the row basis $\bv_i$ into column basis $\bu_i$ by $\sigma_i\bu_i=\bA\bv_i$ for all $i\in \{1, 2, \ldots, r\}$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:svd-four-orthonormal-Basis}]
From Lemma~\ref{lemma:rank-of-symmetric}, for symmetric matrix $\bA^\top\bA$, $\cspace(\bA^\top\bA)$ is spanned by the eigenvectors, thus $\{\bv_1,\bv_2, \ldots, \bv_r\}$ is an orthonormal basis of $\cspace(\bA^\top\bA)$.

Since,

1. $\bA^\top\bA$ is symmetric, then the row space of $\bA^\top\bA$ equals the column space of $\bA^\top\bA$.

2. All rows of $\bA^\top\bA$ are the combinations of the rows of $\bA$, so the row space of $\bA^\top\bA$ $\subseteq$ the row space of $\bA$, i.e., $\cspace(\bA^\top\bA) \subseteq \cspace(\bA^\top)$.

3. Since $rank(\bA^\top\bA) = rank(\bA)$ by Lemma~\ref{lemma:rank-of-ata}, we then have

The row space of $\bA^\top\bA$ = the column space of $\bA^\top\bA$ =  the row space of $\bA$, i.e., $\cspace(\bA^\top\bA) = \cspace(\bA^\top)$. Thus $\{\bv_1, \bv_2,\ldots, \bv_r\}$ is an orthonormal basis of $\cspace(\bA^\top)$.

Further, the space spanned by $\{\bv_{r+1}, \bv_{r+2},\ldots, \bv_n\}$ is an orthogonal complement to the space spanned by $\{\bv_1,\bv_2, \ldots, \bv_r\}$, so $\{\bv_{r+1},\bv_{r+2}, \ldots, \bv_n\}$ is an orthonormal basis of $\nspace(\bA)$.

If we apply this process to $\bA\bA^\top$, we will prove the rest claims in the lemma. Also, we can see that $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is a basis for the column space of $\bA$ by Lemma~\ref{lemma:column-basis-from-row-basis} \footnote{For any matrix $\bA$, let $\{\br_1, \br_2, \ldots, \br_r\}$ be a set of vectors in $\real^n$ which forms a basis for the row space, then $\{\bA\br_1, \bA\br_2, \ldots, \bA\br_r\}$ is a basis for the column space of $\bA$.}, since $\bu_i = \frac{\bA\bv_i}{\sigma_i},\, \forall i \in\{1, 2, \ldots, r\}$.
\end{proof}

\subsection{SVD-Related Orthogonal Projections}\index{Orthogonal projection}
Following from the four subspaces in SVD, there are several important orthogonal projections associated with the SVD. A detailed discussion on the orthogonal projection is provided in Appendix~\ref{section:by-geometry-hat-matrix} (p.~\pageref{section:by-geometry-hat-matrix}). In words, the orthogonal projection matrices are matrices that are symmetric and idempotent. The projection matrix will project any vector onto the column space of it. The idempotency has a geometrical meaning such that projecting twice is equivalent to projecting once. The symmetry also has a geometrical meaning such that the distance between the original vector and the projected vector (which is in the column space of the projection matrix) is minimal. Suppose $\bA=\bU\bSigma\bV^\top$ is the SVD of $\bA$ with rank $r$. If we have the following column partitions
\[
\begin{blockarray}{ccc}
	\begin{block}{c[cc]}
\bU=&	\bU_r & \bU_m   \\
	\end{block}
	&m\times r & m\times (m-r)   \\
\end{blockarray}
,\qquad
\begin{blockarray}{ccc}
	\begin{block}{c[cc]}
\bV=	&	\bV_r & \bV_n   \\
	\end{block}
	&n\times r & n\times (n-r)   \\
\end{blockarray},
\]
where $\bU_r$ and $\bV_r$ are the first $r$ columns of $\bU$ and $\bV$.
Then, the four orthogonal projections can be obtained by
$$
\begin{aligned}
\bV_r\bV_r^\top &= \text{projection onto $\cspace(\bA^\top)$},\\
\bV_n\bV_n^\top &=\text{projection onto $\nspace(\bA)$},\\
\bU_r\bU_r^\top &= \text{projection onto $\cspace(\bA)$},\\
\bU_m\bU_m^\top &= \text{projection onto $\nspace(\bA^\top)$}.\\
\end{aligned}
$$

\subsection{Relationship between Singular Values and Determinant}
Let $\bA\in \real^{n\times n}$ be a square matrix and the singular value decomposition of $\bA$ is given by $\bA = \bU\bSigma\bV^\top$, it follows that
$$
|\det(\bA)| = |\det(\bU\bSigma\bV^\top)| = |\det(\bSigma)| = \sigma_1 \sigma_2\ldots \sigma_n.
$$
If all the singular values $\sigma_i$ are nonzero, then $\det(\bA)\neq 0$. That is, $\bA$ is \textbf{nonsingular}. If there is at least one singular value such that $\sigma_i =0$, then $\det(\bA)=0$, and $\bA$ does not have full rank, and is not invertible. Then the matrix is called \textbf{singular}. This is why $\sigma_i$'s are known as the singular values.

\subsection{Orthogonal Equivalence}
We have defined in Definition~\ref{definition:similar-matrices} (p.~\pageref{definition:similar-matrices}) that $\bA$ and $\bP\bA\bP^{-1}$ are similar matrices for any nonsingular matrix $\bP$ and square matrix $\bA$. The orthogonal equivalence is defined in a similar way for rectangular matrices.

\begin{definition}[Orthogonal Equivalent Matrices\index{Orthogonal equivalent matrices}]
For any orthogonal matrices $\bU$ and $\bV$, the matrices $\bA$ and $\bU\bA\bV$ are called \textit{orthogonal equivalent matrices}. Or \textit{unitary equivalent} in complex domain when $\bU$ and $\bV$ are unitary.
\end{definition}
Then, we have the following property for orthogonal equivalent matrices.

\begin{lemma}[Orthogonal Equivalent Matrices]\label{lemma:orthogonal-equivalent-matrix}
For any orthogonal equivalent matrices $\bA$ and $\bB$, then singular values are the same.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:orthogonal-equivalent-matrix}]
Since $\bA$ and $\bB$ are orthogonal equivalent, there exist orthogonal matrices that $\bB = \bU\bA\bV$. We then have
$$
\bB\bB^\top = (\bU\bA\bV)(\bV^\top\bA^\top\bU^\top) = \bU\bA\bA^\top\bU^\top.
$$
This implies $\bB\bB^\top$ and $\bA\bA^\top$ are similar matrices. By Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}), the eigenvalues of similar matrices are the same, which proves the singular values of $\bA$ and $\bB$ are the same.
\end{proof}

\subsection{SVD for QR}
	\begin{lemma}[Orthogonal Equivalent Matrices]\label{lemma:svd-for-qr}
	Suppose the full QR decomposition for matrix $\bA\in \real^{m\times n}$ with $m\geq n$ is given by $\bA=\bQ\bR$ where $\bQ\in \real^{m\times m}$ is orthogonal and $\bR\in \real^{m\times n}$ is upper triangular. Then $\bA$ and $\bR$ have the same singular values and right singular vectors.
	\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:svd-for-qr}]
We notice that $\bA^\top\bA = \bR^\top\bR$ such that $\bA^\top\bA$ and $\bR^\top\bR$ have the same eigenvalues and eigenvectors, i.e., $\bA$ and $\bR$ have the same singular values and right singular vectors (i.e., the eigenvectors of $\bA^\top\bA$ or $\bR^\top\bR$).
\end{proof}

The above lemma is important show the existence and properties of the rank-revealing QR decomposition (Section~\ref{section:rank-one-qr-revealing}, p.~\pageref{section:rank-one-qr-revealing}).
And it implies that an SVD of a matrix can be constructed by the QR decomposition of itself. Suppose the QR decomposition of $\bA$ is given by $\bA=\bQ\bR$ and the SVD of $\bR$ is given by $\bR=\bU_0 \bSigma\bV^\top$. Therefore, the SVD of $\bA$ can be obtained by
$$
\bA =\underbrace{ \bQ\bU_0}_{\bU} \bSigma\bV^\top.
$$

\subsection{Interlacing Property}
The interlacing property of SVD is from that of the symmetric matrix. We provide the theorem directly and the proof can be found in \citep{wilkinson1971algebraic} and further discussed in \citep{golub2013matrix} (Theorem 8.1.7, p. 443). \index{Interlacing property}
\begin{theorem}[Interlacing Property for Symmetric Matrix]
Suppose $\bA\in \real^{n\times n}$ is symmetric, and $\bA_r$ is defined as the upper left $r\times r$ submatrix of $\bA$ such that $\bA_r=\bA[1:r, 1:r]$. Define $\lambda_i(\bB)$ the $i$-th largest eigenvalue of matrix $\bB$. Let $k=r+1$ and $\bB_k=\bA_{k}$, then we have
$$
\lambda_{r+1}(\bB_{k})\leq \lambda_r(\bA_r) \leq \lambda_r(\bB_k)\leq \ldots \leq \lambda_2(\bB_{k})\leq \lambda_1 (\bA_r)\leq \lambda_1(\bB_k).
$$
\end{theorem}
The interlacing property of for singular value can be derived directly from that of the symmetric matrices:
\begin{theorem}[Interlacing Property for Singular Values]\label{theorem:interlacing-singular}
Suppose $\bA=[\ba_1, \ba_2, \ldots, \ba_n]\in \real^{m\times n}$ with $m\geq n$, and $\bA_r=[\ba_1, \ba_2, \ldots, \ba_r]$. Define $\sigma_i(\bB)$ as the $i$-th largest singular value of matrix $\bB$.

s symmetric, and $\bA_r$ is defined as the upper left $r\times r$ submatrix of $\bA$ such that $\bA_r=\bA[1:r, 1:r]$. Define $\lambda_i(\bB)$ the $i$-th largest eigenvalue of matrix $\bB$. Let $k=r+1$ and $\bB_k=\bA_{k}$, then we have
$$
\sigma_{k}(\bB_{k})\leq \sigma_r(\bA_r) \leq \sigma_r(\bB_k)\leq \ldots \leq \sigma_2(\bB_{k})\leq \sigma_1 (\bA_r)\leq \sigma_1(\bB_k).
$$
\end{theorem}

\section{Computing the SVD}\label{section:comput-svd-in-svd}
Suppose again we have an oracle algorithm to compute the eigenvalues and eigenvectors of $\bA^\top\bA$ which costs $f(m,n)$ flops. Then the computation of SVD is trivial from the steps shown above. The algorithm is shown in Algorithm~\ref{alg:svd-oracle}.

\begin{algorithm}[H]
\caption{A Simple SVD}
\label{alg:svd-oracle}
\begin{algorithmic}[1]
\Require
Rank-$r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ with size $m\times n $;
\State Initially get $\bA^\top\bA \bx_i = \sigma_i^2 \bx_i$ $\forall i\in \{1, 2, \ldots, r\}$; \Comment{ $f(m,n)$ flops}
\State Normalize each eigenvectors $\bv_i = \frac{\bx_i}{||\bx_i||}$;\Comment{$r\times 3n = 3nr$ flops}
\State Normalize each eigenvectors $\bu_i = \frac{\bA \bv_i}{\sigma_i}$; \Comment{$r(m(2n-1)+m)$ flops}
\end{algorithmic}
\end{algorithm}

By completing $\{\bv_{r+1}, \bv_{r+2}, \ldots, \bv_n\}$ into a full orthogonal basis via Gram-Schmidt, we have
\begin{equation*}
	\begin{aligned}
		&\bv_i^\top \bv_i = 1, \gap\forall r+1\leq i \leq n,\\
		&\bv_i^\top \bv_j = 0, \gap\forall 1\leq i \leq n,   r+1\leq j \leq n.
	\end{aligned}
\end{equation*}
And similarly $\{\bu_{r+1}, \bu_{r+2}, \ldots, \bu_m\}$ into a full orthogonal basis via Gram-Schmidt, we have
\begin{equation*}
	\begin{aligned}
		&\bu_i^\top \bu_i = 1, \gap\forall r+1\leq i \leq m,\\
		&\bu_i^\top \bu_j = 0,\gap \forall 1\leq i \leq m,   r+1\leq j \leq m.
	\end{aligned}
\end{equation*}

The detailed analysis of computational complexity for SVD is complicated. It is typically computed numerically by a two-step procedure. In the first step, the matrix is reduced to a bidiagonal matrix. This takes $O(mn^2)$ flops (Section~\ref{section:bidiagonal-decompo}, p.~\pageref{section:bidiagonal-decompo}) and the second step is to compute the SVD of the bidiagonal matrix which takes $O(n)$ iterations with each involving $O(n)$ flops. Thus, the overall cost is $O(mn^2)$ flops. For those who are interested in the computation of SVD, please refer to Section~\ref{section:eigenvalue-problem} or \citep{trefethen1997numerical, golub2013matrix, kishore2017literature} for more details.

\subsection{Randomized Method for Computing the SVD Approximately}
Suppose now the matrix $\bA$ admits rank decomposition (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}):
$$
\underset{m\times n}{\bA} = \underset{m\times r}{\bD}\gap \underset{r\times n}{\bF},
$$
where the columns of $\bD$ span the same column space of $\bA$: $\cspace(\bD)=\cspace(\bF)$. If we orthogonalize the columns of $\bD=[\bd_1, \bd_2, \ldots, \bd_r]$ into $\bQ_r=[\bq_1, \bq_2, \ldots, \bq_r]$ such that
$$
span([\bq_1, \bq_2, \ldots, \bq_k]) = span([\bd_1, \bd_2, \ldots, \bd_k]), \qquad \text{for $k\in \{1,2,\ldots,r\}$}.
$$
This can be done by the \textit{reduced} QR decomposition via the Gram-Schmidt process
where the complexity is $O(mr^2)$ (Section~\ref{section:qr-gram-compute}, p.~\pageref{section:qr-gram-compute}) if $\bD\in \real^{m\times r}$ (where we complete the $\bQ_r$ into an orthogonal one $\bQ=[\bQ_r, \bQ_2]$ such that $\bQ\bQ^\top=\bI$ for further analysis, but note that $\bQ_2$ is not needed for the final algorithm!). One can show that if the SVD of the $m\times n$ matrix $\widetildebE=\bQ^\top \bA \in \real^{m\times n}$ is given by $\widetildebE = \widetildebU\bSigma\bV^\top$, the SVD of $\bA$ can be obtained by
\begin{equation}\label{equation:svd-random-au}
	\bA=\bQ\widetildebE=\underbrace{(\bQ\widetildebU)}_{\bU}\bSigma\bV^\top.
\end{equation}
Now let's decompose the above finding into
$$
\begin{aligned}
\widetildebE &=
\begin{bmatrix}
	\bQ_r^\top \bA \\
	\bQ_2^\top \bA
\end{bmatrix}
=
\widetildebU\bSigma\bV^\top
=
\begin{bmatrix}
	\widetildebU_r & \widetildebU_{12} \\
	\widetildebU_{21} & \widetildebU_{22}
\end{bmatrix}
\begin{bmatrix}
	\bSigma_r & \bzero \\
	\bzero & \bzero
\end{bmatrix}
\begin{bmatrix}
	\bV_r^\top\\
	\bV_2^\top
\end{bmatrix}
=
\begin{bmatrix}
	\widetildebU_r \bSigma_r  \\
	\widetildebU_{21}\bSigma_r
\end{bmatrix}
\bV_r^\top \\
&\leadtosmall
\left\{
\begin{aligned}
	\bQ_r^\top \bA &= \widetildebU_r \bSigma_r\bV_r^\top;\\
	\bQ_2^\top \bA &= \widetildebU_{21}\bSigma_r \bV_r^\top.\\
\end{aligned}\right.
\end{aligned}
$$
By Equation~\eqref{equation:svd-random-au}, we have
$$
\widetildebU = \bQ^\top \bU =
\begin{bmatrix}
	\bQ_r^\top \\
	\bQ_2^\top
\end{bmatrix}
\begin{bmatrix}
	\bU_r & \bU_2
\end{bmatrix}
=
\begin{bmatrix}
\bQ_r^\top \bU_r & \bQ_r^\top\bU_2 \\
\bQ_2^\top \bU_r & \bQ_2^\top\bU_2 \\
\end{bmatrix}
=
\begin{bmatrix}
	\widetildebU_r & \widetildebU_{12} \\
	\widetildebU_{21} & \widetildebU_{22}
\end{bmatrix}.
$$
Since $\cspace(\bQ_r) = \cspace(\bA)=\cspace(\bU_r)$, it follows that $\bQ_2^\top \bU_r=\bzero $ since $\bQ_2$ lies in the orthogonal complement of $\bQ_r$ (which is also the orthogonal complement of $\bU_r$). Therefore,
$$
\begin{aligned}
	\widetildebE &=
	\begin{bmatrix}
		\bQ_r^\top \bA \\
		\bzero
	\end{bmatrix}
\end{aligned}
$$
By Equation~\eqref{equation:svd-random-au} again,
$$
\bA=\bQ\widetildebE=
\begin{bmatrix}
\bQ_r & \bQ_2
\end{bmatrix}
\begin{bmatrix}
	\bQ_r^\top \bA \\
	\bzero
\end{bmatrix}
=
\bQ_r\bQ_r^\top \bA.
$$
We carefully notice that $\bQ_r^\top \bA = \widetildebU_r \bSigma_r\bV_r^\top$ above is the SVD of $\bQ_r^\top \bA \in\real^{r\times n}$.

\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
To conclude the observation, suppose we find a matrix $\bD\in \real^{m\times r}$ that spans the column space of $\bA\in \real^{m\times n}$. The \textit{reduced} QR decomposition of $\bD=\bQ_r\bR$ such that $\bQ_r$ also span the column space of $\bA$. Calculate the SVD of the small matrix $\bE=\bQ_r^\top\bA \in\real^{r\times n} $ which costs $O(nr^2)$ flops: $\bE=\bQ_r^\top\bA=\widetildebU_r\bSigma_r\bV_r^\top$. The \textit{reduced} SVD of the large matrix can be obtained by
$$
\bA = \bQ_r(\bQ_r^\top\bA) = \underbrace{\bQ_r\widetildebU_r}_{\bU_r}\bSigma_r\bV_r^\top.
$$
By completing the matrices $\bU_r\in \real^{m\times r}, \bV_r\in \real^{n\times r}$ into full orthogonal matrices, we find the \textit{full} SVD of $\bA$.
\end{mdframed}
Further, similar to the randomized algorithm for interpolative decomposition in Section~\ref{section:randomi-id} (p.~\pageref{section:randomi-id}), by Lemma~\ref{lemma:column-basis-from-row-basis} (p.~\pageref{lemma:column-basis-from-row-basis}), for any matrix $\bA\in \real^{m\times n}$, suppose that $\{\bg_1, \bg_2, \ldots, \bg_r\}$ is a set of vectors in $\real^n$ which forms a basis for the row space, then $\{\bA\bg_1, \bA\bg_2, \ldots, \bA\bg_r\}$ is a basis for the column space of $\bA$:
\begin{equation}\label{equation:random-svd-column-space1}
	\cspace(\bA\bG) =\cspace(\bA).
\end{equation}
And a small integer $k$ (say $k=10$) should be picked to over-sample such that there is a high probability that $\bG$=$[\bg_1, \ldots,\bg_r, \bg_{r+1}, \ldots, \bg_{r+k}]$ contains the row basis of $\bA$. Other methods to make sure $\bG$ can span the row basis of $\bA$ or $\bA\bG$ can span the column space of $\bA$ are discussed in Remark~\ref{remark:source-row-basis} (p.~\pageref{remark:source-row-basis}). Again, The choice $k=10$ is often good. The procedure is formulated in Algorithm~\ref{alg:svd-randomized} where the end result costs $O(mn(r+k))$ flops compared to the original $O(mn^2)$ flops. We notice that the leading term of $O(mn(r+k))$ flops in the algorithms comes from step 6 to do the matrix multiplication $\bQ_r^\top\bA$. A structured choice on the random matrix $\bG$ can reduce to $O(mn\log(r+k))$ flops \citep{martinsson2019randomized, ailon2006approximate}. We shall not give the details.

\begin{algorithm}[h]
\caption{A Randomized Method to Compute the SVD}
\label{alg:svd-randomized}
\begin{algorithmic}[1]
\Require
Rank-$r$ matrix $\bA$ with size $m\times n $;
\State Decide the over-sampling parameter $k$ (e.g., $k=10$), and let $z=r+k$;
\State Decide the iteration number: $\eta$ (e.g., $\eta=0,1$ or $2$);
\State Generate $r+k$ Gaussian random vectors in $\real^n$ into columns of matrix $\bG\in \real^{n\times (r+k)}$;\Comment{i.e., probably contain the row basis of $\bA$}
\State Initialize $\bD=\bA\bG \in \real^{m\times (r+k)}$; \Comment{probably $\cspace(\bD)=\cspace(\bA)$, $m(2n-1)(r+k)$ flops}
\State Compute full QR decomposition $\bD = \underbrace{\bQ_r}_{m\times (r+k)}\bR$;\Comment{$O(mz^2)$ flops}
\State Form matrix $\bE=\bQ_r^\top \bA \in\real^{(r+k)\times n}$; \Comment{$nz(2m-1)$ flops}
\State Compute the SVD of the small matrix $\bE$: $\bE = \bU_0\bSigma\bV^\top$; \Comment{$O(nz^2)$ flops}
\State Form $\bU=\bQ_r\bU_0$ such that the \textit{reduced} SVD of $\bA=\bU\bSigma\bV^\top$; \Comment{$mz(2z-1)$ flops}
\end{algorithmic}
\end{algorithm}

\section{Polar Decomposition}

	\begin{theoremHigh}[Polar Decomposition]\label{theorem:polar-decomposition}
		For every real $n\times n$ square matrix $\bA$ with rank $r$, then matrix $\bA$ can be factored as
		$$
		\bA = \bQ_l \bS,
		$$
		where $\bQ_l$ is an orthogonal matrix, and $\bS$ is a positive semidefinite matrix. And this form is called the \textbf{left polar decomposition}. Also matrix $\bA$ can be factored as
		$$
		\bA =  \bS\bQ_r,
		$$
		where $\bQ_r$ is an orthogonal matrix, and $\bS$ is a positive semidefinite matrix. And this form is called the \textbf{right polar decomposition}.

		Specially, the left and right polar decomposition of a square matrix $\bA$ is \textbf{unique}.
	\end{theoremHigh}

Since every $n\times n$ square matrix $\bA$ has full SVD $\bA = \bU \bSigma \bV^\top$, where both $\bU$ and $\bV$ are $n\times n$ orthogonal matrix. We then have $\bA = (\bU\bV^\top)( \bV\bSigma \bV^\top) = \bQ_l\bS$ where it can be easily verified that $\bQ_l = \bU\bV^\top$ is an orthogonal matrix and $\bS = \bV\bSigma \bV^\top$ is a symmetric matrix. We notice that the singular values in $\bSigma$ are nonnegative, such that  $\bS=\bV\bSigma \bV^\top = \bV\bSigma^{1/2}\bSigma^{1/2\top} \bV^\top$ showing that $\bS$ is PSD.

Similarly, we have $\bA = \bU \bSigma \bU^\top \bU \bV^\top = (\bU \bSigma \bU^\top)( \bU \bV^\top)=\bS\bQ_r$. And $\bS=\bU \bSigma \bU^\top = \bU\bSigma^{1/2}\bSigma^{1/2\top} \bU^\top$ such that $\bS$ is PSD as well.

For the uniqueness of the right polar decomposition, we suppose the decomposition is not unique, and two of the decompositions are given by
$$
\bA = \bS_1\bQ_1 = \bS_2\bQ_2,
$$
such that
$$
\bS_1= \bS_2\bQ_2\bQ_1^\top.
$$
Since $\bS_1$ and $\bS_2$ are symmetric, we have
$$
\bS_1^2 = \bS_1\bS_1^\top =  \bS_2\bQ_2\bQ_1^\top\bQ_1 \bQ_2^\top\bS_2 = \bS_2^2.
$$
This implies $\bS_1=\bS_2$, and the decomposition is unique (Theorem~\ref{theorem:unique-factor-pd}, p.~\pageref{theorem:unique-factor-pd}). Similarly, the uniqueness of the left polar decomposition can be implied from the context.

\begin{corollary}[Full Rank Polar Decomposition]
When $\bA\in \real^{n\times n}$ has full rank, then the $\bS$ in both the left and right polar decomposition above is a symmetric positive definite matrix.
\end{corollary}

\section{Generalized Singular Value Decomposition (GSVD)*}
Following from \citep{golub2013matrix}, we here give a short review for generalized singular value decomposition (GSVD). See also \citep{zhang2017matrix, bai1993computing, zha1989restricted, golub2013matrix, paige1981towards} for a more detailed discussion on GSVD.
\subsection{CS Decomposition}
\begin{theoremHigh}[CS Decomposition]
Suppose
$$
\begin{blockarray}{cccc}
	\begin{block}{c[cc]c}
			&	\bQ_{11} & \bQ_{12} & m_1  \\
	\bQ=	&	\bQ_{21} & \bQ_{22}  & m_2 \\
	\end{block}
	& n_1 & n_2 &  \\
\end{blockarray},
$$
is an orthogonal matrix with $m_1 \geq n_1$ and $m_1 \geq m_2$. Define the nonnegative integer $p$ and $q$ by $p=\max\{0, n_1-m_2\}$, and $q=\max\{0, m_2-n_1\}$. There exist orthogonal matrices $\bU_1 \in \real^{m_1\times m_1}$, $\bU_2\in \real^{m_2\times m_2}$, $\bV_1\in \real^{n_1\times n_1}$, and $\bV_2\in \real^{n_2\times n_2}$ such that if
$$
\bU = \begin{bmatrix}
	\bU_1 & \bzero \\
	\bzero & \bU_2
\end{bmatrix}
, \qquad
\text{and}
\qquad
\bV = \begin{bmatrix}
\bV_1 & \bzero \\
\bzero & \bV_2
\end{bmatrix},
$$
then
$$
\begin{blockarray}{ccccccc}
	\begin{block}{c[ccccc]c}
		&	\bI & 0 & 0 & 0 & 0 & p  \\
			& 0 & \bC & \bS & 0	 & 0  & n_1-p \\
	\bU^\top\bQ\bV=	& 0 & 0 & 0 &0	 &  \bI  & m_1-n_1 \\
		& 0 & \bS & -\bC & 0	 & 0  & n_1-p \\
		& 0 & 0 & 0 & \bI  & 0  & q \\
	\end{block}
	& p & n_1-p &  n_1-p& q &  m_1-n_1&  \\
\end{blockarray},
$$
where
$$
\begin{aligned}
\bC &= \diag(\cos(\theta_{p+1}), \ldots, \cos(\theta_{n_1})) = \diag(c_{p+1}, \ldots, c_{n_1}),\\
\bS &= \diag(\sin(\theta_{p+1}), \ldots, \sin(\theta_{n_1})) = \diag(s_{p+1}, \ldots, s_{n_1}),
\end{aligned}
$$
and $0\leq \theta_{p+1} \leq \ldots \leq \theta_{n_1} \leq \pi/2$.
\end{theoremHigh}
The proof can be found in \citep{paige1981towards} and a thin version of CS decomposition is provided in \citep{golub2013matrix}.

\subsection{Generalized Singular Value Decomposition (GSVD)}
\begin{theoremHigh}[Generalized Singular Value Decomposition]\label{theorem:gsvd-main}
Assume that $\bA \in \real^{m_1\times n_1}$ and $\bB \in \real^{m_2\times n_1}$ with $m_1 \geq n_1$ and
$$
r = rank(\begin{bmatrix}
	\bA \\
	\bB
\end{bmatrix}).
$$
Then there exist orthogonal matrices $\bU_1 \in \real^{m_1\times m_1}$ and $\bU_2\in \real^{m_2\times m_2}$, and invertible matrix $\bX \in \real^{n_1\times n_1}$ such that
$$
\begin{aligned}
&\begin{blockarray}{ccccc}
	\begin{block}{c[ccc]c}
	 	                        &	\bI & \bzero & \bzero & p  \\
	\bU_1^\top \bA \bX = \bD_A = 	&  \bzero   &	\bS_A & \bzero  & r-p \\
	                         	&	\bzero  &\bzero& \bzero & m_1-r  \\
	\end{block}
	                        & p & r-p &  n_1-r & \\
\end{blockarray}\\
&\begin{blockarray}{ccccc}
	\begin{block}{c[ccc]c}
		&	\bzero & \bzero & \bzero & p  \\
		\bU_2^\top \bB \bX = \bD_B = 	&   \bzero  &	\bS_B & \bzero  & r-p \\
		&	\bzero  &\bzero& \bzero & m_2-r  \\
	\end{block}
	& p & r-p &  n_1-r & \\
\end{blockarray},
  \end{aligned}
$$
where $p = \max\{r-m_2, 0\}$, $\bS_A = \diag(\alpha_{p+1}, \ldots, \alpha_r)$, and
$\bS_B = \diag(\beta_{p+1}, \ldots, \beta_r)$, and
$$
\alpha_i^2 + \beta_i^2 = 1, \qquad \forall i \in \{p+1, \ldots, r\}.
$$
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:gsvd-main}]
Suppose the SVD of $\begin{bmatrix}
	\bA \\
	\bB
\end{bmatrix}$ is given by
$$
\begin{bmatrix}
	\bA \\
	\bB
\end{bmatrix}
=
\begin{bmatrix}
	\bQ_{11} & \bQ_{12} \\
	\bQ_{21} &\bQ_{22}
\end{bmatrix}
\begin{bmatrix}
	\bSigma_r & \bzero \\
	\bzero & \bzero
\end{bmatrix}
\bP^\top,
$$
where $\bSigma_r \in \real^{r\times r}$ is nonsingular, $\bQ_{11}\in \real^{m_1\times r}$, and $\bQ_{21}\in \real^{m_2\times r}$. Apply CS decomposition on $\bQ$, there exist orthogonal matrices $\bU_1\in \real^{m_1\times m_1}$, $\bU_2\in \real^{m_2\times m_2}$, and $\bV_1 \in \real^{r\times r}$ such that
$$
\begin{bmatrix}
	\bU_1 & \bzero \\
	\bzero & \bU_2
\end{bmatrix}^\top
\begin{bmatrix}
	\bQ_{11}\\
	\bQ_{21}
\end{bmatrix}\bV_1
=
\begin{bmatrix}
	\bD_A^r \\
	\bD_B^r
	\end{bmatrix},
$$
where $\bD_A^r$ and $\bD_B^r$ are first $r$ columns of $\bD_A$ and $\bD_B$ (note that row permutation needed here). It then follows that
$$
\begin{aligned}
\begin{bmatrix}
	\bU_1 & \bzero \\
	\bzero & \bU_2
\end{bmatrix}^\top
\begin{bmatrix}
	\bA\\
	\bB
\end{bmatrix}\bP
&=
\begin{bmatrix}
	\bD_A^r & \bU_1\bQ_{12}\\
	\bD_B^r & \bU_2\bQ_{22}
\end{bmatrix}
\begin{bmatrix}
	\bV_1^\top \bSigma_r & \bzero \\
	\bzero  & \bzero
\end{bmatrix} \\
&= \begin{bmatrix}
	\bD_A^r & \bzero\\
	\bD_B^r & \bzero
\end{bmatrix}
\begin{bmatrix}
	\bV_1^\top \bSigma_r & \bzero \\
	\bzero  & \bI_{n_1-r}
\end{bmatrix} \\
&= \begin{bmatrix}
	\bD_A\\
	\bD_B
\end{bmatrix}
\begin{bmatrix}
	\bV_1^\top \bSigma_r & \bzero \\
	\bzero  & \bI_{n_1-r}
\end{bmatrix}. \\
\end{aligned}
$$
By setting
$$
\bX = \bP \begin{bmatrix}
	\bV_1^\top \bSigma_r & \bzero \\
	\bzero  & \bI_{n_1-r}
\end{bmatrix}^{-1},
$$
we complete the proof.
\end{proof}
Note that if $\bB =\bI_{n_1}$, and we set $\bX = \bU_2$, then we obtain the SVD of $\bA$.

\section{Applications}
\subsection{Application: Least Squares via SVD for Rank Deficient Matrices}\label{section:application-ls-svd}
The least squares problem is described in Section~\ref{section:application-ls-qr} (p.~\pageref{section:application-ls-qr}). As a recap,
let's consider the overdetermined system $\bA\bx = \bb$, where $\bA\in \real^{m\times n}$ is the data matrix, $\bb\in \real^m$ with $m>n$ is the observation matrix. Normally $\bA$ will have full column rank since the data from real work has a large chance to be unrelated. And the least squares (LS) solution is given by $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ for minimizing $||\bA\bx-\bb||^2$, where $\bA^\top\bA$ is invertible since $\bA$ has full column rank and $rank(\bA^T\bA)=rank(\bA)$.\index{Least squares}

However, if $\bA$ does not have full column rank, $\bA^\top\bA$ is not invertible. We thus can use the SVD decomposition of $\bA$ to solve the LS problem. And we illustrate this in the following theorem.
\begin{theorem}[LS via SVD for Rank Deficient Matrix]\label{theorem:svd-deficient-rank}
Let $\bA\in \real^{m\times n}$ and $\bA=\bU\bSigma\bV^\top$ is its full SVD decomposition with $\bU\in\real^{m\times m}$ and $\bV\in \real^{n\times n}$ being orthogonal matrices and $rank(\bA)=r$. Suppose $\bU=[\bu_1, \bu_2, \ldots, \bu_m]$, $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ and $\bb\in \real^m$, then the LS solution with the minimal 2-norm to $\bA\bx=\bb$ is given by
\begin{equation}\label{equation:svd-ls-solution}
\bx_{LS} = \sum_{i=1}^{r}\frac{\bu_i^\top \bb}{\sigma_i}\bv_i = \bV\bSigma^+\bU^\top \bb,
\end{equation}
where the upper-left side of $\bSigma^+ \in \real^{n\times m}$ is a diagonal matrix, $\bSigma^+ = \begin{bmatrix}
	\bSigma_1^+ & \bzero \\
	\bzero & \bzero
\end{bmatrix}$ where $\bSigma_1^+=diag(\frac{1}{\sigma_1}, \frac{1}{\sigma_2}, \ldots, \frac{1}{\sigma_r})$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:svd-deficient-rank}]
Write out the loss to be minimized
$$
\begin{aligned}
	||\bA\bx-\bb||^2 &= (\bA\bx-\bb)^\top(\bA\bx-\bb)\\
	&=(\bA\bx-\bb)^\top\bU\bU^\top (\bA\bx-\bb) \qquad &(\text{Since $\bU$ is an orthogonal matrix})\\
	&=||\bU^\top \bA \bx-\bU^\top\bb||^2 \qquad &(\text{Invariant under orthogonal})\\
	&=||\bU^\top \bA \bV\bV^\top \bx-\bU^\top\bb||^2 \qquad &(\text{Since $\bV$ is an orthogonal matrix})\\
	&=||\bSigma\balpha - \bU^\top\bb||^2  \qquad &(\text{Let $\balpha=\bV^\top \bx$})\\
	&=\sum_{i=1}^{r}(\sigma_i\balpha_i - \bu_i^\top\bb)^2 +\sum_{i=r+1}^{m}(\bu_i^\top \bb)^2. \qquad &(\text{Since $\sigma_{r+1}=\sigma_{r+2}= \ldots= \sigma_m=0$})
\end{aligned}
$$
Since $\bx$ only appears in $\balpha$, we just need to set $\balpha_i = \frac{\bu_i^\top\bb}{\sigma_i}$ for all $i\in \{1, 2, \ldots, r\}$ to minimize the above equation. For any value of $\balpha_{r+1}, \balpha_{r+2}, \ldots, \balpha_{n}$, it won't change the result. From the regularization point of view (or here, we want the minimal 2-norm) we can set them to be 0. This gives us the LS solution via SVD:
$$
\bx_{LS} = \sum_{i=1}^{r}\frac{\bu_i^\top \bb}{\sigma_i}\bv_i=\bV\bSigma^+\bU^\top \bb = \bA^+\bb,
$$
where $\bA^+=\bV\bSigma^+\bU^\top\in \real^{n\times m}$ is known as the \textbf{pseudo-inverse} of $\bA$. Please refer to Appendix~\ref{appendix:pseudo-inverse} (p.~\pageref{appendix:pseudo-inverse}) for a detailed discussion about pseudo-inverse where we also prove that the column space of $\bA^+$ is equal to the row space of $\bA$, and the row space of $\bA^+$ is equal to the column space of $\bA$.\index{Pseudo-inverse}
\end{proof}

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.98\textwidth]{imgs/lafundamental4-LS-SVD.pdf}
	\caption{$\bA^+$: Pseudo-inverse of $\bA$.}
	\label{fig:lafundamental4-LS-SVD}
\end{figure}

Let $\bA^+ = \bV\bSigma^+\bU^\top$ be the pseudo-inverse of $\bA$. The pseudo-inverse $\bA^+$ agrees with $\bA^{-1}$ when $\bA$ is invertible. The solution of least squares is to make the error $\bb-\bA\bx$ as small as possible concerning the mean square error. Since $\bA\bx$ can never leave the column space of $\bA$, we should choose the closest point to $\bb$ in the column space \citep{strang1993fundamental}. This point is the projection $\bp$ of $\bb$. Then the error vector $\be=\bb-\bp$ has minimal length. In another word, the best combination $\bp = \bA\bx_{LS}$ is the projection of $\bb$ onto the column space. The error $\be$ is perpendicular to the column space. Therefore, $\be=\bb-\bA\bx_{LS}$ is in the null space of $\bA^\top$:
$$
\bA^\top(\bb-\bA\bx_{LS}) = \bzero  \qquad \text{or} \qquad \bA^\top\bb=\bA^\top\bA\bx_{LS},
$$
which is also known as the normal equation of least squares. The relationship between $\be$ and $\bp$ is shown in Figure~\ref{fig:lafundamental4-LS-SVD} where $\bb$ is split into $\bp+\be$. Since $\be$ is in $\nspace(\bA^\top)$ and perpendicular to $\cspace(\bA)$, and we have shown in Section~\ref{section:property-svd}, $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is an orthonormal basis of $\cspace(\bA)$, then the first $r$ components of $\bU^\top\be$ are all zeros. Therefore, $\bA^+\be = \bV\bSigma^+\bU^\top\be=\bzero$. Moreover, $\bx_{LS}=\bA^+\bb = \bA^+(\bp+\be) = \bA^+\bp$.

Further, we have also shown in Section~\ref{section:property-svd}, $\{\bv_1, \bv_2, \ldots, \bv_r\} $ is an orthonormal basis of $\cspace(\bA^\top)$, $\bx_{LS} = \sum_{i=1}^{r}\frac{\bu_i^\top \bb}{\sigma_i}\bv_i$ thus is in the row space of $\bA$, i.e., it cannot be split into a combination of two components that are in row space of $\bA$ and null space of $\bA$ respectively.

Apart from this LS solution from SVD, in practice, a direct solution of the normal equations can lead to numerical difficulties when $\bX^\top\bX$ is close to singular. In particular, when two or more of the columns in $\bX^\top\bX$ are co-linear, the resulting parameter values can have a large magnitude. Such near degeneracies will not be uncommon when dealing with real data sets. The resulting numerical difficulties can be addressed using the SVD as well \citep{bishop2006pattern}.

\subsection{Application: Least Squares with Norm Ratio Method}\label{section:application-ls-svd-norm-ratio}
We first define the Frobenius norm as follows and a detailed discussion on matrix norm can be found in Appendix~\ref{appendix:matrix-norm} (p.~\pageref{appendix:matrix-norm}).
	\begin{definition}[Frobenius Norm\index{Frobenius norm}]\label{definition:frobernius-in-svd}
		The Frobenius norm of a matrix $\bA\in \real^{m\times n}$ is defined as
		$$
		|| \bA ||_F = \sqrt{\sum_{i=1,j=1}^{m,n} (\bA_{ij})^2}=\sqrt{tr(\bA\bA^\top)}=\sqrt{tr(\bA^\top\bA)} = \sqrt{\sigma_1^2+\sigma_2^2+\ldots+\sigma_r^2}.
		$$

	\end{definition}

Following the setup in the last section. Let $\bA_k \in \real^{m\times n}$ be the rank-$k$ approximation to the original $m\times n$ matrix $\bA$. Define the \textit{Frobenius norm ratio} \citep{zhang2017matrix} as
$$
\nu(k) = \frac{||\bA_k||_F}{||\bA||_F} = \frac{\sqrt{\sigma_1^2+\sigma_2^2+\ldots +\sigma_k^2}}{\sqrt{\sigma_1^2+\sigma_2^2+\ldots +\sigma_h^2}}, \quad h = \min\{m,n\},
$$
 where
 $\bA_k$ is the truncated SVD of $\bA$ with the largest $k$ terms, i.e., $\bA_k = \sum_{i=1}^{k} \sigma_i\bu_i\bv_i^\top$ from SVD of $\bA=\sum_{i=1}^{r} \sigma_i\bu_i\bv_i^\top$.
We choose the minimum integer $k$ satisfying
$$
\nu(k) \geq \alpha
$$
as the \textit{effective rank estimate} $\hat{r}$, where $\alpha$ is is the threshold with maximum value to be 1, and it is usually chosen to be $\alpha=0.997$.
After the effective rank $\hat{r}$ has been determined, we replace the $\hat{r}$ in Equation~\eqref{equation:svd-ls-solution},
$$
\hat{\bx_{LS}} = \sum_{i=1}^{\textcolor{blue}{\hat{r}}}\frac{\bu_i^\top \bb}{\sigma_i}\bv_i ,
$$
which can be regarded as an approximation to the LS solution $\bx_{LS}$. And this solution is the LS solution of the linear equation $\bA_{\hat{r}} = \bb$, where
$$
\bA_{\hat{r}} = \sum_{i=1}^{\hat{r}} \sigma_i \bu_i\bv_i^\top.
$$
This filtering method introduced above is useful when matrix $\bA$ is noisy.

\subsection{Application: Principal Component Analysis (PCA) via the Spectral Decomposition and the SVD}
Given a data set of $n$ observations $\{\bx_1,\bx_2,\ldots,\bx_n\}$ where $\bx_i\in \real^p$ for all $i\in \{1,2,\ldots,n\}$. Our goal is to project the data onto a low-dimensional space, say $m<p$. Define the sample mean vector and sample covariance matrix\index{Principal component analysis}
$$
\overline{\bx} = \frac{1}{n}\sum_{i=1}^{n}\bx_i
\qquad
\text{and}
\qquad
\bS = \frac{1}{n-1}\sum_{i=1}^{n} (\bx_i - \overline{\bx})(\bx_i-\overline{\bx})^\top.
$$
where the $n-1$ term in the covariance matrix is to make it to be an unbiased consistent estimator of the covariance \citep{lu2021rigorous}. Or the covariance matrix can also be defined as $\bS = \frac{1}{\textcolor{blue}{n}}\sum_{i=1}^{n} (\bx_i - \overline{\bx})(\bx_i-\overline{\bx})^\top$ which is also a consistent estimator of covariance matrix \footnote{Consistency: An estimator $\theta_n $ of $\theta$ constructed on the basis of a sample of size $n$ is said to be consistent if $\theta_n\stackrel{p}{\rightarrow} \theta$   as $n \rightarrow \infty $.}.

Each data point $\bx_i$ is then projected onto a scalar value by $\bu_1$ such that $\bu_1^\top\bx_i$. The mean of the projected data is obtained by $\Exp[\bu_1^\top\bx_i] = \bu_1^\top \overline{\bx}$, and the variance of the projected data is given by
$$
\begin{aligned}
\Cov[\bu_1^\top\bx_i] &= \frac{1}{n-1} \sum_{i=1}^{n}( \bu_1^\top \bx_i - \bu_1^\top\overline{\bx})^2=
\frac{1}{n-1} \sum_{i=1}^{n}\bu_1^\top  ( \bx_i -\overline{\bx})( \bx_i -\overline{\bx})^\top\bu_1\\
&=\bu_1^\top\bS\bu_1.
\end{aligned}
$$
We want to maximize the projected variance $\bu_1^\top\bS\bu_1$ with respect to $\bu_1$ where we must constrain $||\bu_1||$ to prevent $||\bu_1|| \rightarrow \infty$ by setting it to be $\bu_1^\top\bu_1=1$. By Lagrange multiplier (see \citep{bishop2006pattern, boyd2004convex}), we have
$$
\bu_1^\top\bS\bu_1 + \lambda_1 (1 - \bu_1^\top\bu_1).
$$
Trivial calculation will lead to
$$
\bS\bu_1 = \lambda_1\bu_1 \leadto \bu_1^\top\bS\bu_1 = \lambda_1.
$$
That is, $\bu_1$ is an eigenvector of $\bS$ corresponding to eigenvalue $\lambda_1$. And the maximum variance projection $\bu_1$ is corresponding to the largest eigenvalues of $\bS$. The eigenvector is known as the \textit{first principal axis}.

Define the other principal axes by decremental eigenvalues until we have $m$ such principal components bring about the dimension reduction. This is known as the \textit{maximum variance formulation} of PCA \citep{hotelling1933analysis, bishop2006pattern, shlens2014tutorial}. A \textit{minimum-error formulation} of PCA is discussed in \citep{pearson1901liii, bishop2006pattern}.

\paragraph{PCA via the spectral decomposition}
Now let's assume the data are centered such that $\overline{\bx}$ is zero, or we can set $\bx_i = \bx_i-\overline{\bx}$ to centralize the data. Let the data matrix $\bX \in \real^{n\times p}$ contain the data observations as rows. The covariance matrix is given by
$$
\bS= \frac{\bX^\top\bX}{n-1},
$$
which is a symmetric matrix, and its spectral decomposition is given by
\begin{equation}\label{equation:pca-equ1}
\bS = \bU\bLambda\bU^\top,
\end{equation}
where $\bU$ is an orthogonal matrix of eigenvectors (columns of $\bU$ are eigenvectors of $\bS$), and $\bLambda=\diag(\lambda_1, \lambda_2,\ldots, \lambda_p)$ is a diagonal matrix with eigenvalues (ordered such that $\lambda_1 \geq \lambda_2 \geq \ldots \geq \lambda_p$). The eigenvectors are called \textit{principal axes} of the data, and they \textit{decorrelate} the the covariance matrix. Projections of the data on the principal axes are called the \textit{principal components}. The $i$-th principal component is given by the $i$-th column of $\bX\bU$. If we want to reduce the dimension from $p$ to $m$, we just select the first $m$ columns of $\bX\bU$.

\paragraph{PCA via the SVD}
If the SVD of $\bX$ is given by $\bX = \bP\bSigma\bQ^\top$, then the covariance matrix can be written as
\begin{equation}\label{equation:pca-equ2}
\bS= \frac{\bX^\top\bX}{n-1} = \bQ \frac{\bSigma^2}{n-1}\bQ^\top,
\end{equation}
where $\bQ\in \real^{p\times p}$ is an orthogonal matrix and contains the right singular vectors of $\bX$, and the upper left of $\bSigma$ is a diagonal matrix containing the singular values $\diag(\sigma_1,\sigma_2,\ldots)$ with $\sigma_1\geq \sigma_2\geq \ldots$. The number of singular values is equal to $\min\{n,p\}$ which will not be larger than $p$ and some of which are zeros.

The above Equation~\eqref{equation:pca-equ2} compared with Equation~\eqref{equation:pca-equ1} implies Equation~\eqref{equation:pca-equ2} is also a spectral decomposition of $\bS$,
since the eigenvalues in $\bLambda$ and singular values in $\bSigma$ are ordered in a descending way and the uniqueness of the spectral decomposition in terms of the eigenspaces (Section~\ref{section:uniqueness-spectral-decomposition}, p.~\pageref{section:uniqueness-spectral-decomposition}).

This results in the right singular vectors $\bQ$ are also the principal axes which decorrelate the covariance matrix,
and the singular values are related to the eigenvalues of the covariance matrix via $\lambda_i = \frac{\sigma_i^2 }{n-1}$. To reduce the dimensionality of the data from $p$ to $m$, we should select largest $m$ singular values and the corresponding right singular vectors. This is also related to the truncated SVD (TSVD) $\bX_m = \sum_{i=1}^{m}\sigma_i \bp_i\bq_i^\top$ as will be shown in the next section, where $\bp_i$'s and $\bq_i$'s are the columns of $\bP$ and $\bQ$.

\paragraph{A byproduct of PCA via the SVD for high-dimensional data} For a principle axis $\bu_i$ of $\bS = \frac{\bX^\top\bX}{n-1}$, we have
$$
\frac{\bX^\top\bX}{n-1} \bu_i = \lambda_i \bu_i.
$$
Left multiply by $\bX$, we obtain
$$
\frac{\bX\bX^\top}{n-1} (\bX\bu_i) = \lambda_i (\bX\bu_i),
$$
which implies $\lambda_i$ is also an eigenvalue of $\frac{\bX\bX^\top}{n-1} \in \real^{n\times n}$, and the corresponding eigenvector is $\bX\bu_i$. This is also stated in the proof of Theorem~\ref{theorem:reduced_svd_rectangular}, the existence of the SVD. If $p \gg n$, instead of finding the eigenvector of $\bS$, i.e., the principle axes of $\bS$, we can find the eigenvector of $\frac{\bX\bX^\top}{n-1}$. This reduces the complexity from $O(p^3)$ to $O(n^3)$. Suppose now, the eigenvector of $\frac{\bX\bX^\top}{n-1}$ is $\bv_i$ corresponding to nonzero eigenvalue $\lambda_i$,
$$
\frac{\bX\bX^\top}{n-1} \bv_i = \lambda_i \bv_i.
$$
Left multiply by $\bX^\top$, we obtain
$$
\frac{\bX^\top\bX}{n-1} (\bX^\top\bv_i) = \bS(\bX^\top\bv_i)   = \lambda_i (\bX^\top\bv_i),
$$
i.e., the eigenvector $\bu_i$ of $\bS$, is proportional to $\bX^\top\bv_i$, where $\bv_i$ is the eigenvector of $\frac{\bX\bX^\top}{n-1}$ corresponding to the same eigenvalue $\lambda_i$. A further normalization step is needed to make $||\bu_i||=1$.

\subsection{Application: Low-Rank Approximation}\label{section:svd-low-rank-approxi}
For a low-rank approximation problem, there are basically two types related due to the interplay of rank and error: \textit{fixed-precision approximation problem} and \textit{fixed-rank
 approximation problem}. In the fixed-precision approximation problem, for a given matrix $\bA$ and a given tolerance $\epsilon$, one wants to find a matrix $\bB$
with rank $r = r(\epsilon)$ such that $||\bA-\bB|| \leq \epsilon$ in an appropriate matrix norm. On the contrary, in the fixed-rank approximation problem, one looks for a matrix $\bB$ with fixed rank $k$ and an error $||\bA-\bB||$ as small as possible. In this section, we will consider the latter. Some excellent examples can also be found in \citep{kishore2017literature, martinsson2019randomized}.

Suppose we want to approximate matrix $\bA\in \real^{m\times n}$ with rank $r$ by a rank $k<r$ matrix $\bB$. The approximation is measured by spectral norm:\index{Low-rank approximation}
$$
\bB = \mathop{\arg\min}_{\bB} \ \ || \bA - \bB||_2,
$$
where the spectral norm is defined as follows:
\begin{definition}[Spectral Norm]\label{definition:spectral_norm}
The spectral norm of a matrix $\bA\in \real^{m\times n}$ is defined as
$$
||\bA||_2 = \mathop{\max}_{\bx\neq\bzero} \frac{||\bA\bx||_2}{||\bx||_2}  =\mathop{\max}_{\bu\in \real^n: ||\bu||_2=1}  ||\bA\bx||_2 ,
$$
which is also the maximal singular value of $\bA$, i.e., $||\bA||_2 = \sigma_1(\bA)$.
\end{definition}

Then, we can recover the best rank-$k$ approximation by the following theorem.
\begin{theorem}[Eckart-Young-Misky Theorem w.r.t. Spectral Norm\index{Eckart-Young-Misky theorem}]\label{theorem:young-theorem-spectral}
Given matrix $\bA\in \real^{m\times n}$ and $1\leq k\leq rank(\bA)=r$, and let $\bA_k$ be the truncated SVD (TSVD) of $\bA$ with the largest $k$ terms, i.e., $\bA_k = \sum_{i=1}^{k} \sigma_i\bu_i\bv_i^\top$ from SVD of $\bA=\sum_{i=1}^{r} \sigma_i\bu_i\bv_i^\top$ by zeroing out the $r-k$ trailing singular values of $\bA$. Then $\bA_k$ is the best rank-$k$ approximation to $\bA$ in terms of the spectral norm.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:young-theorem-spectral}]
We need to show for any matrix $\bB$, if $rank(\bB)=k$, then $||\bA-\bB||_2 \geq ||\bA-\bA_k||_2$.

Since $rank(\bB)=k$, then $dim (\nspace(\bB))=n-k$. As a result, any $k+1$ basis in $\real^n$ intersects $\nspace(\bB)$. As shown in Lemma~\ref{lemma:svd-four-orthonormal-Basis}, $\{\bv_1,\bv_2, \ldots, \bv_r\}$ is an orthonormal basis of $\cspace(\bA^\top)\subset \real^n$, so that we can choose the first $k+1$ $\bv_i$'s as a $k+1$ basis for $\real^n$. Let $\bV_{k+1} = [\bv_1, \bv_2, \ldots, \bv_{k+1}]$, then there is a vector $\bx$ that
$$
\bx \in \nspace(\bB) \cap \cspace(\bV_{k+1}),\qquad s.t.\,\,\,\, ||\bx||_2=1.
$$
That is $\bx = \sum_{i=1}^{k+1} a_i \bv_i$, and $||\sum_{i=1}^{k+1} a_i \bv_i||_2 = \sum_{i=1}^{k+1}a_i^2=1$.
Thus,
$$
\begin{aligned}
	||\bA-\bB||_2^2 &\geq ||(\bA-\bB)\bx||_2^2\cdot ||\bx||_2^2, \qquad &(\text{From defintion of spectral norm}) \\
	&= ||\bA\bx||_2^2, \qquad &(\text{$\bx$ in null space of $\bB$}) \\
	&=\sum_{i=1}^{k+1} \sigma_i^2 (\bv_i^\top \bx)^2, \qquad &(\text{$\bx$ orthogonal to $\bv_{k+2}, \ldots, \bv_r$}) \\
	&\geq \sigma_{k+1}^2\sum_{i=1}^{k+1}  (\bv_i^\top \bx)^2, \qquad &(\sigma_{k+1}\leq \sigma_{k}\leq\ldots\leq \sigma_{1}) \\
	&\geq \sigma_{k+1}^2\sum_{i=1}^{k+1} a_i^2, \qquad &(\bv_i^\top \bx = a_i) \\
	&= \sigma_{k+1}^2.
\end{aligned}
$$
It is trivial that $||\bA-\bA_k||_2^2 = ||\sum_{i=k+1}^{r}\sigma_i\bu_i\bv_i^\top||_2^2=\sigma_{k+1}^2$. Thus, $||\bA-\bA_k||_2 \leq ||\bA-\bB||_2$, which completes the proof.
\end{proof}
\begin{SCfigure
	\centering
	\includegraphics[width=0.6\textwidth]{./imgs/eng300.png}
	\caption{A gray flag image to be compressed.}
	\label{fig:eng300}
\end{SCfigure}

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[$\sigma_1\bu_1\bv_1^\top$, $F_1 = 60,217$.]{\label{fig:svd1}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic1.png}}
	\subfigure[$\sigma_2\bu_2\bv_2^\top$, $F_2 = 120,150$.]{\label{fig:svd2}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic2.png}}
	\subfigure[$\sigma_3\bu_3\bv_3^\top$, $F_3 = 124,141$.]{\label{fig:svd3}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic3.png}}\\
	\subfigure[$\sigma_4\bu_4\bv_4^\top$, $F_4 = 125,937$.]{\label{fig:svd4}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic4.png}}
	\subfigure[$\sigma_5\bu_5\bv_5^\top$, $F_5 = 126,127$.]{\label{fig:svd5}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic5.png}}
	\subfigure[All 5 singular values: $\sum_{i=1}^{5}\sigma_i\bu_i\bv_i^\top$, \protect\newline $F=44,379$.]{\label{fig:svd6}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic6_all.png}}
	\caption{Image compression for gray flag image into a rank-5 matrix via the SVD, and decompose into 5 parts where $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_{5}$, i.e., $F_1\leq F_2\leq \ldots \leq F_5$ with $F_i = ||\sigma_i\bu_i\bv^\top - \bA||_F$ for $i\in \{1,2,\ldots, 5\}$. And reconstruct images by single singular value and its corresponding left and right singular vectors.}
	\label{fig:svdd-by-parts}
\end{figure}

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[$\bc_1\br_1^\top$, $G_1 = 60,464$.]{\label{fig:skeleton1}%
		\includegraphics[width=0.32\linewidth]{./imgs/skeleton_5_1.png}}
	\subfigure[$\bc_2\br_2^\top$, $G_2 = 122,142$.]{\label{fig:skeleton2}%
		\includegraphics[width=0.32\linewidth]{./imgs/skeleton_5_2.png}}
	\subfigure[$\bc_3\br_3^\top$, $G_3 = 123,450$.]{\label{fig:skeleton3}%
		\includegraphics[width=0.32\linewidth]{./imgs/skeleton_5_3.png}}\\
	\subfigure[$\bc_4\br_4^\top$, $G_4 = 125,975$.]{\label{fig:skeleton4}%
		\includegraphics[width=0.32\linewidth]{./imgs/skeleton_5_5.png}}
	\subfigure[$\bc_5\br_5^\top$, $G_5 = 124,794$.]{\label{fig:skeleton5}%
		\includegraphics[width=0.32\linewidth]{./imgs/skeleton_5_4.png}}
	\subfigure[All 5 parts: $\sum_{i=1}^{5}\bc_i\br_i^\top$, \protect\newline $G=45,905$.]{\label{fig:skeleton6}%
		\includegraphics[width=0.32\linewidth]{./imgs/svd_pic6_all.png}}
	\caption{Image compression for gray flag image into a rank-5 matrix via the Pseudoskeleton decomposition, and decompose into 5 parts where $G_i=||\bc_i\br_i^\top-\bA||_F$ for $i\in \{1,2,\ldots, 5\}$ and $G_1\leq G_2 \leq \ldots \leq G_5$. And reconstruct images by $\bc_i\br_i^\top$.}
	\label{fig:skeletond-by-parts}
\end{figure}

Moreover, readers can prove that $\bA_k$ is the best rank-$k$ approximation to $\bA$ in terms of the Frobenius norm. The minimal error is given by the Euclidean norm of the singular values that have been zeroed out in the process $||\bA-\bA_k||_F =\sqrt{\sigma_{k+1}^2 +\sigma_{k+2}^2+\ldots +\sigma_{r}^2}$.

SVD gives the best approximation of a matrix. As mentioned in \citep{stewart1998matrix, kishore2017literature}, \emph{the singular value decomposition is the creme de la creme of rank-reducing decompositions — the decomposition that all others try to beat}. And also \emph{The SVD is the climax of this linear algebra course} in \citep{strang1993introduction}.

Figure~\ref{fig:eng300} shows an example of a gray image to be compressed. The size of the image is $600\times 1200$ with a rank of 402.

In Figure~\ref{fig:svdd-by-parts}, we approximate the image into a rank-5 matrix by truncated SVD: $\bA\approx \sum_{i=1}^{5}\sigma_i\bu_i\bv_i^\top$. It is known that the singular values contain the spectrum information with higher singular values containing lower-frequency information. And low-frequency contains more useful information \citep{leondes1995multidimensional}. We find that the image, $\sigma_1\bu_1\bv_1^\top$, reconstructed by the first singular value $\sigma_1$, first left singular vector $\bu_1$, and first right singular vector $\bv_1$ is very close to the original flag image, and second to the fifth image reconstructed by the corresponding singular values and singular vectors contain more details of the flag to reconstruct it.

Similar results can be observed for the low-rank approximation via the pseudoskeleton decomposition (Section~\ref{section:pseudoskeleton}, p.~\pageref{section:pseudoskeleton}). In Equation~\eqref{equation:skeleton-low-rank} (p.~\pageref{equation:skeleton-low-rank}), we derived the low-rank approximation by $\bA\approx \bC_2\bR_2$ where $\bC_2\in \real^{m\times \gamma}, \bR_2\in \real^{\gamma\times n}$ if $\bA\in \real^{m\times n}$ such that $\bC_2$ and $\bR_2$ are rank-$\gamma$ matrices. Suppose $\gamma=5$, and
$$
\bC_2=[\bc_1, \bc_2, \ldots, \bc_5],
\qquad
\text{and}
\qquad
\bR_2 =
\begin{bmatrix}
\br_1^\top \\
\br_2^\top \\
\vdots \\
\br_5^\top
\end{bmatrix},
$$
are the column and row partitions of $\bC_2, \bR_2$ respectively. Then $\bA$ can be approximated by $\sum_{i=1}^{5}\bc_i\br_i^\top$. The partitions are ordered such that
$$
\underbrace{||\bc_1\br_1^\top-\bA||_F}_{G_1} \leq
\underbrace{||\bc_2\br_2^\top-\bA||_F}_{G_2}
\leq \ldots \leq
\underbrace{||\bc_5\br_5^\top-\bA||_F}_{G_5}.
$$
We observe that $\bc_1\br_1^\top$ works similarly to that of $\sigma_1\bu_1\bv^\top$ where the reconstruction errors measured by the Frobenius norm are very close (60,464 in the pseudoskeleton case compared to that of 60,217 in the SVD case). This is partly because the pseudoskeleton decomposition relies on the SVD (Section~\ref{section:pseudoskeleton}, p.~\pageref{section:pseudoskeleton}) such that $\bc_1\br_1^\top$ internally has the largest ``singular value" in the sense.

\begin{SCfigure
	\centering
	\includegraphics[width=0.5\textwidth]{./imgs/svd_skeleton_fnorm.pdf}
	\caption{Comparison of reconstruction errors measured by Frobenius norm between the SVD and the pseudoskeleton approximation.}
	\label{fig:svd_skeleton_fnorm}
\end{SCfigure}

We finally compare low-rank approximation between the SVD and the pseudoskeleton with different ranks. Figure~\ref{fig:svdd-pseudoskeleton} shows the difference of each compression with ranks of 90, 60, 30, 10. We observe that the SVD does well with ranks of 90, 60, 30. The pseudoskeleton-approximation compresses well in the black horizontal and vertical lines in the image. But it performs poorly in the details of the flag. Figure~\ref{fig:svd_skeleton_fnorm} shows the comparison of the reconstruction errors between the SVD and the pseudoskeleton approximation measured by Frobenius norm ranging from rank $1$ to $100$ where we find in all cases, the truncated SVD does better in terms of Frobenius norm. Similar results can be observed when applied to the spectral norm.

\newpage
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
\subfigure[SVD with rank 90,\gap\gap  \protect\newline Frobenius norm=\textbf{6,498}]{\label{fig:svd90}
	\includegraphics[width=0.47\linewidth]{./imgs/svd90.png}}
\quad
\subfigure[Pseudoskeleton with rank 90,\protect\newline Frobenius norm=13,751]{\label{fig:skeleton90}
	\includegraphics[width=0.47\linewidth]{./imgs/skeleton90.png}}\\
\subfigure[SVD with rank 60,\gap\gap \protect\newline Frobenius norm=\textbf{8,956}]{\label{fig:svd60}
	\includegraphics[width=0.47\linewidth]{./imgs/svd60.png}}
\quad
\subfigure[Pseudoskeleton with rank 60,\protect\newline Frobenius norm=14,217]{\label{fig:skeleton60}
	\includegraphics[width=0.47\linewidth]{./imgs/skeleton60.png}}\\
\subfigure[SVD with rank 30,\gap\gap \protect\newline Frobenius norm=\textbf{14,586}]{\label{fig:svd30}
	\includegraphics[width=0.47\linewidth]{./imgs/svd30.png}}
\quad
\subfigure[Pseudoskeleton with rank 30,\protect\newline Frobenius norm=17,853]{\label{fig:skeleton30}
	\includegraphics[width=0.47\linewidth]{./imgs/skeleton30.png}}

\subfigure[SVD with rank 10,\gap\gap \protect\newline Frobenius norm=\textbf{31,402}]{\label{fig:svd10}
	\includegraphics[width=0.47\linewidth]{./imgs/svd10.png}}
\quad
\subfigure[Pseudoskeleton with rank 10,\protect\newline Frobenius norm=33,797]{\label{fig:skeleton10}
	\includegraphics[width=0.47\linewidth]{./imgs/skeleton10.png}}
\caption{Image compression for gray flag image with different ranks.}
\label{fig:svdd-pseudoskeleton}
\end{figure}

\chapter{CP Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{CP Decomposition}
\begin{theoremHigh}[CP Decomposition]\label{theorem:cp-decomp}
The CP decomposition factorizes a tensor into a sum of component rank-one
tensors. For a general Nth-order tensor, $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, it admits the CP decomposition
$$
\eX \approx \llbracket \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r=1}^{R} \ba_r^{(1)} \circ \ba_r^{(2)} \circ \ldots \circ \ba_r^{(N)} ,
$$
where $\bA^{(n)}=[\ba_1^{(n)}, \ba_2^{(n)}, \ldots, \ba_R^{(n)}] \in \real^{I_n \times R}$ for all $n\in \{1,2,\ldots, N\}$ is the column partition of the matrix $\bA^{(n)}\in \real^{I_n \times R}$.

Or it is often useful to assume that the columns of $\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}$ are normalized to length
one with the weights absorbed into the vector $\blambda\in \real^R$ so that
$$
\eX \approx \llbracket \blambda; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=\sum_{r=1}^{R} \blambda_r \cdot  \ba_r^{(1)} \circ \ba_r^{(2)} \circ \ldots \circ \ba_r^{(N)} ,
$$
where we follow the notation $\llbracket \ldots  \rrbracket$ from \citep{kruskal1977three, kolda2009tensor}.
\end{theoremHigh}

One can find the idea behind the CP decomposition is to express the tensor as a sum of rank-one tensors, i.e., a sum of the outer product of vectors (Section~\ref{section:rank-one-tensor}, p.~\pageref{section:rank-one-tensor}). The CP decomposition is also known as the \textit{Canonical Polyadic Decomposition}, \textit{CANDECOMP-PARAFAC}, or simply \textit{PARAFAC decomposition}.

\begin{figure}[h]
\centering
\resizebox{1.0\textwidth}{!}{%
\begin{tikzpicture}

\draw [very thick] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\filldraw [fill=gray!60!white,draw=green!40!black] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.5,-3.8+0.5) grid (7+0.5,-2.6+0.5);

\draw [very thick] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\filldraw [fill=gray!50!white,draw=green!40!black] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.4,-3.8+0.4) grid (7+0.4,-2.6+0.4);

\draw [very thick] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\filldraw [fill=gray!40!white,draw=green!40!black] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.3,-3.8+0.3) grid (7+0.3,-2.6+0.3);

\draw [very thick] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\filldraw [fill=gray!30!white,draw=green!40!black] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.2,-3.8+0.2) grid (7+0.2,-2.6+0.2);

\draw [very thick] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\filldraw [fill=gray!20!white,draw=green!40!black] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.1,-3.8+0.1) grid (7+0.1,-2.6+0.1);

\draw [very thick] (5.8,-3.8) rectangle (7,-2.6);
\filldraw [fill=gray!10!white,draw=green!40!black] (5.8,-3.8) rectangle (7,-2.6);
\draw [step=0.4/2, very thin, color=gray] (5.8,-3.8) grid (7,-2.6);

\draw [very thick] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\filldraw [fill=gray!6!white,draw=green!40!black] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8-0.1,-3.8-0.1) grid (7-0.1,-2.6-0.1);

\draw[->] (5.6,-2.6) -- ++(0.6,0.6);
\draw[->] (5.55,-2.7) -- ++(0,-1.2);
\draw[->] (5.66,-4.04) -- ++(1.25,0);

\draw (6.35,-4.25) node {{\color{black}\scriptsize{spatial $y$}}};
\draw (5.2-0.3,-3.2) node {{\color{black}\scriptsize{spatial $x$}}};
\draw (5.3,-2.2) node {{\color{black}\scriptsize{temporal}}};

	\draw (6.2,-4.6) node {{\color{black}\scriptsize{$\eX\in\real^{I\times J \times K}$}}};

	\draw (8,-3.2) node {	{\color{black}\large{$\approx$}}};

	\draw [very thick] (8.6,-4.4+0.2) rectangle (8.8,-3.2+0.2);
	\filldraw [fill=WildStrawberry!40!white,draw=green!40!black] (8.6,-4.4+0.2) rectangle (8.8,-3.2+0.2);
	\draw (8.7,-4.8+0.2) node {{\color{black}\scriptsize{$\ba_1 \in \real^I$}}};

	\draw [very thick] (9,-3-0.2+0.2) rectangle (10.2,-2.8-0.2+0.2);
	\filldraw [fill=RubineRed!60!white,draw=green!40!black] (9,-3-0.2+0.2) rectangle (10.2,-2.8-0.2+0.2);
	\draw (9.6,-3.3-0.2+0.2) node {{\color{black}\scriptsize{$\bb_1 \in \real^J$}}};

	\draw[fill=RedOrange!40!white, line width=0.8pt] (9.2,-2.4+0.2) -- (9.4,-2.4+0.2) -- (8.8,-3+0.2) -- (8.6,-3+0.2) -- cycle;
	\draw (10-0.2,-1.9) node {{\color{black}\scriptsize{$\bc_1\in \real^K$}}};

	\draw (10.8,-3.2) node {{\color{black}\large{$+$}}};
	\draw (11.6,-3.2) node {{\color{black}\large{$\dots$}}};
	\draw (12.4,-3.2) node {{\color{black}\large{$+$}}};

	\draw [very thick] (8.6+4.4,-4.4+0.2) rectangle (8.8+4.4,-3.2+0.2);
	\filldraw [fill=WildStrawberry!40!white,draw=green!40!black] (8.6+4.4,-4.4+0.2) rectangle (8.8+4.4,-3.2+0.2);
	\draw (8.7+4.4,-4.8+0.2) node {{\color{black}\scriptsize{$\ba_R\in \mathbb{R}^I $}}};

	\draw [very thick] (9+4.4,-3-0.2+0.2) rectangle (10.2+4.4,-2.8-0.2+0.2);
	\filldraw [fill=RubineRed!60!white,draw=green!40!black] (9+4.4,-3-0.2+0.2) rectangle (10.2+4.4,-2.8-0.2+0.2);
	\draw (9.6+4.4,-3.3-0.2+0.2) node {{\color{black}\scriptsize{$\bb_R\in \real^J$}}};

	\draw[fill=RedOrange!40!white, line width=0.8pt] (9.2+4.4,-2.4+0.2) -- (9.4+4.4,-2.4+0.2) -- (8.8+4.4,-3+0.2) -- (8.6+4.4,-3+0.2) -- cycle;
	\draw (10.1+4,-1.9) node {{\color{black}\scriptsize{$\bc_R\in \real^{K}$}}};

\end{tikzpicture}
}
\caption{The CP decomposition of a third-order tensor: $\eX \approx \llbracket \bA, \bB, \bC   \rrbracket
	=
	\sum_{r=1}^{R} \ba_r\circ \bb_r \circ \bc_r$ where $\eX\in \real^{I\times J\times K}$, $\bA\in \real^{I\times R}, \bB\in \real^{J\times R}$ and $\bC\in \real^{K\times R}$. Compare to the third-order rank-one tensor in Figure~\ref{fig:rank-one-tensor}.}
\label{fig:cp-decom-third}
\end{figure}

\begin{figure}[H]
	\centering
	\resizebox{0.8\textwidth}{!}{%
\begin{tikzpicture}
	\coordinate (O) at (0,0,0);
	\coordinate (A) at (0,2,0);
	\coordinate (B) at (0,2,2);
	\coordinate (C) at (0,0,2);
	\coordinate (D) at (2,0,0);
	\coordinate (E) at (2,2,0);
	\coordinate (F) at (2,2,2);
	\coordinate (G) at (2,0,2);
	\draw[red!60!black,fill=red!5] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[red!60!black,fill=red!5] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[red!60!black,fill=red!5] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[red!60!black,fill=red!5,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[red!60!black,fill=red!5,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[red!60!black,fill=red!5,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\coordinate (O) at (0+1,0+1,0+1);
	\coordinate (A) at (0+1,0.252+1,0+1);
	\coordinate (B) at (0+1,0.252+1,0.252+1);
	\coordinate (C) at (0+1,0+1,0.252+1);
	\coordinate (D) at (0.252+1,0+1,0+1);
	\coordinate (E) at (0.252+1,0.252+1,0+1);
	\coordinate (F) at (0.252+1,0.252+1,0.252+1);
	\coordinate (G) at (0.252+1,0+1,0.252+1);
	\draw[green!80!black,fill=green!10] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle
	\draw (0.2,-1.2,0) node {\scriptsize{\color{gray}$J$}};
	\draw (0.2,-0.95,0) node[rotate = 0] {{\color{gray!65}$\underbrace{\hspace{2cm}}$}};
	\draw (-0.4,1,2) node[rotate = 90] {\scriptsize{\color{gray}$I$}};
	\draw (-0.15,1,2) node[rotate = 270] {{\color{gray!65}$\underbrace{\hspace{2cm}}$}};
	\draw (2.5,-0.2,1.4) node[rotate = 45] {\scriptsize{\color{gray}$K$}};
	\draw (2.2,0,1.2) node[rotate = 45] {{\color{gray!65}$\underbrace{\hspace{1.1cm}}$}};
	\draw [draw=gray!75,thick,->] (0.8,0.1,1) -- (1,0.8,1) node [right] {{\color{green!50!black}$x_{ijk}$}};
	\draw (0.8,0,1) node {\scriptsize{\color{gray}$(i,j,k)$-th}};
	\draw (1,-1.2,1) node {\color{black}$\eX\in\real^{I\times J\times K}$};

	\draw (3,0.5,0) node[rotate = 0] {{\color{black}\LARGE{$\approx$}}};

	\coordinate (O) at (0+4.5,0+-0.3,0+1);
	\coordinate (A) at (0+4.5,2+-0.3,0+1);
	\coordinate (B) at (0+4.5,2+-0.3,0.252+1);
	\coordinate (C) at (0+4.5,0+-0.3,0.252+1);
	\coordinate (D) at (0.52+4.5,0+-0.3,0+1);
	\coordinate (E) at (0.52+4.5,2+-0.3,0+1);
	\coordinate (F) at (0.52+4.5,2+-0.3,0.252+1);
	\coordinate (G) at (0.52+4.5,0+-0.3,0.252+1);
	\draw[red!60,fill=yellow!25] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[red!60,fill=yellow!25] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[red!60,fill=yellow!25] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[red!60,fill=yellow!25,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[red!60,fill=yellow!25,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[red!60,fill=yellow!25,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\coordinate (O) at (0+4.5,0+1-0.1,0+1);
	\coordinate (A) at (0+4.5,0.252+1-0.1,0+1);
	\coordinate (B) at (0+4.5,0.252+1-0.1,0.252+1);
	\coordinate (C) at (0+4.5,0+1-0.1,0.252+1);
	\coordinate (D) at (0.52+4.5,0+1-0.1,0+1);
	\coordinate (E) at (0.52+4.5,0.252+1-0.1,0+1);
	\coordinate (F) at (0.52+4.5,0.252+1-0.1,0.252+1);
	\coordinate (G) at (0.52+4.5,0+1-0.1,0.252+1);
	\draw[green!80!black,fill=green!10] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\draw[red!60] (0.52+4.5,2+-0.3,0.252+1) -- (0.52+4.5,0+-0.3,0.252+1);
	\draw (4.5+0.2,1-1.9,1) node {\color{black}$\bA\in\real^{I\times R}$};
	\draw (4.5-0.3,1-0.1,1) node {\color{green!50!black}$\boldsymbol{a}_{i}$};

	\coordinate (O) at (0+4.5+1.2,0+1.1,0+1);
	\coordinate (A) at (0+4.5+1.2,0.52+1.1,0+1);
	\coordinate (B) at (0+4.5+1.2,0.52+1.1,0.252+1);
	\coordinate (C) at (0+4.5+1.2,0+1.1,0.252+1);
	\coordinate (D) at (2+4.5+1.2,0+1.1,0+1);
	\coordinate (E) at (2+4.5+1.2,0.52+1.1,0+1);
	\coordinate (F) at (2+4.5+1.2,0.52+1.1,0.252+1);
	\coordinate (G) at (2+4.5+1.2,0+1.1,0.252+1);
	\draw[red!60,fill=blue!15] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[red!60,fill=blue!15] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[red!60,fill=blue!15] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[red!60,fill=blue!15,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[red!60,fill=blue!15,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[red!60,fill=blue!15,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\coordinate (O) at (0+4.5+2.5,0+1.1,0+1);
	\coordinate (A) at (0+4.5+2.5,0.52+1.1,0+1);
	\coordinate (B) at (0+4.5+2.5,0.52+1.1,0.252+1);
	\coordinate (C) at (0+4.5+2.5,0+1.1,0.252+1);
	\coordinate (D) at (0.252+4.5+2.5,0+1.1,0+1);
	\coordinate (E) at (0.252+4.5+2.5,0.52+1.1,0+1);
	\coordinate (F) at (0.252+4.5+2.5,0.52+1.1,0.252+1);
	\coordinate (G) at (0.252+4.5+2.5,0+1.1,0.252+1);
	\draw[green!80!black,fill=green!10] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\draw[red!60] (0+4.5+1.2,0.52+1.1,0.252+1) -- (2+4.5+1.2,0.52+1.1,0.252+1);
	\draw (4.5+2.0,1-0.8,1) node {\color{black}$\bB\in\real^{J\times R}$};
	\draw (4.5+2.6,1-0.3,1) node {\color{green!50!black}$\bb_{j}$};

	\coordinate (O) at (0+4.5+1.2,0+1.1+1.5,0+1);
	\coordinate (A) at (0+4.5+1.2,0.252+1.1+1.5,0+1);
	\coordinate (B) at (0+4.5+1.2,0.252+1.1+1.5,2+1);
	\coordinate (C) at (0+4.5+1.2,0+1.1+1.5,2+1);
	\coordinate (D) at (0.52+4.5+1.2,0+1.1+1.5,0+1);
	\coordinate (E) at (0.52+4.5+1.2,0.252+1.1+1.5,0+1);
	\coordinate (F) at (0.52+4.5+1.2,0.252+1.1+1.5,2+1);
	\coordinate (G) at (0.52+4.5+1.2,0+1.1+1.5,2+1);
	\draw[red!60,fill=red!15] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[red!60,fill=red!15] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[red!60,fill=red!15] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[red!60,fill=red!15,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[red!60,fill=red!15,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[red!60,fill=red!15,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\coordinate (O) at (0+4.5+0.8,0+1+1.2,0+1);
	\coordinate (A) at (0+4.5+0.8,0.252+1+1.2,0+1);
	\coordinate (B) at (0+4.5+0.8,0.252+1+1.2,0.252+1);
	\coordinate (C) at (0+4.5+0.8,0+1+1.2,0.252+1);
	\coordinate (D) at (0.52+4.5+0.8,0+1+1.2,0+1);
	\coordinate (E) at (0.52+4.5+0.8,0.252+1+1.2,0+1);
	\coordinate (F) at (0.52+4.5+0.8,0.252+1+1.2,0.252+1);
	\coordinate (G) at (0.52+4.5+0.8,0+1+1.2,0.252+1);
	\draw[green!80!black,fill=green!10] (O) -- (C) -- (G) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (E) -- (D) -- cycle
	\draw[green!80!black,fill=green!10] (O) -- (A) -- (B) -- (C) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (D) -- (E) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.6] (C) -- (B) -- (F) -- (G) -- cycle
	\draw[green!40!black,fill=green!10,opacity=0.8] (A) -- (B) -- (F) -- (E) -- cycle

	\draw[red!60] (0.52+4.5+1.2,0.252+1.1+1.5,0+1) -- (0.52+4.5+1.2,0.252+1.1+1.5,2+1);
	\draw (4.5+2.5,1+1.3,1) node {\color{black}$\bC\in\real^{K\times R}$};
	\draw (4.5+0.5,1+1.4,1) node {\color{green!50!black}$\bc_{k}$};

\end{tikzpicture}
}
\caption{Index of the CP decomposition of a third-order tensor: $\eX\approx\llbracket \bA,\bB,\bC\rrbracket$.}
\label{fig:cp-decom-third-index}
\end{figure}

\begin{figure}[H]
	\centering
	\resizebox{0.7\textwidth}{!}{%
		\begin{tikzpicture}
\pgfmathsetmacro{\bxx}{0.3}
\pgfmathsetmacro{\begupwhite}{0.2}

\pgfmathsetmacro{\bcwd}{0.3}
\pgfmathsetmacro{\bbwd}{0.3}
\pgfmathsetmacro{\bbright}{0.9}
\pgfmathsetmacro{\bbup}{0.225}

\pgfmathsetmacro{\distrightdown}{4.2}
\pgfmathsetmacro{\distl}{1.2}
\pgfmathsetmacro{\aprimeright}{0.4}

			\draw [very thick] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
			\filldraw [fill=gray!60!white,draw=green!40!black] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
			\draw [step=0.4/2, very thin, color=gray] (5.8+0.5,-3.8+0.5) grid (7+0.5,-2.6+0.5);

			\draw [very thick] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
			\filldraw [fill=gray!50!white,draw=green!40!black] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
			\draw [step=0.4/2, very thin, color=gray] (5.8+0.4,-3.8+0.4) grid (7+0.4,-2.6+0.4);

			\draw [very thick] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
			\filldraw [fill=gray!40!white,draw=green!40!black] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
			\draw [step=0.4/2, very thin, color=gray] (5.8+0.3,-3.8+0.3) grid (7+0.3,-2.6+0.3);

			\draw [very thick] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
			\filldraw [fill=gray!30!white,draw=green!40!black] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
			\draw [step=0.4/2, very thin, color=gray] (5.8+0.2,-3.8+0.2) grid (7+0.2,-2.6+0.2);

			\draw [very thick] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
			\filldraw [fill=gray!20!white,draw=green!40!black] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
			\draw [step=0.4/2, very thin, color=gray] (5.8+0.1,-3.8+0.1) grid (7+0.1,-2.6+0.1);

			\draw [very thick] (5.8,-3.8) rectangle (7,-2.6);
			\filldraw [fill=gray!10!white,draw=green!40!black] (5.8,-3.8) rectangle (7,-2.6);
			\draw [step=0.4/2, very thin, color=gray] (5.8,-3.8) grid (7,-2.6);

			\draw [very thick] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
			\filldraw [fill=gray!6!white,draw=green!40!black] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
			\draw [step=0.4/2, very thin, color=gray] (5.8-0.1,-3.8-0.1) grid (7-0.1,-2.6-0.1);

			\draw[->] (5.6,-2.6) -- ++(0.6,0.6);
			\draw[->] (5.55,-2.7) -- ++(0,-1.2);
			\draw[->] (5.66,-4.04) -- ++(1.25,0);

			\draw (6.35,-4.25) node {{\color{black}\scriptsize{spatial $y$}}};
			\draw (5.2-0.3,-3.2) node {{\color{black}\scriptsize{spatial $x$}}};
			\draw (5.3,-2.2) node {{\color{black}\scriptsize{temporal}}};

			\draw (6.2,-4.6) node {{\color{black}\scriptsize{$\eX\in\real^{I\times J \times K}$}}};

			\draw (8,-3.2) node {	{\color{black}\large{$\approx$}}};

\draw [very thick] (8.6+\distl,-5.0+0.2) rectangle (9.2+\distl,-3.2+0.2);
\filldraw [fill=RubineRed!60!white,draw=green!40!black] (8.6+\distl,-5.0+0.2) rectangle (9.2+\distl,-3.2+0.2);
\draw (8.7+\distl,-5.3+0.2) node {{\color{black}\scriptsize{$\bB \in \real^{J\times R}$}}};

\draw [very thick]
(8.8+\bbright+\distl,-2.9+\bbup) rectangle (10.7+\bbright+\distl,-2.3+\bbup);
\filldraw  [fill=WildStrawberry!40!white,draw=green!40!black]
(8.8+\bbright+\distl,-2.9+\bbup) rectangle (10.7+\bbright+\distl,-2.3+\bbup);
\draw (9.6+\bbright+\distl,-3.2+\bbup) node {{\color{black}\scriptsize{$\bA^\top \in \real^{R\times I}$}}};

\draw[fill=RedOrange!40!white, line width=0.8pt]
(8.45+\distl,-2.0-\bxx) -- (8.8+\distl,-1.8-\bxx) -- (8.25+\distl-0.7,-1.6-\bxx+0.6) -- (7.9+\distl-0.7,-1.8-\bxx+0.6) -- cycle;
\draw (8.8+\distl,-1.5) node {{\color{black}\scriptsize{$\bC\in \real^{K\times R}$}}};

\draw[fill=RedOrange!10!white, line width=0.5pt]
(9.2+\distl,-2.0-\bxx) -- (9.3+\bcwd+\distl-0.02,-1.8+0.01-\bxx) -- (8.7+\bcwd+\distl,-1.8-\bxx) -- (8.6+\distl+0.02,-2.0+0.01-\bxx) -- cycle;
\draw[fill=RedOrange!10!white, line width=0.8pt]
(8.6+\distl,-2.0-\bxx) -- (9.2+\distl,-2.0-\bxx) -- (9.2+\distl,-2.6-\bxx) -- (8.6+\distl,-2.6-\bxx) -- cycle;
\draw[fill=RedOrange!10!white, line width=0.8pt]
(9.3+\bcwd+\distl,-1.8-\bxx) -- (9.2+\distl,-2.1+0.1-\bxx) -- (9.2+\distl,-3+0.4-\bxx) -- (9.3+\bcwd+\distl,-2.4-\bxx) -- cycle;
\draw (10-0.2+\distl,-1.5-\bxx) node {{\color{black}\scriptsize{$\eL\in \real^{R\times R\times R}$}}};

\end{tikzpicture}
}
\caption{The CP decomposition of a third-order tensor: $\eX \approx  \llbracket\eL; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r=1}^{R} \ba_r\circ \bb_r \circ \bc_r$ where $\eX\in \real^{I\times J\times K}$, $\eL\in \real^{R\times R\times R}$, and where columns of $\bA\in \real^{I\times R}, \bB\in \real^{J\times R}$ and $\bC\in \real^{K\times R}$ are of \textcolor{blue}{unit length}.
Compare to the first form of the CP decomposition in Figure~\ref{fig:cp-decom-third}.
}
\label{fig:cp-decom-thirr-2ndform}
\end{figure}

\paragraph{Rank of a tensor} From the CP decomposition, the rank of a tensor $\eX$ can be defined as the smallest $R$ for which the CP decomposition holds exactly.

\paragraph{Matricization}
When $\eX$ is a $2$nd-order tensor (i.e., a matrix), the CP decomposition is just the rank decomposition (Theorem~\ref{theorem:rank-decomposition-alternative}, p.~\pageref{theorem:rank-decomposition-alternative}). For simplicity, we consider the CP decomposition for the third-order tensor $\eX\in \real^{I\times J\times K}$ where the situation is illustrated in Figure~\ref{fig:cp-decom-third}:
\begin{equation}\label{equation:third-fact-1}
\eX \approx \llbracket \bA, \bB, \bC   \rrbracket
=
\sum_{r=1}^{R} \ba_r\circ \bb_r \circ \bc_r ,
\end{equation}
where $\bA\in \real^{I\times R}, \bB\in \real^{J\times R}$ and $\bC\in \real^{K\times R}$.
Element-wise, the above Equation~\eqref{equation:third-fact-1} can be written as
$$
x_{ijk} = \sum_{r=1}^{R}a_{ir}b_{jr}c_{kr},
$$
as shown in Figure~\ref{fig:cp-decom-third-index}. Therefore, the matricized form of the third-order CP decomposition can be written as
\begin{equation}\label{equation:cp-matriciza-third-1}
\left\{
\begin{aligned}
	\bX_{(1)}  &\approx \bA (\bC\odot \bB )^\top \in \real^{I\times (JK)}; \\
	\bX_{(2)} &\approx \bB (\bC\odot \bA )^\top \in \real^{J\times (IK)};\\
	\bX_{(3)} &\approx \bC(\bB\odot \bA)^\top \in \real^{K\times (IJ)},
\end{aligned}
\right.
\end{equation}
where $\bX_{(n)}\in \real^{I\times (JK)}$ is the mode-$n$ matricization of the tensor $\eX$ for $n\in \{1,2,3\}$, ``$\odot$" is the Khatri-Rao product of matrices (Definition~\ref{definition:khatri-rao-product}, p.~\pageref{definition:khatri-rao-product}) such that
$$
\left\{
\begin{aligned}
\bC\odot \bB&=[\bc_1\otimes \bb_1, \bc_2\otimes \bb_2, \ldots, \bc_R\otimes \bb_R]\in \real^{JK\times R};\\
\bC\odot \bA&=[\bc_1\otimes \ba_1, \bc_2\otimes \ba_2, \ldots, \bc_R\otimes \ba_R]\in \real^{IK\times R};\\
\bB\odot \bA&=[\bb_1\otimes \ba_1, \bb_2\otimes \ba_2, \ldots, \bb_R\otimes \ba_R]\in \real^{IJ\times R}.\\
\end{aligned}
\right.
$$

In full generality, come back to the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, the mode-$n$ matricized form of $\eX \approx \llbracket \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  $ is given by
$$
\boxed{\underbrace{\bX_{(n)}}_{I_n\times (I_{-n})}\approx
	\underbrace{\bA^{(n)}}_{{I_n\times R}} \underbrace{\left(\bA^{(N)} \odot \bA^{(N-1)} \odot \ldots \odot \bA^{(n+1)}\odot \bA^{(n-1)}\odot \ldots \odot \bA^{(2)}\odot \bA^{(1)}  \right)^\top}_{R\times (I_{-n})}}
$$
where $I_{-n}=I_1 I_2\ldots I_{n-1}I_{n+1}\ldots I_N$.

\paragraph{Vectorization} Similarly, the vectorization of $\eX \approx \llbracket \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  $ is given by
$$
\boxed{
\begin{aligned}
\underbrace{vec(\eX)}_{(I_1 I_2\ldots I_N)\times 1}
 \approx
\underbrace{\left(\bA^{(N)} \odot \bA^{(N-1)} \odot \ldots  \odot \bA^{(2)}\odot \bA^{(1)}  \right)}_{(I_1 I_2\ldots I_N)\times R} \cdot \underbrace{\bm{1}}_{R\times 1},
\end{aligned}
}
$$
where $\bm{1}=[1,1,\ldots, 1]^\top \in \real^{R\times 1}$.

\paragraph{Equivalent forms on the CP decomposition}
We mentioned in the Theorem~\ref{theorem:cp-decomp} when the columns of $\bA^{(n)}$'s are of unit length with the weights absorbed into the vector $\blambda\in \real^R$, then the CP decomposition of the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ can be written as
$$
\eX \approx \llbracket \blambda; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=\sum_{r=1}^{R} \blambda_r \cdot  \ba_r^{(1)} \circ \ba_r^{(2)} \circ \ldots \circ \ba_r^{(N)} ,
$$
Suppose further the Nth-order tensor $\eL\in \real^{R\times R\times \ldots \times R}$ is a diagonal tensor (Definition~\ref{definition:identity-tensors}, p.~\pageref{definition:identity-tensors}) with
\begin{equation}\label{equation:cp-oth3er-fuorm-nth1}
\left\{
\begin{aligned}
	l_{i,i,\ldots,i} &= \lambda_i, \gap \text{if }i\in \{1,2,\ldots, N\};\\
	l_{i_1,i_2,\ldots,i_N}&= 0, \gap \text{if not \{$i_1=i_2=\ldots=i_N$\}},
\end{aligned}
\right.
\end{equation}
then the CP decomposition of the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ can be written as
\begin{equation}\label{equation:cp-oth3er-fuorm-nth2}
\eX \approx \llbracket \eL; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r_1=1}^{R} \sum_{r_2=1}^{R} \ldots \sum_{r_N=1}^{R}
l_{r_1r_2\ldots r_N}
\ba_{\textcolor{blue}{r_1}}^{(1)} \circ \ba_{\textcolor{blue}{r_2}}^{(2)} \circ \ldots \circ \ba_{\textcolor{blue}{r_N}}^{(N)}.
\end{equation}
By the mode-$n$ tensor multiplication in Equation~\eqref{equation:moden-tensor-multi}, the CP decomposition in Equation~\eqref{equation:cp-oth3er-fuorm-nth2} can also be written as:
\begin{equation}\label{equation:cp-oth3er-fuorm-nth3}
	\eX \approx \llbracket\eL; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
	=
	\eL \times_1 \bA^{(1)} \times_2 \bA^{(2)} \ldots \times_N \bA^{(N)}.
\end{equation}
The CP decomposition in the form of Equation~\eqref{equation:cp-oth3er-fuorm-nth3} for a third-order tensor is then shown in Figure~\ref{fig:cp-decom-thirr-2ndform}.

\section{Computing the CP Decomposition}\index{ALS}
\begin{algorithm}[h]
	\caption{CP Decomposition via ALS}
	\label{alg:cp-decomposition-full-gene}
	\begin{algorithmic}[1]
		\Require Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$;
		\State Pick a rank $R$;
		\State Initialize $\bA^{(n)}\in \real^{I_n\times R}$ for all $n\in \{1,2,\ldots, N\}$ randomly;
		\State Choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$iter<C$}
		\State $iter=iter+1$;
		\For{$n=1,2,\ldots, N$}
		\State $\bV= \bA^{(N)\top} \bA^{(N)}\circledast \ldots \circledast \bA^{(n+1)\top} \bA^{(n+1)}\circledast \bA^{(n-1)\top} \bA^{(n-1)}\circledast \ldots \circledast \bA^{(1)\top} \bA^{(1)} \in \real^{R\times R}$;
		\State $\bW=\left(\bA^{(N)} \odot \bA^{(N-1)} \odot \ldots \odot \bA^{(n+1)}\odot \bA^{(n-1)}\odot \ldots \odot \bA^{(2)}\odot \bA^{(1)}  \right)\in \real^{R\times I_{-n}}$;
		\State $\bX_{(n)} \in \real^{I_n\times I_{-n}}$; \Comment{Calculate the mode-$n$ matricization of tensor $\eX$}
		\State $\bA^{(n)} = \bX_{(n)} \bW\bV^+\in \real^{I_n\times R}$;
		\EndFor
		\EndWhile
		\State Output $\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}$;
	\end{algorithmic}
\end{algorithm}

There is no
finite algorithm for determining the rank of a tensor, and the complexity for it is an NP-hard problem \citep{haastad1989tensor}. Therefore, the rank $R$ in the CP decomposition can not be set up front. However, the rank $R$ can be viewed as a hyperparameter and the CP decomposition thus is approximated by a low-rank tensor decomposition.

In the third-order case, $
\eX \approx \llbracket \bA, \bB, \bC   \rrbracket
=
\sum_{r=1}^{R} \ba_r\circ \bb_r \circ \bc_r
$, suppose we fix the tensor rank $R$, we want to find a rank-$R$ tensor $\weX$ such that
$$
\mathop{\min }_{\weX} ||\eX - \weX||, \qquad \text{with $\weX=\sum_{r=1}^{R} \ba_r\circ \bb_r \circ \bc_r $}.
$$
Having fixed all but one matrix, the problem reduces to a least squares
problem from the matricized form. Therefore, the ALS method can be employed.
The ALS approach fixes $\bB$ and $\bC$ to update for $\bA$; then fixes $\bA$ and $\bC$ to update for $\bB$;
then fixes $\bA$ and $\bB$ to update for $\bC$, and continues to repeat the entire procedure until
some convergence criterion is satisfied or it has enough iterations.

\paragraph{Given $\bB$ and $\bC$, update for $\bA$}
For example, suppose that $\bB$ and $\bC$ are fixed. Then we want to solve
$$
\mathop{\min }_{\widehatbA} ||\bX_{(1)} - \widehatbA (\bC\odot\bB)^\top||.
$$
For simplicity, we only consider the ALS without regularization and bias terms. Recall the update of the ALS in the netflix recommender, the update of $\widehatbA$ is just the same update as that of $\bW$ in Equation~\eqref{equation:als-w-update}: for fixed matrices $\bZ,\bD$, we want to find $\bW$ that $\min||\bD-\bW\bZ||^2$, if $\bZ\bZ^\top$ is invertible, the update is given by
$$
\bW = \bD\bZ^\top (\bZ\bZ^\top)^{-1}  \leftarrow \mathop{\arg\min}_{\bW} ||\bD-\bW\bZ||^2.
$$
Therefore, come back to the update in the third-order CP decomposition, it follows that
$$
\begin{aligned}
\widehatbA &\leftarrow \bX_{(1)} (\bC\odot\bB) \left( (\bC\odot\bB)^\top (\bC\odot\bB)\right)^{-1}\\
&=\bX_{(1)} (\bC\odot\bB)\left( (\bC^\top\bC)\circledast (\bB^\top\bB)\right)^{-1},
\end{aligned}
$$
where the last equality comes from Equation~\eqref{equation:two-khatri-rao-pro-equi}. When $\left( (\bC^\top\bC)\circledast (\bB^\top\bB)\right)$ is not invertible, a pseudo inverse can be applied instead (Appendix~\ref{appendix:pseudo-inverse}, p.~\pageref{appendix:pseudo-inverse}):
$$
\begin{aligned}
	\widehatbA\leftarrow \bX_{(1)} (\bC\odot\bB)\left( (\bC^\top\bC)\circledast (\bB^\top\bB)\right)^{+},
\end{aligned}
$$
where $\bA^+$ is the pseudo-inverse of the matrix $\bA$. The full general procedure for computing the CP decomposition of an Nth-order tensor is then formulated in Algorithm~\ref{alg:cp-decomposition-full-gene}. An analogous update for the \textit{nonnegative CP decomposition} follows immediately from the ``similarity" between the ALS and NMF. One can simply change the ALS update by the multiplicative update (see Equation~\eqref{equation:multi-update-w}, p.~\pageref{equation:multi-update-w}).

\chapter{High-Order SVD (HOSVD)}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{High-Order SVD (HOSVD)}
We mentioned that the HOSVD can be utilized as the initialization for the calculation of the Tucker decomposition. We now consider the properties of the HOSVD.
\begin{theoremHigh}[High-Order SVD (HOSVD)]\label{theorem:hosvd-decomp}
The HOSVD factorizes a tensor into a sum of component rank-one
tensors. For a general Nth-order tensor, $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, it admits the HOSVD
$$
\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}
\ba_{r_1}^{(1)} \circ \ba_{r_2}^{(2)} \circ \ldots \circ \ba_{r_N}^{(N)} ,
$$
where
\begin{itemize}
	\item $R_1< I_1, R_2<I_2, \ldots, R_N<I_N$;
	\item  $\eG$ of size ${R_1\times R_2\times \ldots \times R_N}$ is called the \textit{core tensor} so that $\eG$ can be thought of as the compressed version of $\eX$;
	\item $\bA^{(n)}=[\ba_1^{(n)}, \ba_2^{(n)}, \ldots, \ba_{R_n}^{(n)}] \in \real^{I_n \times R_n}$ for all $n\in \{1,2,\ldots, N\}$ is the column partition of the matrix $\bA^{(n)}\in \real^{I_n \times R_n}$;
	\item The $\bA^{(n)}$'s have mutually orthonormal columns and can be thought of as the principal component of each mode. In this sense, the $\bA^{(n)}$'s are \textit{semi-orthogonal matrices} (see the definition in Section~\ref{section:orthogonal-orthonormal-qr}, p.~\pageref{section:orthogonal-orthonormal-qr});
	\item We can complete the semi-orthogonal matrices into \textit{full orthogonal matrices} by adding \textit{silent columns} into $\bA^{(n)}$'s so that $\bA^{(n)}\in \real^{I_n\times I_n}$ is an orthogonal matrix, in which case, $\eG$ will be expanded to a tensor of size $\real^{I_1\times I_2\times \ldots \times I_N}$ where $g_{r_1r_2\ldots r_N} =0$ when either one of $r_n>R_n$ for $n\in \{1,2,\ldots, N\}$. This is known as the \textit{full HOSVD}, and the previous one is also called the \textit{reduced} one to avoid confusion; And we shall only consider the reduced case in most of our discussions.
\end{itemize}
\textbf{Till now, the HOSVD is the same as the Tucker decomposition}. The difference is as follows:
\begin{itemize}
\item \textit{All orthogonality.} The slices in each mode are mutually orthogonal. Suppose $\eG_{r_n=\alpha}$ is the slice of $\eG$ where the $n$-th index is set to $\alpha$, i.e., an $(N-1)$th-order subtensor: $\eG_{r_n=\alpha}=\eG_{:,\ldots,:,r_n=\alpha,:, \ldots,:}$, then it follows that
\begin{equation}\label{equation:hosvd-conditio1}
\langle  \eG_{r_n=\alpha},  \eG_{r_n=\beta}\rangle =0, \gap \alpha\neq \bbeta \in \{1,2,\ldots, R_n\},
\end{equation}
for all possible value of $n\in \{1,2,\ldots, N\}$.
\item \textit{Ordering.} The Frobenius norms of slices in each mode are decreasing with the increase in the running index:
\begin{equation}\label{equation:hosvd-conditio2}
||\eG_{r_n=1}||_F \geq ||\eG_{r_n=2}||_F \geq \ldots \geq ||\eG_{r_n=I_n}||_F \geq 0,
\end{equation}
for all possible value of $n\in \{1,2,\ldots, N\}$.
\end{itemize}
\end{theoremHigh}
The Frobenius norm $||\eG_{r_n=i}||_F$ is usually denoted as $||\eG_{r_n=i}||_F = \sigma_i^{(n)}$, and known as the \textit{mode-$n$ singular values of $\eX$}. And the vector $\ba_i^{(n)}$ is an \textit{$i$-th mode-$n$ singular vector}. Comparison of the matrix and tensor SVD reveals a clear analogy between
the two cases. In matrix language, suppose the reduced SVD of $\bA=\bU\bSigma\bV^\top\in \real^{M\times N}$, the singular values in matrix $\bSigma\in \real^{R\times R}$ is the norm of each row or each column of $\bSigma$ (since $\bSigma$ is diagonal). Left and right singular vectors in the columns of $\bU,\bV$ respectively now are generalized as the mode-$n$ singular vectors.

\paragraph{Equivalent forms on the HOSVD} The equivalent forms on the HOSVD is just the same as the Tucker decomposition. For the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, by the mode-$n$ tensor multiplication in Equation~\eqref{equation:moden-tensor-multi}, the HOSVD can also be written as:
\begin{equation}\label{equation:tucker-decom-in-hosvd}
	\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
	=
	\eG \times_1 \bA^{(1)} \times_2 \bA^{(2)} \ldots \times_N \bA^{(N)}.
\end{equation}
Note the analogy of SVD for a matrix in Equation~\eqref{equation:svd-by-tensor-multi0}:
\begin{equation}\label{equation:svd-by-tensor-multi1}
	\bA = \bU\bSigma\bV^\top = \bSigma\times_1 \bU \times_2 \bV\in \real^{M\times N}.
\end{equation}
By the result in Lemma~\ref{lemma:tensor-multi2}~\eqref{equationbracketlast}, since $\bA^{(n)}$'s are semi-orthogonal, it also follows that
\begin{equation}\label{equation:tucker-decom-in-hosvd2}
	\eG = \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots, \bA^{(N)\top}\rrbracket   = \eX \times_1 \bA^{(1)\top} \times_2 \bA^{(2)\top} \ldots \times_N \bA^{(N)\top}.
\end{equation}
Element-wise, the $(i_1,i_2,\ldots,i_N)$-th element of $\eX$ can be obtained by
$$
\eX_{i_1,i_2,\ldots,i_N} = \sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}
a_{i_1r_1}^{(1)} a_{i_2r_2}^{(2)} \ldots a_{i_Nr_N}^{(N)}.
$$

\paragraph{Matricization}
In full generality, for the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, the mode-$n$ matricized form is given by
$$
\boxed{\underbrace{\bX_{(n)}}_{I_n\times (I_{-n})}\approx
	\underbrace{\bA^{(n)}}_{{I_n\times R_n}}
	\underbrace{\bG_{(n)}}_{R_n\times (R_{-n})}
	\underbrace{\left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots \otimes \bA^{(n+1)}\otimes \bA^{(n-1)}\otimes \ldots \otimes \bA^{(2)}\otimes \bA^{(1)}  \right)^\top}_{(R_{-n})\times (I_{-n})}
}
$$
where $I_{-n}=I_1 I_2\ldots I_{n-1}I_{n+1}\ldots I_N$ and $R_{-n}=R_1 R_2\ldots R_{n-1}R_{n+1}\ldots R_N$. \textbf{Moreover}, since the conditions in Equation~\eqref{equation:hosvd-conditio1} and \eqref{equation:hosvd-conditio2}, this implies $\bG_{(n)}$ has mutually orthogonal rows (not orthonormal), having vector Frobenius norms equal to $\sigma_1^{(n)}, \sigma_2^{(n)},\ldots, \sigma_{R_n}^{(n)}$.

\paragraph{Matrix SVD in HOSVD}
Let further the diagonal matrix
$$
\begin{aligned}
\bSigma^{(n)} = diag\left(\sigma_1^{(n)}, \sigma_2^{(n)}, \ldots, \sigma_{R_n}^{(n)}\right) \in \real^{R_n\times R_n},
\end{aligned}
$$
where $\sigma_i^{(n)} = ||\eG_{r_n=i}||_F$. This implies, for the row normalized version $\widetildebG_{(n)}$ of $\bG_{(n)}$, we have
$$
\underbrace{\bG_{(n)}}_{R_n\times R_{-n}}=
\underbrace{\bSigma^{(n)}}_{R_n\times R_n} \underbrace{\widetildebG_{(n)}}_{R_n\times R_{-n}}.
$$
Let further,
$$
\underbrace{\bV^{(n)\top}}_{R_n \times I_{-n}}
 =
\underbrace{\widetildebG_{(n)}}_{R_n\times R_{-n}}
\underbrace{\left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots \otimes \bA^{(n+1)}\otimes \bA^{(n-1)}\otimes \ldots \otimes \bA^{(2)}\otimes \bA^{(1)}  \right)^\top}_{(R_{-n})\times (I_{-n})},
$$
where the columns of $\bV^{(n)}$ are mutually orthonormal by Equation~\eqref{equation:orthogonal-in-kronecker1} and \eqref{equation:orthogonal-in-kronecker2}.
This reveals the (matrix) reduced SVD of $\bX_{(n)}$:
\begin{equation}\label{equation:hosvd-initia-for-tucker-1}
\boxed{
	\underbrace{\bX_{(n)}}_{I_n\times (I_{-n})} \approx
	\underbrace{\bA^{(n)}}_{{I_n\times R_n}}
	\underbrace{\bSigma^{(n)}}_{R_n\times R_n}
	\underbrace{\bV^{(n)\top}}_{R_n \times I_{-n}}.
}
\end{equation}
And if \{$\eG, \bA^{(N)}, \ldots, \bA^{(n+1)}, \bA^{(n-1)}, \ldots, \bA^{(1)}$\} are fixed, the update on $\bA^{(n)}$ can be obtained by setting the columns of it via the first $R_n$ left singular vectors of $\bX_{(n)}$. This matches the update in the subproblem of Equation~\eqref{equation:hosvd-initia-for-tucker-0}. The uniqueness of the mode-$n$ singular values thus comes from the uniqueness of the (matrix) reduced SVD.

\paragraph{Vectorization} Going further from the matricization for the Nth-order tensor $\eX$, the vectorization is given by
\begin{equation}\label{equation:hosvd-vec-in-theorem}
	\boxed{\underbrace{vec(\eX) }_{(I_1\ldots I_N)\times 1}
		\approx
		\underbrace{\left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots\otimes \bA^{(1)}  \right)}_{(I_1\ldots I_N)\times (R_1\ldots R_N)}
		\underbrace{vec(\eG)}_{(R_1\ldots R_N)\times 1}.
	}
\end{equation}

\paragraph{Third-order case}
For simplicity and a better understanding, we consider the HOSVD for the third-order tensor $\eX\in \real^{I\times J\times K}$:
\begin{equation}\label{equation:tucker-third-fact-1}
	\eX \approx \llbracket \eG; \bA, \bB, \bC   \rrbracket
	=
	\sum_{p=1}^{P}\sum_{q=1}^{Q}\sum_{r=1}^{R}
	g_{pqr} \cdot \ba_p\circ \bb_q \circ \bc_r,
\end{equation}
where $\bA\in \real^{I\times P}, \bB\in \real^{J\times Q}$, $\bC\in \real^{K\times R}$, and $\eG\in \real^{P\times Q\times R}$. Then, it follows that
\begin{itemize}
\item \textit{All orthogonality.}
$$
\langle  \eG_{:,\alpha,:},  \eG_{:,\beta,:}\rangle =0, \gap \alpha\neq \bbeta \in \{1,2,\ldots, J\}.
$$
\item \textit{Ordering.} The Frobenius norms of slices in each mode are decreasing with the increase in the running index:
$$
||\eG_{:,1,:}||_F \geq ||\eG_{:,2,:}||_F \geq \ldots \geq ||\eG_{:,J,:}||_F \geq 0.
$$
\end{itemize}
The illustration of this third-order HOSVD (reduced and full versions) is similar to that of the Tucker decomposition and is shown in Figure~\ref{fig:tucker-decom-third}.
\section{Computing the HOSVD}

The calculation of the HOSVD is just the (matrix) reduced SVD from the matricized form of the HOSVD in Equation~\eqref{equation:hosvd-initia-for-tucker-1}. The procedure is shown in Algorithm~\ref{alg:hosvd-decomposition-full-gene} and we shall not repeat the details. See also \citep{de2000multilinear} for a more detailed discussion.
\begin{algorithm}[h]
\caption{HOSVD via ALS}
\label{alg:hosvd-decomposition-full-gene}
\begin{algorithmic}[1]
\Require Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$;
\State Pick a rank $R_1, R_2, \ldots, R_N$;
\State initialize $\bA^{(n)}\in \real^{I_n\times R_n}$ for all $n\in \{1,2,\ldots, N\}$ randomly;
\State choose maximal number of iterations $C$;
\State $iter=0$;
\While{$iter<C$}
\State $iter=iter+1$;
\For{$n=1,2,\ldots, N$}
\State Find the matricization along mode-$n$: $\bY_{(n)}$;
\State Set the rows of $\bA^{(n)}$ by first $R_n$ leading left singular vectors of $\bX_{(n)}$;
\EndFor
\EndWhile
\State $\eG = \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots, \bA^{(N)\top}\rrbracket$;
\State Output $\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}, \eG$;
\end{algorithmic}
\end{algorithm}

\section{Properties of the HOSVD}

\subsection{Frobenius Norm}
\begin{lemma}[Frobenius Norm of a Tensor]\label{lemma:frobenius-hosvd}
Suppose the HOSVD of the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ is given by $ \eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  $, then it follows that
$$
\begin{aligned}
	||\eX||_F^2 &= ||\eG||_F^2 \\
	&=\sum_{i=1}^{R_1}  (\sigma_i^{(1)})^2
	=\sum_{i=1}^{R_2}  (\sigma_i^{(2)})^2
	=\ldots
	=\sum_{i=1}^{R_N}  (\sigma_i^{(N)})^2 .
\end{aligned}
$$
\end{lemma}
The squared Frobenius norm of a matrix is defined to be the sum of squares of the singular values of the matrix (Definition~\ref{definition:frobernius-in-svd}, p.~\pageref{definition:frobernius-in-svd}). The above lemma tells us that the singular values in each slice of a tensor have a similar property. The proof of the above lemma follows immediately from the (matrix) reduced SVD of the matricized form of the HOSVD in Equation~\eqref{equation:hosvd-initia-for-tucker-1}.

\subsection{Low-Rank Approximation}
We discussed the best rank-$k$ approximation of a matrix is from the truncated SVD (Theorem~\ref{theorem:young-theorem-spectral}, p.~\pageref{theorem:young-theorem-spectral}) where
$$
||\bA - \bA_k||_F^2 = \sigma_{k+1}^2+\sigma_{k+2}^2 +\ldots+\sigma_{\min\{M,N\}}^2
$$
if $\bA\in \real^{M\times N}$. Similar result follows in the lower-rank tensor approximation.
\begin{lemma}[Low-Rank Approximation]\label{lemma:low-rank-hosvd}
Suppose the HOSVD of the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ is given by $ \eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  $. Define a tensor $\weX$ by discarding the smallest mode-$n$ singular values $\sigma_{K_n+1}^{(n)}, \sigma_{K_n+2}^{(n)}, \ldots, \sigma_{R_n}^{(n)}$ for given values of $K_n\leq R_n$ (for all $n\in \{1,2,\ldots, N\}$). Then, it follows that
$$
||\eX-\weX||_F^2 \leq
\sum_{r_1=K_1+1}^{R_1} (\sigma_{r_1}^{(1)})^2+
\sum_{r_2=K_2+1}^{R_2} (\sigma_{r_2}^{(2)})^2+
\ldots
+
\sum_{r_N=K_N+1}^{R_N} (\sigma_{r_N}^{(N)})^2.
$$
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:low-rank-hosvd}]
From Lemma~\ref{lemma:frobenius-hosvd}, we have the following
$$
\begin{aligned}
||\eX-\weX||_F^2&=
\sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}^2
-
\sum_{r_1=1}^{K_1} \sum_{r_2=1}^{K_2} \ldots \sum_{r_N=1}^{K_N}
g_{r_1r_2\ldots r_N}^2 \\
&=\sum_{r_1=K_1+1}^{R_1} \sum_{r_2=K_2+1}^{R_2} \ldots \sum_{r_N=K_N+1}^{R_N}
g_{r_1r_2\ldots r_N}^2 \\
&\leq
\sum_{\textcolor{blue}{r_1=K_1+1}}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}^2 \\
&+
\sum_{r_1=1}^{R_1} \sum_{\textcolor{blue}{r_2=K_2+1}}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}^2 \\
&+
\ldots
+
\sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{\textcolor{blue}{r_N=K_N+1}}^{R_N}
g_{r_1r_2\ldots r_N}^2 \\
&=\sum_{r_1=K_1+1}^{R_1} (\sigma_{r_1}^{(1)})^2+
\sum_{r_2=K_2+1}^{R_2} (\sigma_{r_2}^{(2)})^2+
\ldots
+
\sum_{r_N=K_N+1}^{R_N} (\sigma_{r_N}^{(N)})^2.
\end{aligned}
$$
This completes the proof.
\end{proof}

\part{Tensor Decomposition}

\chapter{Notations and Background}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Matrices to Tensors}
In this section, we briefly overview the background for tensor analysis and we follow the notations largely from the review of \citep{kolda2006multilinear, kolda2009tensor, cichocki2016tensor} and \citep{golub2013matrix}.
A tensor is a multidimensional array where when the dimension is $N$ we call the tensor an \textit{Nth-order tensor}. For example, a first-order tensor is a vector and a second-order tensor is a matrix. In this sense,
tensors are multidimensional extensions of matrices, which are used to represent ubiquitous multidimensional data,
such as RGB images, hyperspectral images, and videos.

Previously, we use \textbf{boldface} lower case letters possibly with subscripts to denote vectors (e.g., $\bmu$, $\bx$, $\bx_n$, $\bz$) and
\textbf{boldface} upper case letters possibly with subscripts to denote matrices (e.g., $\bA$, $\bL_j$). To avoid confusion, we will use a \textbf{boldface} Euler script letters to denote a tensor with order larger than 2, e.g., $\eX, \eY,\eA,\eB,\eG \in \real^{I\times J\times K}$ for third-order tensors.

In a second-order tensor (i.e., a matrix), the first axis is usually taken as the \textbf{coordinate of spatial $x$} or \textbf{height}, and the second axis is then taken as the \textbf{coordinate of spatial $y$} or \textbf{width}, e.g., Figure~\ref{fig:tensor-2dd} is an example of a second-order tensor $\bX\in \real^{I\times J}$. A third-order tensor goes further by adding a third dimension which is usually known as a \textbf{``temporal" coordinate} (or a \textbf{``channel" coordinate} in an RGB picture). The situation of a third-order tensor $\eX\in \real^{I\times J\times K}$ is shown in Figure~\ref{fig:tensor-3dd} in which case $K=3$ in an RGB picture scenario.
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[2nd-order tensor]{\label{fig:tensor-2dd}%
		\includegraphics[width=0.33\linewidth]{./imgs/tensormatrix1.pdf}}%
	\subfigure[3rd-order tensor]{\label{fig:tensor-3dd}%
		\includegraphics[width=0.5\linewidth]{./imgs/tensormatrix2.pdf}}%
	\caption{A 2nd-order tensor into a 3rd-order tensor.}
	\label{fig:tensor-2d-2-3d}
\end{figure}

\section{Tensor Indexing}\index{Tensor indexing}
\textit{Slices} are two-dimensional sections of a tensor, defined by fixing all but two indices. Figure~\ref{fig:thifd-slices} shows the slices of a third-order tensor $\eX\in \real^{I\times J\times K}$ where the \textit{horizontal ones} are the slices by varying the first index and fixing the last two: $\eX_{i,:,:}$ for all $i\in \{1,2,\ldots, I\}$ \footnote{where the indices typically range from 1 to their capital version, here, $i\in \{1,2,\ldots, I\}$.}; the \textit{lateral ones} are the slices by varying the second index: $\eX_{:,j,:}$ for all $j\in \{1,2,\ldots, J\}$; and the \textit{frontal ones} are the slices by varying the third index: $\eX_{:,:,k}$ for all $k\in \{1,2,\ldots, K\}$.

\index{Tensor slices}
\index{Tensor fibers}

Similarly, \textit{Fibers} are the higher-order analogue of matrix rows and columns. A fiber is defined by fixing every index but one. Figure~\ref{fig:thifd-fibers} shows the fibers of a third-order tensor $\eX\in \real^{I\times J\times K}$ where the \textit{column fibers} are the ones by varying only the first index: $\eX_{:,j,k}$; the \textit{row fibers} are the ones by varying only the second index: $\eX_{i,:,k}$; and the \textit{tube fibers} are the ones by varying only the third index: $\eX_{i,j,:}$.

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Horizontal slices $\eX_{i,:,:}$]{\label{fig:third-slice1}%
		\centering
		\begin{minipage}[b]{0.32\linewidth}
			\centering
			\begin{tikzpicture}
				\pgfmathsetmacro{\cubex}{3}
				\pgfmathsetmacro{\cubey}{0.1}
				\pgfmathsetmacro{\cubez}{3}
				\draw[black,fill=gray!70!white] (0,-6*0.5+0.1,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (0,-6*0.5+0.1,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (0,-6*0.5+0.1,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (0,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (0,-5*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (0,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (0,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (0,-4*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (0,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (0,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (0,-3*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (0,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (0,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (0,-2*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (0,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (0,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (0,-0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (0,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (0,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (0,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (0,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
			\end{tikzpicture}
		\end{minipage}%
	}
	\subfigure[Lateral slices $\eX_{:,j,:}$]{\label{fig:third-slice2}%
		\centering
		\begin{minipage}[b]{0.32\linewidth}
			\centering
			\begin{tikzpicture}
				\pgfmathsetmacro{\cubex}{0.1}
				\pgfmathsetmacro{\cubey}{3}
				\pgfmathsetmacro{\cubez}{3}

				\draw[black,fill=gray!10!white] (3-6*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3-5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3-4*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3-3*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3-2*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3-0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
			\end{tikzpicture}
		\end{minipage}%
	}
	\subfigure[Frontal slices $\eX_{:,:,k}$]{\label{fig:thirdslice3}%
		\centering
		\begin{minipage}[b]{0.32\linewidth}
			\centering
			\begin{tikzpicture}
				\pgfmathsetmacro{\cubex}{3}
				\pgfmathsetmacro{\cubey}{3}
				\pgfmathsetmacro{\cubez}{0.1}

				\draw[black,fill=gray!70!white] (6,0,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6,0,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6,0,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (6,0,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6,0,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6,0,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (6,0,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6,0,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6,0,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6,0,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6,0,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6,0,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6,0,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6,0,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6,0,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6,0,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6,0,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6,0,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
			\end{tikzpicture}
		\end{minipage}%
	}
	\caption{Slices of a third-order tensor $\eX\in \real^{I\times J\times K}$. The \textbf{darker}, the larger the index.}
	\label{fig:thifd-slices}
\end{figure}

\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Mode-1 fibers: columns, $\eX_{:,j,k}$]{\label{fig:third-fibdr1}%
		\centering
		\begin{minipage}[b]{0.32\linewidth}
			\centering
			\begin{tikzpicture}
				\pgfmathsetmacro{\cubex}{0.3}
				\pgfmathsetmacro{\cubey}{3}
				\pgfmathsetmacro{\cubez}{0.3}

				\draw[black,fill=gray!10!white] (3-6*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,6*0.008,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3-5*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,6*0.008,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3-4*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,6*0.008,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3-3*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,6*0.008,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3-2*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,6*0.008,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3-0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,6*0.008,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3,6*0.008,-6*0.5)-- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3,6*0.008,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (3-6*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,4.5*0.008,-4.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3-5*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,4.5*0.008,-4.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3-4*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,4.5*0.008,-4.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3-3*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,4.5*0.008,-4.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3-2*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,4.5*0.008,-4.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3-0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,4.5*0.008,-4.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3,4.5*0.008,-4.5*0.5)-- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3,4.5*0.008,-4.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (3-6*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,3*0.008,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3-5*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,3*0.008,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3-4*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,3*0.008,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3-3*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,3*0.008,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3-2*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,3*0.008,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3-0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,5*0.008,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3,3*0.008,-3*0.5)-- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3,3*0.008,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (3-6*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,1.5*0.008,-1.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3-5*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,1.5*0.008,-1.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3-4*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,1.5*0.008,-1.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3-3*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,1.5*0.008,-1.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3-2*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,1.5*0.008,-1.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3-0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,5*0.008,-1.5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3,1.5*0.008,-1.5*0.5)-- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3,1.5*0.008,-1.5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (3-6*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3-6*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3-5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3-5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3-4*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3-4*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3-3*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3-3*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3-2*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3-2*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3-0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3-0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
			\end{tikzpicture}
		\end{minipage}%
	}
	\subfigure[Mode-2 fibers: rows, $\eX_{i,:,k}$]{\label{fig:thirds-fiber2}%
		\centering
		\begin{minipage}[b]{0.32\linewidth}
			\centering
			\begin{tikzpicture}
				\pgfmathsetmacro{\cubex}{3}
				\pgfmathsetmacro{\cubey}{0.3}
				\pgfmathsetmacro{\cubez}{0.3}

				\draw[black,fill=gray!70!white] (6,0,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6,0,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6,0,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6,0,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6,0,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6,0,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6,0,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6,0,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6,0,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6,0,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6,0,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6,0,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6,0,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6,0,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6,0,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6,0,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6,0,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6,0,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (6-1.5*0.008,1.5*0.5-0.1,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6-1.5*0.008,1.5*0.5-0.1,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6-1.5*0.008,1.5*0.5-0.1,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-1.5*0.008,1.5*0.5-0.1,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-1.5*0.008,1.5*0.5-0.1,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6-1.5*0.008,1.5*0.5-0.1,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-1.5*0.008,1.5*0.5-0.1,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-1.5*0.008,1.5*0.5-0.1,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6-1.5*0.008,1.5*0.5-0.1,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-1.5*0.008,1.5*0.5-0.1,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-1.5*0.008,1.5*0.5-0.1,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6-1.5*0.008,1.5*0.5-0.1,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-1.5*0.008,1.5*0.5-0.1,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-1.5*0.008,1.5*0.5-0.1,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6-1.5*0.008,1.5*0.5-0.1,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-1.5*0.008,1.5*0.5-0.1,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-1.5*0.008,1.5*0.5-0.1,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6-1.5*0.008,1.5*0.5-0.1,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-1.5*0.008,1.5*0.5-0.1,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-1.5*0.008,1.5*0.5-0.1,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6-1.5*0.008,1.5*0.5-0.1,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (6-3*0.008,3*0.5-0.17,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6-3*0.008,3*0.5-0.17,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6-3*0.008,3*0.5-0.17,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-3*0.008,3*0.5-0.17,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-3*0.008,3*0.5-0.17,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6-3*0.008,3*0.5-0.17,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-3*0.008,3*0.5-0.17,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-3*0.008,3*0.5-0.17,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6-3*0.008,3*0.5-0.17,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-3*0.008,3*0.5-0.17,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-3*0.008,3*0.5-0.17,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6-3*0.008,3*0.5-0.17,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-3*0.008,3*0.5-0.17,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-3*0.008,3*0.5-0.17,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6-3*0.008,3*0.5-0.17,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-3*0.008,3*0.5-0.17,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-3*0.008,3*0.5-0.17,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6-3*0.008,3*0.5-0.17,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-3*0.008,3*0.5-0.17,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-3*0.008,3*0.5-0.17,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6-3*0.008,3*0.5-0.17,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (6-4.5*0.008,4.5*0.5-0.24,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6-4.5*0.008,4.5*0.5-0.24,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6-4.5*0.008,4.5*0.5-0.24,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-4.5*0.008,4.5*0.5-0.24,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-4.5*0.008,4.5*0.5-0.24,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6-4.5*0.008,4.5*0.5-0.24,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-4.5*0.008,4.5*0.5-0.24,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-4.5*0.008,4.5*0.5-0.24,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6-4.5*0.008,4.5*0.5-0.24,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-4.5*0.008,4.5*0.5-0.24,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-4.5*0.008,4.5*0.5-0.24,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6-4.5*0.008,4.5*0.5-0.24,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-4.5*0.008,4.5*0.5-0.24,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-4.5*0.008,4.5*0.5-0.24,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6-4.5*0.008,4.5*0.5-0.24,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-4.5*0.008,4.5*0.5-0.24,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-4.5*0.008,4.5*0.5-0.24,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6-4.5*0.008,4.5*0.5-0.24,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-4.5*0.008,4.5*0.5-0.24,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-4.5*0.008,4.5*0.5-0.24,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6-4.5*0.008,4.5*0.5-0.24,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (6-6*0.008,6*0.5-0.3,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6-6*0.008,6*0.5-0.3,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6-6*0.008,6*0.5-0.3,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-6*0.008,6*0.5-0.3,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6-6*0.008,6*0.5-0.3,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6-6*0.008,6*0.5-0.3,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-6*0.008,6*0.5-0.3,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6-6*0.008,6*0.5-0.3,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6-6*0.008,6*0.5-0.3,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-6*0.008,6*0.5-0.3,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6-6*0.008,6*0.5-0.3,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6-6*0.008,6*0.5-0.3,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-6*0.008,6*0.5-0.3,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6-6*0.008,6*0.5-0.3,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6-6*0.008,6*0.5-0.3,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-6*0.008,6*0.5-0.3,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6-6*0.008,6*0.5-0.3,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6-6*0.008,6*0.5-0.3,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-6*0.008,6*0.5-0.3,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6-6*0.008,6*0.5-0.3,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6-6*0.008,6*0.5-0.3,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

			\end{tikzpicture}
		\end{minipage}%
	}
	\subfigure[Mode-3 fibers: tubes, $\eX_{i,j,:}$]{\label{fig:third-fiber3}%
		\centering
		\begin{minipage}[b]{0.32\linewidth}
			\centering
			\begin{tikzpicture}
				\pgfmathsetmacro{\cubex}{0.3}
				\pgfmathsetmacro{\cubey}{0.3}
				\pgfmathsetmacro{\cubez}{3}

				\draw[black,fill=gray!70!white] (0,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (0,-6*0.5+0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (0,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (0,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (0,-5*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (0,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (0,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (0,-4*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (0,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (0,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (0,-3*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (0,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (0,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (0,-2*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (0,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (0,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (0,-0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (0,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (0,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (0,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (0,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (1.5*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (1.5*0.5,-6*0.5+0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (1.5*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (1.5*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (1.5*0.5,-5*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (1.5*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (1.5*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (1.5*0.5,-4*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (1.5*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (1.5*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (1.5*0.5,-3*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (1.5*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (1.5*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (1.5*0.5,-2*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (1.5*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (1.5*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (1.5*0.5,-0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (1.5*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (1.5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (1.5*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (1.5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (3*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (3*0.5,-6*0.5+0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (3*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (3*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (3*0.5,-5*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (3*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (3*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (3*0.5,-4*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (3*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (3*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (3*0.5,-3*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (3*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (3*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (3*0.5,-2*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (3*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (3*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (3*0.5,-0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (3*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (3*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (3*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (3*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (4.5*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (4.5*0.5,-6*0.5+0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (4.5*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (4.5*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (4.5*0.5,-5*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (4.5*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (4.5*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (4.5*0.5,-4*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (4.5*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (4.5*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (4.5*0.5,-3*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (4.5*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (4.5*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (4.5*0.5,-2*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (4.5*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (4.5*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (4.5*0.5,-0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (4.5*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (4.5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (4.5*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (4.5*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!70!white] (6*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!70!white] (6*0.5,-6*0.5+0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!70!white] (6*0.5,-6*0.5+0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!60!white] (6*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!60!white] (6*0.5,-5*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!60!white] (6*0.5,-5*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!50!white] (6*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!50!white] (6*0.5,-4*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!50!white] (6*0.5,-4*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!40!white] (6*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!40!white] (6*0.5,-3*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!40!white] (6*0.5,-3*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!30!white] (6*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!30!white] (6*0.5,-2*0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!30!white] (6*0.5,-2*0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!20!white] (6*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!20!white] (6*0.5,-0.5,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!20!white] (6*0.5,-0.5,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

				\draw[black,fill=gray!10!white] (6*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
				\draw[black,fill=gray!10!white] (6*0.5,0,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
				\draw[black,fill=gray!10!white] (6*0.5,0,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

			\end{tikzpicture}
		\end{minipage}%
	}

	\caption{Fibers of a third-order tensor $\eX\in \real^{I\times J\times K}$.}
	\label{fig:thifd-fibers}
\end{figure}

More generally, the $(i,j,k)$-th element of a third-order tensor $\eX\in \real^{I\times J\times K}$ is indexed by $x_{ijk}$. And the general index for an Nth-order tensor can be implied from the context.

\section{Inner Product and Frobernius Norm}
The inner product between two Nth-order tensors with same size $\eX,\eY\in \real^{I_1\times I_2\times \ldots \times I_N}$ is the sum of the product of their entries element-wise:
$$
\langle \eX,\eY\rangle = \sum_{i_1}^{I_1}\sum_{i_2}^{I_2}\ldots \sum_{i_N}^{I_N} (x_{i_1,i_2,\ldots, i_N})\cdot (y_{i_1,i_2,\ldots, i_N}).
$$
Similarly, the Frobenius norm of an Nth-order tensor $\eX\in \real^{I_1\times I_2\times \ldots \times I_N}$ is given by $\sqrt{\langle \eX,\eX\rangle}$:
$$
||\eX||_F = \sqrt{\sum_{i_1}^{I_1}\sum_{i_2}^{I_2}\ldots \sum_{i_N}^{I_N} (x_{i_1,i_2,\ldots, i_N})^2}.
$$

\section{Outer Product and Rank-One Tensor}\label{section:rank-one-tensor}\index{Rank-one tensor}
If given two vectors $\ba\in \real^I, \bb\in \real^J$, the outer product is $\ba\bb^\top \in \real^{I\times J}$ in a trivial vector language for two vectors. Analogously, in tensor language, we use the symbol $``\circ"$ to denote the outer product, e.g., $\ba\bb^\top = \ba \circ\bb$ for the outer product of two vectors. For higher dimensions, the outer product of $N$ vectors $\ba^{(1)}\in \real^{I_1}, \ba^{(2)}\in \real^{I_2}, \ldots, \ba^{(N)}\in \real^{I_N}$ is given by
$$
\ba^{(1)}\circ \ba^{(2)}\circ \ldots \circ \ba^{(N)},
$$
where the $(i_1,i_2,\ldots, i_N)$-th element can be obtained by
$$
(\ba^{(1)}\circ \ba^{(2)}\circ \ldots \circ \ba^{(N)})_{i_1,i_2,\ldots, i_N}=
a^{(1)}_{i_1}a^{(2)}_{i_2}\ldots a^{(N)}_{i_N}.
$$
An Nth-order tensor $\eX \in  \real^{I_1\times I_2\times \ldots \times I_N}$ is of rank-one if it
can be written as the outer product of $N$ vectors, i.e.,

$$
\eX = \ba^{(1)}\circ \ba^{(2)}\circ \ldots \circ \ba^{(N)},
$$
The $(i_1,i_2,\ldots, i_N)$-th element of $\eX$ is thus given by
$$
\eX_{i_1,i_2,\ldots, i_N} = x_{i_1,i_2,\ldots, i_N} = a_{i_1}^{(1)}a_{i_2}^{(2)}\ldots a_{i_N}^{(N)}.
$$
And the situation for a third-order rank-one tensor is shown in Figure~\ref{fig:rank-one-tensor}.

\begin{figure}[htbp]
\centering
\resizebox{0.65\textwidth}{!}{%
\begin{tikzpicture}
\draw [very thick] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\filldraw [fill=gray!60!white,draw=green!40!black] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.5,-3.8+0.5) grid (7+0.5,-2.6+0.5);

\draw [very thick] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\filldraw [fill=gray!50!white,draw=green!40!black] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.4,-3.8+0.4) grid (7+0.4,-2.6+0.4);

\draw [very thick] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\filldraw [fill=gray!40!white,draw=green!40!black] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.3,-3.8+0.3) grid (7+0.3,-2.6+0.3);

\draw [very thick] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\filldraw [fill=gray!30!white,draw=green!40!black] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.2,-3.8+0.2) grid (7+0.2,-2.6+0.2);

\draw [very thick] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\filldraw [fill=gray!20!white,draw=green!40!black] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.1,-3.8+0.1) grid (7+0.1,-2.6+0.1);

\draw [very thick] (5.8,-3.8) rectangle (7,-2.6);
\filldraw [fill=gray!10!white,draw=green!40!black] (5.8,-3.8) rectangle (7,-2.6);
\draw [step=0.4/2, very thin, color=gray] (5.8,-3.8) grid (7,-2.6);

\draw [very thick] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\filldraw [fill=gray!6!white,draw=green!40!black] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8-0.1,-3.8-0.1) grid (7-0.1,-2.6-0.1);

\draw[->] (5.6,-2.6) -- ++(0.6,0.6);
\draw[->] (5.55,-2.7) -- ++(0,-1.2);
\draw[->] (5.66,-4.04) -- ++(1.25,0);

\draw (6.35,-4.25) node {{\color{black}\scriptsize{spatial $y$}}};
\draw (5.2-0.3,-3.2) node {{\color{black}\scriptsize{spatial $x$}}};
\draw (5.3,-2.2) node {{\color{black}\scriptsize{temporal}}};

\draw (6.2,-4.6) node {{\color{black}\scriptsize{$\eX\in\real^{I\times J \times K}$}}};

\draw (8,-3.2) node {{\color{black}\large{$=$}}};

\draw [very thick] (8.6,-4.4+0.2) rectangle (8.8,-3.2+0.2);
\filldraw [fill=WildStrawberry!40!white,draw=green!40!black] (8.6,-4.4+0.2) rectangle (8.8,-3.2+0.2);
\draw (8.7,-4.8+0.2) node {{\color{black}\scriptsize{$\ba \in \real^I$}}};

\draw [very thick] (9,-3-0.2+0.2) rectangle (10.2,-2.8-0.2+0.2);
\filldraw [fill=RubineRed!60!white,draw=green!40!black] (9,-3-0.2+0.2) rectangle (10.2,-2.8-0.2+0.2);
\draw (9.6,-3.3-0.2+0.2) node {{\color{black}\scriptsize{$\bb \in \real^J$}}};

\draw[fill=RedOrange!40!white, line width=0.8pt] (9.2,-2.4+0.2) -- (9.4,-2.4+0.2) -- (8.8,-3+0.2) -- (8.6,-3+0.2) -- cycle;
\draw (10-0.2,-1.9) node {{\color{black}\scriptsize{$\bc\in \real^K$}}};
\end{tikzpicture}
}
\caption{A third-order tensor with rank-one, $\eX=\ba\circ\bb\circ\bc$ where the $(i, j, k)$-th element of $\eX$ is given by $x_{ijk} = a_ib_j c_k$.}
\label{fig:rank-one-tensor}
\end{figure}

\section{Diagonal and Identity Tensors}
Diagonal matrices and identity matrices have their counterparts in tensor language:
\begin{definition}[Diagonal and Identity Tensors]\label{definition:identity-tensors}
A tensor $\eX \in  \real^{I_1\times I_2\times \ldots \times I_N}$ is called a \textit{diagonal tensor} if $x_{i_1i_2\ldots i_N}\neq 0$ if and only if $i_1=i_2=\ldots=i_N$. And when $x_{i_1i_2\ldots i_N}=1$ if and only if $i_1=i_2=\ldots=i_N$, the tensor is known as the \textit{Nth-order identity tensor}.
\end{definition}

\section{Matricization: Matrix Representation of a Higher-Order Tensor}\label{section:matricization-tensor-original}
There are several kinds of matrix representations of an Nth-order tensor $\eX\in \real^{I_1\times I_2\times \ldots \times I_N}$. But for the matrix representation along the $n$-th mode, the size will always be $I_n\times (I_1\ldots I_{n-1} I_{n+1} \ldots  I_N)$, which is also called the \textbf{mode-$n$ matricization of
	the tensor} $\eX$ and is denoted by $\bX_{(n)}$.

To see this, we consider a specific example $\eY\in \real^{2\times 4\times 3}$ containing $\{1,2,\ldots, 24\}$ as entries where each number is stored in ascending order from the first index (height), then the second index (width), and finally the third index (channel) as shown in Figure~\ref{fig:3rd-example}:

\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\textwidth]{imgs/tensormatrix3.pdf}
	\caption{An example of a 3rd-order tensor: $\eY\in \real^{2\times 4\times 3}$ where $y_{ijk}=1+(i-1)+2(j-1)+8(k-1)$.}
	\label{fig:3rd-example}
\end{figure}

\noindent Mathematically, each entry of $\eY$ can be described by
$$
y_{ijk} = 1+(i-1)+2(j-1)+8(k-1).
$$
Now, suppose we want the matricized form along the 1st-mode, $\bY_{(1)}\in \real^{2\times 12}$, the number will be \textbf{\textcolor{blue}{fetched}} first from the 1st index of $\eY$ (height), then the 2nd index (width), and finally the 3rd index (channel). And the number is \textbf{\textcolor{blue}{stored}} into $\bY_{(1)}$ from the first index, and then the second:
$$
\bY_{(1)}=
\begin{bmatrix}
1&3 & 5& 7&9 & 11& 13& 15& 17& 19 \,\,\,\, 21\,\,\,\,23\\
2&4 & 6& 8&10 & 12& 14& 16& 18& 20\,\,\,\, 22\,\,\,\,24\\
\end{bmatrix}.
$$
If we want the model-2 matricization of tensor $\eY$, i.e., $\bY_{(2)}\in \real^{4\times 6}$, the number will be \textbf{\textcolor{blue}{fetched}} first from the 2nd index (width), then the 1st index (height), and finally the 3rd index (channel). \textbf{The storing of the number will always be the same:} the number is \textbf{\textcolor{blue}{stored}} into $\bY_{(2)}$ from the first index, and then the second:
$$
\bY_{(2)}=
\begin{bmatrix}
1& 2& 9& 10& 17& 18 \\
3& 4& 11& 12& 19& 20\\
5& 6& 13& 14& 21& 22\\
7& 8& 15& 16& 23& 24\\
\end{bmatrix}.
$$
If we want the model-3 matricization of tensor $\eY$, i.e., $\bY_{(3)}\in \real^{3\times 8}$, the number will be \textbf{\textcolor{blue}{fetched}} first from the the 3rd index (channel), then 1st index (height), and finally the 2nd index (width). \textbf{The storing of the number will still be the same}:
$$
\bY_{(3)}=
\begin{bmatrix}
1& 2& 3&    4&  5&  6&  7& 8\\
9& 10& 11&  12& 13& 14& 15& 16 \\
17& 18& 19& 20& 21& 22& 23&  24\\
\end{bmatrix}.
$$
More generally, given the Nth-order tensor $\eX\in \real^{I_1\times I_2\times \ldots \times I_N}$, in the model-$n$ matricization, the tensor element $(i_1,i_2,\ldots, i_N)$ is mapped into matrix element $(i_n, j)$:
$$
j= 1+\sum_{k=1,k\neq n}^{N} (i_k-1) J_k, \qquad \text{where }\gap J_k = \prod_{m=1,m\neq n}^{k-1} I_m.
$$

Specially, the \textit{vectorization} of a tensor is defined in a similar way where we fetch the entries in an ordered manner that fetches first from the first index, then the second, \ldots, and finally the last index:
$$
vec(\eX) =
\begin{bmatrix}
1\\2\\ \vdots\\24
\end{bmatrix}.
$$

However, the matricization and vectorization of a tensor are not unique, as long as we keep the entries consistent, they are the same in the analysis. See also \citep{de2000multilinear, kiers2000towards, kolda2009tensor}.

\paragraph{Norm of the difference of two tensors} The matricization or vectorization can help derive the properties of the tensors. For example, given $\eX,\eY\in \real^{I_1\times I_2\times \ldots \times I_N}$, it follows that
\begin{equation}\label{equation:differe-tensor-norm}
\begin{aligned}
	||\eX-\eY||^2 &= ||vec(\eX)-vec(\eY)||^2 = ||vec(\eX)||^2 - 2vec(\eX)^\top vec(\eY) + ||vec(\eY)||^2\\
	&=||\eX||^2 -2\langle \eX,\eY\rangle + ||\eY||^2.
\end{aligned}
\end{equation}

\section{Tensor Multiplication}
We consider the \textit{mode-$n$ tensor multiplication}: multiply the tensor in the $n$-th mode. Given an Nth-order tensor $\eX \in  \real^{I_1\times I_2\times \ldots \times I_N}$ and a matrix $\bA\in \real^{M\times I_n}$, the mode-$n$ tensor multiplication of $\eX$ and $\bA$ will transfer the $n$-th dimension of $\eX$ from $I_n$ to $M$. The mode-$n$ tensor multiplication is denoted by $\eX \times_n \bA \in \real^{I_1\times \ldots \times I_{n-1}\times \textcolor{blue}{M}\times I_{n+1}\times \ldots \times I_N}$, and is given by
\begin{equation}\label{equation:moden-tensor-multi}
	(	\eX \times_n \bA)_{i_1\times \ldots \times i_{n-1}\times \textcolor{blue}{m}\times i_{n+1}\times \ldots \times i_N}=
	\sum_{i_n=1}^{I_n} (x_{i_1i_2\ldots i_N}) (a_{\textcolor{blue}{m} i_n}).
\end{equation}
\paragraph{Matrix multiplication as tensor multiplication} In matrix language, suppose two matrices $\bA\in \real^{I\times K}, \bB \in \real^{K\times J}$, then the matrix multiplication can be equivalently denoted by
$$
\bA\bB = \bB\times_1 \bA.
$$

Suppose again, $\bA\in \real^{I\times K}, \bB \in \real^{J\times K}$, then the matrix multiplication can be equivalently denoted by
$$
\bA\bB^\top = \bA\times_2 \bB.
$$

Going further by supposing $\bA\in \real^{M\times N}$ whose reduced SVD (Theorem~\ref{theorem:reduced_svd_rectangular}, p.~\pageref{theorem:reduced_svd_rectangular}) of $\bA$ is given by
$$
\underset{M\times N}{\bA} = \underset{M\times R}{\bU}
\gap
\underset{R\times R}{\bSigma}
\gap
\underset{R\times N}{\bV^\top}.
$$
Then, the reduced SVD of $\bA$ can be equivalently denoted as
\begin{equation}\label{equation:svd-by-tensor-multi0}
	\bA = \bU\bSigma\bV^\top = \bSigma\times_1 \bU \times_2 \bV.
\end{equation}
The full SVD of $\bA$ can be represented in a similar way.

The matricization of a tensor can show what happens in the tensor multiplication.
\begin{lemma}[Tensor Multiplication in Matricization]\label{lemma:tensor-multi-matriciz1}
Given the Nth-order tensor $\eX \in  \real^{I_1\times I_2\times \ldots \times I_N}$ and the matrix $\bA\in \real^{M\times I_n}$, it follows that
$$
\begin{aligned}
	\eY = \eX\times_n\bA\in \real^{I_1\times \ldots \times I_{n-1}\times \textcolor{blue}{M}\times I_{n+1}\times \ldots \times I_N} \\
	\leadto  \bY_{(n)} = \bA\bX_{(n)} \in \real^{M\times I_{-n}},
\end{aligned}
$$
where $I_{-n}=I_1 I_2\ldots I_{n-1}I_{n+1}\ldots I_N$.
\end{lemma}
From the above lemma, conversely, suppose columns of $\bA$ are mutually orthonormal with $I_n\leq M$ (semi-orthogonal, see definition in Section~\ref{section:orthogonal-orthonormal-qr}, p.~\pageref{section:orthogonal-orthonormal-qr}). Then it follows that
$$
\bA^\top \bY_{(n)} = \underbrace{\bA^\top \bA}_{\bI}\bX_{(n)}  = \bX_{(n)}.
$$
This reveals an important property of tensor multiplication: $\eX =  \eY \times_n \bA^\top $. That is, if $\bA$ is semi-orthogonal, we have
\begin{equation}\label{equation:semiorthogonal-in-tensor-multi}
\eY = \eX\times_n\bA \leadto \eX =  \eY \times_n \bA^\top.
\end{equation}

\begin{lemma}[Tensor Multiplication]\label{lemma:tensor-multi2}
Suppose given the Nth-order tensor $\eX \in  \real^{I_1\times I_2\times \ldots \times I_N}$, we have
\begin{enumerate}
\item \textit{Distributive law.} Given further the matrices $\bA\in \real^{J_m\times I_m}$ and $\bB\in \real^{J_n\times I_n}$ where $m\neq n$, it follows that
$$
\eX \times_m \bA \times_n \bB =  (\eX \times_m \bA) \times_n \bB = (\eX  \times_n \bB) \times_m \bA.
$$
\item Given the matrices $\bA\in \real^{P\times I_m}$ and $\bB\in \real^{Q\times P}$, it follows that
$$
\eX \times_n\bA \times_n \bB = \eX\times_n (\bB\bA)  \in \real^{I_1\times \ldots \times I_{n-1}\times \textcolor{blue}{Q}\times I_{n+1}\times \ldots \times I_N}.
$$
\item \label{equationbracket} Given $\eX\in \real^{I_1\times \ldots \times I_{n-1}\times \textcolor{blue}{M}\times I_{n+1}\times \ldots \times I_N} $, $\eY\in \real^{I_1\times \ldots \times I_{n-1}\times \textcolor{blue}{K}\times I_{n+1}\times \ldots \times I_N} $, and $\bA\in \real^{M\times K}$, it follows that
$$
\langle\eX, \eY\times_n\bA \rangle = \langle \eX\times_n\bA^\top, \eY\rangle .
$$
\item \label{equationbracketlast} Given semi-orthogonal matrix $\bA\in \real^{P\times I_n}$, it follows that
$$
	\eY = \eX\times_n\bA \leadto \eX =  \eY \times_n \bA^\top.
$$
and
$$
||\eY|| = ||\eX||,
$$
i.e., the length preservation under semi-orthogonal.
\end{enumerate}
\end{lemma}

\section{Special Matrix Products} \index{Matrix products}
Several matrix products will be proved important in the illustration of the algorithms in the sequel.

The \textit{Kronecker product} of vectors $\ba \in \real^{I}$ and $\bb\in \real^{K}$ is denoted by $\ba\otimes \bb$:
$$
\ba\otimes \bb=
\begin{bmatrix}
	a_1\bb \\
	a_2\bb \\
	\vdots \\
	a_I\bb
\end{bmatrix},
$$
which is a column vector of size $(IK)$.
\begin{definition}[Matrix Kronecker]\label{definition:kronecker-product}
Similarly, the \textit{Kronecker product} of matrices $\bA \in \real^{I\times J}$ and $\bB\in \real^{K\times L}$ is denoted by $\bA\otimes \bB$:
$$
\begin{aligned}
	\bA\otimes \bB &=
	\begin{bmatrix}
		a_{11} \bB & a_{12}\bB & \ldots & a_{1J}\bB \\
		a_{21} \bB & a_{22}\bB & \ldots & a_{2J}\bB \\
		\vdots  & \vdots  & \ddots & \vdots \\
		a_{I1} \bB & a_{I2}\bB & \ldots & a_{IJ}\bB \\
	\end{bmatrix}\\
	&=
	\begin{bmatrix}
		\ba_1 \otimes \bb_1 \,\,\ldots & \ba_1\otimes \ba_L | \,\,
		\ba_2 \otimes \bb_1   \,\,\ldots &\ba_2\otimes \ba_L |\,\,
		\ba_J \otimes \bb_1   \,\,\ldots &\ba_J\otimes \ba_L\,\,
	\end{bmatrix},
\end{aligned}
$$
which is a matrix of size $(IK)\times (JL)$.
\end{definition}

Specifically, we notice that, when four vectors \{$\ba \in \real^{I}$ and $\bb\in \real^{K}$\} and \{$\bc \in \real^{I}$ and $\bd\in \real^{K}\}$, then
\begin{equation}\label{equation:kronecker-vector-find2}
	(\ba\otimes \bb)^\top(\bc\otimes \bd)=
	\begin{bmatrix}
		a_1\bb^\top &
		a_2\bb^\top &
		\ldots &
		a_I\bb^\top
	\end{bmatrix}
	\begin{bmatrix}
		c_1\bd \\
		c_2\bd \\
		\vdots \\
		c_I\bd
	\end{bmatrix}
	=\sum_{i=1}^{I}a_ic_i \bb^\top\bd=(\ba^\top\bc)(\bb^\top\bd).
\end{equation}
Specially, when $\bc=\ba, \bd=\bb$, it follows that
$$
(\ba\otimes \bb)^\top(\ba\otimes \bb) =||\ba||^2 \cdot ||\bb||^2.
$$
Similarly, for $\bA,\bC\in \real^{I\times J}$ and $\bB,\bD\in \real^{K\times L}$, it follows that
\begin{equation}
	(\bA\otimes \bB)^\top (\bC\otimes \bD) = (\bA^\top\bC) \otimes (\bB^\top\bD).
\end{equation}
Note also, for $\bA \in \real^{I\times J}$, $\bB\in \real^{K\times L}$, $\bC\in \real^{J\times I}$, and $\bD\in \real^{L\times K}$, the above equation reduces to
\begin{equation}
	(\bA\otimes \bB) (\bC\otimes \bD) = (\bA\bC) \otimes (\bB\bD).
\end{equation}
\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={Pseudo Inverse in Kronecker Product}]
Moreover, following from \citep{van2000ubiquitous}, the pseudo inverse of $(\bA\otimes \bB)$ is given by
\begin{equation}\label{equation:otimes-psesudo}
(\bA\otimes \bB)^+ = \bA^+ \otimes \bB^+,
\end{equation}
where $\bA^+$ is the pseudo inverse of matrix $\bA$. Recall that the pseudo inverse of a full column rank matrix $\bA$ is simply $\bA^+=(\bA^\top\bA)^{-1}\bA^\top$.
When $\bA,\bB$ are both semi-orthogonal (see definition in Section~\ref{section:orthogonal-orthonormal-qr}, p.~\pageref{section:orthogonal-orthonormal-qr}), the pseudo inverse is $\bA^+=\bA^\top, \bB^+=\bB^\top$, and it follows that
\begin{equation}\label{equation:otimes-psesudo-semi}
	(\bA\otimes \bB)^+ = \bA^\top \otimes \bB^\top.
\end{equation}
Analogously, the above pseudo inverse can be applied to the Kronecker product of a sequence of matrices.
\end{mdframed}

\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10,frametitle={Orthogonality in Kronecker Product}]
Suppose further $\ba\in \real^I$, and $\bb_1, \bb_2\in \real^K$ with $\bb_1^\top\bb_2=0$, then
\begin{equation}\label{equation:orthogonal-in-kronecker1}
	(\ba\otimes \bb_1)  \perp (\ba\otimes \bb_2).
\end{equation}
Or suppose $\ba_1,\ba_2\in \real^I$ with $\ba_1^\top\ba_2=0$, and $\bb\in \real^K$, then
\begin{equation}\label{equation:orthogonal-in-kronecker2}
(\ba_1\otimes\bb) \perp (\ba_2 \otimes \bb).
\end{equation}
The above two findings imply
$
\bA\otimes \bB
$ contains mutually orthogonal (orthonormal) columns if both $\bA \in \real^{I\times J}$ and $\bB\in \real^{K\times L}$ contain mutually orthogonal (orthonormal) columns.
\end{mdframed}

\begin{definition}[Khatri-Rao Product]\label{definition:khatri-rao-product}
	The \textit{Khatri-Rao product} of matrices  $\bA \in \real^{I\times K}$ and $\bB\in \real^{J\times K}$ is denoted by $\bA\odot \bB$:
	$$
	\bA\odot\bB =
	\begin{bmatrix}
		\ba_1\otimes \bb_1 & \ba_2\otimes \bb_2 & \ldots & \ba_K\otimes \bb_K
	\end{bmatrix},
	$$
	which is a matrix of size $(IJ)\times K$. And it is known as the ``matching columnwise" Kronecker product.
\end{definition}
From the above definition on the Khatri-Rao product, for two vectors $\ba$ and $\bb$, it follows that
$$
\ba \odot \bb = \ba\otimes\bb.
$$
Note that the ``distributive law" follows that
$$
\bA\odot \bB\odot \bC = (\bA\odot \bB)\odot \bC = \bA\odot (\bB\odot \bC).
$$

\begin{definition}[Hadamard Product]
	The \textit{Hadamard product} of matrices $\bA, \bB\in \real^{I\times J}$ is denoted by $\bA\circledast\bB$:
	$$
	\bA\circledast\bB =
	\begin{bmatrix}
		a_{11}b_{11} & a_{12}b_{12} & \ldots & a_{1J}b_{1J}\\
		a_{21}b_{21} & a_{22}b_{22} & \ldots & a_{2J}b_{2J}\\
		\vdots  & \vdots & \ddots & \vdots\\
		a_{I1}b_{I1} & a_{I2}b_{I2} & \ldots & a_{IJ}b_{IJ}\\
	\end{bmatrix},
	$$
	which is a matrix of size $I\times J$.
\end{definition}

Specifically, we further notice that, for two matrices  $\bA \in \real^{I\times K}$ and $\bB\in \real^{J\times K}$, it follows that
\begin{equation}
	\bZ = (\bA\odot \bB )^\top (\bA\odot \bB )=
	\begin{bmatrix}
		(\ba_1\otimes \bb_1)^\top  \\
		(\ba_2\otimes \bb_2)^\top  \\
		\vdots \\
		(\ba_K\otimes \bb_K)^\top
	\end{bmatrix}
	\begin{bmatrix}
		\ba_1\otimes \bb_1 & \ba_2\otimes \bb_2 & \ldots & \ba_K\otimes \bb_K
	\end{bmatrix},
\end{equation}
where $\bZ\in \real^{K\times K}$ and each entry $(i,j)$ element $z_{ij}$ is given by
$$
z_{ij} = (\ba_i\otimes \bb_i)^\top (\ba_j\otimes \bb_j) = (\ba_i^\top\ba_j)(\bb_i^\top\bb_j).
$$
where the last equality comes from Equation~\eqref{equation:kronecker-vector-find2}. Therefore, $\bZ$ can be equivalently written as
\begin{equation}\label{equation:two-khatri-rao-pro-equi}
	\bZ = (\bA\odot \bB )^\top (\bA\odot \bB ) =
	(\bA^\top\bA) \circledast (\bB^\top\bB).
\end{equation}
To conclude, it follows that
\begin{equation}\label{equation:matrix-prod-in-tensor}
\boxed{\left.
	\begin{aligned}
	&	(\ba\otimes \bb)^\top(\bc\otimes \bd)&=&(\ba^\top\bc)(\bb^\top\bd);\\
	&	(\ba\otimes \bb)^\top(\ba\otimes \bb) &=&||\ba||^2 \cdot ||\bb||^2;\\
	&	(\bA\otimes \bB)^\top (\bC\otimes \bD) &=& (\bA^\top\bC) \otimes (\bB^\top\bD), \gap \left(\text{\parbox{11em}{where $\bA,\bC$ same shape, \\$\bB,\bD$ same shape}}\right);\\
	& (\bA\otimes \bB)^+ &=& \bA^+ \otimes \bB^+;\\
	&	\bA\odot \bB\odot \bC &=& (\bA\odot \bB)\odot \bC \\
	&\gap	&=& \bA\odot (\bB\odot \bC);\\
	&	(\bA\odot \bB )^\top (\bA\odot \bB ) &=& 	(\bA^\top\bA) \circledast (\bB^\top\bB).
	\end{aligned}
	\right.}
\end{equation}

\chapter{Tensor-Train (TT) Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Tensor-Train (TT) Decomposition}

\begin{theoremHigh}[Tensor-Train Decomposition \citep{oseledets2011tensor}]\label{theorem:ttrain-decomp}
For a general Nth-order tensor, $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, it admits the tensor-train decomposition
$$
\eX \approx \eG^{(1)} \boxtimes \eG^{(2)} \boxtimes \ldots \boxtimes \eG^{(N)},
$$
where the symbol ``$\boxtimes$" means the elements of $\eX$ can be obtained by a multiplication of $N$ matrices
$$
x_{i_1,i_2,\ldots,i_N} = \eG^{(1)}_{:,:,i_1} \eG^{(2)}_{:,:,i_2}\ldots \eG^{(N)}_{:,:,i_N}.
$$
Note here
\begin{itemize}
\item Each $\eG^{(n)} \in \real^{R_{n-1}\times R_{n}\times I_n}$ for all $n\in \{1,2,\ldots, N\}$  is a third-order tensor, and is referred to as a \textit{TT core};
\item Each $\eG^{(n)}_{:,:,i_n} \in \real^{R_{n-1}\times R_n}$ for all $i_n\in \{1,2,\ldots, I_n\}$ is a frontal slice of $\eG^{(n)}$ (see Figure~\ref{fig:thifd-slices});
\item $R_0, R_N$ are imposed to be 1 for boundary conditions;
\item $R_1,R_2,\ldots, R_{N-1}$ are known as the \textit{tensor ranks} of corresponding dimensions;
\end{itemize}
The illustration of how to extract each element of the decomposition is shown in Figure~\ref{fig:tensortrain-decom}.
\end{theoremHigh}

\begin{figure}[H]
\centering
\resizebox{1.0\textwidth}{!}{%
\begin{tikzpicture}
\pgfmathsetmacro{\bxx}{0.3}
\pgfmathsetmacro{\begupwhite}{0.2}

\pgfmathsetmacro{\bcwd}{0.3}
\pgfmathsetmacro{\bbwd}{0.3}
\pgfmathsetmacro{\bbright}{0.9}
\pgfmathsetmacro{\bbup}{0.225}

\pgfmathsetmacro{\distrightdown}{4.2}
\pgfmathsetmacro{\distl}{1.2}
\pgfmathsetmacro{\aprimeright}{0.4}

\pgfmathsetmacro{\abvd}{3.3}
\pgfmathsetmacro{\abvdd}{3.9}
\draw (6.6,-3.2+\abvd) node {{\color{black}\large{$\eX_{i_1, i_2,\ldots, i_N}$}}};
\draw (8,-3.2+\abvd) node {	{\color{black}\large{$\approx$}}};

\draw [very thick] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\filldraw [fill=gray!60!white,draw=green!40!black] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.5,-3.8+0.5) grid (7+0.5,-2.6+0.5);

\draw [very thick] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\filldraw [fill=gray!50!white,draw=green!40!black] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.4,-3.8+0.4) grid (7+0.4,-2.6+0.4);

\draw [very thick] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\filldraw [fill=gray!40!white,draw=green!40!black] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.3,-3.8+0.3) grid (7+0.3,-2.6+0.3);

\draw [very thick] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\filldraw [fill=gray!30!white,draw=green!40!black] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.2,-3.8+0.2) grid (7+0.2,-2.6+0.2);

\draw [very thick] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\filldraw [fill=gray!20!white,draw=green!40!black] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.1,-3.8+0.1) grid (7+0.1,-2.6+0.1);

\draw [very thick] (5.8,-3.8) rectangle (7,-2.6);
\filldraw [fill=gray!10!white,draw=green!40!black] (5.8,-3.8) rectangle (7,-2.6);
\draw [step=0.4/2, very thin, color=gray] (5.8,-3.8) grid (7,-2.6);

\draw [very thick] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\filldraw [fill=gray!6!white,draw=green!40!black] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8-0.1,-3.8-0.1) grid (7-0.1,-2.6-0.1);

\draw (6.2,-4.6) node {{\color{black}\scriptsize{$\eX\in\real^{I_1\times I_2 \times \ldots \times I_N}$}}};

\draw (8,-3.2) node {	{\color{black}\large{$\approx$}}};

\pgfmathsetmacro{\cubex}{1}
\pgfmathsetmacro{\cubey}{1}
\pgfmathsetmacro{\cubez}{0.1}
\pgfmathsetmacro{0.5}{0.3}
\pgfmathsetmacro{\xa}{3.7}
\pgfmathsetmacro{\ya}{3.1}
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw (7+\xa,0-\ya) node {	{\color{black}\large{$\boxtimes$}}};
\draw (5.7+\xa,-4.8) node {{\color{black}\scriptsize{$\eG^{(1)}\in\real^{R_0\times R_1 \times  I_1}$}}};

\draw[->] (4.9+\xa,-3) -- ++(0.8,0.8);
\draw[->] (4.85+\xa,-3.1) -- ++(0,-1.);
\draw[->] (4.95+\xa,-4.2) -- ++(1.25,0);
\draw (5.5+\xa,-4.45) node {{\color{black}\scriptsize{$R_1$}}};
\draw (4.6+\xa,-3.5) node {{\color{black}\scriptsize{$R_0$}}};
\draw (5.2+\xa,-2.35) node {{\color{black}\scriptsize{$I_1$}}};

\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw (5.8+\xa,-4.7+\abvdd) node {{\color{black}\scriptsize{$\eG^{(1)}_{:,:,i_1}\in\real^{R_0\times R_1}$}}};
\draw (7+\xa,0-\ya+\abvd) node {	{\color{black}\large{$\times$}}};
\draw[-{Computer Modern Rightarrow}] (5.8+\xa,-2.) -- ++(0,0.8);
\draw (5.8+\xa,-1.65) node {{\color{black}\scriptsize{\textcolor{blue}{$i_1$-th slice}}}};
\pgfmathsetmacro{\xa}{6.2}
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-8*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-8*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-8*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-7*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-7*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-7*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw (7.2+\xa,0-\ya) node {	{\color{black}\large{$\boxtimes$}}};
\draw (5.9+\xa,-4.8) node {{\color{black}\scriptsize{$\eG^{(2)}\in\real^{R_1\times R_2 \times  I_2}$}}};
\draw[->] (4.9+\xa,-3) -- ++(0.85,0.85);
\draw[->] (4.85+\xa,-3.1) -- ++(0,-1.);
\draw[->] (4.95+\xa,-4.2) -- ++(1.25,0);
\draw (5.5+\xa,-4.45) node {{\color{black}\scriptsize{$R_2$}}};
\draw (4.6+\xa,-3.6) node {{\color{black}\scriptsize{$R_1$}}};
\draw (5.2+\xa,-2.35) node {{\color{black}\scriptsize{$I_2$}}};

\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw (5.9+\xa,-4.7+\abvdd) node {{\color{black}\scriptsize{$\eG^{(2)}_{:,:,i_2}\in\real^{R_1\times R_2}$}}};
\draw (7+\xa,0-\ya+\abvd) node {	{\color{black}\large{$\times$}}};
\draw[-{Computer Modern Rightarrow}] (5.8+\xa,-2.) -- ++(0,0.8);
\draw (5.8+\xa,-1.65) node {{\color{black}\scriptsize{\textcolor{blue}{$i_2$-th slice}}}};

\draw (7.9+\xa,0-\ya) node {	{\color{black}\large{$\ldots \boxtimes$}}};
\draw (7.9+\xa,0-\ya+\abvd) node {	{\color{black}\large{$\ldots \times$}}};
\pgfmathsetmacro{\xa}{9.9}
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-7*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-7*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-7*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!70!white] (6+\xa,0-\ya,-6*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!60!white] (6+\xa,0-\ya,-5*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;

\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!50!white] (6+\xa,0-\ya,-4*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!30!white] (6+\xa,0-\ya,-2*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!20!white] (6+\xa,0-\ya,-1*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!10!white] (6+\xa,0-\ya,0) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw (5.9+\xa,-4.8) node {{\color{black}\scriptsize{$\eG^{(N)}\in\real^{R_{N-1}\times R_N \times  I_N}$}}};
\draw[->] (4.9+\xa,-3) -- ++(0.8,0.8);
\draw[->] (4.85+\xa,-3.1) -- ++(0,-1.);
\draw[->] (4.95+\xa,-4.2) -- ++(1.25,0);
\draw (5.5+\xa,-4.45) node {{\color{black}\scriptsize{$R_N$}}};
\draw (4.48+\xa,-3.7) node {{\color{black}\scriptsize{$R_{N-1}$}}};
\draw (5.2+\xa,-2.35) node {{\color{black}\scriptsize{$I_N$}}};

\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,-\cubey,0) -- ++(\cubex,0,0) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(0,0,-\cubez) -- ++(0,-\cubey,0) -- ++(0,0,\cubez) -- cycle;
\draw[black,fill=gray!40!white] (6+\xa,0-\ya+\abvd,-3*0.5) -- ++(-\cubex,0,0) -- ++(0,0,-\cubez) -- ++(\cubex,0,0) -- cycle;
\draw (5.9+\xa,-4.7+\abvdd) node {{\color{black}\scriptsize{$\eG^{(N)}_{:,:,i_N}\in\real^{R_{N-1}\times R_N}$}}};
\draw[-{Computer Modern Rightarrow}] (5.8+\xa,-2.) -- ++(0,0.8);
\draw (5.8+\xa,-1.65) node {{\color{black}\scriptsize{\textcolor{blue}{$i_N$-th slice}}}};
\end{tikzpicture}
}
\caption{Tensor-train decomposition of an Nth-order tensor: $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}\approx \eG^{(1)} \boxtimes \eG^{(2)} \boxtimes \ldots \boxtimes \eG^{(N)}$ where each $\eG^{(n)} \in \real^{R_{n-1}\times R_{n}\times I_n}$ for $n\in \{1,2,\ldots,N\}$. Each $(i_1,i_2,\ldots,i_N)$-th element is obtained by the matrix multiplication of the corresponding frontal slices of $\eG^{(n)}$'s.
Note here $R_0=R_N=1$.
}
\label{fig:tensortrain-decom}
\end{figure}

The illustration of the TT decomposition for an Nth order tensor is shown in Figure~\ref{fig:tensortrain-decom}.
In other words, the TT format approximates every entry of the tensor $\eX$ with a multiplication of $N$ matrices, in particular with a sequence of $R_n \times R_{n+1}$ matrices, each indexed by the parameter $i_{n+1}$. The figure looks like a
train with links between them, hence the name ``train". Suppose that the TT-ranks are all equal, $R_1=R_2=\ldots=R_{N-1} = R$, and that $I_1=I_2=\ldots=I_N = I$, then the TT decomposition
requires the storage of $O(N I R^2)$ floats. Thus the memory complexity of the TT decomposition scales linearly with dimension.

\paragraph{The ``$\boxtimes$" notation} The symbol ``$\boxtimes $" is defined to be a special tensor product. Suppose $\eA\in \real^{R_1\times R_2 \times I_1\times I_2\times \ldots\times I_M}, \eB\in \real^{R_2\times R_3\times J_1\times J_2\times \ldots \times J_N}$, then it follows that
$$
\eA\boxtimes \eB \in \real^{R_1\times R_3 \times  I_1\times I_2\times \ldots\times I_M \times J_1\times J_2\times \ldots \times J_N},
$$
where each element is given by
$$
\begin{aligned}
(\eA\boxtimes \eB)_{r_1, r_3 ,  i_1, i_2, \ldots, i_M , j_1, j_2, \ldots , j_N}\\
=\sum_{r_2=1}^{R_2}
(a_{r_1,\textcolor{blue}{r_2},i_1, i_2, \ldots, i_M})
(b_{\textcolor{blue}{r_2},r_3,j_1, j_2, \ldots , j_N}).
\end{aligned}
$$

\section{Computing the TT Decomposition}
\begin{algorithm}[h]
	\caption{TT-SVD}
	\label{alg:tt-decomposi}
	\begin{algorithmic}[1]
		\Require Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$;
		\State Set the initial matrix $\bC = \abovebX{1} \in \real^{I_1 \times I_2I_3\ldots I_N}$;
		\State Set $R_0=R_N=1$;
		\For{$n=1,2,\ldots, N-1$}
		\State $\bC=\text{reshape}\bigg(\bC, (I_nR_{n-1}), (I_{n+1}I_{n+2}\ldots I_N)\bigg)$;
		\State Compute the $\delta$-truncated SVD of $\bC = \bU\bSigma\bV^\top$, numerical rank $R_n=rank(\bC)$;
		\State Set the rows of $\bG^{(n)}_{(2)}\in \real^{R_n \times I_nR_{n-1}}$ by first $R_n$ leading left singular vectors of $\bC$;
		\State Un-matricization: $\eG^{(n)}=\text{reshape}(\bG^{(n)}_{(2)})\in \real^{R_{n-1}\times  R_n\times  I_n}$;
		\State Get the new matrix $\bC = \bSigma\bV^\top\in \real^{R_n\times I_{n+1}I_{n+2}\ldots I_N}$;
		\EndFor
		\State Get the last core tensor: $\eG^{(N)}=\text{reshape}(\bC)\in \real^{R_{N-1}\times  R_N\times  I_N}$;
		\State Output $\eG^{(1)}, \eG^{(2)}, \ldots, \eG^{(N)}$;
	\end{algorithmic}
\end{algorithm}

Define the \textit{tensor unfolding} for an Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$ in the following way
$$
\abovebX{n} \in \real^{(I_1I_2\ldots I_n) \times (I_{n+1}I_{n+2}\ldots I_N)},
$$
where the $\bigg(\{i_1\ldots i_n\},\{i_{n+1}\ldots i_N\}\bigg)$-th element of $\abovebX{n}$ is obtained by  $\abovebX{n}_{i_1\ldots i_n,i_{n+1}\ldots i_N} = \eX_{i_1,i_2,\ldots,i_N}$.
The tensor unfolding can be denoted by a reshape operator:
$$
\abovebX{n} = \text{reshape}\bigg(\eX, (I_1I_2\ldots I_n), (I_{n+1}I_{n+2}\ldots I_N) \bigg).
$$
This unfolding reveals a recursive algorithm to calculate the TT decomposition of $\eX$.

For this tensor unfolding along mode-$1$:
$$
\abovebX{1} \in \real^{I_1 \times I_2I_3\ldots I_N}.
$$
Recall the matricization in mode-$2$ of $\eG^{(1)}$ is given by (Section~\ref{section:matricization-tensor-original}, p.~\pageref{section:matricization-tensor-original}):
$$
\bG^{(1)}_{(2)} \in \real^{ R_1\times R_0I_1 }=\real^{ R_1 \times I_1 }.
$$
Then the $\bG^{(1)}_{(2)}$ can be though of as the data distilled version of $\abovebX{1}$, i.e., the row space of $\bG^{(1)}_{(2)}$ span the same column space as that of $\abovebX{1}$. And therefore, the $R_1$ can be determined by the numerical rank (Definition~\ref{definition:effective-rank-in-svd}, p.~\pageref{definition:effective-rank-in-svd}) of the SVD of $\abovebX{1}$, i.e., by discarding the singular values of $\abovebX{1}$ smaller than $\delta$. Similar to the Tucker and HOSVD, the row of $\bG^{(1)}_{(2)}$ can be obtained by the first $R_1$ left singular vectors of $\abovebX{1}$. For the (truncated) SVD of $\abovebX{1}$:
$$
\begin{aligned}
\abovebX{1} &= \bU_1\bSigma_1\bV_1^\top\\
&\leadto
\boxed{
\bG^{(1)}_{(2)} = \bU_1^\top\in \real^{R_1\times I_1}
}.
\end{aligned}
$$
Now, what's left is $\bSigma_1\bV_1^\top \in \real^{R_1\times I_2I_3\ldots I_N}$. By similar ``matrix unfolding", suppose we reshape $\bSigma_1\bV_1^\top$ into a $I_2R_1 \times I_3I_4\ldots I_N$ matrix $\bC$:
$$
\bC \in \real^{I_2R_1 \times I_3I_4\ldots I_N} = \text{reshape}\bigg(\bSigma_1\bV_1^\top, (I_2R_1 ), (I_3I_4\ldots I_N) \bigg).
$$
The second core tensor $\eG^{(2)}$ can be obtained in the same way via the (truncated) SVD of $\bC$:
$$
\begin{aligned}
\bC &= \bU_2 \bSigma_2\bV_2^\top \\
&\leadto
\boxed{
	\bG^{(2)}_{(2)} = \bU_2^\top\in \real^{ R_2\times I_2R_1}
}.
\end{aligned}
$$
The same process can go on, the eventually all the $N$ core tensors $\{\eG^{(1)}, \eG^{(2)}, \ldots, \eG^{(N)}\}$ will be obtained via the set of (truncated) SVDs. This is known as the TT-SVD algorithm in \citep{oseledets2011tensor}. The full procedure is shown in Algorithm~\ref{alg:tt-decomposi}.

Moreover, if the rank $R_n\leq rank(\bC)$ in each iteration, a low-rank TT best approximation to $\eX$ in Frobenius norm $\eX_{best}$ always exists.
And if further the truncation tolerance for the SVD of each unfolding is set to $\delta = \epsilon/\sqrt{N-1} ||\eX||_F$, the TT-SVD is able to construct the quasi-optimal approximation $\eX_{SVD}$ such that
$$
||\eX-\eX_{SVD}||_F \leq \sqrt{N-1} ||\eX-\eX_{best}||_F.
$$

\paragraph{Complexity and curse of dimensionality} Suppose again that the TT-ranks are all equal, $R_1=R_2=\ldots=R_{N-1} = R$, and that $I_1=I_2=\ldots=I_N = I$. And the complexity of the SVD calculation is $O(MN^2)$ if the matrix is of size $M\times N$ (Section~\ref{section:comput-svd-in-svd}, p.~\pageref{section:comput-svd-in-svd}). The complexity of each step $n$ in Algorithm~\ref{alg:tt-decomposi} can be shown to be
$$
f(n) = (R I) (I^{(N-n)})^2.
$$
A simple summation shows that the complexity of the TT-SVD algorithm is
$$
\text{cost}= f(1)+f(2)+\ldots+f(N-1)= RI (I^{2(N-1)} + I^{2(N-2)}+ I^{2}) =RI^{N^2-N+1}.
$$
So the complexity grows exponentially with dimension $I$ and thus the curse of dimensionality is not evolved.

\paragraph{Further calculation methods} We notice that the calculation of the TT decomposition relies on the rank-revealing decomposition, SVD. Other methods, such as the rank-revealing QR (Section~\ref{section:rank-r-qr}, p.~\pageref{section:rank-r-qr}), column-pivoted QR (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}), CUR (Theorem~\ref{theorem:skeleton-decomposition}, p.~\pageref{theorem:skeleton-decomposition}), UTV (Theorem~\ref{theorem:ulv-decomposition}, p.~\pageref{theorem:ulv-decomposition}), Column ID (Theorem~\ref{theorem:interpolative-decomposition}, p.~\pageref{theorem:interpolative-decomposition}) can be applied in each iteration to do the matrix decomposition that finds the spanning columns. We shall not go to the details. See also \citep{oseledets2011tensor, bigoni2016spectral}.

\chapter{Tucker Decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{Tucker Decomposition}
\begin{theoremHigh}[Tucker Decomposition]\label{theorem:tucker-decomp}
The Tucker decomposition factorizes a tensor into a sum of component rank-one
tensors. For a general Nth-order tensor, $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, it admits the Tucker decomposition
$$
\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}
\ba_{r_1}^{(1)} \circ \ba_{r_2}^{(2)} \circ \ldots \circ \ba_{r_N}^{(N)} ,
$$
where
\begin{itemize}
\item $R_1< I_1, R_2<I_2, \ldots, R_N<I_N$;
\item  $\eG$ of size ${R_1\times R_2\times \ldots \times R_N}$ is called the \textit{core tensor} so that $\eG$ can be thought of as the compressed version of $\eX$;
\item $\bA^{(n)}=[\ba_1^{(n)}, \ba_2^{(n)}, \ldots, \ba_{R_n}^{(n)}] \in \real^{I_n \times R_n}$ for all $n\in \{1,2,\ldots, N\}$ is the column partition of the matrix $\bA^{(n)}\in \real^{I_n \times R_n}$;
\item The $\bA^{(n)}$'s usually have mutually orthonormal columns and can be thought of as the principal component of each mode. In this sense, the $\bA^{(n)}$'s are \textit{semi-orthogonal matrices} (see the definition in Section~\ref{section:orthogonal-orthonormal-qr}, p.~\pageref{section:orthogonal-orthonormal-qr});
\item We can complete the semi-orthogonal matrices into \textit{full orthogonal matrices} by adding \textit{silent columns} into $\bA^{(n)}$'s so that $\bA^{(n)}\in \real^{I_n\times I_n}$ is an orthogonal matrix, in which case, $\eG$ will be expanded to a tensor of size $\real^{I_1\times I_2\times \ldots \times I_N}$ where $g_{r_1r_2\ldots r_N} =0$ when either one of $r_n>R_n$ for $n\in \{1,2,\ldots, N\}$. This is known as the \textit{full Tucker decomposition}, and the previous one is also called the \textit{reduced} one to avoid confusion; And we shall only consider the reduced case in most of our discussions.
\end{itemize}

\end{theoremHigh}

\paragraph{Equivalent forms on the Tucker decomposition} By the mode-$n$ tensor multiplication in Equation~\eqref{equation:moden-tensor-multi}, the Tucker decomposition can also be written as:
\begin{equation}\label{equation:tucker-decom-in-tuckeropera}
\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\eG \times_1 \bA^{(1)} \times_2 \bA^{(2)} \ldots \times_N \bA^{(N)}.
\end{equation}
By the result in Lemma~\ref{lemma:tensor-multi2}~\eqref{equationbracketlast}, since $\bA^{(n)}$'s are semi-orthogonal, it also follows that
\begin{equation}\label{equation:tucker-decom-in-tuckeropera2}
\eG = \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots, \bA^{(N)\top}\rrbracket   = \eX \times_1 \bA^{(1)\top} \times_2 \bA^{(2)\top} \ldots \times_N \bA^{(N)\top}.
\end{equation}
The operator defined in Equation~\eqref{equation:tucker-decom-in-tuckeropera} is sometimes referred to as the \textit{Tucker operator} \citep{kolda2006multilinear}.
Element-wise, the $(i_1,i_2,\ldots,i_N)$-th element of $\eX$ can be obtained by
$$
x_{i_1,i_2,\ldots,i_N} = \sum_{r_1=1}^{R_1} \sum_{r_2=1}^{R_2} \ldots \sum_{r_N=1}^{R_N}
g_{r_1r_2\ldots r_N}
a_{i_1r_1}^{(1)} a_{i_2r_2}^{(2)} \ldots a_{i_Nr_N}^{(N)}.
$$

\begin{figure}[htbp]
\centering
\resizebox{1.0\textwidth}{!}{%
\begin{tikzpicture}
\pgfmathsetmacro{\bxx}{0.3}
\pgfmathsetmacro{\begupwhite}{0.2}

\pgfmathsetmacro{\bcwd}{0.3}
\pgfmathsetmacro{\bbwd}{0.3}
\pgfmathsetmacro{\bbright}{0.9}
\pgfmathsetmacro{\bbup}{0.225}

\pgfmathsetmacro{\distrightdown}{4.2}
\pgfmathsetmacro{\dist}{4}
\pgfmathsetmacro{\distl}{0}
\pgfmathsetmacro{\aprimeright}{0.4}

\draw [very thick] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\filldraw [fill=gray!60!white,draw=green!40!black] (5.8+0.5,-3.8+0.5) rectangle (7+0.5,-2.6+0.5);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.5,-3.8+0.5) grid (7+0.5,-2.6+0.5);

\draw [very thick] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\filldraw [fill=gray!50!white,draw=green!40!black] (5.8+0.4,-3.8+0.4) rectangle (7+0.4,-2.6+0.4);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.4,-3.8+0.4) grid (7+0.4,-2.6+0.4);

\draw [very thick] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\filldraw [fill=gray!40!white,draw=green!40!black] (5.8+0.3,-3.8+0.3) rectangle (7+0.3,-2.6+0.3);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.3,-3.8+0.3) grid (7+0.3,-2.6+0.3);

\draw [very thick] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\filldraw [fill=gray!30!white,draw=green!40!black] (5.8+0.2,-3.8+0.2) rectangle (7+0.2,-2.6+0.2);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.2,-3.8+0.2) grid (7+0.2,-2.6+0.2);

\draw [very thick] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\filldraw [fill=gray!20!white,draw=green!40!black] (5.8+0.1,-3.8+0.1) rectangle (7+0.1,-2.6+0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8+0.1,-3.8+0.1) grid (7+0.1,-2.6+0.1);

\draw [very thick] (5.8,-3.8) rectangle (7,-2.6);
\filldraw [fill=gray!10!white,draw=green!40!black] (5.8,-3.8) rectangle (7,-2.6);
\draw [step=0.4/2, very thin, color=gray] (5.8,-3.8) grid (7,-2.6);

\draw [very thick] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\filldraw [fill=gray!6!white,draw=green!40!black] (5.8-0.1,-3.8-0.1) rectangle (7-0.1,-2.6-0.1);
\draw [step=0.4/2, very thin, color=gray] (5.8-0.1,-3.8-0.1) grid (7-0.1,-2.6-0.1);

\draw[->] (5.6,-2.6) -- ++(0.6,0.6);
\draw[->] (5.55,-2.7) -- ++(0,-1.2);
\draw[->] (5.66,-4.04) -- ++(1.25,0);

\draw (6.35,-4.25) node {{\color{black}\scriptsize{spatial $y$}}};
\draw (5.2-0.3,-3.2) node {{\color{black}\scriptsize{spatial $x$}}};
\draw (5.3,-2.2) node {{\color{black}\scriptsize{temporal}}};

\draw (6.2,-4.6) node {{\color{black}\scriptsize{$\eX\in\real^{I\times J \times K}$}}};

\draw (8,-3.2) node {	{\color{black}\large{$\approx$}}};

\draw [very thick] (8.6+\distl,-4+0.2) rectangle (9.2+\distl,-3.2+0.2);
\filldraw [fill=RubineRed!60!white,draw=green!40!black]  (8.6+\distl,-4+0.2) rectangle (9.2+\distl,-3.2+0.2);
\draw (8.7+\distl,-4.4+0.2) node {{\color{black}\scriptsize{$\bB \in \real^{J\times Q}$}}};

\draw [very thick]
(9+\bbright+\distl,-2.9+\bbup) rectangle (9.5+\bbright+\distl,-2.6+\bbup);
\filldraw [fill=WildStrawberry!40!white,draw=green!40!black]
(9+\bbright+\distl,-2.9+\bbup) rectangle (9.5+\bbright+\distl,-2.6+\bbup);
\draw (9.6+\bbright+\distl,-3.2+\bbup) node {{\color{black}\scriptsize{$\bA^\top \in \real^{P\times I}$}}};

\draw[fill=RedOrange!40!white, line width=0.8pt]
(8.45+\distl,-2.3-\bxx) -- (8.9+\distl,-2.1-\bxx) -- (8.35+\distl,-1.9-\bxx) -- (7.9+\distl,-2.1-\bxx) -- cycle;
\draw (8.4+\distl,-1.8) node {{\color{black}\scriptsize{$\bC\in \real^{K\times R}$}}};

\draw[fill=RedOrange!10!white, line width=0.5pt]
(9.2+\distl,-2.4+0.1-\bxx) -- (9.4+\bcwd+\distl-0.02,-2.1+0.01-\bxx) -- (8.8+\bcwd+\distl,-2.1-\bxx) -- (8.6+\distl+0.02,-2.4+0.1+0.01-\bxx) -- cycle;
\draw[fill=RedOrange!10!white, line width=0.8pt]
(8.6+\distl,-2.4+0.1-\bxx) -- (9.2+\distl,-2.4+0.1-\bxx) -- (9.2+\distl,-3+0.4-\bxx) -- (8.6+\distl,-3+0.4-\bxx) -- cycle;
\draw[fill=RedOrange!10!white, line width=0.8pt]
(9.4+\bcwd+\distl,-2.1-\bxx) -- (9.2+\distl,-2.4+0.1-\bxx) -- (9.2+\distl,-3+0.4-\bxx) -- (9.4+\bcwd+\distl,-2.4-\bxx) -- cycle;
\draw (10-0.2+\distl,-1.8-\bxx) node {{\color{black}\scriptsize{$\eG\in \real^{P\times Q\times R}$}}};

\draw (11.6,-3.2) node {{\color{black}\large{$=$}}};

\draw [very thick]
(8.6+\dist,-4+0.2) rectangle (9.4+\dist,-3.2+0.2);
\filldraw [fill=blue!70,draw=green!40!black]
(8.6+\dist,-4+0.2) rectangle (9.4+\dist,-3.2+0.2);

\draw [very thick] (8.6+\dist,-4+0.2) rectangle (9.2+\dist,-3.2+0.2);
\filldraw [fill=RubineRed!60!white,draw=green!40!black]  (8.6+\dist,-4+0.2) rectangle (9.2+\dist,-3.2+0.2);
\draw (8.7+\dist,-4.4+0.2) node {{\color{black}\scriptsize{$\bB^\prime \in \real^{J\times \textcolor{blue}{J}}$}}};

\draw [very thick]
(9+\bbright+\dist+\aprimeright,-2.8+\bbup) rectangle (9.5+\bbright+\dist+\aprimeright,-2.3+\bbup);
\filldraw [fill=blue!70,draw=green!40!black]
(9+\bbright+\dist+\aprimeright,-2.8+\bbup) rectangle (9.5+\bbright+\dist+\aprimeright,-2.3+\bbup);

\draw [very thick]
(9+\bbright+\dist+\aprimeright,-2.8+\bbup) rectangle (9.5+\bbright+\dist+\aprimeright,-2.5+\bbup);
\filldraw [fill=WildStrawberry!40!white,draw=green!40!black]
(9+\bbright+\dist+\aprimeright,-2.8+\bbup) rectangle (9.5+\bbright+\dist+\aprimeright,-2.5+\bbup);
\draw (9.6+\bbright+\dist+\aprimeright,-3.1+\bbup) node {{\color{black}\scriptsize{$\bA^{\prime\top }\in \real^{ \textcolor{blue}{I}\times I}$}}};

\draw[fill=blue!70, line width=0.8pt]
(8.45+\dist,-2.3-\bxx+\begupwhite) -- (8.9+\dist+0.3,-2.1-\bxx+\begupwhite+0.14) -- (8.35+\dist+0.3,-1.9-\bxx+\begupwhite+0.15) -- (7.9+\dist,-2.1-\bxx+\begupwhite) -- cycle;

\draw[fill=RedOrange!40!white, line width=0.8pt]
(8.45+\dist,-2.3-\bxx+\begupwhite) -- (8.9+\dist,-2.1-\bxx+\begupwhite) -- (8.35+\dist,-1.9-\bxx+\begupwhite) -- (7.9+\dist,-2.1-\bxx+\begupwhite) -- cycle;
\draw (8.4+\dist,-1.8+\begupwhite) node {{\color{black}\scriptsize{$\bC^\prime\in \real^{K\times \textcolor{blue}{K}}$}}};

\draw[fill=RedOrange!1!white, line width=0.8pt,opacity=0.5]
(8.6+\dist,-2.4+0.1-\bxx) -- (9.4+\dist,-2.4+0.1-\bxx) -- (9.4+\dist,-3+0.4-\bxx) -- (8.6+\dist,-3+0.4-\bxx) -- cycle;

\draw[fill=RedOrange!10!white, line width=0.8pt]
(9.2+\dist,-2.4+0.1-\bxx+\begupwhite) -- (9.4+\bcwd+\dist,-2.1-\bxx+\begupwhite) -- (8.8+\bcwd+\dist+0.04,-2.1-\bxx+\begupwhite+0.01) -- (8.6+\dist,-2.4+0.1-\bxx+\begupwhite) -- cycle;
\draw[fill=RedOrange!10!white, line width=0.8pt]
(8.6+\dist,-2.4+0.1-\bxx+\begupwhite) -- (9.2+\dist,-2.4+0.1-\bxx+\begupwhite) -- (9.2+\dist,-3+0.4-\bxx+\begupwhite) -- (8.6+\dist,-3+0.4-\bxx+\begupwhite) -- cycle;
\draw[fill=RedOrange!10!white, line width=0.8pt]
(9.4+\bcwd+\dist,-2.1-\bxx+\begupwhite) -- (9.2+\dist,-2.4+0.1-\bxx+\begupwhite) -- (9.2+\dist,-3+0.4-\bxx+\begupwhite) -- (9.4+\bcwd+\dist,-2.4-\bxx+\begupwhite) -- cycle;
\draw (10+\dist,-1.7-\bxx+\begupwhite) node {{\color{black}\scriptsize{$\eG^\prime\in \real^{\textcolor{blue}{I\times J\times K}}$}}};

\draw[fill=RedOrange!1!white, line width=0.8pt,opacity=0.5]
(9.4+\dist,-2.4+0.1-\bxx+\begupwhite) -- (9.6+\bcwd+\dist+0.24,-2.1-\bxx+\begupwhite+0.12) -- (8.8+\bcwd+\dist+0.26,-2.1-\bxx+\begupwhite+0.12) -- (8.6+\dist,-2.4+0.1-\bxx+\begupwhite) -- cycle;
\draw[fill=RedOrange!1!white, line width=0.8pt,opacity=0.5]
(9.6+\bcwd+\dist+0.25,-2.1-\bxx+\begupwhite+0.12) -- (9.4+\dist,-2.4+0.1-\bxx+\begupwhite) -- (9.4+\dist,-3+0.4-\bxx) -- (9.6+\bcwd+\dist+0.25,-2.4-\bxx+0.12) -- cycle;

\end{tikzpicture}
}
\caption{The Tucker decomposition of a third-order tensor: $\eX \approx \llbracket \eG; \bA, \bB, \bC   \rrbracket
	=
	\sum_{p=1}^{P}\sum_{q=1}^{Q}\sum_{r=1}^{R}
	g_{pqr} \cdot \ba_p\circ \bb_q \circ \bc_r$. Middle: \textbf{reduced Tucker decomposition}. Right: \textbf{full Tucker decomposition} where the \textcolor{blue}{blue} entries are silent \textit{orthogonal} columns and the white entries of $\eG^\prime$ are zero.}
\label{fig:tucker-decom-third}
\end{figure}

\paragraph{Matricization}
For simplicity, we consider the Tucker decomposition for the third-order tensor $\eX\in \real^{I\times J\times K}$ where the situation is illustrated in Figure~\ref{fig:tucker-decom-third}:
\begin{equation}\label{equation:tucker-third-fact-1}
\eX \approx \llbracket \eG; \bA, \bB, \bC   \rrbracket
=
\sum_{p=1}^{P}\sum_{q=1}^{Q}\sum_{r=1}^{R}
g_{pqr} \cdot \ba_p\circ \bb_q \circ \bc_r,
\end{equation}
where $\bA\in \real^{I\times P}, \bB\in \real^{J\times Q}$, $\bC\in \real^{K\times R}$, and $\eG\in \real^{P\times Q\times R}$. Analogously, following the matricization of the third-order CP decomposition in Equation~\eqref{equation:cp-matriciza-third-1}, we have
\begin{equation}\label{equation:tucker-matriciza-third-1}
	\left\{
	\begin{aligned}
		\bX_{(1)}  &\approx \bA\bG_{(1)} (\bC\otimes \bB )^\top \in \real^{I\times (JK)}; \\
		\bX_{(2)} &\approx \bB\bG_{(2)} (\bC\otimes \bA )^\top \in \real^{J\times (IK)};\\
		\bX_{(3)} &\approx \bC\bG_{(3)}(\bB\otimes \bA)^\top \in \real^{K\times (IJ)},
	\end{aligned}
	\right.
\end{equation}
where now ``$\otimes$" is the Kronecker product (Definition~\ref{definition:kronecker-product}, p.~\pageref{definition:kronecker-product}).

In full generality, come back to the Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, the mode-$n$ matricized form is given by
$$
\boxed{\underbrace{\bX_{(n)}}_{I_n\times (I_{-n})}\approx
	\underbrace{\bA^{(n)}}_{{I_n\times R_n}}
	\underbrace{\bG_{(n)}}_{R_n\times (R_{-n})}
	\underbrace{\left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots \otimes \bA^{(n+1)}\otimes \bA^{(n-1)}\otimes \ldots \otimes \bA^{(2)}\otimes \bA^{(1)}  \right)^\top}_{(R_{-n})\times (I_{-n})}}
$$
where $I_{-n}=I_1 I_2\ldots I_{n-1}I_{n+1}\ldots I_N$ and $R_{-n}=R_1 R_2\ldots R_{n-1}R_{n+1}\ldots R_N$.
\paragraph{Vectorization} Going further from the matricization for the Nth-order tensor $\eX$, the vectorization is given by
\begin{equation}\label{equation:tucker-vec-in-theorem}
\boxed{\underbrace{vec(\eX) }_{(I_1\ldots I_N)\times 1}
	\approx
	\underbrace{\left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots\otimes \bA^{(1)}  \right)}_{(I_1\ldots I_N)\times (R_1\ldots R_N)}
	\underbrace{vec(\eG)}_{(R_1\ldots R_N)\times 1}.
}
\end{equation}

\paragraph{Counterpart in $\eG$} For the problem in Equation~\eqref{equation:tucker-decom-in-tuckeropera2}, it also has the matricization and vectorization in a similar way:
$$
\boxed{
\begin{aligned}
\bG_{(n)}\approx
\bA^{(n)\top}
\bG_{(n)}\left(\bA^{(N)\top} \otimes \bA^{(N-1)\top} \otimes \ldots \otimes \bA^{(n+1)\top}\otimes \bA^{(n-1)\top}\otimes \ldots \otimes \bA^{(2)\top}\otimes \bA^{(1)\top}  \right)^\top
\end{aligned}
}
$$

\begin{equation}\label{equation:tucker-vec-in-theorem-eg}
\boxed{\underbrace{vec(\eG) }_{(R_1\ldots R_N)\times 1 }
\approx
\underbrace{\left(\bA^{(N)\top} \otimes \bA^{(N-1)\top} \otimes \ldots\otimes \bA^{(1)\top}  \right)}_{ (R_1\ldots R_N)  \times (I_1\ldots I_N)}
\underbrace{vec(\eX)}_{(I_1\ldots I_N)\times 1}.
}
\end{equation}

\paragraph{Connection to the CP decomposition} We notice that when $\eG$ is the identity tensor (Definition~\ref{definition:identity-tensors}, p.~\pageref{definition:identity-tensors}), $R_1=R_2=\ldots=R_N$, and we do not further restrict the columns of $\bA^{(n)}$'s are mutually orthonormal, then the Tucker decomposition reduces to a CP decomposition. Note that the $\eG$ has to be an identity tensor in this case and it cannot be simply treated as a tensor with all 1's elements where the Tucker decomposition is given by:
$$
\eX \approx \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket
=
\sum_{r_1=1}^{R} \sum_{r_2=1}^{R} \ldots \sum_{r_N=1}^{R}
1\cdot
\ba_{r_1}^{(1)} \circ \ba_{r_2}^{(2)} \circ \ldots \circ \ba_{r_N}^{(N)} ,
$$
which is slightly different to the CP decomposition that has only one summation.

\section{Computing the Tucker Decomposition}
\begin{algorithm}[h]
	\caption{Tucker Decomposition via ALS}
	\label{alg:tucker-decomposition-full-gene}
	\begin{algorithmic}[1]
		\Require Nth-order tensor $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$;
		\State Pick a rank $R_1, R_2, \ldots, R_N$;
		\State Initialize $\bA^{(n)}\in \real^{I_n\times R_n}$ for all $n\in \{1,2,\ldots, N\}$ randomly;
		\State Choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$iter<C$}
		\State $iter=iter+1$;
		\For{$n=1,2,\ldots, N$}
		\State Set $\eY= \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots,\bA^{(n-1)}, \textcolor{blue}{\bI},\bA^{(n+1)},\ldots,  \bA^{(N)\top}\rrbracket$;
		\State Find the matricization along mode-$n$: $\bY_{(n)}$;
		\State Set the rows of $\bA^{(n)}$ by first $R_n$ leading left singular vectors of $\bY_{(n)}$;
		\EndFor
		\EndWhile
		\State $\eG = \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots, \bA^{(N)\top}\rrbracket$;\Comment{by vectorize and un-vectorize, Eq.~\eqref{equation:tucker-eg-vector-undo}}
		\State Output $\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}, \eG$;
	\end{algorithmic}
\end{algorithm}

To compute the Tucker decomposition of $\eX \in \real^{I_1\times I_2\times \ldots \times I_N}$, we now consider algorithms for solving the problem:\index{ALS}
$$
\boxed{\{{\eG,\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}}\}
	=
	\mathop{\arg \gap \min}_{\eG,\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}}
	\left\Vert\eX - \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  \right\Vert^2,}
$$
where $\eG$ is a tensor of size ${R_1\times R_2\times \ldots \times R_N}$, $\bA^{(n)}$'s are semi-orthogonal of size ${I_n\times R_n}$ for $n\in \{1,2,\ldots, N\}$. Similar to the CP decomposition, an \textit{alternating descent} algorithm can be employed to find the solution approximately.

\subsubsection*{\textbf{Given $\bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}$, Update $\eG$:}}
The update on $\eG$ follows immediately since we mentioned in Equation~\eqref{equation:tucker-decom-in-tuckeropera2} that
$$
\eG = \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots, \bA^{(N)\top}\rrbracket   = \eX \times_1 \bA^{(1)\top} \times_2 \bA^{(2)\top} \ldots \times_N \bA^{(N)\top}.
$$
To simplify matters, from the vectorized form in Eequation~\eqref{equation:tucker-vec-in-theorem}, we have
$$
\begin{aligned}
&\left\Vert\eX - \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  \right\Vert^2\\
&=
\left\Vert
vec(\eX)
-
\left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots\otimes \bA^{(1)}  \right)
vec(\eG)
\right\Vert^2,
\end{aligned}
$$
which is just the least squares problem and the solution is given by
\begin{equation}\label{equation:tucker-eg-vector-undo}
	\begin{aligned}
		vec(\eG) &\leftarrow \left(\bA^{(N)} \otimes \bA^{(N-1)} \otimes \ldots\otimes \bA^{(1)}  \right)^+ vec(\eX)\\
		&=\left(\bA^{(N)\top} \otimes \bA^{(N-1)\top} \otimes \ldots\otimes \bA^{(1)\top}  \right)vec(\eX),
	\end{aligned}
\end{equation}
where the last equality comes from Equation~\eqref{equation:otimes-psesudo-semi} since $\bA^{(n)}$'s are semi-orthogonal. As long as we update for the vectorized version, an un-vectorization operation can be applied to find the updated $\eG$.

\subsubsection*{\textbf{Given $\bA^{(1)},\ldots, \bA^{(n-1)}, \bA^{(n+1)}, \ldots, \bA^{(N)}$ and $\eG$, Update $\bA^{(n)}$:}}
When all but $\bA^{(n)}$ are fixed, by Equation~\eqref{equation:differe-tensor-norm}, the Frobenius norm of the difference is given by
$$
\begin{aligned}
&\gap \left\Vert\eX - \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  \right\Vert^2\\
&=\left\Vert\eX \right\Vert^2
-2\langle \eX,\llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket\rangle
+ \left\Vert\llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)},   \rrbracket\right\Vert^2,
\end{aligned}
$$
where by Lemma~\ref{lemma:tensor-multi2} (\ref{equationbracket}) and Equation~\eqref{equation:tucker-decom-in-tuckeropera}~\eqref{equation:tucker-decom-in-tuckeropera2}, it follows that
$$
\begin{aligned}
&\gap\langle \eX,\llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket\rangle\\
&=
\langle \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots, \bA^{(N)\top}   \rrbracket, \eG\rangle \gap &\text{(Lemma~\ref{lemma:tensor-multi2} (\ref{equationbracket}), Equation~\eqref{equation:tucker-decom-in-tuckeropera})}\\
&= \langle \eG, \eG \rangle=\left\Vert\eG \right\Vert^2,  &\gap \text{(Equation~\eqref{equation:tucker-decom-in-tuckeropera2})}
\end{aligned}
$$
and where by Equation~\eqref{equation:tucker-decom-in-tuckeropera} and Lemma~\ref{lemma:tensor-multi2}~\eqref{equationbracketlast}, the length preservation under semi-orthogonal, we obtain
$$
\left\Vert\llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)},   \rrbracket\right\Vert^2 = ||\eG||^2.
$$
Combine all the findings,
$$
\begin{aligned}
\left\Vert\eX - \llbracket\eG; \bA^{(1)}, \bA^{(2)}, \ldots, \bA^{(N)}   \rrbracket  \right\Vert^2
=\left\Vert\eX \right\Vert^2- \left\Vert\eG \right\Vert^2.
\end{aligned}
$$
Hence, minimizing the left of the above equation is equivalent to maximizing $\left\Vert\eG \right\Vert^2$. Therefore, to update $\bA^{(n)}$, the problem becomes
$$
\boxed{\begin{aligned}
		&\mathop{\max}_{\bA^{(n)}} \,\,\,\,\eG = \mathop{\max}_{\bA^{(n)}} \,\,\,\, \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots,\bA^{(n-1)},\bA^{(n)},\bA^{(n+1)},\ldots,  \bA^{(N)\top}\rrbracket\\
		&\text{subject to \gap  $\bA^{(n)} \in \real^{I_n\times R_n}$ is semi-orthogonal.}
\end{aligned} }
$$
By defining $\eY= \llbracket\eX; \bA^{(1)\top}, \bA^{(2)\top}, \ldots,\bA^{(n-1)}, \textcolor{blue}{\bI},\bA^{(n+1)},\ldots,  \bA^{(N)\top}\rrbracket$. From Lemma~\ref{lemma:tensor-multi-matriciz1}, this again, is equivalent to find the solution of
$$
\boxed{
\begin{aligned}
&\mathop{\max}_{\bA^{(n)}} \,\,\,\,||\eY \times_n \bA^{(n)}||= \mathop{\max}_{\bA^{(n)}} \,\,\,\, ||\bA^{(n)}\bY_{(n)}||_F\\
&\text{subject to \gap  $\bA^{(n)} \in \real^{I_n\times R_n}$ is semi-orthogonal,}
\end{aligned}
}
$$
where $\bY_{(n)} \in \real^{R_n\times R_{1}\ldots R_{n-1}R_{n+1}\ldots R_N}$ is the matricization along the mode-$n$ of $\eY$. The solution is by setting the rows of $\bA^{(n)}$ as the first $R_n$ leading left singular vectors of $\bY_{(n)}$, \citep{kolda2006multilinear, kolda2009tensor}. We notice the above update on $\bA^{(n)}$ is not dependent on $\eG$ finally, and thus we can update the $\eG$ once after the convergence of $\bA^{(n)}$'s. The procedure is shown in Algorithm~\ref{alg:tucker-decomposition-full-gene}.

\paragraph{Initialization by High-Order SVD (HOSVD)}\index{HOSVD}
In Algorithm~\ref{alg:tucker-decomposition-full-gene}, we initialize $\bA^{(n)}$'s randomly. However, the High-Order Singular Value Decomposition can be utilized as the initial point. The problem is given by
$$
\boxed{\begin{aligned}
&\mathop{\max}_{\bA^{(n)}} \,\,\,\, \llbracket\eX; \bI, \bI, \ldots,\bI,\bA^{(n)\top},\bI,\ldots,  \bI\rrbracket\\
&\text{subject to \gap  $\bA^{(n)} \in \real^{I_n\times R_n}$ is semi-orthogonal.}
\end{aligned} }
$$
Similarly, the problem is equal to finding the solution of
\begin{equation}\label{equation:hosvd-initia-for-tucker-0}
\boxed{\begin{aligned}
		&\mathop{\max}_{\bA^{(n)}}||\eX\times_n \bA^{(n)\top}||=||\bA^{(n)\top} \bX_{(n)}||_F\\
		&\text{subject to \gap  $\bA^{(n)} \in \real^{I_n\times R_n}$ is semi-orthogonal,}
\end{aligned} }
\end{equation}
which is solved by setting columns of $\bA^{(n)}$ as the first $R_n$ leading left singular vectors of $\bX_{(n)}$. And we shall shortly see an equivalent derivation by the matricization of the HOSVD in the next section, see Equation~\eqref{equation:hosvd-initia-for-tucker-1}.

\chapter{UTV Decomposition: ULV and URV Decomposition}\label{section:ulv-urv-decomposition}
\begingroup
\hypersetup{linkcolor=winestain}
\minitoc \newpage
\endgroup

\section{UTV Decomposition}
The UTV decomposition goes further by factoring the matrix into two orthogonal matrices $\bA=\bU\bT\bV$, where $\bU, \bV$ are orthogonal, whilst $\bT$ is (upper/lower) triangular.\footnote{These decompositions fall into a category known as the \textit{double-sided orthogonal decomposition}. We will see, the UTV decomposition, complete orthogonal decomposition, and singular value decomposition are all in this notion.}
The resulting $\bT$ supports rank estimation. The matrix $\bT$ can be lower triangular which results in the ULV decomposition, or it can be upper triangular which results in the URV decomposition. The UTV framework shares a similar form as the singular value decomposition (SVD, see Section~\ref{section:SVD}, p.~\pageref{section:SVD}) and can be regarded as inexpensive alternatives to the SVD.
\begin{theoremHigh}[Full ULV Decomposition]\label{theorem:ulv-decomposition}
	Every $m\times n$ matrix $\bA$ with rank $r$ can be factored as
	$$
	\bA = \bU \begin{bmatrix}
		\bL & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV,
	$$
	where $\bU\in \real^{m\times m}$ and $\bV\in \real^{n\times n}$ are two orthogonal matrices, and $\bL\in \real^{r\times r}$ is a lower triangular matrix.
\end{theoremHigh}

The existence of the ULV decomposition is from the QR and LQ decomposition.
\begin{proof}[of Theorem~\ref{theorem:ulv-decomposition}]
	For any rank $r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$, we can use a column permutation matrix $\bP$ (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}) such that the linearly independent columns of $\bA$ appear in the first $r$ columns of $\bA\bP$. Without loss of generality, we assume $\bb_1, \bb_2, \ldots, \bb_r$ are the $r$ linearly independent columns of $\bA$ and
	$$
	\bA\bP = [\bb_1, \bb_2, \ldots, \bb_r, \bb_{r+1}, \ldots, \bb_n].
	$$
	Let $\bZ = [\bb_1, \bb_2, \ldots, \bb_r] \in \real^{m\times r}$. Since any $\bb_i$ is in the column space of $\bZ$, we can find a $\bE\in \real^{r\times (n-r)}$ such that
	$$
	[\bb_{r+1}, \bb_{r+2}, \ldots, \bb_n] = \bZ \bE.
	$$
	That is,
	$$
	\bA\bP = [\bb_1, \bb_2, \ldots, \bb_r, \bb_{r+1}, \ldots, \bb_n] = \bZ
	\begin{bmatrix}
		\bI_r & \bE
	\end{bmatrix},
	$$
	where $\bI_r$ is an $r\times r$ identity matrix. Moreover, $\bZ\in \real^{m\times r}$ has full column rank such that its full QR decomposition is given by $\bZ = \bU\begin{bmatrix}
		\bR \\
		\bzero
	\end{bmatrix}$, where $\bR\in \real^{r\times r}$ is an upper triangular matrix with full rank and $\bU$ is an orthogonal matrix. This implies
	\begin{equation}\label{equation:ulv-smpl}
		\bA\bP = \bZ
		\begin{bmatrix}
			\bI_r & \bE
		\end{bmatrix}
		=
		\bU\begin{bmatrix}
			\bR \\
			\bzero
		\end{bmatrix}
		\begin{bmatrix}
			\bI_r & \bE
		\end{bmatrix}
		=
		\bU\begin{bmatrix}
			\bR & \bR\bE \\
			\bzero & \bzero
		\end{bmatrix}.
	\end{equation}
	Since $\bR$ has full rank, this means
	$\begin{bmatrix}
		\bR & \bR\bE
	\end{bmatrix}$ also has full rank such that its full LQ decomposition is given by
	$\begin{bmatrix}
		\bL & \bzero
	\end{bmatrix} \bV_0$ where $\bL\in \real^{r\times r}$ is a lower triangular matrix and $\bV_0$ is an orthogonal matrix. Substitute into Equation~\eqref{equation:ulv-smpl}, we have
	$$
	\bA = \bU \begin{bmatrix}
		\bL & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV_0 \bP^{-1}.
	$$
	Let $\bV =\bV_0 \bP^{-1}$ which is a product of two orthogonal matrices, and is also an orthogonal matrix. This completes the proof.
\end{proof}
A second way to see the proof of the ULV decomposition will be discussed in the proof of Theorem~\ref{theorem:complete-orthogonal-decom} shortly via the rank-revealing QR decomposition and trivial QR decomposition.

Now suppose the ULV decomposition of matrix $\bA$ is
$$
\bA = \bU \begin{bmatrix}
	\bL & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV.
$$
Let $\bU_0 = \bU_{:,1:r}$ and $\bV_0 = \bV_{1:r,:}$, i.e., $\bU_0$ contains only the first $r$ columns of $\bU$, and $\bV_0$ contains only the first $r$ rows of $\bV$. Then, we still have $\bA = \bU_0 \bL\bV_0$. This is known as the \textbf{reduced ULV decomposition}.
The comparison between the reduced and the full ULV decomposition is shown in Figure~\ref{fig:ulv-comparison} where white entries are zero and blue entries are not necessarily zero.
\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Reduced ULV decomposition]{\label{fig:ulvhalf}
		\includegraphics[width=0.47\linewidth]{./imgs/ulvreduced.pdf}}
	\quad
	\subfigure[Full ULV decomposition]{\label{fig:ulvall}
		\includegraphics[width=0.47\linewidth]{./imgs/ulvfull.pdf}}
	\quad
	\subfigure[Reduced URV decomposition]{\label{fig:urvhalf}
		\includegraphics[width=0.47\linewidth]{./imgs/urvreduced.pdf}}
	\quad
	\subfigure[Full URV decomposition]{\label{fig:urvall}
		\includegraphics[width=0.47\linewidth]{./imgs/urvfull.pdf}}
	\caption{Comparison between the reduced and full ULV, and between the reduced and full URV.}
	\label{fig:ulv-comparison}
\end{figure}

Similarly, we can also claim the URV decomposition as follows.
\begin{theoremHigh}[Full URV Decomposition]\label{theorem:urv-decomposition}
	Every $m\times n$ matrix $\bA$ with rank $r$ can be factored as
	$$
	\bA = \bU \begin{bmatrix}
		\bR & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV,
	$$
	where $\bU\in \real^{m\times m}$ and $\bV\in \real^{n\times n}$ are two orthogonal matrices, and $\bR\in \real^{r\times r}$ is an upper triangular matrix.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:urv-decomposition}]
	For any rank $r$ matrix $\bA=[\ba_1^\top; \ba_2^\top; \ldots; \ba_m^\top]$ where $\ba_1, \ba_2, \ldots$ are the rows of $\bA$, we can again construct a row permutation matrix $\bP$ such that the independent rows of $\bA$ appear in the first $r$ rows of $\bP\bA$. Without loss of generality, we assume $\bb_1^\top, \bb_2^\top, \ldots, \bb_r^\top$ are the $r$ linearly independent rows of $\bA$ and
	$$
	\bP\bA = \begin{bmatrix}
		\bb_1^\top \\
		\bb_2^\top\\
		\vdots\\
		\bb_r^\top\\
		\bb_{r+1}^\top\\
		\vdots \\
		\bb_m^\top
	\end{bmatrix}.
	$$
	Let $\bZ = [\bb_1^\top; \bb_2^\top; \ldots; \bb_r^\top] \in \real^{r\times n}$. Since any $\bb_i$ is in the row space of $\bZ$, we can find a $\bE\in \real^{(m-r)\times r}$ such that
	$$
	\begin{bmatrix}
		\bb_{r+1}^\top\\
		\bb_{r+2}^\top \\
		\vdots\\
		\bb_m^\top
	\end{bmatrix} =  \bE\bZ.
	$$
	That is,
	$$
	\bP\bA = \begin{bmatrix}
		\bb_1^\top \\
		\bb_2^\top\\
		\vdots\\
		\bb_r^\top\\
		\bb_{r+1}^\top\\
		\vdots \\
		\bb_m^\top
	\end{bmatrix}=
	\begin{bmatrix}
		\bI_r \\
		\bE
	\end{bmatrix}
	\bZ
	$$
	where $\bI_r$ is an $r\times r$ identity matrix. Moreover, $\bZ\in \real^{r\times n}$ has full row rank such that its full LQ decomposition is given by $\bZ = [\bL, \bzero]\bV$, where $\bL\in \real^{r\times r}$ is a lower triangular matrix with full rank and $\bV$ is an orthogonal matrix. This implies
	\begin{equation}\label{equation:urv-smpl}
		\bP\bA = \begin{bmatrix}
			\bI_r \\
			\bE
		\end{bmatrix}
		\bZ
		=
		\begin{bmatrix}
			\bI_r \\
			\bE
		\end{bmatrix}
		\begin{bmatrix}
			\bL & \bzero
		\end{bmatrix}
		\bV
		=
		\begin{bmatrix}
			\bL & \bzero \\
			\bE\bL & \bzero
		\end{bmatrix}\bV.
	\end{equation}
	Since $\bL$ has full rank, this means
	$\begin{bmatrix}
		\bL \\
		\bE\bL
	\end{bmatrix}$ also has full rank such that its full QR decomposition is given by
	$\bU_0\begin{bmatrix}
		\bR \\
		\bzero
	\end{bmatrix}$ where $\bR\in \real^{r\times r}$ is upper triangular and $\bU_0$ is an orthogonal matrix. Substitute into Equation~\eqref{equation:urv-smpl}, we have
	$$
	\bA =\bP^{-1} \bU_0 \begin{bmatrix}
		\bR & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV .
	$$
	Let $\bU =\bP^{-1} \bU_0$, which is a product of two orthogonal matrices, and is also an orthogonal matrix, from which the result follows.
\end{proof}

Again, there is a version of reduced URV decomposition and the difference between the full and reduced URV can be implied from the context as shown in Figure~\ref{fig:ulv-comparison}. The ULV and URV sometimes are referred to as the UTV decomposition framework \citep{fierro1997low, golub2013matrix}.
\begin{figure}[htbp]
	\centering

	\centering
	\resizebox{1.0\textwidth}{!}{%
		\begin{tikzpicture}[>=latex]

			\tikzstyle{state} = [draw, very thick, fill=white, rectangle, minimum height=3em, minimum width=6em, node distance=8em, font={\sffamily\bfseries}]
			\tikzstyle{stateEdgePortion} = [black,thick];
			\tikzstyle{stateEdge} = [stateEdgePortion,->];
			\tikzstyle{stateEdge2} = [stateEdgePortion,<->];
			\tikzstyle{edgeLabel} = [pos=0.5, text centered, font={\sffamily\small}];

			\node[ellipse, name=subspace, draw,font={\sffamily\bfseries},  node distance=7em, xshift=-9em, yshift=-1em,fill={colorals}]  {Subspaces};
			\node[state, name=qr, below of=subspace, xshift=-9em, yshift=0em, fill={colorlu}] {Full QR};
			\node[state, name=lq, below of=subspace, xshift=9em, yshift=0em, fill={colorlu}] {Full LQ};
			\node[state, name=rqr, left of=qr, xshift=-3em, fill={colorqr}] {Reduced QR};
			\node[state, name=rlq, right of=lq, xshift=3em, fill={colorqr}] {Reduced LQ};
			\node[state, name=utv, below of=subspace, xshift=0em, yshift=-9em, fill={colorlu}] {Full URV/ULV};
			\node[state, name=rutv, draw, right of=utv,xshift=7em, yshift=0em,fill={colorqr}]
			{Reduced URV/ULV};

			\coordinate (qr2inter3) at ($(subspace.west -| qr.north) + (-0em,0em)$);
			\draw (subspace.west) edge[stateEdgePortion] (qr2inter3);
			\draw (qr2inter3) edge[stateEdge]
			node[edgeLabel, text width=8em, yshift=0.8em]{\parbox{2em}{Column\\Space}} (qr.north);

			\coordinate (lq2inter3) at ($(subspace.east -| lq.north) + (-0em,0em)$);
			\draw (subspace.east) edge[stateEdgePortion] (lq2inter3);
			\draw (lq2inter3) edge[stateEdge]
			node[edgeLabel, text width=7.25em, yshift=0.8em]{\parbox{2em}{Row\\Space}} (lq.north);

			\coordinate (rqr2inter1) at ($(subspace.north) + (0,1em)$);
			\coordinate (rqr2inter3) at ($(rqr2inter1-| rqr.north) + (-0em,0em)$);
			\draw (subspace.north) edge[stateEdgePortion] (rqr2inter1);
			\draw (rqr2inter1) edge[stateEdgePortion] (rqr2inter3);
			\draw (rqr2inter3) edge[stateEdge]
			node[edgeLabel, text width=8em, yshift=0.8em]{\parbox{2em}{Column\\Space}} (rqr.north);

			\coordinate (rlr2inter3) at ($(rqr2inter1-| rlq.north) + (-0em,0em)$);
			\draw (rqr2inter1) edge[stateEdgePortion] (rlr2inter3);
			\draw (rlr2inter3) edge[stateEdge]
			node[edgeLabel, text width=8em, yshift=0.8em]{\parbox{2em}{Row\\Space}} (rlq.north);

			\draw (rqr.east)
			edge[stateEdge] node[edgeLabel,yshift=0.5em]{Complete}
			(qr.west) ;

			\draw (rlq.west)
			edge[stateEdge] node[edgeLabel,yshift=0.5em]{Complete}
			(lq.east) ;

			\draw (utv.east)
			edge[stateEdge] node[edgeLabel,yshift=0.5em]{Deleting}
			(rutv.west) ;

			\coordinate (qr2utv) at ($(qr.south) + (-0em,-3em)$);
			\coordinate (qr2utv2) at ($(qr2utv -| utv.north) + (-0em,0em)$);
			\coordinate (qr2utv3) at ($(lq.south) + (-0em,-3em)$);
			\draw (qr.south) edge[stateEdgePortion] (qr2utv);
			\draw (qr2utv) edge[stateEdgePortion] (qr2utv2);
			\draw (lq.south) edge[stateEdgePortion] (qr2utv3);
			\draw (qr2utv3) edge[stateEdgePortion] (qr2utv2);
			\draw (qr2utv2) edge[stateEdge] (utv.north);

			\begin{pgfonlayer}{background}
				\draw [join=round,cyan,dotted,fill={colormiddleleft}] ($(subspace.north west -|rqr.north west) + (-0.6em, 2em)$) rectangle ($(rutv.east -| rlq.south east) + (0.6em, -2.0em)$);
			\end{pgfonlayer}
		\end{tikzpicture}
	}

	\caption{Derive the ULV/URV by the QR and LR.}
	\label{fig:decomQR}
\end{figure}

The relationship between the QR and ULV/URV is depicted in Figure~\ref{fig:decomQR}. Furthermore, the ULV and URV decomposition can be utilized to prove an important property in linear algebra that we shall shortly see in the sequel: the row rank and column rank of any matrix are the same. Whereas, an elementary construction to prove this claim is provided in Appendix~\ref{append:row-equal-column} (p.~\pageref{append:row-equal-column}).

We will shortly see that the forms of ULV and URV are very close to the singular value decomposition (SVD). All of the three factor the matrix $\bA$ into two orthogonal matrices. Specially, there exists a set of basis for the four subspaces of $\bA$ in the fundamental theorem of linear algebra via the ULV and the URV. Taking ULV as an example, the first $r$ columns of $\bU$ form an orthonormal basis of $\cspace(\bA)$, and the last $(m-r)$ columns of $\bU$ form an orthonormal basis of $\nspace(\bA^\top)$. The first $r$ rows of $\bV$ form an orthonormal basis for the row space $\cspace(\bA^\top)$, and the last $(n-r)$ rows form an orthonormal basis for $\nspace(\bA)$ (similar to the two-sided orthogonal decomposition):
$$
\begin{aligned}
	\cspace(\bA) &= span\{\bu_1, \bu_2, \ldots, \bu_r\}, \\
	\nspace(\bA) &= span\{\bv_{r+1}, \bv_{r+2}, \ldots, \bv_n\}, \\
	\cspace(\bA^\top) &= span\{ \bv_1, \bv_2, \ldots,\bv_r \} ,\\
	\nspace(\bA^\top) &=span\{\bu_{r+1}, \bu_{r+2}, \ldots, \bu_m\}.
\end{aligned}
$$
The SVD goes further that there is a connection between the two pairs of orthonormal basis, i.e., transforming from column basis into row basis, or left null space basis into right null space basis. We will get more details in the SVD section.

\section{Complete Orthogonal Decomposition}
What is related to the UTV decomposition is called the \textit{complete orthogonal decomposition} which factors into two orthogonal matrices as well.
\begin{theoremHigh}[Complete Orthogonal Decomposition]\label{theorem:complete-orthogonal-decom}
	Every $m\times n$ matrix $\bA$ with rank $r$ can be factored as
	$$
	\bA = \bU \begin{bmatrix}
		\bT & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV,
	$$
	where $\bU\in \real^{m\times m}$ and $\bV\in \real^{n\times n}$ are two orthogonal matrices, and $\bT\in \real^{r\times r}$ is an rank-$r$ matrix.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:complete-orthogonal-decom}]
	By rank-revealing QR decomposition (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}), $\bA$ can be factored as
	$$
	\bQ_1^\top \bA\bP =
	\begin{bmatrix}
		\bR_{11} & \bR_{12} \\
		\bzero   & \bzero
	\end{bmatrix},
	$$
	where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ_1\in \real^{m\times m}$ is an orthogonal matrix, and $\bP$ is a permutation matrix.

	\begin{mdframed}[hidealllines=true,backgroundcolor=yellow!10]
		Then, it is not hard to find a decomposition such that
		\begin{equation}\label{equation:orthogonal-complete-qr-or-not}
			\begin{bmatrix}
				\bR_{11}^\top \\
				\bR_{12}^\top
			\end{bmatrix}
			=
			\bQ_2
			\begin{bmatrix}
				\bS \\
				\bzero
			\end{bmatrix},
		\end{equation}
		where $\bQ_2$ is an orthogonal matrix, $\bS$ is an rank-$r$ matrix. The decomposition is reasonable in the sense the matrix $ \begin{bmatrix}
			\bR_{11}^\top \\
			\bR_{12}^\top
		\end{bmatrix} \in \real^{n\times r}$ has rank $r$ of which the columns stay in a subspace of $\real^{n}$. Nevertheless, the columns of $\bQ_2$ span the whole space of $\real^n$, where we can assume the first $r$ columns of $\bQ_2$ span the same space as that of  $ \begin{bmatrix}
			\bR_{11}^\top \\
			\bR_{12}^\top
		\end{bmatrix}$. The matrix $\begin{bmatrix}
			\bS \\
			\bzero
		\end{bmatrix}$ is to transform $\bQ_2$ into $ \begin{bmatrix}
			\bR_{11}^\top \\
			\bR_{12}^\top
		\end{bmatrix}$.
	\end{mdframed}
	Then, it follows that
	$$
	\bQ_1^\top \bA\bP \bQ_2 =
	\begin{bmatrix}
		\bS^\top & \bzero \\
		\bzero & \bzero
	\end{bmatrix}.
	$$
	Let $\bU = \bQ_1$, $\bV=\bQ_2^\top\bP^\top$ and $\bT = \bS^\top$, we complete the proof.
\end{proof}
We can find that when Equation~\eqref{equation:orthogonal-complete-qr-or-not} is taken to be the reduced QR decomposition of $ \begin{bmatrix}
	\bR_{11}^\top \\
	\bR_{12}^\top
\end{bmatrix}$, then the complete orthogonal decomposition reduces to the ULV decomposition.

\section{Applications}
\subsection{Application: Least Squares via ULV/URV for Rank Deficient Matrices}\label{section:ls-utv}
In Section~\ref{section:application-ls-qr} (p.~\pageref{section:application-ls-qr}), we introduced the LS via the full QR decomposition for full rank matrices.
However, if often happens that the matrix may be rank-deficient. If $\bA$ does not have full column rank, $\bA^\top\bA$ is not invertible. We can then use the ULV/URV decomposition to find the least squares solution as illustrated in the following theorem.\index{Least squares}\index{Rank deficient}
\begin{theorem}[LS via ULV/URV for Rank Definient Matrix]\label{theorem:qr-for-ls-urv}
	Let $\bA\in \real^{m\times n}$ with rank $r$ and $m\geq n$. Suppose $\bA=\bU\bT\bV$ is its full ULV/URV decomposition with $\bU\in\real^{m\times m}, \bV\in \real^{n\times n}$ being orthogonal matrix matrices, and
	$$
	\bT = \begin{bmatrix}
		\bT_{11} & \bzero \\
		\bzero & \bzero
	\end{bmatrix}
	$$
	where $\bT_{11} \in \real^{r\times r}$ is a lower triangular matrix or an upper triangular matrix.

		Suppose $\bb\in \real^m$, then the LS solution with minimal 2-norm to $\bA\bx=\bb$ is given by
		$$
		\bx_{LS} = \bV^\top
		\begin{bmatrix}
			\bT_{11}^{-1}\bc\\
			\bzero
		\end{bmatrix},
		$$
		where $\bc$ is the first $r$ components of $\bU^\top\bb$.
	\end{theorem}

	\begin{proof}[of Theorem~\ref{theorem:qr-for-ls-urv}]
		Since $\bA=\bQ\bR$ is the full QR decomposition of $\bA$ and $m>n$, the last $m-n$ rows of $\bR$ are zero as shown in Figure~\ref{fig:qr-comparison}. Then $\bR_1 \in \real^{n\times n}$ is the square upper triangular in $\bR$ and
		$
		\bQ^\top \bA = \bR = \begin{bmatrix}
			\bR_1 \\
			\bzero
		\end{bmatrix}.
		$
		Thus,
		$$
		\begin{aligned}
			||\bA\bx-\bb||^2 &= (\bA\bx-\bb)^\top(\bA\bx-\bb)\\
			&=(\bA\bx-\bb)^\top\bU\bU^\top (\bA\bx-\bb) \qquad &(\text{Since $\bU$ is an orthogonal matrix})\\
			&=||\bU^\top \bA \bx-\bU^\top\bb||^2 \qquad &(\text{Invariant under orthogonal})\\
			&=|| \bU^\top\bU\bT\bV \bx-\bU^\top\bb||^2\\
			&=|| \bT\bV \bx-\bU^\top\bb||^2\\
			&=||\bT_{11}\be - \bc||^2+||\bd||^2,
		\end{aligned}
		$$
		where $\bc$ is the first $r$ components of $\bU^\top\bb$ and $\bd$ is the last $m-r$ components of $\bU^\top\bb$, $\be$ is the first $r$ components of $\bV\bx$ and $\bff$ is the last $n-r$ components of $\bV\bx$:
		$$
		\bU^\top\bb
		= \begin{bmatrix}
			\bc \\
			\bd
		\end{bmatrix},
		\qquad
		\bV\bx
		= \begin{bmatrix}
			\be \\
			\bff
		\end{bmatrix}
		$$
		And the LS solution can be calculated by back/forward substitution of the upper/lower triangular system $\bT_{11}\be = \bc$, i.e., $\be = \bT_{11}^{-1}\bc$. For $\bx$ to have minimal 2-norm, $\bff$ must be zero. That is
		$$
		\bx_{LS} = \bV^\top
		\begin{bmatrix}
			\bT_{11}^{-1}\bc\\
			\bzero
		\end{bmatrix}.
		$$
		This completes the proof.
	\end{proof}

	Moreover, we will shortly find that a similar argument can also be made via the singular value decomposition (SVD) in Section~\ref{section:application-ls-svd} since SVD shares similar form as the ULV/URV, both sandwiched by two orthogonal matrices. And SVD goes further by diagonalizing the matrix. Readers can also derive the LS solution via the rank-revealing QR decomposition.

	\paragraph{A word on the minimal 2-norm LS solution} For the least squares problem, the set of all minimizers
	$$
	\mathcal{X} = \{\bx\in \real^n: ||\bA\bx-\bb|| =\min \}
	$$
	is convex \citep{golub2013matrix}. And if $\bx_1, \bx_2 \in \mathcal{X}$ and $\lambda \in [0,1]$, then
	$$
	||\bA(\lambda\bx_1 + (1-\lambda)\bx_2) -\bb|| \leq \lambda||\bA\bx_1-\bb|| +(1-\lambda)||\bA\bx_2-\bb|| = \mathop{\min}_{\bx\in \real^n} ||\bA\bx-\bb||.
	$$
	Thus $\lambda\bx_1 + (1-\lambda)\bx_2 \in \mathcal{X}$. In above proof, if we do not set $\bff=\bzero$, we will find more least squares solutions. However, the minimal 2-norm least squares solution is unique. For full-rank case in the previous section, the least squares solution is unique and it must have a minimal 2-norm. See also \citep{foster2003solving, golub2013matrix} for a more detailed discussion on this topic.

	\subsection{Application: Row Rank equals Column Rank Again via UTV}
	As mentioned above, the UTV framework can prove the important theorem in linear algebra that the row rank and column rank of a matrix are equal.

	Notice that to apply the UTV in the proof, a slight modification on the claim of the existence of the UTV decomposition needs to be taken care of. For example, in Theorem~\ref{theorem:ulv-decomposition}, the assumption of the matrix $\bA$ is to have rank $r$. Since rank $r$ already admits the fact that row rank equals column rank. A better claim here to this aim is to say matrix $\bA$ has column rank $r$ in Theorem~\ref{theorem:ulv-decomposition}. See
	\citep{lu2021column} for a detailed discussion.

	\begin{theorem}[Row Rank Equals Column Rank\index{Matrix rank}]\label{lemma:equal-dimension-rank}
		The dimension of the column space of a matrix $\bA\in \real^{m\times n}$ is equal to the dimension of its
		row space, i.e., the row rank and the column rank of a matrix $\bA$ are equal.
	\end{theorem}

	\begin{proof}[\textbf{of Theorem~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}, A First Way}]
		Any $m\times n$ matrix $\bA$ with rank $r$ can be factored as
		$$
		\bA = \bU_0 \begin{bmatrix}
			\bL & \bzero \\
			\bzero & \bzero
		\end{bmatrix}\bV_0,
		$$
		where $\bU_0\in \real^{m\times m}$ and $\bV_0\in \real^{n\times n}$ are two orthogonal matrices, and $\bL\in \real^{r\times r}$ is a lower triangular matrix \footnote{Instead of using the ULV decomposition, in some texts, the authors use elementary transformations $\bE_1, \bE_2$ such that $
			\bA = \bE_1 \begin{bmatrix}
				\bI_r & \bzero \\
				\bzero & \bzero
			\end{bmatrix}\bE_2,
			$ to prove the result.}. Let
		$$\bD = \begin{bmatrix}
			\bL & \bzero \\
			\bzero & \bzero
		\end{bmatrix},
		$$
		the row rank and column rank of $\bD$ are apparently the same. If we could prove the column rank of $\bA$ equals the column rank of $\bD$, and the row rank of $\bA$ equals the row rank of $\bD$, then we complete the proof.

		Let $\bU = \bU_0^\top$, $\bV=\bV_0^\top$, then $\bD = \bU\bA\bV$. Decompose the above idea into two steps, a moment of reflexion reveals that, if we could first prove the row rank and column rank of $\bA$ are equal to those of $\bU\bA$, and then, if we further prove the row rank and column rank of $\bU\bA $ are equal to those of $\bU\bA\bV$, we could also complete the proof.

		\paragraph{Row rank and column rank of $\bA$ are equal to those of $\bU\bA$} Let $\bB = \bU\bA$, and let further $\bA=[\ba_1,\ba_2,\ldots, \ba_n]$ and $\bB=[\bb_1,\bb_2,\ldots,\bb_n]$ be the column partitions of $\bA$ and $\bB$ respectively. Therefore, $[\bb_1,\bb_2,\ldots,\bb_n] = [\bU\ba_1,\bU\ba_2,\ldots, \bU\ba_n]$. If $x_1\ba_1+x_2\ba_2+\ldots+x_n\ba_n=0$, then we also have
		$$
		\bU(x_1\ba_1+x_2\ba_2+\ldots+x_n\ba_n) = x_1\bb_1+x_2\bb_2+\ldots+x_n\bb_n = 0.
		$$
		Let $j_1, j_2, \ldots, j_r$ be distinct indices between 1 and $n$, if the set $\{\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}\}$ is independent, the set $\{\bb_{j_1}, \bb_{j_2}, \ldots, \bb_{j_r}\}$ must also be linearly independent. This implies
		$$
		dim(\cspace(\bB)) \leq dim(\cspace(\bA)).
		$$
		Similarly, by $\bA = \bU^\top\bB$, it follows that
		$$
		dim(\cspace(\bA)) \leq dim(\cspace(\bB)).
		$$
		This implies
		$$
		dim(\cspace(\bB)) = dim(\cspace(\bA)).
		$$
		Apply the process onto $\bB^\top$ and $\bA^\top$, we have
		$$
		dim(\cspace(\bB^\top)) = dim(\cspace(\bA^\top)).
		$$
		This implies the row rank and column rank of $\bA$ and $\bB=\bU\bA$ are the same.
		\paragraph{Row rank and column rank of $\bU\bA$ are equal to those of $\bU\bA\bV$}
		Similarly, by applying above discussion on $\bU\bA$ and $\bU\bA\bV$, we can also show that the row rank and column rank of $\bU\bA$ and $\bU\bA\bV$ are the same. This completes the proof.
	\end{proof}

\chapter*{\centering \begin{normalsize}Abstract\end{normalsize}}
In 1954, Alston S. Householder published \textit{Principles of Numerical Analysis}, one of the first modern treatments on matrix decomposition that favored a (block) LU
decomposition-the factorization of a matrix
into the product of lower and upper triangular
matrices.
And now, matrix decomposition has become a core technology in machine learning, largely due to the development of the back propagation algorithm in fitting a neural network. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, Hilbert space, and things in the complex domain. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields.

This survey is primarily a summary of purpose, significance of important matrix decomposition methods, e.g., LU, QR, and SVD, and the origin and complexity of the methods which shed light on their modern applications.
Most importantly, this article presents improved procedures for most of the calculations of the decomposition algorithms which potentially reduce the complexity they induce.
Again, this is a decomposition-based context, thus we will introduce the related background when it is needed and necessary. In many other textbooks on linear algebra, the principal ideas are discussed and the matrix decomposition methods serve as ``byproduct". However, we focus on the decomposition methods instead and the principal ideas serve as fundamental tools for them. The mathematical prerequisite is a first course in linear algebra. Other than this modest background, the development is self-contained, with rigorous proof provided throughout.

\paragraph{Keywords: }
Existence and computing of matrix decompositions, Complexity, Floating point operations (flops), Low-rank approximation, Pivot, LU decomposition for nonzero leading principal minors, Data distillation, CR decomposition, CUR/Skeleton decomposition, Interpolative decomposition, Biconjugate decomposition, Coordinate transformation, Hessenberg decomposition, ULV decomposition, URV decomposition, Rank decomposition, Gram-Schmidt process, Householder reflector, Givens rotation, Rank revealing decomposition, Cholesky decomposition and update/downdate, Eigenvalue problems, Alternating least squares, Randomized algorithm, Tensor decomposition, CP decomposition, Tucker decomposition, High-order SVD.

\input{chapter-worldmap.tex}
\input{chapter-intro.tex}
\input{chapter-LU.tex}
\input{chapter-cholesky.tex}
\input{chapter-qr.tex}
\input{chapter-utv.tex}
\input{chapter-cr.tex}
\input{chapter-skeleton.tex}
\input{chapter-id.tex}
\input{chapter-hessenberg.tex}
\input{chapter-eigenvalue.tex}
\input{chapter-spectral.tex}
\input{chapter-svd.tex}
\input{chapter-eigencalc.tex}
\input{chapter-coordinate.tex}
\input{chapter-als.tex}
\input{chapter-NMF.tex}
\input{chapter-biconjugate.tex}
\input{chapter-applications.tex}
\input{chapter-tensornotation.tex}
\input{chapter-tensorcp.tex}
\input{chapter-tensortucker.tex}
\input{chapter-tensorhosvd.tex}
\input{chapter-tensorttdecom.tex}
\input{chapter-append.tex}

\newpage
\vskip 0.2in

History and Overview (math.HO); Numerical Analysis (math.NA)

Numerical Matrix Decomposition