Matrix Decomposition and Applications

Abstract: In 1954, Alston S. Householder published Principles of Numerical Analysis, one of the first modern treatments on matrix decomposition that favored a (block) LU decomposition-the factorization of a matrix into the product of lower and upper triangular matrices. And now, matrix decomposition has become a core technology in machine learning, largely due to the development of the back propagation algorithm in fitting a neural network. The sole aim of this survey is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, Hilbert space, and things in the complex domain. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields.

\part{Gaussian Elimination}

\newpage
\chapter{LU Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{LU Decomposition}
Perhaps the best known and the first matrix decomposition we should know about is the LU decomposition. We now illustrate the results in the following theorem and the proof of the existence of which will be delayed in the following sections.
\begin{theoremHigh}[LU Decomposition with Permutation]\label{theorem:lu-factorization-with-permutation}
Every nonsingular $n\times n$ square matrix $\bA$ can be factored as
\begin{equation}
	\bA = \bP\bL\bU, \nonumber
\end{equation}
where $\bP$ is a permutation matrix, $\bL$ is a unit lower triangular matrix (i.e., lower triangular matrix with all 1's on the diagonal), and $\bU$ is a nonsingular upper triangular matrix.
\end{theoremHigh}

Note that, in the remainder of this text, we will put the decomposition-related results in the blue box. And other claims will be in a gray box. This rule will be applied for the rest of the book without special mention.

\begin{remark}[Decomposition Notation]
The above decomposition applies to any nonsingular matrix $\bA$. We will see that this decomposition arises from the elimination steps in which case row operations of subtraction and exchange of two rows are allowed, the subtractions are recorded in matrix $\bL$ and the row exchanges are recorded in matrix $\bP$. To make this row exchange explicit, the common form for the above decomposition is $\bQ\bA=\bL\bU$ where $\bQ=\bP^\top$ records the exact row exchanges of the rows of $\bA$. Otherwise, the $\bP$ would record the row exchanges of $\bL\bU$. In our case, we will make the decomposition to be clear for matrix $\bA$ rather than for $\bQ\bA$. For this reason, we will put the permutation matrix on the right-hand side of the equation for the remainder of the text without special mention.
\end{remark}

Particularly, there are circumstances where we won't require the permutation matrix. This decomposition relies on the leading principal minors. We provide a rigorous definition which is important for the illustration in the sequel.

\begin{definition}[Leading Principal Minors\index{Leading principal minors}]\label{definition:leading-principle-minors}
	Let $\bA$ be an $n\times n$ square matrix. A $k \times k$ submatrix of $\bA$ obtained by deleting the \textbf{last} $n-k$ columns and the \textbf{last} $n-k$ rows from $\bA$ is called a $k$-th order \textbf{leading principal submatrix} of $\bA$, that is, the $k\times k$ submatrix taken from the top left corner of $\bA$. The determinant of the $k \times k$ leading principal submatrix is called a $k$-th order \textbf{leading principal minor} of $\bA$.
\end{definition}

Under mild conditions on the leading principal minors of matrix $\bA$, the LU decomposition will not involve the permutation matrix.
\begin{theoremHigh}[LU Decomposition without Permutation]\label{theorem:lu-factorization-without-permutation}
For any $n\times n$ square matrix $\bA$, if all the leading principal minors are nonzero, i.e., $\det(\bA_{1:k,1:k})\neq 0$, for all $k\in \{1,2,\ldots, n\}$, then $\bA$ can be factored as
\begin{equation}
	\bA = \bL\bU, \nonumber
\end{equation}
where $\bL$ is a unit lower triangular matrix (i.e., lower triangular matrix with all 1's on the diagonal), and $\bU$ is a \textbf{nonsingular} upper triangular matrix.

\item Specifically, this decomposition is \textbf{unique}. See Corollary~\ref{corollary:unique-lu-without-permutation} (p.~\pageref{corollary:unique-lu-without-permutation}).
\end{theoremHigh}

\begin{remark}[Other Forms of the LU Decomposition without Permutation]
The leading principal minors are nonzero, in another word, this means the leading principal submatrices are nonsingular.

\paragraph{Singular $\bA$} In the above theorem, we assume $\bA$ is nonsingular as well. The LU decomposition also exists for singular matrix $\bA$. However, the matrix $\bU$ will be singular as well in this case. This can be shown in the following section that, if matrix $\bA$ is singular, some pivots will be zero, and the corresponding diagonal values of $\bU$ will be zero.

\paragraph{Singular leading principal submatrices} Even if we assume matrix $\bA$ is nonsingular, the leading principal submatrices might be singular. Suppose further that some of the leading principal minors are zero, the LU decomposition also exists, but if so, it is then not unique.
\end{remark}

We will discuss where this decomposition comes from in the next section. There are also generalizations of LU decomposition to non-square or singular matrices, such as rank-revealing LU decomposition. Please refer to \citet{pan2000existence, miranian2003strong, dopico2006multiple} for further discussion or we will have a short discussion in Section~\ref{section:rank-reveal-lu-short} (p.~\pageref{section:rank-reveal-lu-short}).

\index{Backward substitution}
\index{Gaussian elimination}
\section{Relation to Gaussian Elimination}\label{section:gaussian-elimination}
Solving linear system equation $\bA\bx=\bb$ is the basic problem in linear algebra.
Gaussian elimination transforms a linear system into an upper triangular one by applying simple \textit{elementary row transformations} on the left of the linear system in $n-1$ stages if $\bA\in \real^{n\times n}$. As a result, it is much easier to solve by a \textit{backward substitution}. The elementary transformation is defined rigorously as follows.
\begin{definition}[Elementary Transformation\index{Elementary transformation}]
For square matrix $\bA$, the following three transformations are referred to as \textbf{elementary row/column transformations}:
\item 1. Interchanging two rows (or columns) of $\bA$;
\item 2. Multiplying all elements of a row (or a column) of $\bA$ by some nonzero values;
\item 3. Adding any row (or column) of $\bA$ multiplied by a nonzero number to any other row (or column);
\end{definition}
Specifically, the elementary row transformations of $\bA$ are unit lower triangular to multiply $\bA$ on the left, and the elementary column transformations of $\bA$ are unit upper triangular to multiply $\bA$ on the right.

The Gaussian elimination is then described by the third type (elementary row transformation) above. Suppose the upper triangular matrix obtained by Gaussian elimination is given by $\bU = \bE_{n-1}\bE_{n-2}\ldots\bE_1\bA$ ($n-1$ steps), and in the $k$-th stage, the $k$-th column of $\bE_{k-1}\bE_{k-2}\ldots\bE_1\bA$ is $\bx\in \real^n$. Gaussian elimination will introduce zeros below the diagonal of $\bx$ by
$$
\bE_k = \bI - \bz_k \be_k^\top,
$$
where $\be_k \in \real^n$ is the $k$-th unit basis vector, and $\bz_k\in \real^n$ is given by
$$
\bz_k = [0, \ldots, 0, z_{k+1}, \ldots, z_n]^\top, \qquad z_i= \frac{x_{i}}{x_{k}}, \gap \forall i \in \{k+1,\ldots, n\}.
$$
We realize that $\bE_k$ is a unit lower triangular matrix (with $1$'s on the diagonal) with only the $k$-th column of the lower submatrix being nonzero,
$$
\bE_k=
\begin{bmatrix}
1 & \ldots & 0&  0 & \ldots & 0\\
\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
0 & \ldots & 1  & 0 & \ldots & 0 \\
0 & \ldots & -z_{k+1}  & 1 & \ldots & 0\\
\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
0 & \ldots & -z_n & 0 & \ldots & 1
\end{bmatrix},
$$
and multiplying on the left by $\bE_k$ will introduce zeros below the diagonal:
$$
\bE_k \bx =
\begin{bmatrix}
	1 & \ldots & 0&  0 & \ldots & 0\\
	\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
	0 & \ldots & 1  & 0 & \ldots & 0 \\
	0 & \ldots & -z_{k+1}  & 1 & \ldots & 0\\
	\vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
	0 & \ldots & -z_n & 0 & \ldots & 1
\end{bmatrix}
\begin{bmatrix}
	x_1 \\
	\vdots \\
	x_k\\
	x_{k+1} \\
	\vdots \\
	x_n
\end{bmatrix}=
\begin{bmatrix}
	x_1 \\
	\vdots \\
	x_k\\
0 \\
	\vdots \\
0
\end{bmatrix}.
$$
For example, we write out the Gaussian elimination steps for a $4\times 4$ matrix. For simplicity, we assume there are no row permutations. And in the following matrix, $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\paragraph{A Trivial Gaussian Elimination For a $4\times 4$ Matrix:}
\begin{equation}\label{equation:elmination-steps}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	 0 & \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	 0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
      0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bE_3}{\longrightarrow}
\begin{sbmatrix}{\bE_3\bE_2\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
    0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & 0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix},
\end{equation}
where $\bE_1, \bE_2, \bE_3$ are lower triangular matrices. Specifically, as discussed above, Gaussian transformation matrices $\bE_i$'s are unit lower triangular matrices with $1$'s on the diagonal. This can be explained that for the $k$-th transformation $\bE_k$, working on the matrix $\bE_{k-1}\ldots\bE_1\bA$, the transformation subtracts multiples of the $k$-th row from rows $\{k+1, k+2, \ldots, n\}$ to get zeros below the diagonal in the $k$-th column of the matrix. And never use rows $\{1, 2, \ldots, k-1\}$.

To be more concrete, for the transformation example above in step $1$, we multiply on the left by $\bE_1$ so that multiples of the $1$-st row are subtracted from rows $2, 3, 4$ and the first entries of rows $2, 3, 4$ are set to zero. Similar situations for step 2 and step 3. By setting $\bL=\bE_1^{-1}\bE_2^{-1}\bE_3^{-1}$ and letting the matrix after elimination be $\bU$, \footnote{The inverses of unit lower triangular matrices are also unit lower triangular matrices. And the products of unit lower triangular matrices are also unit lower triangular matrices.} we get $\bA=\bL\bU$. Thus we obtain an LU decomposition for this $4\times 4$ matrix $\bA$.

\begin{definition}[Pivot\index{Pivot}]\label{definition:pivot}
First nonzero entry in the row after each elimination step is called a \textbf{pivot}. For example, the \textcolor{blue}{blue} crosses in Equation~\eqref{equation:elmination-steps} are pivots.
\end{definition}

However, it is possible for $a_{11}$ (the entry (1,1) of matrix $\bA$) to have a value of zero on occasion.
No such matrix $\bE_1$ can make the next elimination step successful.
So we need to interchange the first row and the second row via a permutation matrix $\bP_1$. This is known as the \textit{pivoting}, or simply \textit{permutation}.
\paragraph{Gaussian Elimination With a Permutation In the Beginning:}
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	0 & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\bP_1}{\longrightarrow}
\begin{sbmatrix}{\bP_1\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bP_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
&\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bE_1\bP_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bE_3}{\longrightarrow}
\begin{sbmatrix}{\bE_3\bE_2\bE_1\bP_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & 0  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix}.
\end{aligned}
$$
By setting $\bL=\bE_1^{-1}\bE_2^{-1}\bE_3^{-1}$ and $\bP=\bP_1^{-1}$, we get $\bA=\bP\bL\bU$. Therefore we obtain a full LU decomposition with permutation for this $4\times 4$ matrix $\bA$.

In some circumstances, other permutation matrices $\bP_2, \bP_3, \ldots$ will appear in between the lower triangular $\bE_i$'s. An example is shown as follows.
\paragraph{Gaussian Elimination With a Permutation In Between:}
$$
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\end{sbmatrix}
\stackrel{\bE_1}{\longrightarrow}
\begin{sbmatrix}{\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bP_1}{\longrightarrow}
\begin{sbmatrix}{\bP_1\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bE_2}{\longrightarrow}
\begin{sbmatrix}{\bE_2\bP_1\bE_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
	0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
	0 & \bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
\end{sbmatrix}.
$$
In this case, we find $\bU=\bE_2\bP_1\bE_1\bA$. In Section~\ref{section:lu-perm} (p.~\pageref{section:lu-perm}) or Section~\ref{section:partial-pivot-lu} (p.~\pageref{section:partial-pivot-lu}), we will show that the permutations in-between will result in the same form $\bA=\bP\bL\bU$ where $\bP$ takes account of all the permutations.

The above examples can be easily extended to any $n\times n$ matrix if we assume there are no row permutations in the process. And we will have $n-1$ such lower triangular transformations.
The $k$-th transformation $\bE_k$ introduces zeros below the diagonal in the $k$-th column of $\bA$ by subtracting multiples of the $k$-th row from rows $\{k+1, k+2, \ldots, n\}$.
Finally, by setting $\bL=\bE_1^{-1}\bE_2^{-1}\ldots \bE_{n-1}^{-1}$ we obtain the LU decomposition $\bA=\bL\bU$ (without permutation).

\section{Existence of the LU Decomposition without Permutation}\label{section:exist-lu-without-perm}
The Gaussian elimination or Gaussian transformation reveals the origin of the LU decomposition. We then prove Theorem~\ref{theorem:lu-factorization-without-permutation} rigorously, i.e., the existence of the LU decomposition without permutation by induction.
\begin{proof}[\textbf{of Theorem~\ref{theorem:lu-factorization-without-permutation}: LU Decomposition without Permutation}]
We will prove by induction that every $n\times n$ square matrix $\bA$ with nonzero leading principal minors has a decomposition $\bA=\bL\bU$. The $1\times 1$ case is trivial by setting $L=1, U=A$, thus, $A=LU$.

Suppose for any $k\times k$ matrix $\bA_k$ with all the leading principal minors being nonzero has an LU decomposition without permutation. If we prove any $(k+1)\times(k+1)$ matrix $\bA_{k+1}$ can also be factored as this LU decomposition without permutation, then we complete the proof.

For any $(k+1)\times(k+1)$ matrix $\bA_{k+1}$, suppose the $k$-th order leading principal submatrix of $\bA_{k+1}$ is $\bA_k$ with the size of $k\times k$. Then $\bA_k$ can be factored as $\bA_k =  \bL_k\bU_k$ with $\bL_k$ being a unit lower triangular matrix and $\bU_k$ being a nonsingular upper triangular matrix from the assumption. Write out $\bA_{k+1}$ as
$
\bA_{k+1} = \begin{bmatrix}
	\bA_k & \bb \\
	\bc^\top & d
\end{bmatrix}.
$
Then it admits the factorization:
$$
\bA_{k+1} = \begin{bmatrix}
	\bA_k & \bb \\
	\bc^\top & d
\end{bmatrix}
=
\begin{bmatrix}
	\bL_k &\bzero \\
	\bx^\top  & 1
\end{bmatrix}
\begin{bmatrix}
	\bU_k & \by\\
	\bzero & z
\end{bmatrix} = \bL_{k+1}\bU_{k+1},
$$
where $\bb = \bL_k\by$, $\bc^\top = \bx^\top\bU_k$, $d = \bx^\top\by + z$, $\bL_{k+1}=\begin{bmatrix}
	\bL_k &\bzero \\
	\bx^\top  & 1
\end{bmatrix}$, and $\bU_{k+1}=\begin{bmatrix}
\bU_k & \by\\
\bzero & z
\end{bmatrix}$.
From the assumption, $\bL_k$ and $\bU_k$ are nonsingular. Therefore
$$
\by = \bL_k^{-1}\bb, \qquad \bx^\top=\bc^\top\bU_k^{-1}, \qquad  z=d - \bx^\top\by.
$$
If further, we could prove $z$ is nonzero such that $\bU_{k+1}$ is nonsingular, we complete the proof.

Since all the leading principal minors of $\bA_{k+1}$ are nonzero,
we have $\det(\bA_{k+1})=$
\footnote{By the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bA & \bB \\
		\bC & \bD
	\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$.}
$\det(\bA_k)\cdot$ $\det(d-\bc^\top\bA_k^{-1}\bb)
=\det(\bA_k)\cdot(d-\bc^\top\bA_k^{-1}\bb) \neq 0$, where $d-\bc^\top\bA_k^{-1}\bb$ is a scalar.
As $\det(\bA_k)\neq 0$ from the assumption, we obtain $d-\bc^\top\bA_k^{-1}\bb \neq 0$. Substitute $\bb = \bL_k\by$ and $\bc^\top = \bx^\top\bU_k$ into the formula, we have $d-\bx^\top\bU_k\bA_k^{-1}\bL_k\by =d-\bx^\top\bU_k(\bL_k\bU_k)^{-1}\bL_k\by =d-\bx^\top\by \neq 0$ which is exactly the form of $z\neq 0$. Thus we find $\bL_{k+1}$ with all the values on the diagonal being 1, and $\bU_{k+1}$ with all the values on the diagonal being nonzero which means $\bL_{k+1}$ and $\bU_{k+1}$ are nonsingular. \footnote{A triangular matrix (upper or lower) is nonsingular if and only if all the entries on its main diagonal are nonzero.}
This completes the proof.
\end{proof}

We further prove that if no permutation involves, the LU decomposition is unique.
\begin{corollary}[Uniqueness of the LU Decomposition without Permutation]\label{corollary:unique-lu-without-permutation}
Suppose the $n\times n$ square matrix $\bA$ has nonzero leaning principal minors. Then, the LU decomposition is unique.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:unique-lu-without-permutation}]
Suppose the LU decomposition is not unique, then we can find two decompositions such that $\bA=\bL_1\bU_1 = \bL_2\bU_2$ which implies $\bL_2^{-1}\bL_1=\bU_2\bU_1^{-1}$. The left of the equation is a unit lower triangular matrix and the right of the equation is an upper triangular matrix.
This implies both sides of the above equation are diagonal matrices. Since the inverse of a unit lower triangular matrix is also a unit lower triangular matrix, and the product of unit lower triangular matrices is also a unit lower triangular matrix, this results in that $\bL_2^{-1}\bL_1 = \bI$.
The equality implies that both sides are identity matrices such that $\bL_1=\bL_2$ and $\bU_1=\bU_2$ which leads to a contradiction.
\end{proof}

In the proof of Theorem~\ref{theorem:lu-factorization-without-permutation}, we have shown that the diagonal values of the upper triangular matrix are all nonzero if the leading principal minors of $\bA$ are all nonzero. We then can formulate this decomposition in another form if we divide each row of $\bU$ by each diagonal value of $\bU$. This is called the \textit{LDU decomposition}.
\begin{corollaryHigh}[LDU Decomposition]\label{corollary:ldu-decom}
For any $n\times n$ square matrix $\bA$, if all the leading principal minors are nonzero, i.e., $\det(\bA_{1:k,1:k})\neq 0$, for all $k\in \{1,2,\ldots, n\}$, then $\bA$ can be \textbf{uniquely} factored as
\begin{equation}
	\bA = \bL\bD\bU, \nonumber
\end{equation}
where $\bL$ is a unit lower triangular matrix, $\bU$ is a \textbf{unit} upper triangular matrix, and $\bD$ is a diagonal matrix.
\end{corollaryHigh}
The proof is trivial that from the LU decomposition of $\bA=\bL\bR$, we can find a diagonal matrix $\bD=diag(r_{11}, r_{22}, \ldots, r_{nn})$ such that $\bD^{-1}\bR = \bU$ is a unit upper triangular matrix. And the uniqueness comes from the uniqueness of the LU decomposition.

\section{Existence of the LU Decomposition with Permutation}\label{section:lu-perm}
In Theorem~\ref{theorem:lu-factorization-without-permutation}, we require that matrix $\bA$ has nonzero leading principal minors. However, this is not necessarily. Even when the leading principal minors are zero, nonsingular matrices still have an LU decomposition, but with an additional permutation. The proof is still from induction.
\begin{proof}[\textbf{of Theorem~\ref{theorem:lu-factorization-with-permutation}: LU Decomposition with Permutation}]
We note that any $1\times 1$ nonsingular matrix has a full LU decomposition $A=PLU$ by simply setting $P=1$, $L=1$, $U=A$.
We will show that if every $(n-1)\times (n-1)$ nonsingular matrix has a full LU decomposition, then this is also true for every $n\times n$ nonsingular matrix. By induction, we prove that every nonsingular matrix has a full LU decomposition.

We will formulate the proof in the following order. \textcolor{black}{If $\bA$ is nonsingular, then its row permuted matrix $\bB$ is also nonsingular}. And the \textit{Schur complement} of $b_{11}$ in $\bB$ is also nonsingular. Finally, we formulate the decomposition of $\bA$ by $\bB$ from this property.

We notice that at least one element in the first column of $\bA$ must be nonzero, otherwise $\bA$ will be singular. We can then apply a row permutation that makes the element in entry $(1,1)$ to be nonzero. That is, there exists a permutation $\bP_1$ such that $\bB = \bP_1 \bA$ in which case $b_{11} \neq 0$. Since $\bA$ and $\bP_1$ are both nonsingular and the product of nonsingular matrices is also nonsingular, then $\bB$ is also nonsingular.

\begin{mdframed}[hidealllines=\mdframehidelineNote,backgroundcolor=\mdframecolorSkip,frametitle={Schur complement of $\bB$ is also nonsingular:}]
Now consider the Schur complement of $b_{11}$ in $\bB$ with size $(n-1)\times (n-1)$:
$$
\widehat{\bB} = \bB_{2:n,2:n} -\frac{1}{b_{11}} \bB_{2:n,1} \bB_{1,2:n}.
$$
Suppose there is an $(n-1)$-vector $\bx$ satisfies
\begin{equation}\label{equ:lu-pivot1}
	\widehat{\bB} \bx = 0.
\end{equation}
Then $\bx$ and $y=-\frac{1}{b_{11}}\bB_{1,2:n} \cdot \bx  $ satisfy
$$
\bB
\left[
\begin{matrix}
	\bx \\
	y
\end{matrix}
\right]
=
\left[
\begin{matrix}
	b_{11} & \bB_{1,2:n} \\
	\bB_{2:n,1} & \bB_{2:n,2:n}
\end{matrix}
\right]
\left[
\begin{matrix}
	\bx \\
	y
\end{matrix}
\right]
=
\left[
\begin{matrix}
	\bzero \\
	0
\end{matrix}
\right].
$$
Since $\bB$ is nonsingular, $\bx$ and $y$ must be zero. Hence, Equation~\eqref{equ:lu-pivot1} holds only if $\bx=\bzero$ which means that the null space of $\widehat{\bB}$ is of dimension 0 and thus $\widehat{\bB}$ is nonsingular with size $(n-1)\times(n-1)$.
\end{mdframed}

By the induction assumption that any $(n-1)\times(n-1)$ nonsingular matrix can be factorized as the full LU decomposition form
$$
\widehat{\bB} = \bP_2\bL_2\bU_2.
$$
We then factor $\bA$ as
\begin{equation*}
	\begin{aligned}
		\bA &= \bP_1^\top
		\left[
		\begin{matrix}
			b_{11} & \bB_{1,2:n} \\
			\bB_{2:n,1} & \bB_{2:n,2:n}
		\end{matrix}
		\right]\\
		&= \bP_1^\top
		\left[
		\begin{matrix}
			1 & 0 \\
			0 & \bP_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			b_{11} & \bB_{1,2:n} \\
			\bP_2^\top \bB_{2:n,1} &\bP_2^\top \bB_{2:n,2:n}
		\end{matrix}
		\right]\\
		&= \bP_1^\top
		\left[
		\begin{matrix}
			1 & 0 \\
			0 & \bP_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			b_{11} & \bB_{1,2:n} \\
			\bP_2^\top \bB_{2:n,1} & \textcolor{blue}{\bL_2\bU_2}+\bP_2^\top \textcolor{blue}{\frac{1}{b_{11}} \bB_{2:n,1} \bB_{1,2:n}}
		\end{matrix}
		\right]\\
		&= \bP_1^\top
		\left[
		\begin{matrix}
			1 & 0 \\
			0 & \bP_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			1 & 0 \\
			\frac{1}{b_{11}}\bP_2^\top \bB_{2:n,1} & \bL_2
		\end{matrix}
		\right]
		\left[
		\begin{matrix}
			b_{11} & \bB_{1,2:n} \\
			\bzero & \bU_2
		\end{matrix}
		\right].\\
	\end{aligned}
\end{equation*}
Therefore, we find the full LU decomposition of $\bA=\bP\bL\bU$ by defining
$$
\bP = \bP_1^\top
\left[
\begin{matrix}
	1 & 0 \\
	0 & \bP_2
\end{matrix}
\right], \qquad
\bL=\left[
\begin{matrix}
	1 & 0 \\
	\frac{1}{b_{11}}\bP_2^\top \bB_{2:n,1} & \bL_2
\end{matrix}
\right], \qquad
\bU=
\left[
\begin{matrix}
	b_{11} & \bB_{1,2:n} \\
	\bzero & \bU_2
\end{matrix}
\right],
$$
from which the result follows.
\end{proof}

\section{Bandwidth Preserving in the LU Decomposition without Permutation}

For any matrix, the bandwidth of it can be defined as follows.
\begin{definition}[Matrix Bandwidth\index{Matrix bandwidth}]\label{defin:matrix-bandwidth}
For any matrix $\bA\in \real^{n\times n}$ with entry $(i,j)$ element denoted by $a_{ij}$. Then $\bA$ has \textbf{upper bandwidth $q$} if $a_{ij} =0$ for $j>i+q$, and \textbf{lower bandwidth $p$} if $a_{ij}=0$ for $i>j+p$.

An example of a $6\times 6$ matrix with upper bandwidth $2$ and lower bandwidth $3$ is shown as follows:
$$
\begin{bmatrix}
\boxtimes & \boxtimes & \boxtimes & 0& 0 & 0\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes& 0 & 0\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes & 0\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes & \boxtimes\\
0 & 0 & \boxtimes & \boxtimes& \boxtimes & \boxtimes\\
\end{bmatrix}.
$$
\end{definition}

Then, we prove that the bandwidth after the LU decomposition without permutation is preserved.
\begin{lemma}[Bandwidth Preserving]\label{lemma:lu-bandwidth-presev}
For any matrix $\bA\in \real^{n\times n}$ with upper bandwidth $q$ and lower bandwidth $p$. If $\bA$ has an LU decomposition $\bA=\bL\bU$, then $\bU$ has upper bandwidth $q$ and $\bL$ has lower bandwidth $p$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:lu-bandwidth-presev}]
The LU decomposition without permutation can be obtained as follows:
$$
\bA=\left[
\begin{matrix}
	a_{11} & \bA_{1,2:n} \\
	\bA_{2:n,1} & \bA_{2:n,2:n}
\end{matrix}
\right]
=\left[
\begin{matrix}
	1 & 0 \\
\frac{1}{a_{11}} \bA_{2:n,1}  & \bI_{n-1}
\end{matrix}
\right]
\left[
\begin{matrix}
	a_{11} & \bA_{1,2:n}\\
	0 & \bS
\end{matrix}
\right] = \bL_1 \bU_1,
$$
where $\bS =\bA_{2:n,2:n} - \frac{1}{a_{11}}\bA_{2:n,1}\bA_{1,2:n}$ is the Schur complement of $a_{11}$ in $\bA$.
We can name this decomposition of $\bA$ the $s$-decomposition of $\bA$.
The first column of $\bL_1$ and the first row of $\bU_1$ have the required structure (bandwidth $p$ and $q$ respectively), and the Schur complement $\bS$ of $a_{11}$ has upper bandwidth $q-1$ and lower bandwidth $p-1$ respectively. The result follows by induction on the $s$-decomposition of $\bS$.
\end{proof}

\section{Block LU Decomposition}
Another form of the LU decomposition is to factor the matrix into  block triangular matrices.

\begin{theoremHigh}[Block LU Decomposition without Permutation]\label{theorem:block-lu-factorization-without-permutation}
For any $n\times n$ square matrix $\bA$, if the first $m$ leading principal block submatrices are nonsingular, then $\bA$ can be factored as
\begin{equation}
\bA = \bL\bU
=
\begin{bmatrix}
\bI  & & & \\
\bL_{21} & \bI & & \\
\vdots & & \ddots & \\
\bL_{m1} & \ldots & \bL_{m,m-1} & \bI
\end{bmatrix}
\begin{bmatrix}
\bU_{11}  &\bU_{12} & \ldots & \bU_{1m}\\
  & \bU_{22} & &  \vdots \\
  & & \ddots & \bU_{m-1,m}\\
  & & & \bU_{mm}\\
\end{bmatrix}
, \nonumber
\end{equation}
where $\bL_{i,j}$'s and $\bU_{ij}$'s are some block matrices.

Specifically, this decomposition is unique.
\end{theoremHigh}
Note that the $\bU$ in the above theorem is not necessarily upper triangular. An example can be shown as follows:
$$
\bA =
\left[\begin{array}{cc;{2pt/2pt}cc}
0& 1 &   1 & 1\\
-1& 2 &   -1 & 2\\\hdashline[2pt/2pt]
2& 1 &   4 & 2\\
1& 2 &   3 & 3\\
\end{array}\right]
=
\left[\begin{array}{cc;{2pt/2pt}cc}
	1& 0 &   0& 0\\
	0& 1 &   0 & 0\\\hdashline[2pt/2pt]
	5& -2 &   1 & 0\\
	4& -1 &   0 & 1\\
\end{array}\right]
\left[\begin{array}{cc;{2pt/2pt}cc}
	0& 1 &   1 & 1\\
	-1& 2 &   -1 & 2\\\hdashline[2pt/2pt]
	0& 0 &   -3 & 1\\
	0& 0 &   -2 & 1\\
\end{array}\right].
$$
The trivial non-block LU decomposition fails on $\bA$ since the entry $(1,1)$ is zero. However, the block LU decomposition exists.

\section{Application: Linear System via the LU Decomposition}\label{section:lu-linear-sistem}
 Consider the well-determined linear system $\bA\bx = \bb$ with $\bA$ of size $n\times n $ and nonsingular. Avoid solving the system by computing the inverse of $\bA$, we solve linear equation by the LU decomposition. Suppose $\bA$ admits the LU decomposition $\bA = \bP\bL\bU$. The solution is given by the following algorithm.

\begin{algorithm}[H]
	\caption{Solving Linear Equations by LU Decomposition}
	\label{alg:linear-equation-by-LU}
	\begin{algorithmic}[1]
		\Require
		matrix $\bA$ is nonsingular and square with size $n\times n $, solve $\bA\bx=\bb$;
		\State LU Decomposition: factor $\bA$ as $\bA=\bP\bL\bU$; \Comment{(2/3)$n^3$ flops}
		\State Permutation: $\bw = \bP^\top\bb$; \Comment{0 flops }
		\State Forward substitution: solve $\bL\bv = \bw$; \Comment{$1+3+\ldots + (2n-1)=n^2$ flops}
		\State Backward substitution: solve $\bU\bx= \bv$; \Comment{$1+3+\ldots + (2n-1)=n^2$ flops}
	\end{algorithmic}
\end{algorithm}

The complexity of the decomposition step is $(2/3)n^3$ floating point operations (flops) \citep{lu2021numerical}, the backward and forward substitution steps both cost $1+3+\ldots + (2n-1)=n^2$ flops. Therefore, the total cost for computing the linear system via the LU factorization is $(2/3)n^3 + 2n^2$ flops. If we keep only the leading term, the Algorithm~\ref{alg:linear-equation-by-LU} costs $(2/3)n^3$ flops where the most complexity comes from the LU decomposition.

\paragraph{Linear system via the block LU decomposition} For a block LU decomposition of $\bA=\bL\bU$, we need to solve $\bL\bv = \bw$ and $\bU\bx = \bv$. But the latter system is not triangular and requires some extra computations.

\index{Inverse of a matrix}
\section{Application: Computing the Inverse of Nonsingular Matrices}\label{section:inverse-by-lu}
By Theorem~\ref{theorem:lu-factorization-with-permutation}, for any nonsingular matrix $\bA\in \real^{n\times n}$, we have a full LU factorization $\bA=\bP\bL\bU$. Then the inverse can be obtained by solving the matrix equation
$$
\bA\bX = \bI,
$$
which contains $n$ linear systems computation: $\bA\bx_i = \be_i$ for all $i \in \{1, 2, \ldots, n\}$ where $\bx_i$ is the $i$-the column of $\bX$ and $\be_i$ is the $i$-th column of $\bI$ (i.e., the $i$-th unit vector).

\begin{theorem}[Inverse of Nonsingular Matrix by Linear System]
Computing the inverse of a nonsingular matrix $\bA \in \real^{n\times n}$ by $n$ linear systems needs $\sim (2/3)n^3 + n(2n^2)=(8/3)n^3$ flops where $(2/3)n^3$ comes from the computation of the LU decomposition of $\bA$.
\end{theorem}
The proof is trivial by using Algorithm~\ref{alg:linear-equation-by-LU}.
However, the complexity can be reduced by taking the advantage of the structures of $\bU, \bL$.
We find that the inverse of the nonsingular matrix is $\bA^{-1} = \bU^{-1}\bL^{-1}\bP^{-1}=\bU^{-1}\bL^{-1}\bP^{T}$. By taking this advantage, the complexity is reduced from $(8/3)n^3$ to $2n^3$ flops.

\section{Application: Computing the Determinant via the LU Decomposition}
We can find the determinant easily by using the LU decomposition.
If $\bA=\bL\bU$, then $\det(\bA) = \det(\bL\bU) = \det(\bL)\det(\bU) = u_{11}u_{22}\ldots u_{nn}$ where $u_{ii}$ is the $i$-th diagonal of $\bU$ for $i\in \{1,2,\ldots,n\}$. \footnote{The determinant of a lower triangular matrix (or an upper triangular matrix) is the product of the diagonal entries.}

Further, for the LU decomposition with permutation $\bA=\bP\bL\bU$, $\det(\bA) = \det(\bP\bL\bU) = \det(\bP)u_{11}u_{22}\ldots u_{nn}$.
The determinant of a permutation matrix is either 1 or –1 because after
changing rows around (which changes the sign of the determinant \footnote{The determinant changes sign when two rows are exchanged (sign reversal).}) a permutation matrix
becomes identity matrix $\bI$, whose determinant is one.

 \index{Partial pivoting}
\section{Pivoting}
\subsection{Partial Pivoting}\label{section:partial-pivot-lu}
In practice, it is desirable to pivot even when it is not necessary. When dealing with a linear system via the LU decomposition as shown in Algorithm~\ref{alg:linear-equation-by-LU}, if the diagonal entries of $\bU$ are small, it can lead to inaccurate solutions for the linear solution. Thus, it is common to pick the largest entry to be the pivot. This is known as the \textit{partial pivoting}. For example,
\paragraph{Partial Pivoting For a $4\times 4$ Matrix:}
	\begin{equation}\label{equation:elmination-steps2}
		\begin{sbmatrix}{\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes
		\end{sbmatrix}
		\stackrel{\bE_1}{\longrightarrow}
		\begin{sbmatrix}{\bE_1\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \bm{2} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & \bm{5} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & \bm{7} & \bm{\boxtimes} & \bm{\boxtimes}
		\end{sbmatrix}
		\stackrel{\bP_1}{\longrightarrow}
		\begin{sbmatrix}{\bP_1\bE_1\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \textcolor{blue}{7} & \textcolor{blue}{\boxtimes} & \textcolor{blue}{\boxtimes} \\
			0  & 5 & \boxtimes & \boxtimes \\
			0 & \textcolor{blue}{2} & \textcolor{blue}{\boxtimes} & \textcolor{blue}{\boxtimes}
		\end{sbmatrix}
		\stackrel{\bE_2}{\longrightarrow}
		\begin{sbmatrix}{\bE_2\bP_1\bE_1\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 &  7 & \boxtimes & \boxtimes \\
			0 & \bm{0}  & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
			0 & \bm{0} & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
		\end{sbmatrix},
	\end{equation}
in which case, we pick 7 as the pivot after transformation by $\bE_1$ even when it is not necessary. This interchange permutation can guarantee that no multiplier is greater than 1 in absolute value during the Gaussian elimination.
More generally, the procedure for computing the LU decomposition with partial pivoting of $\bA\in\real^{n\times n}$ is given in Algorithm~\ref{alg:lu-partial-pivot}.

\begin{algorithm}[H]
	\caption{LU Decomposition with Partial Pivoting}
	\label{alg:lu-partial-pivot}
	\begin{algorithmic}[1]
		\Require
		Matrix $\bA$ with size $n\times n$;
		\State Let $\bU = \bA$;
		\For{$k=1$ to $n-1$} \Comment{i.e., get the $k$-th column of $\bU$}
		\State Find a row permutation $\bP_k$ that swaps $u_{kk}$ with the largest element in $|\bU_{k:n,k}|$;
		\State $\bU =\bP_k\bU$;
		\State Determine the Gaussian transformation $\bE_k $ to introduce zeros below the diagonal of the $k$-th column of $\bU$;
		\State $\bU = \bE_k\bU$;
		\EndFor
		\State Output $\bU$;
	\end{algorithmic}
\end{algorithm}
The algorithm requires $~2/3(n^3)$ flops and $(n-1)+(n-2)+\ldots + 1 \sim O(n^2)$ comparisons resulting from the pivoting procedure.
Upon completion, the upper triangular matrix $\bU$ is given by
$$
\bU = \bE_{n-1}\bP_{n-1} \ldots \bE_2\bP_2\bE_1\bP_1\bA.
$$

\paragraph{Computing the final $\bL$} And we here show that Algorithm~\ref{alg:lu-partial-pivot} computes the LU decomposition in the following form
$$
\bA = \bP\bL\bU,
$$
where $\bP=\bP_1 \bP_2\ldots\bP_{n-1}$, $\bU$ is the upper triangular matrix results directly from the algorithm, $\bL$ is unit lower triangular with $|\bL_{ij}|\leq 1$ for all $1 \leq i,j\leq n$. $\bL_{k+1:n,k}$ is a permuted version of $\bE_k$'s multipliers. To see this, we notice that the permutation matrices used in the algorithm fall into a special kind of permutation matrix since we only interchange two rows of the matrix. \textit{This implies the $\bP_k$'s are symmetric and $\bP_k^2 = \bI$ for $k\in \{1,2,\ldots,n-1\}$}.
Suppose
$$
\bM_k = (\bP_{n-1} \ldots \bP_{k+1}) \bE_k (\bP_{k+1} \ldots \bP_{n-1}).
$$
Then, $\bU$ can be written as
$$
\bU = \bM_{n-1}\ldots \bM_2\bM_1 \bP^\top \bA.
$$
To see what $\bM_k$ is, we realize that $\bP_{k+1}$ is a permutation with the upper left $k\times k$ block being an identity matrix. And thus we have
$$
\begin{aligned}
\bM_k &= (\bP_{n-1} \ldots \bP_{k+1}) (\bI_n-\bz_k\be_k^\top) (\bP_{k+1} \ldots \bP_{n-1})\\
&=\bI_n - (\bP_{n-1} \ldots \bP_{k+1})(\bz_k\be_k^\top)(\bP_{k+1} \ldots \bP_{n-1}) \\
&=\bI_n - (\bP_{n-1} \ldots \bP_{k+1}\bz_k) (\be_k^\top\bP_{k+1} \ldots \bP_{n-1}) \\
&=\bI_n - (\bP_{n-1} \ldots \bP_{k+1}\bz_k)\be_k^\top.  \qquad &\text{(since $\be_k^\top\bP_{k+1} \ldots \bP_{n-1} = \be_k^\top$)}
\end{aligned}
$$
This implies that $\bM_k$ is unit lower triangular with the $k$-th column being the permuted version of $\bE_k$. And the final lower triangular $\bL$ is thus given by
$$
\bL = \bM_1^{-1}\bM_2^{-1} \ldots \bM_{n-1}^{-1}.
$$

\index{Complete pivoting}
\subsection{Complete Pivoting}\label{section:complete-pivoting}
In partial pivoting, when introducing zeros below the diagonal of the $k$-th column of $\bU$, the $k$-th pivot is determined by scanning the current subcolumn $\bU_{k:n,k}$. In complete pivoting, the largest absolute entry in the current submatrix $\bU_{k:n,k:n}$ is interchanged into the entry $(k,k)$ of $\bU$. Therefore, an additional \textit{column permutation} $\bQ_k$ is applied in each step. The final upper triangular matrix $\bU$ is obtained by
$$
\bU = \bE_{n-1}\bP_{n-1}\ldots (\bE_2\bP_2(\bE_1\bP_1\bA\bQ_1)\bQ_2) \ldots \bQ_{n-1}.
$$
Similarly, the complete pivoting algorithm is formulated in Algorithm~\ref{alg:lu-complete-pivot}.
\begin{algorithm}[H]
	\caption{LU Decomposition with Complete Pivoting}
	\label{alg:lu-complete-pivot}
	\begin{algorithmic}[1]
		\Require
		Matrix $\bA$ with size $n\times n$;
		\State Let $\bU = \bA$;
		\For{$k=1$ to $n-1$} \Comment{the value $k$ is to get the $k$-th column of $\bU$}
		\State Find a row permutation matrix $\bP_k$, and a column permutation $\bQ_k$ that swaps $\bU_{kk}$ with the largest element in $|\bU_{k:n,k:n}|$, say $\bU_{u,v} = \max{|\bU_{k:n,k:n}|}$;
		\State $\bU =\bP_k\bU\bQ_k$;
		\State Determine the Gaussian transformation $\bE_k $ to introduce zeros below the diagonal of the $k$-th column of $\bU$;
		\State $\bU = \bE_k\bU$;
		\EndFor
		\State Output $\bU$;
	\end{algorithmic}
\end{algorithm}
The algorithm requires $~2/3(n^3)$ flops and $(n^2+(n-1)^2+\ldots +1^2)\sim O(n^3)$ comparisons resulting from the pivoting procedure. Again, let $\bP=\bP_1 \bP_2\ldots\bP_{n-1}$, $\bQ = \bQ_1 \bQ_2\ldots\bQ_{n-1}$,
$$
\bM_k = (\bP_{n-1} \ldots \bP_{k+1}) \bE_k (\bP_{k+1} \ldots \bP_{n-1}), \qquad \text{for all $k\in \{1,2,\ldots,n-1\}$}
$$
and
$$
\bL = \bM_1^{-1}\bM_2^{-1} \ldots \bM_{n-1}^{-1}.
$$
We have $\bA = \bP\bL\bU \bQ^\top$ or $\bP^\top\bA\bQ = \bL\bU$ as the final decomposition.

\index{Pivoting}
\index{Rook pivoting}
\subsection{Rook Pivoting}
The \textit{rook pivoting} provides an alternative to the partial and complete pivoting. Instead of choosing the larges value in $|\bU_{k:n,k:n}|$ in the $k$-th step, it searches for an element of $\bU_{k:n,k:n}$ that is \textit{maximal in both its row and column}. Apparently, the rook pivoting is not unique such that we could find many entries that satisfy the criteria. For example, for a submatrix $\bU_{k:n,k:n}$ as follows
$$
\bU_{k:n,k:n} =
\begin{bmatrix}
1 & 2 & 3 & 4 \\
2 & 3 & 7 & 3 \\
5 & 2 & 1 & 2 \\
2 & 1 & 2 & 1 \\
\end{bmatrix},
$$
where the $7$ will be chosen by complete pivoting. And one of $5, 4, 7$ will be identified as a rook pivot.

\section{Rank-Revealing LU Decomposition}\label{section:rank-reveal-lu-short}
In many applications, a factorization produced by Gaussian elimination with pivoting when $\bA$ has rank $r$ will reveal rank in the following form\index{Rank-revealing}\index{Rank-revealing LU}
$$
\bP\bA\bQ =
\begin{bmatrix}
\bL_{11} & \bzero \\
\bL_{21}^\top & \bI
\end{bmatrix}
\begin{bmatrix}
\bU_{11} & \bU_{12} \\
\bzero & \bzero
\end{bmatrix},
$$
where $\bL_{11}\in \real^{r\times r}$ and $\bU_{11}\in \real^{r\times r}$ are nonsingular, $\bL_{21}, \bU_{21}\in \real^{r\times (n-r)}$, and $\bP,\bQ$ are permutations. Gaussian elimination with rook pivoting or complete pivoting can result in such decomposition \citep{hwang1992rank, higham2002accuracy}.

\newpage
\chapter{Cholesky Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Cholesky Decomposition}
\begin{theoremHigh}[Cholesky Decomposition]\label{theorem:cholesky-factor-exist}
Every positive definite matrix $\bA\in \real^{n\times n}$ can be factored as
$$
\bA = \bR^\top\bR,
$$
where $\bR \in \real^{n\times n}$ is an upper triangular matrix \textbf{with positive diagonal elements}. This decomposition is known as the \textit{Cholesky decomposition}  of $\bA$. $\bR$ is known as the \textit{Cholesky factor} or \textit{Cholesky triangle} of $\bA$.

\item Alternatively, $\bA$ can be factored as $\bA=\bL\bL^\top$ where $\bL=\bR^\top$ is a lower triangular matrix \textit{with positive diagonals}.

\item Specifically, the Cholesky decomposition is unique (Corollary~\ref{corollary:unique-cholesky-main}, p.~\pageref{corollary:unique-cholesky-main}).
\end{theoremHigh}

The Cholesky decomposition is named after a French military officer and mathematician, Andr\'{e}-Louis Cholesky (1875-1918), who developed the Cholesky decomposition in his surveying work. Similar to the LU decomposition for solving linear systems, the Cholesky decomposition is further used primarily to solve positive definite linear systems. The development of the solution is similar to that of the LU decomposition in Section~\ref{section:lu-linear-sistem} (p.~\pageref{section:lu-linear-sistem}), and we shall not repeat the details.

\section{Existence of the Cholesky Decomposition via Recursive Calculation}

In this section, we prove the existence of the Cholesky decomposition via recursive calculation. In Section~\ref{section:cholesky-by-qr-spectral} (p.~\pageref{section:cholesky-by-qr-spectral}), we will also prove the existence of the Cholesky decomposition via the QR decomposition and spectral decomposition.
Before showing the existence of Cholesky decomposition, we need the following definitions and lemmas.
\begin{definition}[Positive Definite and Positive Semidefinite\index{Positive definite}\index{Positive semidefinite}]\label{definition:psd-pd-defini}
A matrix $\bA\in \real^{n\times n}$ is positive definite (PD) if $\bx^\top\bA\bx>0$ for all nonzero $\bx\in \real^n$.
And a matrix $\bA\in \real^{n\times n}$ is positive semidefinite (PSD) if $\bx^\top\bA\bx \geq 0$ for all $\bx\in \real^n$.
\end{definition}

One of the prerequisites for the Cholesky decomposition is the definition of the above positive definiteness of a matrix. We sketch several properties of this PD matrix as follows:
\begin{tcolorbox}[title={Positive Definite Matrix Property 1 of 5}]
We will show the equivalent definition on the positive definiteness of a matrix $\bA$ is that $\bA$ only has positive eigenvalues, or on the positive semidefiniteness of a matrix $\bA$ is that $\bA$ only has nonnegative eigenvalues. The proof is provided in Section~\ref{section:equivalent-pd-psd} (p.~\pageref{section:equivalent-pd-psd}) as a consequence of the spectral theorem.
\end{tcolorbox}

\begin{tcolorbox}[title={Positive Definite Matrix Property 2 of 5}]
\begin{lemma}[Positive Diagonals of Positive Definite Matrices]\label{lemma:positive-in-pd}
	The diagonal elements of a positive definite matrix $\bA$ are all \textbf{positive}. And similarly, the diagonal elements of a positive semidefinite matrix $\bB$ are all \textbf{nonnegative}.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:positive-in-pd}]
From the definition of positive definite matrices, we have $\bx^\top\bA \bx >0$ for all nonzero $\bx$. In particular, let $\bx=\be_i$ where $\be_i$ is the $i$-th unit vector with the $i$-th entry  equal to 1 and other entries equal to 0. Then,
$$
\be_i^\top\bA \be_i = a_{ii}>0, \qquad \forall i \in \{1, 2, \ldots, n\},
$$
where $a_{ii}$ is the $i$-th diagonal component. The proof for the second part follows similarly. This completes the proof.
\end{proof}
\begin{tcolorbox}[title={Positive Definite Matrix Property 3 of 5}]
\begin{lemma}[Schur Complement of Positive Definite Matrices\index{Schur complement}]\label{lemma:pd-of-schur}
For any positive definite matrix $\bA\in \real^{n\times n}$, its Schur complement of $a_{11}$ is given by $\bS_{n-1}=\bA_{2:n,2:n}-\frac{1}{a_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top$ which is also positive definite.

\paragraph{A word on the notation} Note that the subscript $n-1$ of $\bS_{n-1}$ means it is of size $(n-1)\times (n-1)$ and it is a Schur complement of an $n\times n$ positive definite matrix. We will use this notation in the following sections.
\end{lemma}
\end{tcolorbox}

\begin{proof}[of Lemma~\ref{lemma:pd-of-schur}]
For any nonzero vector $\bv\in \real^{n-1}$, we can construct a vector $\bx\in \real^n$ by the following equation:
$$
\bx =
\begin{bmatrix}
	-\frac{1}{a_{11}} \bA_{2:n,1}^\top  \bv \\
	\bv
\end{bmatrix},
$$
which is nonzero. Then
$$
\begin{aligned}
\bx^\top\bA\bx
&= \left[-\frac{1}{a_{11}} \bv^\top \bA_{2:n,1}\qquad \bv^\top\right]
\begin{bmatrix}
a_{11} & \bA_{2:n,1}^\top \\
\bA_{2:n,1} & \bA_{2:n,2:n}
\end{bmatrix}
\begin{bmatrix}
	-\frac{1}{a_{11}} \bA_{2:n,1}^\top  \bv \\
	\bv
\end{bmatrix} \\
&= \left[-\frac{1}{a_{11}} \bv^\top \bA_{2:n,1}\qquad \bv^\top\right]
\begin{bmatrix}
	0 \\
	\bS_{n-1}\bv
\end{bmatrix} \\
&= \bv^\top\bS_{n-1}\bv.
\end{aligned}
$$
Since $\bA$ is positive definite, we have $\bx^\top\bA\bx = \bv^\top\bS_{n-1}\bv >0$ for all nonzero $\bv$. Thus, the Schur complement $\bS_{n-1}$ is positive definite as well.
\end{proof}

The above argument can be extended to PSD matrices as well. If $\bA$ is PSD, then the Schur complement $\bS_{n-1}$ is also PSD.

\begin{mdframed}[hidealllines=\mdframehidelineNote,backgroundcolor=\mdframecolorSkip]
\paragraph{A word on the Schur complement} In the proof of Theorem~\ref{theorem:lu-factorization-with-permutation} (p.~\pageref{theorem:lu-factorization-with-permutation}), we have shown this Schur complement $\bS_{n-1}=\bA_{2:n,2:n}-\frac{1}{a_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top$ is also nonsingular if $\bA$ is nonsngular and $a_{11}\neq 0$. Similarly, the Schur complement of $a_{nn}$ in $\bA$ is $\widehat{\bS}_{n-1} =\bA_{1:n-1,1:n-1} - \frac{1}{a_{nn}}\bA_{1:n-1,n} \bA_{1:n-1,n}^\top$ which is also positive definite if $\bA$ is positive definite. This property can help prove the fact that the leading principal minors of positive definite matrices are all positive. See Section~\ref{appendix:leading-minors-pd} (p.~\pageref{appendix:leading-minors-pd}) for more details.
\end{mdframed}

\index{Recursive algorithm}
We then prove the existence of Cholesky decomposition using these lemmas.
\begin{proof}[\textbf{of Theorem~\ref{theorem:cholesky-factor-exist}: Existence of Cholesky Decomposition Recursively}]
For any positive definite matrix $\bA$, we can write out (since $a_{11}$ is positive by Lemma~\ref{lemma:positive-in-pd})
$$
\begin{aligned}
\bA &=
\begin{bmatrix}
a_{11} & \bA_{2:n,1}^\top \\
\bA_{2:n,1} & \bA_{2:n,2:n}
\end{bmatrix} \\
&=\begin{bmatrix}
\sqrt{a_{11}} &\bzero\\
\frac{1}{\sqrt{a_{11}}} \bA_{2:n,1} &\bI
\end{bmatrix}
\begin{bmatrix}
\sqrt{a_{11}} & \frac{1}{\sqrt{a_{11}}}\bA_{2:n,1}^\top \\
\bzero & \bA_{2:n,2:n}-\frac{1}{a_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top
\end{bmatrix}\\
&=\begin{bmatrix}
	\sqrt{a_{11}} &\bzero\\
	\frac{1}{\sqrt{a_{11}}} \bA_{2:n,1} &\bI
\end{bmatrix}
\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bA_{2:n,2:n}-\frac{1}{a_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top
\end{bmatrix}
\begin{bmatrix}
	\sqrt{a_{11}} & \frac{1}{\sqrt{a_{11}}}\bA_{2:n,1}^\top \\
	\bzero & \bI
\end{bmatrix}\\
&=\bR_1^\top
\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bS_{n-1}
\end{bmatrix}
\bR_1.
\end{aligned}
$$
where
$$\bR_1 =
\begin{bmatrix}
	\sqrt{a_{11}} & \frac{1}{\sqrt{a_{11}}}\bA_{2:n,1}^\top \\
	\bzero & \bI
\end{bmatrix}.
$$
Since we proved the Schur complement $\bS_{n-1}$ is positive definite in Lemma~\ref{lemma:pd-of-schur}, then we can factor it in the same way as
$$
\bS_{n-1}=
\hat{\bR}_2^\top
\begin{bmatrix}
	1 & \bzero \\
	\bzero & \bS_{n-2}
\end{bmatrix}
\hat{\bR}_2.
$$
Therefore, we have
$$
\begin{aligned}
\bA &= \bR_1^\top
\begin{bmatrix}
	1 & \bzero \\
	\bzero & \hat{\bR}_2^\top
	\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bS_{n-2}
	\end{bmatrix}
	\hat{\bR}_2.
\end{bmatrix}
\bR_1\\
&=
\bR_1^\top
\begin{bmatrix}
1 &\bzero \\
\bzero &\hat{\bR}_2^\top
\end{bmatrix}
\begin{bmatrix}
	1 &\bzero \\
	\bzero &\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bS_{n-2}
	\end{bmatrix}
\end{bmatrix}
\begin{bmatrix}
	1 &\bzero \\
	\bzero &\hat{\bR}_2
\end{bmatrix}
\bR_1\\
&=
\bR_1^\top \bR_2^\top
\begin{bmatrix}
	1 &\bzero \\
	\bzero &\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bS_{n-2}
	\end{bmatrix}
\end{bmatrix}
\bR_2 \bR_1.
\end{aligned}
$$
The same formula can be recursively applied. This process gradually continues down to the bottom-right corner giving us the decomposition
$$
\begin{aligned}
\bA &= \bR_1^\top\bR_2^\top\ldots \bR_n^\top \bR_n\ldots \bR_2\bR_1\\
&= \bR^\top \bR,
\end{aligned}
$$
where $\bR_1, \bR_2, \ldots, \bR_n$ are upper triangular matrices with positive diagonal elements and $\bR=\bR_1\bR_2\ldots\bR_n$ is also an upper triangular matrix with positive diagonal elements from which the result follows.
\end{proof}
The process in the proof can also be used to calculate the Cholesky decomposition and compute the complexity of the algorithm.
\begin{lemma}[$\bR^\top\bR$ is PD]\label{lemma:r-to-pd}\index{Positive definite}
Given any upper triangular matrix $\bR$ with positive diagonal elements, then
$
\bA = \bR^\top\bR
$
is positive definite.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:r-to-pd}]
If an upper triangular matrix $\bR$ has positive diagonals, it has full column rank, and the null space of $\bR$ is of dimension 0 by the fundamental theorem of linear algebra (Theorem~\ref{theorem:fundamental-linear-algebra}, p.~\pageref{theorem:fundamental-linear-algebra}). As a result, $\bR\bx \neq \bzero$ for any nonzero vector $\bx$. Thus $\bx^\top\bA\bx = ||\bR\bx||^2 >0$ for any nonzero vector $\bx$.
\end{proof}
This lemma above works not only for the upper triangular matrices $\bR$, but can be extended to any $\bR$ with linearly independent columns.

\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolorSkip]
\paragraph{A word on the two claims} Combine Theorem~\ref{theorem:cholesky-factor-exist} and Lemma~\ref{lemma:r-to-pd}, we can claim that matrix $\bA$ is positive definite if and only if $\bA$ can be factored as $\bA=\bR^\top\bR$ where $\bR$ is an upper triangular matrix with positive diagonals.
\end{mdframed}

\section{Sylvester's Criterion: Leading Principal Minors of PD Matrices}\label{appendix:leading-minors-pd}

In Lemma~\ref{lemma:pd-of-schur}, we proved for any positive definite matrix $\bA\in \real^{n\times n}$, its Schur complement of $a_{11}$ is $\bS_{n-1}=\bA_{2:n,2:n}-\frac{1}{a_{11}} \bA_{2:n,1}\bA_{2:n,1}^\top$ and it is also positive definite.
This is also true for its Schur complement of $a_{nn}$, i.e., $\bS_{n-1}^\prime = \bA_{1:n-1,1:n-1} -\frac{1}{a_{nn}} \bA_{1:n-1,n}\bA_{1:n-1,n}^\top$ is also positive definite.

We then claim that all the leading principal minors (Definition~\ref{definition:leading-principle-minors}, p.~\pageref{definition:leading-principle-minors}) of a positive definite matrix $\bA \in \real^{n\times n}$ are positive. This is also known as Sylvester's criterion \citep{swamy1973sylvester, gilbert1991positive}. Recall that these positive leading principal minors imply the existence of the LU decomposition for positive definite matrix $\bA$ by Theorem~\ref{theorem:lu-factorization-without-permutation} (p.~\pageref{theorem:lu-factorization-without-permutation}).

To show Sylvester's criterion, we need the following lemma.
\begin{tcolorbox}[title={Positive Definite Matrix Property 4 of 5}]
\begin{lemma}[Quadratic PD]\label{lemma:quadratic-pd}
Let $\bE$ be any invertible matrix. Then $\bA$ is positive definite if and only if $\bE^\top\bA\bE$ is also positive definite.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:quadratic-pd}]
	We will prove by forward implication and reverse implication separately as follows.
\paragraph{Forward implication}
Suppose $\bA$ is positive definite, then for any nonzero vector $\bx$, we have $\bx^\top \bE^\top\bA\bE \bx = \by^\top\bA\by > 0$, since $\bE$ is invertible such that $\bE\bx$ is nonzero. \footnote{Since the null space of $\bE$ is of dimension 0 and the only solution for $\bE\bx=\bzero$ is the trivial solution $\bx=0$.} This implies $\bE^\top\bA\bE$ is PD.

\paragraph{Reverse implication}
Conversely, suppose $\bE^\top\bA\bE$ is positive definite, for any nonzero $\bx$, we have $\bx^\top \bE^\top\bA\bE \bx>0$. For any nonzero $\by$, there exists a nonzero $\bx$ such that $\by =\bE\bx$ since $\bE$ is invertible. This implies $\bA$ is PD as well.
\end{proof}

We then provide a rigorous proof for Sylvester's criterion.
\begin{tcolorbox}[title={Positive Definite Matrix Property 5 of 5}]
\begin{theorem}[Sylvester's Criterion\index{Sylvester's criterion}]\label{lemma:sylvester-criterion}
The real symmetric matrix $\bA\in \real^{n\times n}$ is positive definite if and only if all the leading principal minors of $\bA$ are positive.
\end{theorem}
\end{tcolorbox}

\begin{proof}[of Theorem~\ref{lemma:sylvester-criterion}]
We will prove by forward implication and reverse implication separately as follows.
\paragraph{Forward implication: } We will prove by induction for the forward implication. Suppose $\bA$ is positive definite. Since all the components on the diagonal of positive definite matrices are all positive (Lemma~\ref{lemma:positive-in-pd}, p.~\pageref{lemma:positive-in-pd}). The case for $n=1$ is trivial that $\det(\bA)> 0$ if $\bA$ is a scalar.

	Suppose all the leading principal minors for $k\times k$ matrices are all positive. If we could prove this is also true for $(k+1)\times (k+1)$ PD  matrices, then we complete the proof.

	For a $(k+1)\times (k+1)$ matrix with the block form $\bM=\begin{bmatrix}
		\bA & \bb\\
		\bb^\top & d
	\end{bmatrix}$, where $\bA$ is a $k\times k$ submatrix. Then its Schur complement of $d$, $\bS_{k} = \bA - \frac{1}{d} \bb\bb^\top$ is also positive definite and its determinant is positive from the assumption. Therefore, $\det(\bM) = \det(d)\det( \bA - \frac{1}{d} \bb\bb^\top) $=
	\footnote{By the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
			\bA & \bB \\
			\bC & \bD
		\end{bmatrix}$, then $\det(\bM) = \det(\bD)\det(\bA-\bB\bD^{-1}\bC)$.}
	$d\cdot \det( \bA - \frac{1}{d} \bb\bb^\top)>0$, which completes the proof.

\paragraph{Reverse implication:} Conversely, suppose all the leading principal minors of $\bA\in \real^{n\times n}$ are positive, i.e., leading principal submatrices are nonsingular. Suppose further the $(i,j)$-th entry of $\bA$ is denoted by $a_{ij}$, we realize that $a_{11}>0$ by the assumption.
Subtract multiples of the first row of $\bA$ to zero out the entries in the first column of $\bA$ below the first diagonal $a_{11}$. That is,
$$
\bA =
\begin{bmatrix}
a_{11} & a_{12} & \ldots &a_{1n}\\
a_{21} & a_{22} & \ldots &a_{2n}\\
\vdots & \vdots & \ddots &\vdots\\
a_{n1} & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}
\stackrel{\bE_1 \bA}{\longrightarrow}
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	0 & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	0 & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}.
$$
This operation preserves the values of the principal minors of $\bA$. The $\bE_1$ might be mysterious to the readers. Actually, the $\bE_1$ contains two steps $\bE_1 = \bZ_{12}\bZ_{11}$. The first step $\bZ_{11}$ is to subtract the 2-nd row to the $n$-th row by a multiple of the first row, that is
$$
\begin{aligned}
\bA =
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	a_{21} & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	a_{n1} & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}
&\stackrel{\bZ_{11} \bA}{\longrightarrow}
\begin{bmatrix}
	1 & 0 & \ldots &0\\
	-\frac{a_{21}}{a_{11}} & 1 & \ldots &0\\
	\vdots & \vdots & \ddots &\vdots\\
	-\frac{a_{n1}}{a_{11}} & 0 & \ldots &1\\
\end{bmatrix}
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	a_{21} & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	a_{n1} & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}\\
&=
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	0 & (a_{22} -\frac{a_{21}}{a_{11}} a_{12})  & \ldots &(a_{2n}-\frac{a_{21}}{a_{11}} a_{1n})\\
	\vdots & \vdots & \ddots &\vdots\\
	0 & (a_{n2}-\frac{a_{n1}}{a_{11}}a_{12}) & \ldots &(a_{nn}-\frac{a_{n1}}{a_{11}}a_{1n})\\
\end{bmatrix},
\end{aligned}
$$
where we subtract the bottom-right $(n-1)\times (n-1)$ by some terms additionally. $\bZ_{12}$ is to add back these terms.
$$
\begin{aligned}
\bZ_{11}\bA &=
\begin{bmatrix}
a_{11} & a_{12} & \ldots &a_{1n}\\
0 & (a_{22} -\frac{a_{21}}{a_{11}} a_{12})  & \ldots &(a_{2n}-\frac{a_{21}}{a_{11}} a_{1n})\\
\vdots & \vdots & \ddots &\vdots\\
0 & (a_{n2}-\frac{a_{n1}}{a_{11}}a_{12}) & \ldots &(a_{nn}-\frac{a_{n1}}{a_{11}}a_{1n})\\
\end{bmatrix}\\
&\stackrel{\bZ_{12}(\bZ_{11} \bA)}{\longrightarrow}
\begin{bmatrix}
1 & 0 & \ldots &0\\
\frac{a_{21}}{a_{11}} & 1 & \ldots &0\\
\vdots & \vdots & \ddots &\vdots\\
\frac{a_{n1}}{a_{11}} & 0 & \ldots &1\\
\end{bmatrix}
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	0 & (a_{22} -\frac{a_{21}}{a_{11}} a_{12})  & \ldots &(a_{2n}-\frac{a_{21}}{a_{11}} a_{1n})\\
	\vdots & \vdots & \ddots &\vdots\\
	0 & (a_{n2}-\frac{a_{n1}}{a_{11}}a_{12}) & \ldots &(a_{nn}-\frac{a_{n1}}{a_{11}}a_{1n})\\
\end{bmatrix}\\
&=\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	0 & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	0 & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}=\bE_1\bA.
\end{aligned}
$$

Now subtract multiples of the first column of $\bE_1\bA$, from the other columns of $\bE_1\bA$ to zero out the entries in the first row of $\bE_1\bA$ to the right of the first column. Since $\bA$ is symmetric, we can multiply on the right by $\bE_1^\top$ to get what we want. We then have
$$
\bA =
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	a_{21} & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	a_{n1} & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}
\stackrel{\bE_1 \bA}{\longrightarrow}
\begin{bmatrix}
	a_{11} & a_{12} & \ldots &a_{1n}\\
	0 & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	0 & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}
\stackrel{\bE_1 \bA\bE_1^\top}{\longrightarrow}
\begin{bmatrix}
	a_{11} & 0 & \ldots &0\\
	0 & a_{22} & \ldots &a_{2n}\\
	\vdots & \vdots & \ddots &\vdots\\
	0 & a_{n2} & \ldots &a_{nn}\\
\end{bmatrix}.
$$
This operation also preserves the values of the principal minors of $\bA$. The leading principal minors of $\bE_1 \bA\bE_1^\top$ are exactly the same as those of $\bA$.

Continue this process, we will transform $\bA$ into a diagonal matrix $\bE_n \ldots \bE_1 \bA\bE_1^\top\ldots\bE_n^\top$ whose diagonal values are exactly the same as the diagonals of $\bA$ and are positive. Let $\bE = \bE_n \ldots \bE_1$, which is an invertible matrix. Apparently, $\bE \bA \bE^\top$ is PD, which implies $\bA$ is PD as well from Lemma~\ref{lemma:quadratic-pd}.
\end{proof}

\section{Existence of the Cholesky Decomposition via the LU Decomposition without Permutation}
By Theorem~\ref{lemma:sylvester-criterion} on Sylvester's criterion and the existence of LU decomposition without permutation in Theorem~\ref{theorem:lu-factorization-without-permutation} (p.~\pageref{theorem:lu-factorization-without-permutation}), there is a unique LU decomposition for positive definite matrix $\bA=\bL\bU_0$ where $\bL$ is a unit lower triangular matrix and $\bU_0$ is an upper triangular matrix. Since \textit{the signs of the pivots of a symmetric matrix are the same as the signs of the eigenvalues} \citep{strang1993introduction}: \index{Pivot}
$$
\text{number of positive pivots = number of positive eigenvalues. }
$$
And $\bA = \bL\bU_0$ has the following form
$$
\begin{aligned}
\bA = \bL\bU_0 &=
\begin{bmatrix}
	1 & 0 & \ldots & 0 \\
	l_{21} & 1 & \ldots & 0\\
	\vdots & \vdots & \ddots & \vdots\\
	l_{n1} & l_{n2} & \ldots & 1
\end{bmatrix}
\begin{bmatrix}
	u_{11} & u_{12} & \ldots & u_{1n} \\
	0 & u_{22} & \ldots & u_{2n}\\
	\vdots & \vdots & \ddots & \vdots\\
	0 & 0 & \ldots & u_{nn}
\end{bmatrix}.\\
\end{aligned}
$$
This implies that the diagonals of $\bU_0$ contain the pivots of $\bA$. And all the eigenvalues of PD matrices are positive (see Lemma~\ref{lemma:eigens-of-PD-psd}, p.~\pageref{lemma:eigens-of-PD-psd}, which is a consequence of the spectral decomposition). Thus the diagonals of $\bU_0$ are positive.

Taking the diagonal of $\bU_0$ out into a diagonal matrix $\bD$, we can rewrite $\bU_0=\bD\bU$ as shown in the following equation
$$
\begin{aligned}
	\bA = \bL\bU_0 =
	\begin{bmatrix}
		1 & 0 & \ldots & 0 \\
		l_{21} & 1 & \ldots & 0\\
		\vdots & \vdots & \ddots & \vdots\\
		l_{n1} & l_{n2} & \ldots & 1
	\end{bmatrix}
	\begin{bmatrix}
		u_{11} & 0 & \ldots & 0 \\
		0 & u_{22} & \ldots & 0\\
		\vdots & \vdots & \ddots & \vdots\\
		0 & 0 & \ldots & u_{nn}
	\end{bmatrix}
	\begin{bmatrix}
		1 & u_{12}/u_{11} & \ldots & u_{1n}/u_{11} \\
		0 & 1 & \ldots & u_{2n}/u_{22}\\
		\vdots & \vdots & \ddots & \vdots\\
		0 & 0 & \ldots & 1
	\end{bmatrix}=\bL\bD\bU,
\end{aligned}
$$
 where $\bU$ is a \textit{unit} upper triangular matrix.
 By the uniqueness of the LU decomposition without permutation in Corollary~\ref{corollary:unique-lu-without-permutation} (p.~\pageref{corollary:unique-lu-without-permutation}) and the symmetry of $\bA$, it follows that $\bU = \bL^\top$, and $\bA = \bL\bD\bL^\top$. Since the diagonals of $\bD$ are positive, we can set $\bR = \bD^{1/2}\bL^\top$ where $\bD^{1/2}=\diag(\sqrt{u_{11}}, \sqrt{u_{22}}, \ldots, \sqrt{u_{nn}})$ such that $\bA = \bR^\top\bR$ is the Cholesky decomposition of $\bA$, and $\bR$ is upper triangular with positive diagonals.

\index{Upper triangular}
\subsection{Diagonal Values of the Upper Triangular Matrix}\label{section:cholesky-diagonals}

Suppose $\bA$ is a PD matrix, take $\bA$ as a block matrix $\bA = \begin{bmatrix}
\bA_{k} & \bA_{12} \\
\bA_{21} & \bA_{22}
\end{bmatrix}$ where $\bA_{k}\in \real^{k\times k}$, and its block LU decomposition is given by
$$
\begin{aligned}
\bA &= \begin{bmatrix}
	\bA_{k} & \bA_{12} \\
	\bA_{21} & \bA_{22}
\end{bmatrix}
=\bL\bU_0=
\begin{bmatrix}
	\bL_{k} & \bzero \\
	\bL_{21} & \bL_{22}
\end{bmatrix}
\begin{bmatrix}
	\bU_{k} & \bU_{12} \\
	\bzero & \bU_{22}
\end{bmatrix} \\
&=\begin{bmatrix}
	\bL_{k}\bU_{k} & \bL_{k}\bU_{12} \\
	\bL_{21}\bU_{k11}  & \bL_{21}\bU_{12}+\bL_{22}\bU_{22}
\end{bmatrix}.
\end{aligned}
$$
Then the leading principal minor (Definition~\ref{definition:leading-principle-minors}, p.~\pageref{definition:leading-principle-minors}), $\Delta_k=\det(\bA_{1:k,1:k} ) = \det(\bA_{k})$  is given by
$$
\Delta_k = \det(\bA_{k}) = \det(\bL_{k}\bU_{k} ) = \det(\bL_{k} )\det(\bU_{k}).
$$
We notice that $\bL_{k}$ is a unit lower triangular matrix and $\bU_{k}$ is an upper triangular matrix. By the fact that the determinant of a lower triangular matrix (or an upper triangular matrix) is the product of the diagonal entries, we obtain
$$
\Delta_k = \det(\bU_{k})= u_{11} u_{22}\ldots u_{kk},
$$
i.e., the $k$-th leading principal minor of $\bA$ is the determinant of the $k\times k$ submatrix of $\bU_0$. That is also the product of the first $k$ diagonals of $\bD$ ($\bD$ is the matrix from $\bA = \bL\bD\bL^\top$). Let $\bD = \diag(d_1, d_2, \ldots, d_n)$, therefore, we have
$$
\Delta_k = d_1 d_2\ldots d_k = \Delta_{k-1}d_k.
$$
This gives us an alternative form of $\bD$, i.e., the \textbf{squared} diagonal values of $\bR$ ($\bR$ is the Cholesky factor from $\bA=\bR^\top\bR$), and it is given by
$$
\bD = \diag\left(\Delta_1, \frac{\Delta_2}{\Delta_1}, \ldots, \frac{\Delta_n}{\Delta_{n-1}}\right),
$$
where $\Delta_k$ is the $k$-th leading principal minor of $\bA$, for all $k\in \{1,2,\ldots, n\}$. That is, the diagonal values of $\bR$ are given by
$$
\diag\left(\sqrt{\Delta_1}, \sqrt{\frac{\Delta_2}{\Delta_1}}, \ldots, \sqrt{\frac{\Delta_n}{\Delta_{n-1}}}\right).
$$

\subsection{Block Cholesky Decomposition}
Following from the last section, suppose $\bA$ is a PD matrix, take $\bA$ as a block matrix $\bA = \begin{bmatrix}
	\bA_k & \bA_{12} \\
	\bA_{21} & \bA_{22}
\end{bmatrix}$ where $\bA_k\in \real^{k\times k}$, and its block LU decomposition is given by
$$
\begin{aligned}
	\bA &= \begin{bmatrix}
		\bA_k & \bA_{12} \\
		\bA_{21} & \bA_{22}
	\end{bmatrix}
	=\bL\bU_0=
	\begin{bmatrix}
		\bL_k & \bzero \\
		\bL_{21} & \bL_{22}
	\end{bmatrix}
	\begin{bmatrix}
		\bU_k & \bU_{12} \\
		\bzero & \bU_{22}
	\end{bmatrix} \\
	&=\begin{bmatrix}
		\bL_k\bU_k & \bL_{k}\bU_{12} \\
		\bL_{21}\bU_{k}  & \bL_{21}\bU_{12}+\bL_{22}\bU_{22}
	\end{bmatrix}.\\
\end{aligned}
$$
where the $k$-th leading principal submatrix $\bA_k$ of $\bA$ also has its LU decomposition,  $\bA_k=\bL_k\bU_k$. Then, it is trivial that the Cholesky decomposition of an $n\times n$ matrix contains $n-1$ other Cholesky decompositions within it: $\bA_k = \bR_k^\top\bR_k$, for all $k\in \{1,2,\ldots, n-1\}$. This is particularly true that any leading principal submatrix $\bA_k$ of the positive definite matrix $\bA$ is also positive definite. This can be shown that for positive definite matrix $\bA_{k+1} \in \real^{(k+1)\times(k+1)}$, and any nonzero vector $\bx_k\in\real^k$ appended by a zero element $\bx_{k+1} = \begin{bmatrix}
	\bx_k\\
	0
\end{bmatrix}$. It follows that
$$
\bx_k^\top\bA_k\bx_k = \bx_{k+1}^\top\bA_{k+1}\bx_{k+1} >0,
$$
and $\bA_k$ is positive definite. If we start from $\bA\in \real^{n\times n}$, we will recursively get that $\bA_{n-1}$ is PD, $\bA_{n-2}$ is PD, $\ldots$. And all of them admit a Cholesky decomposition.

\section{Existence of the Cholesky Decomposition via Induction}
In the last section, we proved the existence of the Cholesky decomposition via the LU decomposition without permutation. Following the proof of the LU decomposition in Section~\ref{section:exist-lu-without-perm} (p.~\pageref{section:exist-lu-without-perm}), we realize that the existence of Cholesky decomposition can be a direct consequence of induction as well.
\begin{proof}[\textbf{of Theorem~\ref{theorem:cholesky-factor-exist}: Existence of Cholesky Decomposition by Induction\index{Induction}}]
We will prove by induction that every $n\times n$ positive definite matrix $\bA$ has a decomposition $\bA=\bR^\top\bR$. The $1\times 1$ case is trivial by setting $R=\sqrt{A}$, thus, $A=R^2$.

Suppose for any $k\times k$ PD matrix $\bA_k$ has a Cholesky decomposition. If we prove any $(k+1)\times(k+1)$ PD matrix $\bA_{k+1}$ can also be factored as this Cholesky decomposition, then we complete the proof.

For any $(k+1)\times(k+1)$ PD matrix $\bA_{k+1}$, Write out $\bA_{k+1}$ as
$$
\bA_{k+1} = \begin{bmatrix}
	\bA_k & \bb \\
	\bb^\top & d
\end{bmatrix}.
$$
We note that $\bA_k$ is PD from the last section.
By the inductive hypothesis, it admits a Cholesky decomposition $\bA_k = \bR_k^\top\bR_k$. We can construct the upper triangular matrix
$$
\bR_{k+1}=\begin{bmatrix}
\bR_k & \br\\
0 & s
\end{bmatrix},
$$
such that it follows that
$$
\bR_{k+1}^\top\bR_{k+1} =
\begin{bmatrix}
\bR_k^\top\bR_k & \bR_k^\top \br\\
\br^\top \bR_k & \br^\top\br+s^2
\end{bmatrix}.
$$
Therefore, if we can prove $\bR_{k+1}^\top \bR_{k+1} = \bA_{k+1}$ is the Cholesky decomposition of $\bA_{k+1}$ (which requires the value $s$ to be positive), then we complete the proof. That is, we need to prove
$$
\begin{aligned}
\bb &= \bR_k^\top \br, \\
d &= \br^\top\br+s^2.
\end{aligned}
$$
Since $\bR_k$ is nonsingular, we have a unique solution for $\br$ and $s$ that
$$
\begin{aligned}
\br &= \bR_k^{-\top}\bb, \\
s &= \sqrt{d - \br^\top\br} = \sqrt{d - \bb^\top\bA_k^{-1}\bb},
\end{aligned}
$$
since we assume $s$ is nonnegative. However, we need to further prove that $s$ is not only nonnegative, but also positive. Since $\bA_k$ is PD, from Sylvester's criterion, and the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
	\bA & \bB \\
	\bC & \bD
\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$. We have
$$
\det(\bA_{k+1}) = \det(\bA_k)\det(d- \bb^\top\bA_k^{-1}\bb) =  \det(\bA_k)(d- \bb^\top\bA_k^{-1}\bb)>0.
$$
Since $ \det(\bA_k)>0$, we then obtain that $(d- \bb^\top\bA_k^{-1}\bb)>0$ and this implies $s>0$.
We complete the proof.
\end{proof}

\section{Uniqueness of the Cholesky Decomposition}
\begin{corollary}[Uniqueness of Cholesky Decomposition\index{Uniqueness}]\label{corollary:unique-cholesky-main}
The Cholesky decomposition $\bA=\bR^\top\bR$ for any positive definite matrix $\bA\in \real^{n\times n}$ is unique.
\end{corollary}
The uniqueness of the Cholesky decomposition can be an immediate consequence of the uniqueness of the LU decomposition without permutation. Or, an alternative rigorous proof is provided as follows.
\begin{proof}[of Corollary~\ref{corollary:unique-cholesky-main}]
Suppose the Cholesky decomposition is not unique, then we can find two decompositions such that $\bA=\bR_1^\top\bR_1 = \bR_2^\top\bR_2$ which implies
$$
\bR_1\bR_2^{-1}= \bR_1^{-\top} \bR_2^\top.
$$
From the fact that the inverse of an upper triangular matrix is also an upper triangular matrix, and the product of two upper triangular matrices is also an upper triangular matrix, \footnote{Same for lower triangular matrices: the inverse of a lower triangular matrix is also a lower triangular matrix, and the product of two lower triangular matrices is also a lower triangular matrix.} we realize that the left-side of the above equation is an upper triangular matrix and the right-side of it is a lower triangular matrix. This implies $\bR_1\bR_2^{-1}= \bR_1^{-\top} \bR_2^\top$ is a diagonal matrix, and $\bR_1^{-\top} \bR_2^\top= (\bR_1^{-\top} \bR_2^\top)^\top = \bR_2\bR_1^{-1}$.
Let $\bLambda = \bR_1\bR_2^{-1}= \bR_2\bR_1^{-1}$ be the diagonal matrix. We notice that the diagonal value of $\bLambda$ is the product of the corresponding diagonal values of $\bR_1$ and $\bR_2^{-1}$ (or $\bR_2$ and $\bR_1^{-1}$). That is, for
$$
\bR_1=\begin{bmatrix}
	r_{11} & r_{12} & \ldots & r_{1n} \\
	0 & r_{22} & \ldots & r_{2n}\\
	\vdots & \vdots & \ddots & \vdots\\
	0 & 0 & \ldots & r_{nn}
\end{bmatrix},
\qquad
\bR_2=
\begin{bmatrix}
	s_{11} & s_{12} & \ldots & s_{1n} \\
	0 & s_{22} & \ldots & s_{2n}\\
	\vdots & \vdots & \ddots & \vdots\\
	0 & 0 & \ldots & s_{nn}
\end{bmatrix},
$$
we have,
$$
\begin{aligned}
\bR_1\bR_2^{-1}=
\begin{bmatrix}
	\frac{r_{11}}{s_{11}} & 0 & \ldots & 0 \\
	0 & \frac{r_{22}}{s_{22}} & \ldots & 0\\
	\vdots & \vdots & \ddots & \vdots\\
	0 & 0 & \ldots & \frac{r_{nn}}{s_{nn}}
\end{bmatrix}
=
\begin{bmatrix}
	\frac{s_{11}}{r_{11}} & 0 & \ldots & 0 \\
	0 & \frac{s_{22}}{r_{22}} & \ldots & 0\\
	\vdots & \vdots & \ddots & \vdots\\
	0 & 0 & \ldots & \frac{s_{nn}}{r_{nn}}
\end{bmatrix}
=\bR_2\bR_1^{-1}.
\end{aligned}
$$
Since both $\bR_1$ and $\bR_2$ have positive diagonals, this implies $r_{11}=s_{11}, r_{22}=s_{22}, \ldots, r_{nn}=s_{nn}$. And $\bLambda = \bR_1\bR_2^{-1}= \bR_2\bR_1^{-1}  =\bI$.
That is, $\bR_1=\bR_2$ and this leads to a contradiction. The Cholesky decomposition is thus unique.
\end{proof}

\section{Last Words on Positive Definite Matrices}
In Section~\ref{section:equivalent-pd-psd} (p.~\pageref{section:equivalent-pd-psd}), we will prove that a matrix $\bA$ is PD if and only if $\bA$ can be factored as $\bA=\bP^\top\bP$ where $\bP$ is nonsingular. And in Section~\ref{section:unique-posere-pd} (p.~\pageref{section:unique-posere-pd}), we will prove that PD matrix $\bA$ can be uniquely factored as $\bA =\bB^2$ where $\bB$ is also PD. The two results are both consequences of the spectral decomposition of PD matrices.

To conclude, for PD matrix $\bA$, we can factor it into $\bA=\bR^\top\bR$ where $\bR$ is an upper triangular matrix with positive diagonals as shown in Theorem~\ref{theorem:cholesky-factor-exist} by Cholesky decomposition, $\bA = \bP^\top\bP$ where $\bP$ is nonsingular in Theorem~\ref{lemma:nonsingular-factor-of-PD} (p.~\pageref{lemma:nonsingular-factor-of-PD}), and $\bA = \bB^2$ where $\bB$ is PD in Theorem~\ref{theorem:unique-factor-pd} (p.~\pageref{theorem:unique-factor-pd}). For clarity, the different factorizations of positive definite matrix $\bA$ are summarized in Figure~\ref{fig:pd-summary}.

\begin{figure}[htbp]
	\centering
	\begin{widepage}
		\centering
		\resizebox{0.75\textwidth}{!}{%
			\begin{tikzpicture}[>=latex]

				\tikzstyle{state} = [draw, very thick, fill=white, rectangle, minimum height=3em, minimum width=6em, node distance=8em, font={\sffamily\bfseries}]
				\tikzstyle{stateEdgePortion} = [black,thick];
				\tikzstyle{stateEdge} = [stateEdgePortion,->];
				\tikzstyle{stateEdge2} = [stateEdgePortion,<->];
				\tikzstyle{edgeLabel} = [pos=0.5, text centered, font={\sffamily\small}];

				\node[ellipse, name=pdmatrix, draw,font={\sffamily\bfseries},  node distance=7em, xshift=-9em, yshift=-1em,fill={colorals}]  {PD Matrix $\bA$};
				\node[state, name=bsqure, below of=pdmatrix, xshift=0em, yshift=1em, fill={colorlu}] {$\bB^2$};
				\node[state, name=lq, right of=bsqure, xshift=3em, fill={colorlu}] {$\bP^\top\bP$};
				\node[state, name=rsqure, left of=bsqure, xshift=-3em, fill={colorlu}] {$\bR^\top\bR$};
				\node[ellipse, name=utv, below of=pdmatrix,draw,  node distance=7em, xshift=0em, yshift=-4em,font={\tiny},fill={coloruppermiddle}]  {PD $\bB$};
				\node[ellipse, name=upperr, left of=utv, draw, node distance=8em, xshift=-3em,font={\tiny},fill={coloruppermiddle}]  {\parbox{6em}{Upper \\Triangular $\bR$}};
				\node[ellipse, name=nonp, right of=utv,draw,  node distance=8em, xshift=3em, font={\tiny},fill={coloruppermiddle}]  {Nonsingular $\bP$};

				\coordinate (lq2inter3) at ($(pdmatrix.east -| lq.north) + (-0em,0em)$);
				\draw (pdmatrix.east) edge[stateEdgePortion] (lq2inter3);
				\draw (lq2inter3) edge[stateEdge]
				node[edgeLabel, text width=7.25em, yshift=0.8em]{\parbox{5em}{Spectral\\Decomposition}} (lq.north);

				\coordinate (rqr2inter1) at ($(pdmatrix.west) + (0,0em)$);
				\coordinate (rqr2inter3) at ($(rqr2inter1-| rsqure.north) + (-0em,0em)$);
				\draw (rqr2inter1) edge[stateEdgePortion] (rqr2inter3);
				\draw (rqr2inter3) edge[stateEdge]
				node[edgeLabel, text width=8em, yshift=0.8em]{\parbox{2em}{LU/\\Spectral/\\Recursive}} (rsqure.north);

				\draw (pdmatrix.south)
				edge[stateEdge] node[edgeLabel,yshift=0.5em]{\parbox{5em}{Spectral\\Decomposition} }
				(bsqure.north);

				\draw (upperr.north)
				edge[stateEdge] node[edgeLabel,yshift=0.5em]{}
				(rsqure.south);

				\draw (utv.north)
				edge[stateEdge] node[edgeLabel,yshift=0.5em]{}
				(bsqure.south);

				\draw (nonp.north)
				edge[stateEdge] node[edgeLabel,yshift=0.5em]{}
				(lq.south);

			\end{tikzpicture}
		}
	\end{widepage}
	\caption{Demonstration of different factorizations on positive definite matrix $\bA$.}
	\label{fig:pd-summary}
\end{figure}

\section{Decomposition for Semidefinite Matrices}
For positive semidefinite matrices, the Cholesky decomposition also exists with slight modification.
	\begin{theoremHigh}[Semidefinite Decomposition\index{Positive semidefinite}]\label{theorem:semidefinite-factor-exist}
		Every positive semidefinite matrix $\bA\in \real^{n\times n}$ can be factored as
		$$
		\bA = \bR^\top\bR,
		$$
		where $\bR \in \real^{n\times n}$ is an upper triangular matrix with possible \textbf{zero} diagonal elements.
	\end{theoremHigh}
For such decomposition, the diagonal of $\bR$ may not display the rank of $\bA$ \citep{higham2009cholesky}.
More generally, a rank-revealing decomposition for semidefinite decomposition is provided as follows.\index{Rank-revealing}\index{Semidefinite rank-revealing}
	\begin{theoremHigh}[Semidefinite Rank-Revealing Decomposition\index{Rank-revealing}]\label{theorem:semidefinite-factor-rank-reveal}
		Every positive semidefinite matrix $\bA\in \real^{n\times n}$ with rank $r$ can be factored as
		$$
		\bP^\top \bA\bP  = \bR^\top\bR, \qquad \mathrm{with} \qquad
		\bR = \begin{bmatrix}
			\bR_{11} & \bR_{12}\\
			\bzero &\bzero
		\end{bmatrix} \in \real^{n\times n},
		$$
		where $\bR_{11} \in \real^{r\times r}$ is an upper triangular matrix with positive diagonal elements, and $\bR_{12}\in \real^{r\times (n-r)}$.
	\end{theoremHigh}
The proof for the existence of the above rank-revealing decomposition for semidefinite matrices is delayed in Section~\ref{section:semi-rank-reveal-proof} (p.~\pageref{section:semi-rank-reveal-proof}) as a consequence of the spectral decomposition (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}) and the column-pivoted QR decomposition (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}).

\section{Application: Rank-One Update/Downdate}\label{section:cholesky-rank-one-update}
Updating linear systems after low-rank modifications of the system matrix is widespread in machine learning, statistics, and many other fields. However, it is well known that this update can lead to serious instabilities in the presence
of round-off error \citep{seeger2004low}. If the system matrix is positive definite, it is almost always
possible to use a representation based on the Cholesky decomposition which is
much more numerically stable. We will shortly provide the proof for this rank one update/downdate via Cholesky decomposition in this section.
\subsection{Rank-One Update}\index{Rank-one update}
A rank-one update $\bA^\prime$ of matrix $\bA$ by vector $\bx$ is of the form:
\begin{equation*}
\begin{aligned}
	\bA^\prime &= \bA + \bv \bv^\top\\
		\bR^{\prime\top}\bR^\prime &= \bR^\top\bR + \bv \bv^\top.
\end{aligned}
\end{equation*}
If we have already calculated the Cholesky factor $\bR$ of $\bA \in \real^{n\times n}$, then the Cholesky factor $\bR^\prime$ of $\bA^\prime$ can be obtained efficiently.  Note that $\bA^\prime$ differs from $\bA$ only via the symmetric rank-one matrix. Hence we can compute $\bR^\prime$ from $\bR$ using the rank-one Cholesky update, which takes $O(n^2)$ operations each saving from $O(n^3)$ if we do know $\bR$, the Cholesky decomposition of $\bA$ up front, i.e., we want to compute the Cholesky decomposition of $\bA^\prime$ via that of $\bA$.
To see this,
suppose there is a set of orthogonal matrices $\bQ_n \bQ_{n-1}\ldots \bQ_1$ such that that
$$
\bQ_n \bQ_{n-1}\ldots \bQ_1
\begin{bmatrix}
\bv^\top \\
\bR
\end{bmatrix}
=
\begin{bmatrix}
\bzero \\
\bR^\prime
\end{bmatrix}.
$$
Then we find out the expression for the Cholesky factor of $\bA^\prime$ by $\bR^\prime$.
Specifically, multiply the left-hand side (l.h.s.,) of the above equation by its transpose,
$$
\begin{bmatrix}
	\bv & \bR^\top
\end{bmatrix}
 \bQ_1 \ldots \bQ_{n-1}\bQ_n
\bQ_n \bQ_{n-1}\ldots \bQ_1
\begin{bmatrix}
	\bv^\top \\
	\bR
\end{bmatrix}
= \bR^\top\bR + \bv \bv^\top.
$$
And multiply the right-hand side (r.h.s.,) by its transpose,
$$
\begin{bmatrix}
	\bzero & \bR^{\prime\top}
\end{bmatrix}
\begin{bmatrix}
	\bzero \\
	\bR^\prime
\end{bmatrix}=\bR^{\prime\top}\bR^\prime,
$$
which agrees with the l.h.s., equation. \textit{Givens rotations} are such orthogonal matrices that can transfer $\bR,\bv$ into $\bR^\prime$. We will discuss the fundamental meaning of Givens rotation shortly to prove the existence of QR decomposition in Section~\ref{section:qr-givens} (p.~\pageref{section:qr-givens}). Here, we only introduce the definition of it and write out the results directly. Feel free to skip this section for a first reading.

	\begin{definition}[$n$-th Order Givens Rotation]\label{definition:givens-rotation-in-qr}
		A Givens rotation is represented by a matrix of the following form

		$$
		\bG_{kl}=
		\begin{bmatrix}
			1 &          &   &  &   &   & &  & &\\
			& \ddots  &  &  &  & && & &\\
			&      & 1 &  & & &  && &\\
			&      &  & c &  &  &  & s & &\\
			&& &   & 1 & & && &\\
			&& &   &   &\ddots &  && &\\
			&& &  &   &  & 1&& &\\
			&& & -s &  &  & &c& &\\
			&& & &  &  & & &1 & \\
			&& & &  &  & & & &\ddots
		\end{bmatrix}_{n\times n},
		$$
		where the $(k,k), (k,l), (l,k), (l,l)$ entries are $c, s, -s, c$ respectively, and $s = \cos \theta$ and $c=\cos \theta$ for some $\theta$.

		Let $\bdelta_k \in \real^n$ be the zero vector except that the $k$-th entry is 1. Then mathematically, the Givens rotation defined above can be denoted by
		$$
		\bG_{kl}= \bI + (c-1)(\bdelta_k\bdelta_k^\top + \bdelta_l\bdelta_l^\top) + s(\bdelta_k\bdelta_l^\top -\bdelta_l\bdelta_k^\top ).
		$$
		where the subscripts $k,l$ indicate the \textbf{rotation is in plane $k$ and $l$}.
		Specifically, one can also define the $n$-th order Givens rotation where $(k,k), (k,l), (l,k), (l,l)$ entries are $c, \textcolor{blue}{-s, s}, c$ respectively. The ideas are the same.
	\end{definition}

It can be easily verified that the $n$-th order Givens rotation is an orthogonal matrix and its determinant is 1. For any vector $\bx =[x_1, x_2, \ldots, x_n]^\top \in \real^n$, we have $\by = \bG_{kl}\bx$, where
$$
\left\{
\begin{aligned}
	&y_k = c \cdot x_k + s\cdot x_l,   \\
	&y_l = -s\cdot x_k +c\cdot x_l,  \\
	&y_j = x_j , &  (j\neq k,l)
\end{aligned}
\right.
$$
That is, a Givens rotation applied to $\bx$ rotates two components of $\bx$ by some angle $\theta$ and leaves all other components the same.

Now, suppose we have an $(n+1)$-th order Givens rotation indexed from $0$ to $n$, and it is given by
$$
\bG_k = \bI + (c_k-1)(\bdelta_0\bdelta_0^\top + \bdelta_k\bdelta_k^\top) + s_k(\bdelta_0\bdelta_k^\top -\bdelta_k\bdelta_0^\top ),
$$
where $c_k = \cos \theta_k, s_k=\sin\theta_k$ for some $\theta_k$, $\bG_k \in \real^{(n+1)\times (n+1)}$, $\delta_k\in \real^{n+1}$ is a zero vector except that the $(k+1)$-th entry is 1.

\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolorSkip]
	Taking out the $k$-th column of the following equation
	$$
	\begin{bmatrix}
		\bv^\top \\
		\bR
	\end{bmatrix}
	=
	\begin{bmatrix}
		\bzero \\
		\bR^\prime
	\end{bmatrix},
	$$
	where we let the $k$-th element of $\bv$ be $v_k$, and the $k$-th diagonal of $\bR$ be $r_{kk}$.
	We realize that $\sqrt{v_k^2 + r_{kk}^2} \neq 0$,
	let $c_k = \frac{r_{kk}}{\sqrt{v_k^2 + r_{kk}^2}}$, $s_k=-\frac{v_k}{\sqrt{v_k^2 + r_{kk}^2}}$. Then,
	$$
	\left\{
	\begin{aligned}
		&v_k \rightarrow c_kv_k+s_kr_{kk}=0;   \\
		&r_{kk}\rightarrow -s_k v_k +c_kr_{kk}= \sqrt{v_k^2 + r_{kk}^2} = r^\prime_{kk} .  \\

	\end{aligned}
	\right.
	$$
That is, $G_k$ will introduce zero value to the $k$-th element to $\bv$ and nonzero value to $r_{kk}$.
\end{mdframed}
This finding above is essential for the rank-one update. And we obtain
$$
\bG_n \bG_{n-1}\ldots \bG_1
\begin{bmatrix}
	\bv^\top \\
	\bR
\end{bmatrix}
=
\begin{bmatrix}
	\bzero \\
	\bR^\prime
\end{bmatrix}.
$$
For each Givens rotation, it takes $6n$ flops. And there are $n$ such rotations, which requires $6n^2$ flops if keeping only the leading term. The complexity to calculate the Cholesky factor of $\bA^\prime$ is thus reduced from $\frac{1}{3} n^3$ to $6n^2$ flops if we already know the Cholesky factor of $\bA$ by the rank-one update. The above algorithm is essential to reduce the complexity of the posterior calculation in the Bayesian inference for Gaussian mixture model \citep{lu2021bayes}.

\subsection{Rank-One Downdate}
Now suppose we have calculated the Cholesky factor of $\bA$, and the $\bA^\prime$ is the downdate of $\bA$ as follows:
\begin{equation*}
	\begin{aligned}
		\bA^\prime &= \bA - \bv \bv^\top\\
		\bR^{\prime\top}\bR^\prime &= \bR^\top\bR - \bv \bv^\top.
	\end{aligned}
\end{equation*}
The algorithm is similar by proceeding as follows:
\begin{equation}\label{equation:rank-one-downdate}
\bG_1 \bG_{2}\ldots \bG_n
\begin{bmatrix}
	\bzero \\
	\bR
\end{bmatrix}
=
\begin{bmatrix}
	\bv^\top \\
	\bR^\prime
\end{bmatrix}.
\end{equation}
Again, $
\bG_k = \bI + (c_k-1)(\bdelta_0\bdelta_0^\top + \bdelta_k\bdelta_k^\top) + s_k(\bdelta_0\bdelta_k^\top -\bdelta_k\bdelta_0^\top ),
$ can be constructed as follows:
\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolorSkip]
	Taking out the $k$-th column of the following equation
	$$
	\begin{bmatrix}
		\bzero \\
		\bR
	\end{bmatrix}
	=
	\begin{bmatrix}
		\bv^\top \\
		\bR^\prime
	\end{bmatrix}.
	$$
	We realize that $r_{kk} \neq 0$,
	let $c_k=\frac{\sqrt{r_{kk}^2 - v_k^2}}{r_{kk}}$, $s_k = \frac{v_k}{r_{kk}}$. Then,
	$$
	\left\{
	\begin{aligned}
		& 0 \rightarrow s_kr_{kk}=v_k;   \\
		&r_{kk}\rightarrow c_k r_{kk}= \sqrt{r_{kk}^2-v_k^2  }=r^\prime_{kk} .  \\

	\end{aligned}
	\right.
	$$
This requires $r^2_{kk} > v_k^2$ to make $\bA^\prime$ to be positive definite. Otherwise, $c_k$ above will not exist.
\end{mdframed}
Again, one can check that, multiply the l.h.s., of Equation~\eqref{equation:rank-one-downdate} by its transpose, we have
$$
\begin{bmatrix}
	\bzero & \bR^\top
\end{bmatrix}
\bG_n \ldots \bG_{2}\bG_1
\bG_1 \bG_{2}\ldots \bG_n
\begin{bmatrix}
	\bzero \\
	\bR
\end{bmatrix} =\bR^\top\bR.
$$
And multiply the r.h.s., by its transpose, we have
$$
\begin{bmatrix}
	\bv & \bR^{\prime\top}
\end{bmatrix}
\begin{bmatrix}
	\bv^\top \\
	\bR^\prime
\end{bmatrix}=\bv\bv^\top + \bR^{\prime\top}\bR^\prime.
$$
This results in $\bR^{\prime\top}\bR^\prime = \bR^\top\bR - \bv \bv^\top$.

\section{Application: Indefinite Rank Two Update}\index{Rank-two update}

Let $\bA = \bR^\top\bR$ be the Cholesky decomposition of $\bA$, \citet{goldfarb1976factorized, seeger2004low} introduce a stable method for the indefinite rank-two update of the form
$$
\bA^\prime = (\bI+\bv\bu^\top)\bA(\bI+\bu\bv^\top).
$$
Let
$$
\bigg\{
\begin{aligned}
	\bz &= \bR^{-\top}\bv,   \\
	\bw &= \bR\bu,
\end{aligned}
\qquad
\rightarrow
\qquad
\bigg\{
\begin{aligned}
	\bv &= \bR^{\top}\bz,   \\
	\bu &= \bR^{-1}\bw.
\end{aligned}
$$
And suppose the LQ decomposition \footnote{We will shortly introduce in Theorem~\ref{theorem:lq-decomposition} (p.~\pageref{theorem:lq-decomposition}).} of $\bI+\bz\bw^\top$ is given by $\bI+\bz\bw^\top =\bL\bQ$, where $\bL$ is lower triangular and $\bQ$ is orthogonal. Thus, we have
$$
\begin{aligned}
\bA^\prime &= (\bI+\bv\bu^\top)\bA(\bI+\bu\bv^\top)\\
&= (\bI+\bR^{\top}\bz   \bw^\top \bR^{-\top })\bA(\bI+\bR^{-1}\bw \bz^\top\bR)\\
&= \bR^\top (\bI+\bz\bw^\top)(\bI+\bw\bz^\top)\bR\\
&= \bR^\top\bL \bQ\bQ^\top \bL^\top\bR \\
&= \bR^\top\bL \bL^\top\bR.
\end{aligned}
$$
Let $\bR^\prime = \bR^\top\bL$ which is lower triangular, we find the Cholesky decomposition of $\bA^\prime$.

\chapter{Nonnegative Matrix Factorization (NMF)}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Nonnegative Matrix Factorization (NMF)}\index{NMF}\label{section:nmf}
Following  the matrix factorization via the ALS, we now consider algorithms for solving the nonnegative matrix factorization (NMF) problem:
\begin{itemize}
\item Given a nonnegative matrix $\bA\in \real^{M\times N}$, find nonnegative matrix factors $\bW\in \real^{M\times K}$ and $\bZ\in \real^{K\times N}$ such that:
$$
\bA\approx\bW\bZ.
$$
\end{itemize}
To measure the approximation, the loss to evaluate is still from the Frobenius norm of the difference between the two matrices:
$$
L(\bW,\bZ) = ||\bW\bZ-\bA||^2.
$$

\section{NMF via Multiplicative Update}
Following Section~\ref{section:als-netflix} (p.~\pageref{section:als-netflix}), given $\bW\in \real^{M\times K}$, we want to update $\bZ\in \real^{K\times N}$, the gradient with respect to $\bZ$ is given by Equation~\eqref{equation:givenw-update-z-allgd} (p.~\pageref{equation:givenw-update-z-allgd}):
$$
\begin{aligned}
	\frac{\partial L(\bZ|\bW)}{\partial \bZ} =2  \bW^\top(\bW\bZ-\bA) \in \real^{K\times N}.
\end{aligned}
$$
Applying the gradient descent idea in Section~\ref{section:als-gradie-descent} (p.~\pageref{section:als-gradie-descent}), the trivial update on $\bZ$ can be done by
$$
(\text{GD on $\bZ$})\gap \bZ \leftarrow \bZ - \eta \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)=\bZ - \eta \left(2  \bW^\top\bW\bZ-2\bW^\top\bA\right),
$$
where $\eta$ is a small positive step size. Now if we suppose a different step size for each entry of $\bZ$ and incorporate the constant 2 into the step size, the update can be obtained by
$$
(\text{GD$^\prime$ on $\bZ$})\gap
\begin{aligned}
	a_{kn} &\leftarrow z_{kn} - \frac{\eta_{kn}}{2} \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)_{kn}\\
	&=z_{kn} - \eta_{kn}(\bW^\top\bW\bZ-\bW^\top\bA)_{kn}, \gap k\in \{1,\ldots,K\}, n\in\{1,\ldots,N\},
\end{aligned}
$$
where $z_{kn}$ is the $(k,n)$-th entry of $\bZ$. Now if we rescale the step size:
$$
\eta_{kn} = \frac{z_{kn}}{(\bW^\top\bW\bZ)_{kn}},
$$
then we obtain the update rule:
$$
(\text{Multiplicative update on $\bZ$})\gap
z_{kn} \leftarrow z_{kn}\frac{(\bW^\top\bA)_{kn}}{(\bW^\top\bW\bZ)_{kn}}, \gap k\in \{1,\ldots,K\}, n\in\{1,\ldots,N\},
$$
which is known as the \textit{multiplicative update} and is first developed in \citet{lee2001algorithms} and further discussed in \citet{pauca2006nonnegative}. Analogously, the multiplicative update on $\bW$ can be obtained by
$$
(\text{Multiplicative update on $\bW$})\gap
w_{mk} \leftarrow w_{mk} \frac{(\bA\bZ^\top)_{mk}}{(\bW\bZ\bZ^\top)_{mk}}, \,\, m\in\{1,\ldots,M\}, k\in\{1,\ldots,K\}.
$$

\begin{theorem}[Convergence of Multiplicative Update]
The loss $L(\bW,\bZ)=||\bW\bZ-\bA||^2$ is non-increasing under the multiplicative update rules:
$$
\left\{
\begin{aligned}
z_{kn} &\leftarrow z_{kn}\frac{(\bW^\top\bA)_{kn}}{(\bW^\top\bW\bZ)_{kn}}, \gap k\in \{1,\ldots,K\}, n\in\{1,\ldots,N\},;\\
w_{mk} &\leftarrow w_{mk} \frac{(\bA\bZ^\top)_{mk}}{(\bW\bZ\bZ^\top)_{mk}}, \gap  m\in\{1,\ldots,M\}, k\in\{1,\ldots,K\}.
\end{aligned}
\right.
$$
\end{theorem}
We refer the proof of the above theorem to \citep{lee2001algorithms}.  Clearly the approximations $\bW$ and $\bZ$ remain nonnegative during the updates.
It
is generally best to update $\bW$ and $\bZ$ ``simultaneously”, instead of updating each
matrix completely before the other. In this case, after updating a row of $\bZ$, we update the corresponding column of $\bW$.
In
the implementation, a small positive quantity, say the square root of the machine
precision, should be added to the denominators in the approximations of $\bW$ and $\bZ$
at each iteration step. And a trivial $\epsilon=10^{-9}$ will suffice. The full procedure is shown in Algorithm~\ref{alg:nmf-multiplicative}.

\begin{algorithm}[h]
\caption{NMF via Multiplicative Updates}
\label{alg:nmf-multiplicative}
\begin{algorithmic}[1]
\Require $\bA\in \real^{M\times N}$;
\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ randomly with nonnegative entries.
\State choose a stop criterion on the approximation error $\delta$;
\State choose maximal number of iterations $C$;
\State $iter=0$;
\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
\State $iter=iter+1$;
\For{$k=1$ to $K$}
\For{$n=1$ to $N$} \Comment{update $k$-th row of $\bZ$}
\State $z_{kn} \leftarrow z_{kn}\frac{(\bW^\top\bA)_{kn}}{(\bW^\top\bW\bZ)_{kn}+\textcolor{blue}{\epsilon}}$;
\EndFor
\For{$m=1$ to $M$} \Comment{update $k$-th column of $\bW$}
\State $w_{mk} \leftarrow w_{mk} \frac{(\bA\bZ^\top)_{mk}}{(\bW\bZ\bZ^\top)_{mk}+\textcolor{blue}{\epsilon}}$;
\EndFor
\EndFor
\EndWhile
\State Output $\bW,\bZ$;
\end{algorithmic}
\end{algorithm}
\section{Regularization}

Similar to the ALS with regularization in Section~\ref{section:regularization-extention-general} (p.~\pageref{section:regularization-extention-general}), recall that the regularization helps employ the ALS into general matrices. We can also add a regularization in the context of NMF:
$$
	L(\bW,\bZ)  =||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2, \qquad \lambda_w>0, \lambda_z>0,
$$
where the induced matrix norm is still the Frobenius norm. The gradient with respective to $\bZ$ given $\bW$ is the same as that in Equation~\eqref{equation:als-regulari-gradien} (p.~\pageref{equation:als-regulari-gradien}):
$$
\begin{aligned}
	\frac{\partial L(\bZ|\bW)}{\partial \bZ} =2\bW^\top(\bW\bZ-\bA) + \textcolor{blue}{2\lambda_z\bZ}  \in \real^{K\times N}.
\end{aligned}
$$
The trivial gradient descent update can be obtained by
$$
(\text{GD on }\bZ) \gap \bZ \leftarrow \bZ - \eta \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)=\bZ - \eta \left(2  \bW^\top\bW\bZ-2\bW^\top\bA+\textcolor{blue}{2\lambda_z\bZ}\right),
$$
Analogously, if we suppose a different step size for each entry of $\bZ$ and incorporate the constant 2 into the step size, the update can be obtained by
$$
(\text{GD$^\prime$ on $\bZ$})\gap
\begin{aligned}
	z_{kn} &\leftarrow z_{kn} - \frac{\eta_{kn}}{2} \left(\frac{\partial L(\bZ|\bW)}{\partial \bZ}\right)_{kn}\\
	&=z_{kn} - \eta_{kn}(\bW^\top\bW\bZ-\bW^\top\bA+\textcolor{blue}{\lambda_z\bZ})_{kn}, \\
	&\gap\gap\qquad \gap k\in \{1,\ldots,K\}, n\in\{1,\ldots,N\}.
\end{aligned}
$$
Now if we rescale the step size:
$$
\eta_{kn} = \frac{z_{kn}}{(\bW^\top\bW\bZ)_{kn}},
$$
then we obtain the update rule:
$$
\begin{aligned}
(\text{Multiplicative update on $\bZ$})\gap
z_{kn} &\leftarrow z_{kn}\frac{(\bW^\top\bA)_{kn}-
	\textcolor{blue}{\lambda_z z_{kn}}
}{(\bW^\top\bW\bZ)_{kn}}, \\
&\gap\gap\qquad \gap k\in \{1,\ldots,K\}, n\in\{1,\ldots,N\}.
\end{aligned}
$$
Similarly, the multiplicative update on $\bW$ can be obtained by
$$
\begin{aligned}
(\text{Multiplicative update on $\bW$})\gap
w_{mk} &\leftarrow w_{mk} \frac{(\bA\bZ^\top)_{mk}
	-\textcolor{blue}{\lambda_w w_{mk}}
}{(\bW\bZ\bZ^\top)_{mk}}, \\
&\gap\gap\qquad \gap  m\in\{1,\ldots,M\}, k\in\{1,\ldots,K\}.
\end{aligned}
$$
In the collaborative filtering context, it is also known that the NMF via multiplicative update can result in overfitting though the convergence results are good and the overfitting can be partly reduced via the regularization. However, the out-of-sample performance is still low. While Bayesian optimization via generative models can present overfitting to a large extent \citep{lu2022flexible}.
\section{Initialization}
In the above discussion, we initialize $\bW$ and $\bZ$ randomly. Whereas, there are also alternative strategies designed to obtain better initial estimates in the hope of converging more rapidly to a good solution \citep{boutsidis2008svd, gillis2014and}. We sketch the methods as follows:
\begin{itemize}
\item \textit{Clustering techniques.} Use some clustering methods on the columns of $\bA$, make the cluster means of the top $K$ clusters as the columns of $\bW$, and initialize $\bZ$ as a proper scaling of
the cluster indicator matrix (that is, $z_{kn}\neq 0$ indicates $\ba_n$ belongs to the $k$-th cluster);
\item \textit{Subset selection.} Pick $K$ columns of $\bA$ and set those as the initial columns for $\bW$, and analogously, $K$ rows of $\bA$ are selected to form the rows of $\bZ$;
\item \textit{SVD-based.} Suppose the SVD of $\bA=\sum_{i=1}^{r}\sigma_i\bu_i\bv_i^\top$ where each factor $\sigma_i\bu_i\bv_i^\top$ is a rank-one matrix with possible negative values in $\bu_i, \bv_i$, and nonnegative $\sigma_i$. Denote $[x]_+=\max(x, 0)$, we notice
$$
\bu_i\bv_i^\top = [\bu_i]_+[\bv_i]_+^\top+[-\bu_i]_+[-\bv_i]_+^\top-[-\bu_i]_+[\bv_i]_+^\top-[\bu_i]_+[-\bv_i]_+^\top.
$$
Either $[\bu_i]_+[\bv_i]_+^\top$ or $[-\bu_i]_+[-\bv_i]_+^\top$ can be selected as a column and a row in $\bW,\bZ$.
\end{itemize}
However, these techniques are not guaranteed to perform better theoretically.
We recommend readers to the aforementioned papers for more information.

\section{Movie Recommender Context}
Both the NMF and the ALS approximate the matrix and reconstruct the entries in the matrix with a set of basis vectors. The basis in the NMF is composed of vectors with nonnegative elements while the basis in the ALS can have positive or negative values.
The difference then is that the NMF reconstructs each vector as a positive summation of the basis vectors with a ``relative" small component in the direction of each basis vector.
Whereas, In the ALS the data is modeled as a linear combination of the basis you can add or subtract vectors as needed and the components in the direction of each basis vector can be large positive values or negative values. Therefore, depending on the application one or the other factorization can be utilized to describe the data with different meanings.

In the context of a movie recommender system then the rows of $\bW$ represent the feature of the movies and columns of $\bZ$ represent the features of a user.
In the NMF you can say that a movie is 0.5 comedy, 0.002 action, and 0.09 romantic. However, In the ALS you can get combinations such as 4 comedy, -0.05 comedy, and -3 drama, i.e., a positive or negative component on that feature.

The ALS and NMF are similar in the sense that the importance of each basis vector is not in a hierarchical manner. Whereas, the key difference between the ALS and the SVD is that, in the SVD, the importance of each vector in the basis is relative to the value of the singular value associated with that vector. For the SVD of $\bA=\sum_{i=1}^{r}\sigma_i\bu_i\bv_i^\top$,
this usually means that the reconstruction $\sigma_1\bu_1\bv_i^\top$ via the first set of basis vectors dominates and is the most used set to reconstruct data, then the second set, and so on. So the basis in the SVD has an implicit hierarchy and that doesn't happen in the ALS or the NMF.

\chapter{Alternating Least Squares (ALS)}\label{section:als}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Netflix Recommender and Matrix Factorization}\label{section:als-netflix}
In the Netflix prize \citep{bennett2007netflix}, the goal is to predict the ratings of users for different movies, given the existing ratings of those users for other movies.
We index $M$ movies with $m= 1, 2,\ldots,M$ and $N$ users
by $n = 1, 2,\ldots,N$. We denote the rating of the $n$-th user for the $m$-th movie by $a_{mn}$. Define $\bA$ to be an $M \times N$ rating matrix with columns $\ba_n \in \real^M$ containing ratings of the $n$-th user. Note that many ratings $a_{mn}$ are missing and our goal is to predict those missing ratings accurately.

We formally consider algorithms for solving the following problem: The matrix $\bA$ is approximately factorized into an $M\times K$ matrix $\bW$ and a $K \times  N$ matrix $\bZ$. Usually $K$ is chosen to be smaller than $M$ or $N$, so that $\bW$ and $\bZ$ are smaller than the original
matrix $\bA$. This results in a compressed version of the original data matrix.
An
appropriate decision on the value of $K$ is critical in practice, but the choice of $K$ is very
often problem dependent.
The factorization is significant in the sense, suppose $\bA=[\ba_1, \ba_2, \ldots, \ba_N]$ and $\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$ are the column partitions of $\bA,\bZ$ respectively, then $\ba_n = \bW\bz_n$, i.e., each column $\ba_n$ is approximated by a linear combination of the columns of $\bW$ weighted by the components in $\bz_n$. Therefore, columns of $\bW$ can be thought of as containing the column basis of $\bA$. This is similar to the factorization in the data interpretation part (Part~\ref{part:data-interation}, p.~\pageref{part:data-interation}). What's different is that we are not restricting $\bW$ to be exact columns from $\bA$.

To find the approximation $\bA\approx\bW\bZ$, we need to define a loss function such that the distance between $\bA$ and $\bW\bZ$ can be measured. The loss function is selected to be the Frobenius norm between two matrices which vanishes to zero if $\bA=\bW\bZ$ where the advantage will be seen shortly.

To simplify the problem, let us
assume that there are no missing
ratings first. Project data vectors $\ba_n$ to a smaller
dimension $\bz_n \in \real^K$  with $K<M$,
such that the \textit{reconstruction error} measured by Frobenius norm is minimized (assume $K$ is known):
\begin{equation}\label{equation:als-per-example-loss}
\mathop{\min}_{\bW,\bZ}  \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2,
\end{equation}
where $\bW=[\bw_1^\top; \bw_2^\top; \ldots; \bw_M^\top]\in \real^{M\times K}$ and $\bZ=[\bz_1, \bz_2, \ldots, \bz_N] \in \real^{K\times N}$ containing $\bw_m$'s and $\bz_n$'s as \textbf{rows and columns} respectively. The loss form in Equation~\eqref{equation:als-per-example-loss} is known as the \textit{per-example loss}. It can be equivalently written as
$$
L(\bW,\bZ) = \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2 = ||\bW\bZ-\bA||^2.
$$
Moreover, the loss $L(\bW,\bZ)=\sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)$ is convex with respect to $\bZ$ given $\bW$ and vice versa. Therefore, we can first minimize with respect to $\bZ$
given $\bW$ and then minimize with respect to $\bW$
given $\bZ$:
$$
\left\{
\begin{aligned}
	\bZ &\leftarrow \mathop{\arg \min}_{\bZ} L(\bW,\bZ);    \qquad \text{(ALS1)} \\
	\bW &\leftarrow \mathop{\arg \min}_{\bW} L(\bW,\bZ). \qquad \text{(ALS2)}
\end{aligned}
\right.
$$
This is known as the \textit{coordinate descent algorithm} in which case we employ the least squares alternatively. Hence it is also called the \textit{alternating least squares (ALS)} \citep{comon2009tensor, takacs2012alternating, giampouras2018alternating}. The convergence is guaranteed if the loss function $L(\bW,\bZ)$ decreases at each iteration and we shall discuss more on this in the sequel.\index{ALS}

\begin{remark}[Convexity and Global Minimum]
Although the loss function defined by Frobenius norm $|| \bW\bZ-\bA||^2$ is convex in $\bW$ given $\bZ$ or vice versa, it is not convex in both variables together. Therefore we are not able to find the global minimum. However, the convergence is assured to find local minima.
\end{remark}

\paragraph{Given $\bW$, Optimizing $\bZ$}

Now, let's see what is in the problem of $\bZ \leftarrow \mathop{\arg \min}_{\bZ} L(\bW,\bZ)$. When there exists a unique minimum of the loss function $L(\bW,\bZ)$ with respect to $\bZ$, we speak of the \textit{least squares} minimizer of $\mathop{\arg \min}_{\bZ} L(\bW,\bZ)$.
Given $\bW$, $L(\bW,\bZ)$ can be written as $L(\bZ|\bW)$ to emphasize on the variable of $\bZ$:
$$
\begin{aligned}
	L(\bZ|\bW) &= ||\bW\bZ-\bA||^2= \left\Vert\bW[\bz_1,\bz_2,\ldots, \bz_N]-[\ba_1,\ba_2,\ldots,\ba_N]\right\Vert^2=\left\Vert
	\begin{bmatrix}
		\bW\bz_1 - \ba_1 \\
		\bW\bz_2 - \ba_2\\
		\vdots \\
		\bW\bz_N - \ba_N
	\end{bmatrix}
	\right\Vert^2.
\end{aligned}\footnote{The matrix norm used here is the Frobenius norm such that $|| \bA ||= \sqrt{\sum_{i=1,j=1}^{m,n} (\bA_{ij})^2}$ if $\bA\in \real^{m\times n}$. And the vector norm used here is the $l_2$ norm such that $||\bx||_2  = \sqrt{\sum_{i=1}^{n}x_i^2}$ if $\bx\in \real^n$.}
$$
Now, if we define
$$
\widetildebW =
\begin{bmatrix}
	\bW & \bzero & \ldots & \bzero\\
	\bzero & \bW & \ldots & \bzero\\
	\vdots & \vdots & \ddots & \vdots \\
	\bzero & \bzero & \ldots & \bW
\end{bmatrix}
\in \real^{MN\times KN},
\gap
\widetildebz=
\begin{bmatrix}
	\bz_1 \\ \bz_2 \\ \vdots \\ \bz_N
\end{bmatrix}
\in \real^{KN},
\gap
\widetildeba=
\begin{bmatrix}
	\ba_1 \\ \ba_2 \\ \vdots \\ \ba_N
\end{bmatrix}
\in \real^{MN},
$$
then the (ALS1) problem can be reduced to the normal least squares for minimizing  $||\widetildebW \widetildebz - \widetildeba||^2$ with respect to $\widetildebz$. And the solution is given by
$$
\widetildebz = (\widetildebW^\top\widetildebW)^{-1} \widetildebW^\top\widetildeba.
$$
Alternatively,
a direct way to solve (ALS1) is to find the differential of $L(\bZ|\bW)$ with respect to $\bZ$:
\begin{equation}\label{equation:givenw-update-z-allgd}
\begin{aligned}
	\frac{\partial L(\bZ|\bW)}{\partial \bZ} &=
	\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial \bZ}\\
	&=\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial (\bW\bZ-\bA)}
	\frac{\partial (\bW\bZ-\bA)}{\partial \bZ}\\
	&\stackrel{\star}{=}2  \bW^\top(\bW\bZ-\bA) \in \real^{K\times N},
\end{aligned}
\end{equation}
where the first equality is from the definition of Frobenius such that $|| \bA || = \sqrt{\sum_{i=1,j=1}^{m,n} (\bA_{ij})^2}=\sqrt{tr(\bA\bA^\top)}$, and equality ($\star$) comes from the fact that $\frac{\partial tr(\bA\bA^\top)}{\partial \bA} = 2\bA$. When the loss function is a differentiable function of $\bZ$, we may determine the least squares solution by differential calculus, and a minimum of the function
$L(\bZ|\bW)$ must be a root of the equation:
$$
\frac{\partial L(\bZ|\bW)}{\partial \bZ}  = \bzero.
$$
By finding the root of the above equation, we have the ``candidate" update on $\bZ$ that find the minimizer of $L(\bZ|\bW)$
\begin{equation}\label{equation:als-z-update}
	\bZ = (\bW^\top\bW)^{-1} \bW^\top \bA  \leftarrow \mathop{\arg \min}_{\bZ} L(\bZ|\bW).
\end{equation}
Before we declare a root of the above equation is actually a minimizer rather than a maximizer (that's why we call the update a ``candidate" update above), we need to verify the function is convex such that if the function is twice differentiable, this can be equivalently done by verifying
$$
\frac{\partial^2 L(\bZ|\bW)}{\partial \bZ^2} > 0,
$$
i.e., the Hessian matrix is positive definite (recall the definition of positive definiteness, Definition~\ref{definition:psd-pd-defini}, p.~\pageref{definition:psd-pd-defini}). To see this, we write out the twice differential
$$
\frac{\partial^2 L(\bZ|\bW)}{\partial \bZ^2}= 2\bW^\top\bW \in \real^{K\times K},
$$
which has full rank if $\bW\in \real^{M\times K}$ has full rank (Lemma~\ref{lemma:rank-of-ata}, p.~\pageref{lemma:rank-of-ata}) and $K<M$.
\begin{remark}[Positive definite Hessian if $\bW$ has full rank]
We here claim that if $\bW$ has full rank, then $\frac{\partial^2 L(\bZ|\bW)}{\partial \bZ^2}$ is positive definite.
This can be done by checking that when $\bW$ has full rank, $\bW\bx=\bzero$ only when $\bx=\bzero$ since the null space of $\bW$ is of dimension 0. Therefore,
$$
\bx^\top (2\bW^\top\bW)\bx >0, \qquad \text{for any nonzero vector $\bx\in \real^K$}.
$$
\end{remark}

Now, the thing is that we need to check \textcolor{blue}{if $\bW$ has full rank so that the Hessian of $L(\bZ|\bW)$ is positive definiteness}, otherwise, we cannot claim the update of $\bZ$ in Equation~\eqref{equation:als-z-update} decreases the loss (due to convexity) so that the matrix decomposition is going into the right way to better approximate the original matrix $\bA$ by $\bW\bZ$ in each iteration.
We will shortly come back to the positive definiteness of the Hessian matrix in the sequel which relies on the following lemma
\begin{lemma}[Rank of $\bZ$ after Updating]\label{lemma:als-update-z-rank}
	Suppose $\bA\in \real^{M\times N}$ has full rank with $M\leq N$ and $\bW\in \real^{M\times K}$ has full rank with $K<M$, then the update of $\bZ=(\bW^\top\bW)^{-1} \bW^\top \bA \in \real^{K\times N}$ in Equation~\eqref{equation:als-z-update} has full rank.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:als-update-z-rank}]
	Since $\bW^\top\bW\in \real^{K\times K}$ has full rank if $\bW$ has full rank (Lemma~\ref{lemma:rank-of-ata}, p.~\pageref{lemma:rank-of-ata}) such that $(\bW^\top\bW)^{-1} $ has full rank.

	Suppose $\bW^\top\bx=\bzero$, this implies $(\bW^\top\bW)^{-1} \bW^\top\bx=\bzero$. Thus
	$$
	\nspace(\bW^\top) \subseteq \nspace\left((\bW^\top\bW)^{-1} \bW^\top\right).
	$$
	Moreover, suppose $(\bW^\top\bW)^{-1} \bW^\top\bx=\bzero$, and since $(\bW^\top\bW)^{-1} $ is invertible. This implies $ \bW^\top\bx=(\bW^\top\bW)\bzero=\bzero$, and
	$$
	\nspace\left((\bW^\top\bW)^{-1} \bW^\top\right)\subseteq \nspace(\bW^\top).
	$$
	As a result, by ``sandwiching", it follows that
	\begin{equation}\label{equation:als-z-sandiwch1}
		\nspace(\bW^\top) = \nspace\left((\bW^\top\bW)^{-1} \bW^\top\right).
	\end{equation}
	Therefore, $(\bW^\top\bW)^{-1} \bW^\top$ has full rank $K$. Let $\bT=(\bW^\top\bW)^{-1} \bW^\top\in \real^{K\times M}$, and suppose $\bT^\top\bx=\bzero$. This implies $\bA^\top\bT^\top\bx=\bzero$, and
	$$
	\nspace(\bT^\top) \subseteq \nspace(\bA^\top\bT^\top).
	$$
	Similarly, suppose $\bA^\top(\bT^\top\bx)=\bzero$. Since $\bA$ has full rank with the dimension of the null space being 0: $dim\left(\nspace(\bA^\top)\right)=0$, $(\bT^\top\bx)$ must be zero. The claim follows since $\bA$ has full rank $M$ with the row space of $\bA^\top$ being equal to the column space of $\bA$ where $dim\left(\cspace(\bA)\right)=M$ and the $dim\left(\nspace(\bA^\top)\right) = M-dim\left(\cspace(\bA)\right)=0$. Therefore, $\bx$ is in the null space of $\bT^\top$ if $\bx$ is in the null space of $\bA^\top\bT^\top$:
	$$
	\nspace(\bA^\top\bT^\top)\subseteq \nspace(\bT^\top).
	$$
	By ``sandwiching" again,
	\begin{equation}\label{equation:als-z-sandiwch2}
		\nspace(\bT^\top) = \nspace(\bA^\top\bT^\top).
	\end{equation}
	Since $\bT^\top$ has full rank $K<M<N$, $dim\left(\nspace(\bT^\top) \right) = dim\left(\nspace(\bA^\top\bT^\top)\right)=0$.
	Therefore,
	$\bZ^\top=\bA^\top\bT^\top$ has full rank $K$.
	We complete the proof.
\end{proof}

\paragraph{Given $\bZ$, Optimizing $\bW$}

Given $\bZ$, $L(\bW,\bZ)$ can be written as $L(\bW|\bZ)$ to emphasize the variable of $\bW$:
$$
\begin{aligned}
	L(\bW|\bZ) &= ||\bW\bZ-\bA||^2.
\end{aligned}
$$
A direct way to solve (ALS2) is to find the differential of $L(\bW|\bZ)$ with respect to $\bW$:
$$
\begin{aligned}
	\frac{\partial L(\bW|\bZ)}{\partial \bW} &=
	\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial \bW}\\
	&=\frac{\partial \,\,tr\left((\bW\bZ-\bA)(\bW\bZ-\bA)^\top\right)}{\partial (\bW\bZ-\bA)}
	\frac{\partial (\bW\bZ-\bA)}{\partial \bW}\\
	&=  2(\bW\bZ-\bA)\bZ^\top \in \real^{M\times K}.
\end{aligned}
$$
The ``candidate" update on $\bW$ is similarly to find the root of the differential $\frac{\partial L(\bW|\bZ)}{\partial \bW}$:
\begin{equation}\label{equation:als-w-update}
	\bW^\top = (\bZ\bZ^\top)^{-1}\bZ\bA^\top  \leftarrow \mathop{\arg\min}_{\bW} L(\bW|\bZ).
\end{equation}
Again, we emphasize that the update is only a ``candidate" update. We need to further check whether the Hessian is positive definite or not.
The Hessian matrix is given by
$$
\begin{aligned}
	\frac{\partial^2 L(\bW|\bZ)}{\partial \bW^2} =2\bZ\bZ^\top \in \real^{K\times K}.
\end{aligned}
$$
Therefore, by analogous analysis, if $\bZ$ has full rank with $K<N$, the Hessian matrix is positive definite.

\begin{lemma}[Rank of $\bW$ after Updating]\label{lemma:als-update-w-rank}
	Suppose $\bA\in \real^{M\times N}$ has full rank with $M\leq N$ and $\bZ\in \real^{K\times N}$ has full rank with $K<N$, then the update of $\bW^\top = (\bZ\bZ^\top)^{-1}\bZ\bA^\top$ in Equation~\eqref{equation:als-w-update} has full rank.
\end{lemma}
The proof of Lemma~\ref{lemma:als-update-w-rank} is similar to that of  Lemma~\ref{lemma:als-update-z-rank}, and we shall not repeat the details.

Combine the observations in Lemma~\ref{lemma:als-update-z-rank} and Lemma~\ref{lemma:als-update-w-rank}, as long as we \textcolor{blue}{initialize $\bZ, \bW$ to have full rank}, the updates in Equation~\eqref{equation:als-z-update} and Equation~\eqref{equation:als-w-update} are reasonable. \textcolor{blue}{The requirement on the $M\leq N$ is reasonable in that there are always more users than the number of movies}. We conclude the process in Algorithm~\ref{alg:als}.

\begin{algorithm}[H]
	\caption{Alternating Least Squares}
	\label{alg:als}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$ \textcolor{blue}{with $M\leq N$};
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{with full rank and $K<M\leq N$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$||\bA-\bW\bZ||>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\State $\bZ = (\bW^\top\bW)^{-1} \bW^\top \bA  \leftarrow \mathop{\arg \min}_{\bZ} L(\bZ|\bW)$;
		\State $\bW^\top = (\bZ\bZ^\top)^{-1}\bZ\bA^\top  \leftarrow \mathop{\arg\min}_{\bW} L(\bW|\bZ)$;
		\EndWhile
		\State Output $\bW,\bZ$;
	\end{algorithmic}
\end{algorithm}

\section{Regularization: Extension to General Matrices}\label{section:regularization-extention-general}
We can add a regularization to minimize the following loss:
\begin{equation}\label{equation:als-regularion-full-matrix}
L(\bW,\bZ)  =||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2, \qquad \lambda_w>0, \lambda_z>0,
\end{equation}
where the differential with respect to $\bZ, \bW$ are given respectively by
\begin{equation}\label{equation:als-regulari-gradien}
\left\{
\begin{aligned}
	\frac{\partial L(\bW,\bZ) }{\partial \bZ} &= 2\bW^\top(\bW\bZ-\bA) + 2\lambda_z\bZ \in \real^{K\times N};\\
	\frac{\partial L(\bW,\bZ) }{\partial \bW} &= 2(\bW\bZ-\bA)\bZ^\top + 2\lambda_w\bW \in \real^{M\times K}.
\end{aligned}
\right.
\end{equation}
The Hessian matrices are given respectively by
$$
\left\{
\begin{aligned}
	\frac{\partial^2 L(\bW,\bZ) }{\partial \bZ^2} &= 2\bW^\top\bW+ 2\lambda_z\bI \in \real^{K\times K};\\
	\frac{\partial^2 L(\bW,\bZ) }{\partial \bW^2} &= 2\bZ\bZ^\top + 2\lambda_w\bI \in \real^{K\times K}, \\
\end{aligned}
\right.
$$
which are positive definite due to the perturbation by the regularization. To see this,
$$
\left\{
\begin{aligned}
	\bx^\top (2\bW^\top\bW +2\lambda_z\bI)\bx &= \underbrace{2\bx^\top\bW^\top\bW\bx}_{\geq 0} + 2\lambda_z ||\bx||^2>0, \gap \text{for nonzero $\bx$};\\
	\bx^\top (2\bZ\bZ^\top +2\lambda_w\bI)\bx &= \underbrace{2\bx^\top\bZ\bZ^\top\bx}_{\geq 0} + 2\lambda_w ||\bx||^2>0,\gap \text{for nonzero $\bx$}.
\end{aligned}
\right.
$$
\textcolor{blue}{The regularization makes the Hessian matrices positive definite even if $\bW, \bZ$ are rank deficient}. And now the matrix decomposition can be extended to any matrix even when $M>N$. In rare cases, $K$ can be chosen as $K>\max\{M, N\}$ such that a high-rank approximation of $\bA$ is obtained. However, in most scenarios, we want to find the low-rank approximation of $\bA$ such that $K<\min\{M, N\}$. For example, the ALS can be utilized to find the low-rank neural networks to reduce the memory of the neural networks whilst increase the performance \citep{lu2021numerical}.

Therefore, the minimizers are given by finding the roots of the differential:
\begin{equation}\label{equation:als-regular-final-all}
	\left\{
	\begin{aligned}
		\bZ &= (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \bA  ;\\
		\bW^\top &= (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bA^\top .
	\end{aligned}
	\right.
\end{equation}
The regularization parameters $\lambda_z, \lambda_w\in \real$ are used to balance the trade-off
between the accuracy of the approximation and the smoothness of the computed solution. The selection of the parameters is typically problem dependent and can be obtained by \textit{cross-validation}.
Again, we conclude the process in Algorithm~\ref{alg:als-regularizer}.
\begin{algorithm}[h]
	\caption{Alternating Least Squares with Regularization}
	\label{alg:als-regularizer}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$;
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose regularization parameters $\lambda_w, \lambda_z$;
		\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$||\bA-\bW\bZ||>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\State $\bZ = (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \bA  \leftarrow \mathop{\arg \min}_{\bZ} L(\bZ|\bW)$;
		\State $\bW^\top = (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bA^\top  \leftarrow \mathop{\arg\min}_{\bW} L(\bW|\bZ)$;
		\EndWhile
		\State Output $\bW,\bZ$;
	\end{algorithmic}
\end{algorithm}

\section{Missing Entries}\label{section:alt-columb-by-column}
Since the matrix decomposition via the ALS is extensively used in the Netflix recommender data, where many entries are missing since many users have not watched some movies or they will not rate the movies for some reason. We can employ an additional mask matrix $\bM\in \real^{M\times N}$ where $m_{mn}\in \{0,1\}$ means if the user $n$ has rated the movie $m$ or not. Therefore, the loss function can be defined as
$$
L(\bW,\bZ) = ||\bM\odot  \bA- \bM\odot (\bW\bZ)||^2,
$$
where $\odot$ is the \textit{Hadamard product} between matrices. For example, the Hadamard product for a $3 \times 3$ matrix $\bA$ with a $3\times 3$ matrix $\bB$ is
$$
\bA\odot \bB =
\begin{bmatrix}
	a_{11} & a_{12} & a_{13} \\
	a_{21} & a_{22} & a_{23} \\
	a_{31} & a_{32} & a_{33}
\end{bmatrix}
\odot
\begin{bmatrix}
	b_{11} & b_{12} & b_{13} \\
	b_{21} & b_{22} & b_{23} \\
	b_{31} & b_{32} & b_{33}
\end{bmatrix}
=
\begin{bmatrix}
	a_{11}b_{11} &  a_{12}b_{12} & a_{13}b_{13} \\
	a_{21}b_{21} & a_{22}b_{22} & a_{23}b_{23} \\
	a_{31}b_{31} & a_{32}b_{32} & a_{33}b_{33}
\end{bmatrix}.
$$
To find the solution of the problem, let's decompose the updates in Equation~\eqref{equation:als-regular-final-all} into:
\begin{equation}\label{equation:als-ori-all-wz}
	\left\{
	\begin{aligned}
		\bz_n &= (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \ba_n, &\gap& \text{for $n\in \{1,2,\ldots, N\}$}  ;\\
		\bw_m &= (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bb_m,  &\gap& \text{for $m\in \{1,2,\ldots, M\}$} ,
	\end{aligned}
	\right.
\end{equation}
where $\bZ=[\bz_1, \bz_2, \ldots, \bz_N], \bA=[\ba_1,\ba_2, \ldots, \ba_N]$ are the column partitions of $\bZ, \bA$ respectively. And $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M], \bA^\top=[\bb_1,\bb_2, \ldots, \bb_M]$ are the column partitions of $\bW^\top, \bA^\top$ respectively. The factorization of the updates indicates the update can be done in a column by column fashion.

\paragraph{Given $\bW$}
Let $\bo_n\in \real^M$ denote the movies rated by user $n$ where $o_{nm}=1$ if user $n$ has rated movie $m$, and $o_{nm}=1$ otherwise. Then the $n$-th column of $\bA$ without missing entries can be denoted as the Matlab style notation $\ba_n[\bo_n]$. And we want to approximate the existing $n$-th column by $\ba_n[\bo_n] \approx \bW[\bo_n, :]\bz_n$ which is actually a rank-one least squares problem:
\begin{equation}\label{equation:als-ori-all-wz-modif-z}
	\begin{aligned}
		\bz_n &= \left(\bW[\bo_n, :]^\top\bW[\bo_n, :]+ \lambda_z\bI\right)^{-1} \bW[\bo_n, :]^\top \ba_n[\bo_n], &\gap& \text{for $n\in \{1,2,\ldots, N\}$} .
	\end{aligned}
\end{equation}
Moreover, the loss function with respect to $\bz_n$ can be described by
$$
L(\bz_n|\bW) =\sum_{m\in \bo_n} \left(a_{mn} - \bw_m^\top\bz_n\right)^2
$$
and if we are concerned about the loss for all users:
$$
L(\bZ|\bW) =\sum_{n=1}^N\ \sum_{m\in \bo_n} \left(a_{mn} - \bw_m^\top\bz_n\right)^2.
$$

\paragraph{Given $\bZ$}
Similarly, if $\bp_m \in \real^{N}$ denotes the users that have rated the movie $m$ with $p_{mn}=1$ if the movie $m$ has been rated by user $n$. Then the $m$-th row of $\bA$ without missing entries can be denoted as the Matlab style notation $\bb_m[\bp_m]$. And we want to approximate the existing $m$-th row by $\bb_m[\bp_m] \approx \bZ[:, \bp_m]^\top\bw_m$,
\footnote{Note that $\bZ[:, \bp_m]^\top$ is the transpose of $\bZ[:, \bp_m]$, which is equal to $\bZ^\top[\bp_m,:]$, i.e., transposing first and then selecting.}
which again is a rank-one least squares problem:
\begin{equation}\label{equation:als-ori-all-wz-modif-w}
	\begin{aligned}
		\bw_m &= (\bZ[:, \bp_m]\bZ[:, \bp_m]^\top+\lambda_w\bI)^{-1}\bZ[:, \bp_m]\bb_m[\bp_m],  &\gap& \text{for $m\in \{1,2,\ldots, M\}$} .
	\end{aligned}
\end{equation}
Moreover, the loss function with respect to $\bw_n$ can be described by
$$
L(\bw_n|\bZ) =\sum_{n\in \bp_m} \left(a_{mn} - \bw_m^\top\bz_n\right)^2
$$
and if we are concerned about the loss for all users:
$$
L(\bW|\bZ) =\sum_{m=1}^M  \sum_{n\in \bp_m} \left(a_{mn} - \bw_m^\top\bz_n\right)^2 .
$$
The procedure is again formulated in Algorithm~\ref{alg:als-regularizer-missing-entries}.
\begin{algorithm}[H]
\caption{Alternating Least Squares with Missing Entries and Regularization}
\label{alg:als-regularizer-missing-entries}
\begin{algorithmic}[1]
\Require $\bA\in \real^{M\times N}$;
\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
\State choose a stop criterion on the approximation error $\delta$;
\State choose regularization parameters $\lambda_w, \lambda_z$;
\State compute the mask matrix $\bM$ from $\bA$;
\State choose maximal number of iterations $C$;
\State $iter=0$;
\While{\textcolor{blue}{$||\bM\circledast  \bA- \bM\circledast (\bW\bZ)||^2>\delta $} and $iter<C$}
\State $iter=iter+1$;
\For{$n=1,2,\ldots, N$}
\State $\bz_n = \left(\bW[\bo_n, :]^\top\bW[\bo_n, :]+ \lambda_z\bI\right)^{-1} \bW[\bo_n, :]^\top \ba_n[\bo_n]$; \Comment{$n$-th column of $\bZ$}
\EndFor

\For{$m=1,2,\ldots, M$}
\State $\bw_m = (\bZ[:, \bp_m]\bZ[:, \bp_m]^\top+\lambda_w\bI)^{-1}\bZ[:, \bp_m]\bb_m[\bp_m]$;\Comment{$m$-th column of $\bW^\top$}
\EndFor
\EndWhile
\State Output $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$;
\end{algorithmic}
\end{algorithm}

\section{Vector Inner Product}\label{section:als-vector-product}
We have seen that the ALS is to find matrices $\bW, \bZ$ such that $\bW\bZ$ can approximate $\bA\approx \bW\bZ$ in terms of minimum least squared loss:
$$
\mathop{\min}_{\bW,\bZ}  \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2,
$$
that is, each entry $a_{mn}$ in $\bA$ can be approximated by the inner product between the two vectors $\bw_m^\top\bz_n$. The geometric definition of vector inner product is given by
$$
\bw_m^\top\bz_n = ||\bw||\cdot ||\bz|| \cos \theta,
$$
where $\theta$ is the angle between $\bw$ and $\bz$. So if the vector norms of $\bw, \bz$ are determined, the smaller the angle, the larger the inner product.

Come back to the Netflix data, where the rating are ranging from 0 to 5, and the larger the better. If $\bw_m$ and $\bz_n$ fall ``close" enough, then $\bw^\top\bz$ will have a larger value. This reveals the meaning behind the ALS where $\bw_m$ represents the features of movie $m$, whilst $\bz_n$ contains the features of user $n$. And each element in $\bw_m$ and $\bz_n$ represents a same feature. For example, it could be that the second feature $w_{m2}$ \footnote{$w_{m2}$ is the second element of vector $\bw_{m}$.} represents if the movie is an action movie or not, and $z_{n2}$ denotes if the user $n$ likes action movies or not. If it happens the case, then $\bw_m^\top\bz_n$ will be large and approximates $a_{mn}$ well.

Note that, in the decomposition $\bA\approx \bW\bZ$, we know the rows of $\bW$ contain the hidden features of the movies, and the columns of $\bZ$ contain the hidden features of the users. However, we cannot identify what are the meanings of the rows of $\bW$ or the columns of $\bZ$. We know they could be something like categories or genres of the movies, that provide some underlying connections between the users and the movies, but we cannot be sure what exactly they are. This is where the terminology ``hidden" comes from.

\section{Gradient Descent}\label{section:als-gradie-descent}
In Equation~\eqref{equation:als-ori-all-wz}, we obtain the column-by-column update directly from the full matrix way in Equation~\eqref{equation:als-regular-final-all} (with regularization considered). Now let's see what's behind the idea. Following Equation~\eqref{equation:als-regularion-full-matrix}, the loss under the regularization:
\begin{equation}
	L(\bW,\bZ)  =||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2, \qquad \lambda_w>0, \lambda_z>0,
\end{equation}
Since we are now considering the minimization of the above loss with respect to $\bz_n$, we can decompose the loss into
\begin{equation}\label{als:gradient-regularization-zn}
\begin{aligned}
L(\bz_n)  &=||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2\\\
&= ||\bW\bz_n-\ba_n||^2 + \lambda_z ||\bz_n||^2 +
\underbrace{\sum_{i\neq n} ||\bW\bz_i-\ba_i||^2  + \lambda_z \sum_{i\neq n}||\bz_i||^2   +    \lambda_w ||\bW||^2 }_{C_{z_n}},
\end{aligned}
\end{equation}
where $C_{z_n}$ is a constant with respect to $\bz_n$, and $\bZ=[\bz_1, \bz_2, \ldots, \bz_N], \bA=[\ba_1,\ba_2, \ldots, \ba_N]$ are the column partitions of $\bZ, \bA$ respectively. Taking the differential
$$
\frac{\partial L(\bz_n)}{\partial \bz_n} = 2\bW^\top\bW\bz_n - 2\bW^\top\ba_n + 2\lambda_z\bz_n,
$$
under which the root is exactly the first update of the column fashion in Equation~\eqref{equation:als-ori-all-wz}:
$$
\bz_n = (\bW^\top\bW+ \lambda_z\bI)^{-1} \bW^\top \ba_n, \gap \text{for $n\in \{1,2,\ldots, N\}$}.
$$
Similarly, we can decompose the loss with respect to $\bw_m$,
\begin{equation}\label{als:gradient-regularization-wd}
\begin{aligned}
L(\bw_m )  &=
||\bW\bZ-\bA||^2 +\lambda_w ||\bW||^2 + \lambda_z ||\bZ||^2\\
&=||\bZ^\top\bW-\bA^\top||^2 +\lambda_w ||\bW^\top||^2 + \lambda_z ||\bZ||^2\\\
&= ||\bZ^\top\bw_m-\bb_n||^2 + \lambda_w ||\bw_m||^2 +
\underbrace{\sum_{i\neq m} ||\bZ^\top\bw_i-\bb_i||^2  + \lambda_w \sum_{i\neq m}||\bw_i||^2   +    \lambda_z ||\bZ||^2 }_{C_{w_m}},
\end{aligned}
\end{equation}
where $C_{w_m}$ is a constant with respect to $\bw_m$, and $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],$ $\bA^\top=[\bb_1,\bb_2,$ $\ldots, \bb_M]$ are the column partitions of $\bW^\top, \bA^\top$ respectively.
Analogously, taking the differential with respect to $\bw_m$, it follows that
$$
\frac{\partial L(\bw_m)}{\partial \bw_m} = 2\bZ\bZ^\top\bw_m - 2\bZ\bb_n + 2\lambda_w\bw_m,
$$
under which the root is exactly the second update of the column fashion in Equation~\eqref{equation:als-ori-all-wz}:
$$
\bw_m = (\bZ\bZ^\top+\lambda_w\bI)^{-1}\bZ\bb_m,  \gap \text{for $m\in \{1,2,\ldots, M\}$} .
$$

Now suppose we write out the iteration number ($k=1,2,\ldots$) as the superscript and we want to find the updates $\{\bz^{(k+1)}_n, \bw^{(k+1)}_m\}$ in the $(k+1)$-th iteration base on $\{\bZ^{(k)}, \bW^{(k)}\}$ in the $k$-th iteration:
$$
\left\{
\begin{aligned}
	\bz^{(k+1)}_n    &\leftarrow \mathop{\arg \min}_{\bz_n^{(k)}} L(\bz_n^{(k)});\\
	\bw_m^{(k+1)}    &\leftarrow \mathop{\arg\min}_{\bw_m^{(k)}} L(\bw_m^{(k)}).
\end{aligned}
\right.
$$
For simplicity, we will be looking at $\bz^{(k+1)}_n    \leftarrow \mathop{\arg \min}_{\bz_n^{(k)}} L(\bz_n^{(k)})$, and the derivation for the update on $\bw_m^{(k+1)}$ will be the same. Suppose we want to approximate $\bz^{(k+1)}_n$ by a \textit{linear update} on $\bz^{(k)}_n$:
$$
\text{Linear Update: }\gap\boxed{\bz^{(k+1)}_n = \bz^{(k)}_n + \eta \bv.}
$$
The problem now turns to the solution of $\bv$ such that
$$
\bv=\mathop{\arg \min}_{\bv} L(\bz^{(k)}_n + \eta \bv) .
$$
By Taylor's formula, $L(\bz^{(k)}_n + \eta \bv)$ can be approximated by
$$
L(\bz^{(k)}_n + \eta \bv) \approx L(\bz^{(k)}_n ) + \eta \bv^\top \nabla  L(\bz^{(k)}_n ),
$$
where $\eta$ is small enough and $\nabla  L(\bz^{(k)}_n )$ is the gradient of $L(\bz)$ at $\bz^{(k)}_n$. Then a search under the condition $||\bv||=1$ given positive $\eta$ is as follows:
$$
\bv=\mathop{\arg \min}_{||\bv||=1} L(\bz^{(k)}_n + \eta \bv) \approx\mathop{\arg \min}_{||\bv||=1}
 \left\{L(\bz^{(k)}_n ) + \eta \bv^\top \nabla  L(\bz^{(k)}_n )\right\}.
$$
This is known as the \textit{greedy search}. The optimal $\bv$ can be obtained by
$$
\bv = -\frac{\nabla L(\bz^{(k)}_n )}{||\nabla L(\bz^{(k)}_n )||},
$$
i.e., $\bv$ is in the opposite direction of $\nabla L(\bz^{(k)}_n )$. Therefore, the update of $\bz_n^{(k+1)}$ is reasonable to be taken as
$$
\bz^{(k+1)}_n =\bz^{(k)}_n + \eta \bv = \bz^{(k)}_n - \eta \frac{\nabla L(\bz^{(k)}_n )}{||\nabla L(\bz^{(k)}_n )||},
$$
which is usually called the \textit{gradient descent}. Similarly, the gradient descent of $\bw_m^{(k+1)}$ is given by
$$
\bw^{(k+1)}_m =\bw^{(k)}_m + \eta \bv = \bw^{(k)}_m - \eta \frac{\nabla L(\bw^{(k)}_m )}{||\nabla L(\bw^{(k)}_m )||}.
$$
The updated procedure on Algorithm~\ref{alg:als-regularizer} by a gradient descent way is then formulated in Algorithm~\ref{alg:als-regularizer-missing-stochas-gradient}.

\begin{algorithm}[h]
	\caption{Alternating Least Squares with Full Entries and Gradient Descent}
	\label{alg:als-regularizer-missing-stochas-gradient}
	\begin{algorithmic}[1]
		\Require $\bA\in \real^{M\times N}$;
		\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
		\State choose a stop criterion on the approximation error $\delta$;
		\State choose regularization parameters $\lambda_w, \lambda_z$, and step size $\eta_w, \eta_z$;
		\State choose maximal number of iterations $C$;
		\State $iter=0$;
		\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
		\State $iter=iter+1$;
		\For{$n=1,2,\ldots, N$}
		\State $\bz^{(k+1)}_n =\bz^{(k)}_n - \eta_z \frac{\nabla L(\bz^{(k)}_n )}{||\nabla L(\bz^{(k)}_n )||}$; \Comment{$n$-th column of $\bZ$}
		\EndFor

		\For{$m=1,2,\ldots, M$}
		\State $\bw^{(k+1)}_m  = \bw^{(k)}_m - \eta_w \frac{\nabla L(\bw^{(k)}_m )}{||\nabla L(\bw^{(k)}_m )||}$;\Comment{$m$-th column of $\bW^\top$}
		\EndFor
		\EndWhile
		\State Output $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$;
	\end{algorithmic}
\end{algorithm}

\paragraph{Geometrical Interpretation of Gradient Descent}
\begin{lemma}[Direction of Gradients]\label{lemm:direction-gradients}
An important fact is that gradients are orthogonal to level curves (a.k.a., level surface).
\end{lemma}
See \citet{lu2021numerical, lu2022gradient} for a proof.

The lemma above reveals the geometrical interpretation of gradient descent. For finding a solution to minimize a convex function $L(\bz)$, gradient descent goes to the negative gradient direction that can decrease the loss. Figure~\ref{fig:alsgd-geometrical} depicts a $2$-dimensional case, where $-\nabla L(\bz)$ pushes the loss to decrease for the convex function $L(\bz)$.
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[A 2-dimensional convex function $L(\bz)$]{\label{fig:alsgd1}
		\includegraphics[width=0.47\linewidth]{./imgs/alsgd1.pdf}}
	\subfigure[$L(\bz)=c$ is a constant]{\label{fig:alsgd2}
		\includegraphics[width=0.44\linewidth]{./imgs/alsgd2.pdf}}
	\caption{Figure~\ref{fig:alsgd1} shows a function ``density" and a contour plot (\textcolor{bluepigment}{blue}=low, \textcolor{canaryyellow}{yellow}=high) where the upper graph is the ``density", and the lower one is the projection of it (i.e., contour). Figure~\ref{fig:alsgd2}: $-\nabla L(\bz)$ pushes the loss to decrease for the convex function $L(\bz)$.}
	\label{fig:alsgd-geometrical}
\end{figure}
\section{Regularization: A Geometrical Interpretation}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.95\textwidth]{./imgs/alsgd3.pdf}
	\caption{Constrained gradient descent with $\bz^\top\bz\leq C$. The \textcolor{green}{green} vector $\bw$ is the projection of $\bv_1$ into $\bz^\top\bz\leq C$ where $\bv_1$ is the component of $-\nabla l(\bz)$ perpendicular to $\bz_1$. The right picture is the next step after the update in the left picture. $\bz^\star$ denotes the optimal solution of \{$\min l(\bz)$\}.}
	\label{fig:alsgd3}
\end{figure}
We have seen in Section~\ref{section:regularization-extention-general} (p.~\pageref{section:regularization-extention-general}) that the regularization can extend the ALS to general matrices. The gradient descent can reveal the geometric meaning of the regularization. To avoid confusion, we denote the loss function without regularization by $l(\bz)$ and the loss with regularization by $L(\bz) = \l(\bz)+\lambda_z ||\bz||^2$ where $l(\bz): \real^n \rightarrow \real$. When minimizing $l(\bz)$, descent method will search in $\real^n$ for a solution. However, in machine learning, searching in the whole space $\real^n$ can cause overfitting. A partial solution is to search in a subset of the vector space, e.g., searching in $\bz^\top\bz < C$ for some constant $C$. That is
$$
\mathop{\arg\min}_{\bz} \gap l(\bz), \qquad s.t., \gap \bz^\top\bz\leq C.
$$
As shown above, a trivial gradient descent method will go further in the direction of $-\nabla l(\bz)$, i.e., update $\bz$ by $\bz\leftarrow \bz-\eta \nabla l(\bz)$ for a small step size $\eta$. When the level curve is $l(\bz)=c_1$ and the current position of $\bz=\bz_1$ where $\bz_1$ is the intersection of $\bz^\top\bz=C$ and $l(\bz)=c_1$, the descent direction $-\nabla l (\bz_1)$ will be perpendicular to the level curve of $l(\bz_1)=c_1$ as shown in the left picture of Figure~\ref{fig:alsgd3} (by Lemma~\ref{lemm:direction-gradients}). However, if we further restrict the optimal value can only be in $\bz^\top\bz\leq C$, the trivial descent direction $-\nabla l(\bz_1)$ will lead $\bz_2=\bz_1-\eta \nabla l(\bz_1)$ outside of $\bz^\top\bz\leq C$. A solution is to decompose the step $-\nabla l(\bz_1)$ into
$$
-\nabla l(\bz_1) = a\bz_1 + \bv_1,
$$
where $a\bz_1$ is the component perpendicular to the curve of $\bz^\top\bz=C$, and $\bv_1$ is the component parallel to the curve of $\bz^\top\bz=C$. Keep only the step $\bv_1$, then the update
$$
\bz_2 = \text{project}(\bz_1+\eta \bv_1) = \text{project}\left(\bz_1 + \eta
\underbrace{(-\nabla l(\bz_1) -a\bz_1)}_{\bv_1}\right)\footnote{where the project($\bx$)
will project the vector $\bx$ to the closest point inside $\bz^\top\bz\leq C$. Notice here the direct update $\bz_2 = \bz_1+\eta \bv_1$ can still make $\bz_2$ outside the curve of $\bz^\top\bz\leq C$.}
$$
will lead to a smaller loss from $l(\bz_1)$ to $l(\bz_2)$ and still match the requirement of $\bz^\top\bz\leq C$. This is known as the \textit{projection gradient descent}. It is not hard to see that the update $\bz_2 = \text{project}(\bz_1+\eta \bv_1)$ is equivalent to finding a vector $\bw$ (shown by the \textcolor{green}{green} vector in the left picture of Figure~\ref{fig:alsgd3}) such that $\bz_2=\bz_1+\bw$ is inside the curve of $\bz^\top\bz\leq C$. Mathematically, the $\bw$ can be obtained by $-\nabla l(\bz_1) -2\lambda \bz_1$ for some $\lambda$ as shown in the middle picture of Figure~\ref{fig:alsgd3}. This is exactly the negative gradient of $L(\bz)=l(\bz)+\lambda||\bz||^2$ such that
$$
\nabla L(\bz) = \nabla l(\bz) + 2\lambda \bz,
$$
and
$$
\begin{aligned}
\bw &= -\nabla L(\bz) \leadto \bz_2 &= \bz_1+ \bw =\bz_1 -  \nabla L(\bz).
\end{aligned}
$$
And in practice, a small step size $\eta$ can avoid going outside the curve of $\bz^\top\bz\leq C$:
$$
\bz_2  =\bz_1 -  \eta\nabla L(\bz),
$$
which is exactly what we have discussed in Section~\ref{section:regularization-extention-general} (p.~\pageref{section:regularization-extention-general}), the regularization term.

\paragraph{Sparsity}
In rare cases, we want to find sparse solution $\bz$ such that $l(\bz)$ is minimized. Regularization to be constrained in $||\bz||_1 \leq C$ exists to this purpose where $||\cdot||_1$ is the $l_1$ norm of a vector or a matrix. Similar to the previous case, the $l_1$ constrained optimization pushes the gradient descent towards the border of the level of $||\bz||_1=C$. The situation in the 2-dimensional case is shown in Figure~\ref{fig:alsgd4}. In a high-dimensional case, many elements in $\bz$ will be pushed into the breakpoint of $||\bz||_1=C$ as shown in the right picture of Figure~\ref{fig:alsgd4}.
\begin{figure}[h]
	\centering
	\includegraphics[width=0.95\textwidth]{./imgs/alsgd4.pdf}
	\caption{Constrained gradient descent with $||\bz||_1\leq C$, where the \textcolor{red}{red} dot denotes the breakpoint in $l_1$ norm. The right picture is the next step after the update in the left picture. $\bz^\star$ denotes the optimal solution of \{$\min l(\bz)$\}.}
	\label{fig:alsgd4}
\end{figure}

\section{Stochastic Gradient Descent}
We consider again the per-example loss:
$$
L(\bW,\bZ)=  \sum_{n=1}^N \sum_{m=1}^{M} \left(a_{mn} - \bw_m^\top\bz_n\right)^2 + \lambda_w||\bw_m||^2 +\lambda_z||\bz_n||^2.
$$
And when we iteratively decrease the per-example loss term $l(\bw_m, \bz_n)=\left(a_{mn} - \bw_m^\top\bz_n\right)^2$ for all $m\in \{1,2,\ldots, M\}, n\in\{1,2,\ldots,N\}$, the full loss $L(\bW,\bZ)$ can also be decreased. This is known as the \textit{stochastic coordinate descent}. The differentials with respect to $\bw_m, \bz_n$, and their roots are given by
$$
\left\{
\begin{aligned}
\nabla l(\bz_n)=\frac{\partial l(\bw_m,\bz_n)}{\partial \bz_n} &= 2\bw_m\bw_m^\top \bz_n + 2\lambda_w\bw_m -2a_{mn} \bw_m \\
&\qquad \leadtosmall \bz_n= a_{mn}(\bw_m\bw_m^\top+\lambda_z\bI)^{-1}\bw_m;\\
\nabla l(\bw_m)=\frac{\partial l(\bw_m, \bz_n)}{\partial \bw_m} &= 2\bz_n\bz_n^\top\bw_m +2\lambda_z\bz_n - 2a_{mn}\bz_n\\
&\qquad \leadtosmall \bw_m= a_{mn}(\bz_n\bz_n^\top+\lambda_w\bI)^{-1}\bw_n.
\end{aligned}
\right.
$$
or analogously, the update can be done by gradient descent, and since we update by per-example loss, it is also known as the \textit{stochastic gradient descent}
$$
\left\{
\begin{aligned}
\bz_n&= \bz_n - \eta_z \frac{\nabla l(\bz_n)}{||\nabla l(\bz_n)||}; \\
\bw_m&= \bw_m - \eta_w \frac{\nabla l(\bw_m)}{||\nabla l(\bw_m)||}.
\end{aligned}
\right.
$$
The stochastic gradient descent update for ALS is formulated in Algorithm~\ref{alg:als-regularizer-missing-stochas-gradient-realstoch}. And in practice, the $m,n$ in the algorithm can be randomly produced, that's where the name \textit{stochastic} comes from.
\begin{algorithm}[h]
\caption{Alternating Least Squares with Full Entries and Stochastic Gradient Descent}
\label{alg:als-regularizer-missing-stochas-gradient-realstoch}
\begin{algorithmic}[1]
\Require $\bA\in \real^{M\times N}$;
\State initialize $\bW\in \real^{M\times K}$, $\bZ\in \real^{K\times N}$ \textcolor{blue}{randomly without condition on the rank and the relationship between $M, N, K$};
\State choose a stop criterion on the approximation error $\delta$;
\State choose regularization parameters $\lambda_w, \lambda_z$, and step size $\eta_w, \eta_z$;
\State choose maximal number of iterations $C$;
\State $iter=0$;
\While{$||  \bA- (\bW\bZ)||^2>\delta $ and $iter<C$}
\State $iter=iter+1$;
\For{$n=1,2,\ldots, N$}
\For{$m=1,2,\ldots, M$} \Comment{in practice, $m,n$ can be randomly produced}
\State $\bz_n= \bz_n - \eta_z \frac{\nabla l(\bz_n)}{||\nabla l(\bz_n)||}$;\Comment{$n$-th column of $\bZ$}
\State $\bw_m= \bw_m - \eta_w \frac{\nabla l(\bw_m)}{||\nabla l(\bw_m)||}$;\Comment{$m$-th column of $\bW^\top$}
\EndFor
\EndFor

\EndWhile
\State Output $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M],\bZ=[\bz_1, \bz_2, \ldots, \bz_N]$;
\end{algorithmic}
\end{algorithm}

\section{Bias Term}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.95\textwidth]{./imgs/als-bias.pdf}
	\caption{Bias terms in alternating least squares where the \textcolor{canaryyellow}{yellow} entries denote ones (which are fixed) and \textcolor{cyan}{cyan} entries denote the added features to fit the bias terms. The dotted boxes give an example of how the bias terms work.}
	\label{fig:als-bias}
\end{figure}
In ordinary least squares, a bias term is added to the raw matrix. A similar idea can be applied to the ALS problem. We can add a fixed column with all 1's to the last column of $\bW$, thus an extra row should be added to the last row of $\bZ$ to fit the features introduced by the bias term in $\bW$. Analogously, a fixed row with all 1's can be added to the first row of $\bZ$, and an extra column in the first column of $\bW$ to fit the features. The situation is shown in Figure~\ref{fig:als-bias}.

Following the loss with respect to the columns of $\bZ$ in Equation~\eqref{als:gradient-regularization-zn}, suppose $\widetildebz_n
=
\begin{bmatrix}
	1\\
	\bz_n
\end{bmatrix}
$ is the $n$-th column of $\widetildebZ$, we have
\begin{equation}
	\begin{aligned}
		L(\bz_n)  &=||\widetildebW\widetildebZ-\bA||^2 +\lambda_w ||\widetildebW||^2 + \lambda_z ||\widetildebZ||^2\\\
		&= \left\Vert
		\widetildebW
		\begin{bmatrix}
			1 \\
			\bz_n
		\end{bmatrix}-\ba_n
		\right\Vert^2
		+
		\underbrace{\lambda_z ||\widetildebz_n||^2}_{=\lambda_z ||\bz_n||^2+\lambda_z}
		+
		\sum_{i\neq n} ||\widetildebW\widetildebz_i-\ba_i||^2  + \lambda_z \sum_{i\neq n}||\widetildebz_i||^2   +    \lambda_w ||\widetildebW||^2 \\
		&= \left\Vert
		\begin{bmatrix}
			\widebarbw_0 & \widebarbW
		\end{bmatrix}
		\begin{bmatrix}
			1 \\
			\bz_n
		\end{bmatrix}-\ba_n
		\right\Vert^2
		+ \lambda_z ||\bz_n||^2 + C_{z_n}
		= \left\Vert
		\widebarbW \bz_n -
		\underbrace{(\ba_n-\widebarbw_0)}_{\widebarba_n}
		\right\Vert^2
		+ \lambda_z ||\bz_n||^2 + C_{z_n},
	\end{aligned}
\end{equation}
where $\widebarbw_0$ is the first column of $\widetildebW$ and $C_{z_n}$ is a constant with respect to $\bz_n$. Let $\widebarba_n = \ba_n-\widebarbw_0$, the update of $\bz_n$ is just like the one in Equation~\eqref{als:gradient-regularization-zn} where the differential is given by:
$$
\frac{\partial L(\bz_n)}{\partial \bz_n} = 2\widebarbW^\top\widebarbW\bz_n - 2\widebarbW^\top\widebarba_n + 2\lambda_z\bz_n.
$$
Therefore the update on $\bz_n$ is given by the root of the above differential:
$$
\text{update on $\widetildebz_n$ is }
\left\{
\begin{aligned}
	\bz_n &= (\widebarbW^\top\widebarbW+ \lambda_z\bI)^{-1} \widebarbW^\top \widebarba_n, \gap \text{for $n\in \{1,2,\ldots, N\}$};\\
	\widetildebz_n &= \begin{bmatrix}
		1\\\bz_n
	\end{bmatrix}.
\end{aligned}
\right.
$$
Similarly, following the loss with respect to each row of $\bW$ in Equation~\eqref{als:gradient-regularization-wd}, suppose $\widetildebw_m =
\begin{bmatrix}
	\bw_m \\
	1
\end{bmatrix}$ is the $m$-th row of $\widetildebW$ (or $m$-th column of $\widetildebW^\top$), we have
\begin{equation}
	\begin{aligned}
		L(\bw_m )
		&=||\widetildebZ^\top\widetildebW-\bA^\top||^2 +\lambda_w ||\widetildebW^\top||^2 + \lambda_z ||\widetildebZ||^2\\\
		&=
		||\widetildebZ^\top\widetildebw_m-\bb_m||^2 +
		\underbrace{\lambda_w ||\widetildebw_m||^2}_{=\lambda_w ||\bw_m||^2+\lambda_w}
		+
		\sum_{i\neq m} ||\widetildebZ^\top\widetildebw_i-\bb_i||^2  + \lambda_w \sum_{i\neq m}||\widetildebw_i||^2   +    \lambda_z ||\widetildebZ||^2 \\
		&=
		\left\Vert
		\begin{bmatrix}
			\widebarbZ^\top&
			\widebarbz_0
		\end{bmatrix}
		\begin{bmatrix}
			\bw_m \\
			1
		\end{bmatrix}
		-\bb_m\right\Vert^2 +
		\lambda_w ||\bw_m||^2
		+
		C_{w_m}\\
		&=
		\left\Vert
		\widebarbZ^\top\bw_m
		-(\bb_m-\widebarbz_0) \right\Vert^2+
		\lambda_w ||\bw_m||^2
		+
		C_{w_m},
	\end{aligned}
\end{equation}
where $\widebarbz_0$ is the last column of $\widetildebZ^\top$ and $\widebarbZ^\top$ contains the remaining columns of it,
$C_{w_m}$ is a constant with respect to $\bw_m$, and $\bW^\top=[\bw_1, \bw_2, \ldots, \bw_M], \bA^\top=[\bb_1,\bb_2, \ldots, \bb_M]$ are the column partitions of $\bW^\top, \bA^\top$ respectively. Let $\widebarbb_m = \bb_m-\widebarbz_0$, the update of $\bw_m$ is again just like the one in Equation~\eqref{als:gradient-regularization-wd} where the differential is given by:
$$
\frac{\partial L(\bw_md)}{\partial \bw_m} = 2\widebarbZ\cdot \widebarbZ^\top\bw_m - 2\widebarbZ\cdot \widebarbb_m + 2\lambda_w\bw_m.
$$
Therefore the update on $\bw_m$ is given by the root of the above differential
$$
\text{update on $\widetildebw_m$ is }
\left\{
\begin{aligned}
	\bw_m&=(\widebarbZ\cdot \widebarbZ^\top+\lambda_w\bI)^{-1}\widebarbZ\cdot \widebarbb_m,  \gap \text{for $m\in \{1,2,\ldots, M\}$} ;\\
	\widetildebw_m &= \begin{bmatrix}
		\bw_m \\ 1
	\end{bmatrix}.
\end{aligned}
\right.
$$
Similar updates by gradient descent under the bias terms or treatment on missing entries can be deduced and we shall not repeat the details (see Section~\ref{section:als-gradie-descent}, p.~\pageref{section:als-gradie-descent} and \ref{section:alt-columb-by-column}, p.~\pageref{section:alt-columb-by-column} for a reference).

\chapter{Biconjugate Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Existence of the Biconjugate Decomposition}
The biconjugate decomposition was proposed in \citet{chu1995rank} and discussed in \citet{yang2000matrix}.
A variety of matrix decomposition methods can be unified via this biconjugate decomposition.
And the existence of the biconjugate decomposition relies on the rank-one reduction theorem shown below.
\begin{theorem}[Rank-One Reduction\index{Rank-one reduction}]\label{theorem:rank-1-reduction}
Given $m\times n$ matrix $\bA\in \real^{m\times n}$ with rank $r$, a pair of vectors $\bx\in \real^n$ and $\by\in \real^m$ such that $w=\by^\top\bA\bx \neq 0$, then the matrix $\bB=\bA-w^{-1}\bx\by^\top\bA$ has rank $r-1$ which has exactly one less than the rank of $\bA$.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:rank-1-reduction}]
If we can show that the dimension of the null space $\nspace(\bB)$ is one larger than that of $\bA$. Then this implicitly shows $\bB$ has a rank exactly one less than the rank of $\bA$.

For any vector $\bn \in \nspace(\bA)$, i.e., $\bA\bn=\bzero$, we then have $\bB\bn =\bA\bn-w^{-1}\bx\by^\top\bA\bn=\bzero $ which means $\nspace(\bA)\subseteq \nspace(\bB)$.

Now for any vector $\bmm \in \nspace(\bB)$, then $\bB\bmm = \bA\bmm-w^{-1}\bx\by^\top\bA\bmm =\bzero$.

Let $k=w^{-1}\by^\top\bA\bmm$, which is a scalar, thus $\bA(\bmm - k\bx)=\bzero$, i.e., for any vector $\bn\in \nspace(\bA)$, we could find a vector $\bmm\in \nspace(\bB)$ such that $\bn=(\bmm - k\bx)\in \nspace(\bA)$. Note that $\bA\bx\neq \bzero$ from the definition of $w$. Thus, the null space of $\bB$ is therefore obtained from the null space of $\bA$ by adding $\bx$ to its basis which will increase the order of the space by 1. Thus the dimension of $\nspace(\bA)$ is smaller than the dimension of $\nspace(\bB)$ by 1 which completes the proof.
\end{proof}

Suppose matrix $\bA\in \real^{m\times n}$ has rank $r$, we can define a rank reducing process to generate a sequence of \textit{Wedderburn matrices} $\{\bA_k\}$:
$$
\bA_1 = \bA, \qquad \text{and}\qquad \bA_{k+1} = \bA_k-w_k^{-1}\bA_k\bx_k\by_k^\top\bA_k,
$$
where $\bx_k \in \real^n$ and $\by_k\in \real^m$ are any vectors satisfying $w_k = \by_k^\top\bA\bx_k \neq 0$. The sequence will terminate in $r$ steps since the rank of $\bA_k$ decreases by exactly one at each step. Write out the sequence:
$$
\begin{aligned}
	\bA_1 &= \bA, \\
\bA_1-\bA_{2} &= w_1^{-1}\bA_1\bx_1\by_1^\top\bA_1,\\
\bA_2-\bA_3 &=w_2^{-1}\bA_2\bx_2\by_2^\top\bA_2, \\
\bA_3-\bA_4 &=w_3^{-1}\bA_3\bx_3\by_3^\top\bA_3, \\
\vdots &=\vdots\\
\bA_{r-1}-\bA_{r} &=w_{r-1}^{-1}\bA_{r-1}\bx_{r-1}\by_{r-1}^\top\bA_{r-1}, \\
\bA_r-\bzero &=w_{r}^{-1}\bA_{r}\bx_{r}\by_{r}^\top\bA_{r}.
\end{aligned}
$$
By adding the sequence we get
$$
(\bA_1-\bA_2)+(\bA_2-\bA_3)+\ldots+(\bA_{r-1}-\bA_{r})+(\bA_r-\bzero ) =\bA= \sum_{i=1}^{r}w_i^{-1}\bA_i\bx_i\by_i^\top\bA_i.
$$
\begin{theoremHigh}[Biconjugate Decomposition: Form 1]\label{theorem:biconjugate-form1}
This equality from rank-reducing process implies the following matrix decomposition
$$
\bA = \bPhi \bOmega^{-1} \bPsi^\top,
$$
where $\bOmega=diag(w_1, w_2, \ldots, w_r)$, $\bPhi=[\bphi_1,\bphi_2, \ldots, \bphi_r]\in \real^{m\times r}$ and $\bPsi=[\bpsi_1, \bpsi_2, \ldots, \bpsi_r]$ with
$$
\bphi_k = \bA_k\bx_k, \qquad \text{and}\qquad \bpsi_k=\bA_k^\top \by_k.
$$
\end{theoremHigh}
Obviously, different choices of $\bx_k$'s and $\by_k$'s will result in different factorizations. So this factorization is rather general and we will show its connection to some well-known decomposition methods.\index{Wedderburn sequence}

\begin{remark}
For the vectors $\bx_k, \by_k$ in the Wedderburn sequence, we have the following property
$$
\begin{aligned}
\bx_k &\in \nspace(\bA_{k+1}) \bot \cspace(\bA_{k+1}^\top), \\
\by_k &\in \nspace(\bA_{k+1}^\top) \bot \cspace(\bA_{k+1}).
\end{aligned}
$$
\end{remark}

\begin{lemma}[General Term Formula of Wedderburn Sequence: V1]\label{lemma:wedderburn-sequence-general}
For each matrix with $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$, then $\bA_{k+1} $ can be written as
$$
\bA_{k+1} = \bA - \sum_{i=1}^{k}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA,
$$
where
$$
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{w_i}\bu_i,\qquad \text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\by_k^\top \bA\bu_i}{w_i}\bv_i.
$$
\end{lemma}

The proof of this lemma is delayed in Section~\ref{appendix:wedderburn-general-term} (p.~\pageref{appendix:wedderburn-general-term}). We notice that $w_i =\by_k^\top\bA_k\bx_k$ in the general term formula is related to $\bA_k$. So it's not the true general term formula. We will reformulate $w_i$ to be related to $\bA$ rather than $\bA_k$ later.
From the general term formula of the Wedderburn sequence, we have
$$
\begin{aligned}
\bA_{k+1} &= \bA - \sum_{i=1}^{k}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA \\
\bA_{k} &= \bA - \sum_{i=1}^{k-1}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA.
\end{aligned}
$$
Thus, $\bA_{k+1} - \bA_{k} = -w_{k}^{-1} \bA\bu_k \bv_k^\top \bA$. Since we define the sequence by $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$. We then find $w_{k}^{-1} \bA\bu_k \bv_k^\top \bA = w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$. It is trivial to see
\begin{equation}\label{equation:wedderburn-au-akxk}
\begin{aligned}
\bA\bu_k &=\bA_k\bx_k,\\
\bv_k^\top \bA&=\by_k^\top \bA_k.
\end{aligned}
\end{equation}

Let $z_{k,i} = \frac{\bv_i^\top \bA\bx_k}{w_i}$ which is a scalar. From the definition of $\bu_k$ and $\bv_k$ in the above lemma, then

$\bullet$ $\bu_1=\bx_1$;

$\bullet$ $\bu_2 = \bx_2 - z_{2,1}\bu_1$;

$\bullet$ $\bu_3 = \bx_3 - z_{3,1}\bu_1-z_{3,2}\bu_2$;

$\bullet$ $\ldots$.

This process is just similar to the Gram-Schmidt process (Section~\ref{section:gram-schmidt-process}, p.~\pageref{section:gram-schmidt-process}). But now, we do not project $\bx_2$ onto $\bx_1$ with the smallest distance. The vector of $\bx_2$ along $\bx_1$ is now defined by $z_{2,1}$. This process is shown in Figure~\ref{fig:projection-wedd}. In Figure~\ref{fig:project-line-wedd}, $\bu_2$ is not perpendicular to $\bu_1$. But $\bu_2$ does not lie on the same line as $\bu_1$ so that $\bu_1, \bu_2$ still could span a $\real^2$ subspace. Similarly, in Figure~\ref{fig:project-space-wedd}, $\bu_3= \bx_3 - z_{3,1}\bu_1-z_{3,2}\bu_2$ does not lie in the space spanned by $\bu_1, \bu_2$ so that $\bu_1, \bu_2, \bu_3$ could still span a $\real^3$ subspace.

A moment of reflexion would reveal that the span of $\bx_2, \bx_1$ is the same as the span of $\bu_2, \bu_1$. Similarly for $\bv_i$'s. We have the following property:
\begin{equation}\label{equation:wedderburn-span-same}
	\left\{
\begin{aligned}
	\textrm{span}\{\bx_1, \bx_2, \ldots, \bx_j\} &= \textrm{span}\{\bu_1, \bu_2, \ldots, \bu_j\};\\
	\textrm{span}\{\by_1, \by_2, \ldots, \by_j\} &= \textrm{span}\{\bv_1, \bv_2, \ldots, \bv_j\}.\\
\end{aligned}
\right.
\end{equation}

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[``Project" onto a line]{\label{fig:project-line-wedd}
		\includegraphics[width=0.47\linewidth]{./imgs/projectline_wedderburn.pdf}}
	\quad
	\subfigure[``Project" onto a space]{\label{fig:project-space-wedd}
		\includegraphics[width=0.47\linewidth]{./imgs/projectspace_wedderburn.pdf}}
	\caption{``Project" a vector onto a line and onto a space.}
	\label{fig:projection-wedd}
\end{figure}

Further, from the rank-reducing property in the Wedderburn sequence, we have
$$
\left\{
\begin{aligned}
	\cspace(\bA_1) &\supset \cspace(\bA_2) \supset \cspace(\bA_3) \supset \ldots;\\
	\nspace(\bA_1^\top) &\subset \nspace(\bA_2^\top) \subset \nspace(\bA_3^\top) \subset \ldots.
\end{aligned}
\right.
$$
Since $\by_k \in \nspace(\bA_{k+1}^\top)$, it then follows that $\by_j \in \nspace(\bA_{k+1}^\top)$ for all $j<k+1$, i.e., $\bA_{k+1}^\top \by_j=\bzero$ for all $j<k+1$. Which also holds true for $\bx_{k+1}^\top \bA_{k+1}^\top \by_j=0$ for all $j<k+1$. From Equation~\eqref{equation:wedderburn-au-akxk}, we also have $\bu_{k+1}^\top \bA^\top \by_j=0$ for all $j<k+1$. Following Equation~\eqref{equation:wedderburn-span-same}, we obtain
$$
\bv_j^\top\bA\bu_{k+1}=0 \text{ for all } j<k+1.
$$
Similarly, we can prove
$$
\bv_{k+1}^\top\bA\bu_{j}=0 \text{ for all } j<k+1.
$$
Moreover, we defined $w_k = \by_k^\top \bA_k\bx_k$. By Equation~\eqref{equation:wedderburn-au-akxk}, we can write the $w_k$ as:
$$
\begin{aligned}
	w_k &= \by_k^\top \bA_k\bx_k\\
	&=\bv_k^\top \bA\bx_k \\
	&=\bv_k^\top\bA (\bu_k +\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{w_i}\bu_i) \qquad &(\text{by the definition of }\bu_k \text{ in Lemma~\ref{lemma:wedderburn-sequence-general}})\\
	&=\bv_k^\top\bA \bu_k,\qquad &(\text{by } \bv_{k}^\top\bA\bu_{j}=0 \text{ for all } j<k)
\end{aligned}
$$
which can be utilized to substitute the $w_k$ in Lemma~\ref{lemma:wedderburn-sequence-general} and we then have the full version of the general term formula of the Wedderburn sequence such that the formula does not depend on $\bA_k$'s (in the form of $w_k$'s) with
\begin{equation}\label{equation:uk-vk-to-mimic-gram-process}
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{\textcolor{blue}{\bv_i^\top\bA\bu_i}}\bu_i,\qquad \text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\by_k^\top \bA\bu_i}{\textcolor{blue}{\bv_i^\top\bA\bu_i}}\bv_i.
\end{equation}

\paragraph{Gram-Schmidt Process from Wedderburn Sequence} If $\bX=[\bx_1,
\bx_2, \ldots, \bx_r]\in \real^{n\times r}$, $\bY=[\by_1, \by_2, \ldots, \by_r]\in \real^{m\times r}$ effect a rank-reducing process for $\bA$. Let $\bA$ be the identity matrix and $(\bX, \bY)$ be identical and contain the vectors for which an orthogonal basis is desired, then $(\bU = \bV)$ gives the resultant orthogonal basis.

This form of $\bu_k$ and $\bv_k$ in Equation~\eqref{equation:uk-vk-to-mimic-gram-process} is very close to the projection to the perpendicular space of the Gram-Schmidt process in Equation~\eqref{equation:gram-schdt-eq2} (p.~\pageref{equation:gram-schdt-eq2}). We then define \fbox{$<\bx, \by>:=\by^\top\bA\bx$} to explicitly mimic the form of projection in Equation~\eqref{equation:gram-schdt-eq2}.
We formulate the results so far in the following lemma which can help us have a clear vision of what we have been working on and we will use these results extensively in the sequel:
\begin{lemma}[Properties of Wedderburn Sequence]\label{lemma:wedderburn-sequence-general-v2}
For each matrix with $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$, then $\bA_{k+1} $ can be written as
$$
\bA_{k+1} = \bA - \sum_{i=1}^{k}w_{i}^{-1} \bA\bu_i \bv_i^\top \bA,
$$
where
\begin{equation}\label{equation:properties-of-wedderburn-ukvk}
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\textcolor{blue}{<\bx_k, \bv_i>}}{\textcolor{blue}{<\bu_i,\bv_i>}}\bu_i,\qquad
\text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\textcolor{blue}{<\bu_i,\by_k>}}{\textcolor{blue}{<\bu_i,\bv_i>}}\bv_i.
\end{equation}
Further, we have the following properties:
\begin{equation}\label{equation:wedderburn-au-akxk-2}
	\begin{aligned}
		\bA\bu_k &=\bA_k\bx_k,\\
		\bv_k^\top \bA&=\by_k^\top \bA_k.
	\end{aligned}
\end{equation}
\begin{equation}
<\bu_k, \bv_j>=<\bu_j, \bv_k>=0 \text{ for all } j<k.
\end{equation}
\begin{equation}\label{equation:wk-by-ukvk}
	w_k = \by_k^\top\bA_k\bx_k = <\bu_k, \bv_k>
\end{equation}

\end{lemma}

By substituting Equation~\eqref{equation:wedderburn-au-akxk-2} into Form 1 of biconjugate decomposition, and using Equation~\eqref{equation:wk-by-ukvk} which implies $w_k = \bv_k^\top\bA\bu_k$, we have the Form 2 and Form 3 of this decomposition:
\begin{theoremHigh}[Biconjugate Decomposition: Form 2 and Form 3]\label{theorem:biconjugate-form2}
The equality from rank-reducing process implies the following matrix decomposition
$$
\bA = \bA\bU_r \bOmega_r^{-1} \bV_r^\top\bA,
$$
where $\bOmega_r=diag(w_1, w_2, \ldots, w_r)$, $\bU_r=[\bu_1,\bu_2, \ldots, \bu_r]\in \real^{m\times r}$ and  $\bV_r=[\bv_1, \bv_2,$ $\ldots,$ $\bv_r] \in \real^{n\times r}$ with
\begin{equation}\label{equation:properties-of-wedderburn-ukvk2-inform2}
\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{<\bx_k, \bv_i>}{<\bu_i,\bv_i>}\bu_i,\qquad
\text{and}\qquad \bv_k=\by_k -\sum_{i=1}^{k-1}\frac{<\bu_i,\by_k>}{<\bu_i,\bv_i>}\bv_i.
\end{equation}
And also the following decomposition
\begin{equation}\label{equation:wedderburn-vgamma-ugamma}
\bV_\gamma^\top \bA \bU_\gamma = \bOmega_\gamma,
\end{equation}
where $\bOmega_\gamma=diag(w_1, w_2, \ldots, w_\gamma)$, $\bU_\gamma=[\bu_1,\bu_2, \ldots, \bu_\gamma]\in \real^{m\times \gamma}$ and $\bV_\gamma=[\bv_1, \bv_2,$ $\ldots,$ $\bv_\gamma]\in \real^{n\times \gamma}$. Note the difference between the subscripts $r$ and $\gamma$ we used here with $\gamma \leq r$.
\end{theoremHigh}
We notice that, in these two forms of biconjugate decomposition, they are independent of the Wedderburn matrices $\{\bA_k\}$.

\paragraph{A word on the notation} We will use the subscript to indicate the dimension of the matrix to avoid confusion in the sequel, e.g., the $r, \gamma$ in the above theorem.

\section{Properties of the Biconjugate Decomposition}
\begin{corollary}[Connection of $\bU_\gamma$ and $\bX_\gamma$]\label{corollary:biconjugate-connection-u-x}
If $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$ effects a rank-reducing process for $\bA$, then there are unique unit upper triangular matrices $\bR_\gamma^{(x)}\in\real^{\gamma\times \gamma}$ and $\bR_\gamma^{(y)}\in\real^{\gamma\times \gamma}$ such that
$$
\bX_\gamma = \bU_\gamma \bR_\gamma^{(x)}, \qquad \text{and} \qquad \bY_\gamma=\bV_\gamma\bR_\gamma^{(y)},
$$
where $\bU_\gamma$ and $\bV_\gamma$ are matrices with columns resulting from the Wedderburn sequence as in Equation~\eqref{equation:wedderburn-vgamma-ugamma}.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:biconjugate-connection-u-x}]
The proof is trivial from the definition of $\bu_k$ and $\bv_k$ in Equation~\eqref{equation:properties-of-wedderburn-ukvk} or Equation~\eqref{equation:properties-of-wedderburn-ukvk2-inform2} by setting the $j$-th column of $\bR_\gamma^{(x)}$ and $\bR_\gamma^{(y)}$ as
$$
\left[\frac{<\bx_j,\bv_1>}{<\bu_1,\bv_1>}, \frac{<\bx_j,\bv_2>}{<\bu_2,\bv_2>}, \ldots, \frac{<\bx_j,\bv_{j-1}>}{<\bu_{j-1},\bv_{j-1}>}, 1, 0, 0, \ldots, 0  \right]^\top,
$$
and
$$
\left[\frac{<\bu_1, \by_j>}{<\bu_1,\bv_1>},\frac{<\bu_2, \by_j>}{<\bu_2,\bv_2>}, \ldots, \frac{<\bu_{j-1}, \by_j>}{<\bu_{j-1},\bv_{j-1}>}, 1, 0, 0, \ldots, 0  \right]^\top.
$$
This completes the proof.
\end{proof}

The $(\bU_\gamma, \bV_\gamma) \in \real^{m\times \gamma}\times \real^{n\times \gamma}$ in Theorem~\ref{theorem:biconjugate-form2} is called a \textbf{biconjugate pair} with respect to $\bA$ if $\Omega_\gamma$ is nonsingular and diagonal. And let $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$ effect a rank-reducing process for $\bA$, then $(\bX_\gamma, \bY_\gamma)$ is said to be \textbf{biconjugatable} and \textbf{biconjugated into a biconjugate pair} of matrices $(\bU_\gamma, \bV_\gamma)$, if there exist unit upper triangular matrices $\bR_\gamma^{(x)},\bR_\gamma^{(y)}$ such that $\bX_\gamma = \bU_\gamma \bR_\gamma^{(x)}$ and $\bY_\gamma=\bV_\gamma\bR_\gamma^{(y)}$.

\section{Connection to Well-Known Decomposition Methods}

\subsection{LDU Decomposition}
\begin{theorem}[LDU, \cite{chu1995rank} Theorem 2.4]\label{theorem:biconjugate-ldu}
If $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$
and $\bA\in \real^{m\times n}$ with $\gamma$ in $\{1, 2, \ldots, r\}$. Then $(\bX_\gamma, \bY_\gamma)$ can be biconjugated if and only if $\bY_\gamma^\top\bA\bX_\gamma$ has an LDU decomposition.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:biconjugate-ldu}]
Suppose $\bX_\gamma$ and $\bY_\gamma$ are biconjugatable, then, there exists unit upper triangular matrices $\brx$ and $\bry$ such that $\bxgamma = \bugamma\brx$, $\bygamma = \bvgamma\bry$ and $\bvgamma^\top\bA\bugamma = \bomegagamma$ is a nonsingular diagonal matrix. Then, it follows that
$$
\bygamma^\top\bA\bxgamma = \bryt \bvgamma^\top \bA \bugamma\brx = \bryt \bomegagamma \brx
$$
is the unique unit triangular LDU decomposition of $\bygamma^\top\bA\bxgamma$. This form above can be seen as the \textbf{fourth form of biconjugate decomposition}.

Conversely, suppose $\bygamma^\top\bA\bxgamma = \bR_2^\top \bD\bR_1$ is an LDU decomposition with both $\bR_1$ and $\bR_2$ being unit upper triangular matrices. Then since $\bR_1^{-1}$ and $\bR_2^{-1}$ are also unit upper triangular matrices, and $(\bX_\gamma, \bY_\gamma)$ biconjugates into $(\bX_\gamma\bR_1^{-1}, \bY_\gamma\bR_2^{-1})$.
\end{proof}

\begin{corollary}[Determinant]\label{corollary:lu-determinant}
Suppose $(\bX_\gamma, \bY_\gamma) \in \real^{n\times \gamma}\times \real^{m\times \gamma}$ are biconjugatable. Then
$$
\det(\bygamma^\top \bA\bxgamma) = \prod_{i=1}^{\gamma} w_i.
$$
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:lu-determinant}]
By Theorem~\ref{theorem:biconjugate-ldu}, since $(\bX_\gamma, \bY_\gamma)$ are biconjugatable, then there are unit upper triangular matrices $\brx$ and $\bry$ such that $\bygamma^\top\bA\bxgamma = \bryt \bomegagamma \brx$. The determinant is just the product of the trace.
\end{proof}

\begin{lemma}[Biconjugatable in Principal Minors]\label{lemma:Biconjugatable-in-Principal-Minors}
Let $r=rank(\bA) \geq \gamma$ with $\bA\in \real^{m\times n}$. In the Wedderburn sequence, take $\bx_i$ as the $i$-th basis in $\real^n$ for $i \in \{1, 2, \ldots, \gamma\}$ (i.e., $\bx_i = \be_i \in \real^n$) and $\by_i$ as the $i$-th basis in $\real^m$ for $i \in \{1, 2, \ldots, \gamma\}$ (i.e., $\by_i=\be_i \in
\real^m$). That is $\bygamma^\top \bA\bxgamma$ is the leading principal submatrix of $\bA$, i.e., $\bygamma^\top \bA\bxgamma = \bA_{1:\gamma, 1:\gamma}$. Then, $(\bX_\gamma, \bY_\gamma)$ is biconjugatable if and only if the $\gamma$-th leading principal minor of $\bA$ is nonzero. In this case, the $\gamma$-th leading principal minor of $\bA$ is given by $\prod_{i=1}^{\gamma} w_i$.\index{Leading principal minors}
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:Biconjugatable-in-Principal-Minors}]
The proof is trivial that the $\gamma$-th leading principal minor of $\bA$ is nonzero will imply that $w_i \neq 0$ for all $i\leq \gamma$. Thus the Wedderburn sequence can be successfully obtained. The converse holds since Corollary~\ref{corollary:lu-determinant} implies $\det(\bygamma^\top \bA\bxgamma)$ is nonzero.
\end{proof}

We thus finally come to the LDU decomposition for square matrices.
\begin{theorem}[LDU: Biconjugate Decomposition for Square Matrices]\label{theorem:biconjugate-square-ldu}
For any matrix $\bA\in \real^{n\times n}$, $(\bI_n, \bI_n)$ is biconjugatable if and only if all the leading principal minors of $\bA$ are nonzero. In this case, $\bA$ can be factored as
$$
\bA = \bV_n^{-\top} \bOmega_n \bU_n^{-1} = \bL\bD\bU,
$$
where $\bOmega_n = \bD$ is a diagonal matrix with nonzero values on the diagonal, $\bV_n^{-\top} = \bL$ is a unit lower triangular matrix, and $\bU_n^{-1} = \bU$ is a unit upper triangular matrix.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:biconjugate-square-ldu}]
From Lemma~\ref{lemma:Biconjugatable-in-Principal-Minors}, it is trivial that $(\bI_n, \bI_n)$ is biconjugatable. From Corollary~\ref{corollary:biconjugate-connection-u-x}, we have $\bU_n \bR_n^{(x)} = \bI_n$ and $\bI_n=\bV_n\bR_n^{(y)}$, thus $\bR_n^{(x)} = \bU_n^{-1}$ and $\bR_n^{(y)} = \bV_n^{-1}$ are well defined and we complete the proof.
\end{proof}

\subsection{Cholesky Decomposition}
For symmetric and positive definite matrices, the leading principal minors are positive for sure. The proof is provided in Section~\ref{appendix:leading-minors-pd} (p.~\pageref{appendix:leading-minors-pd}).
\begin{theorem}[Cholesky: Biconjugate Decomposition for PD Matrices]\label{theorem:biconjugate-square-cholesky}
For any symmetric and positive definite matrix $\bA\in \real^{n\times n}$, the Cholesky decomposition of $\bA$ can be obtained from the Wedderburn sequence applied to $(\bI_n, \bI_n)$ as $(\bX_n, \bY_n)$.
In this case, $\bA$ can be factored as
$$
\bA = \bU_n^{-\top} \bOmega_n \bU_n^{-1} = (\bU_n^{-\top} \bOmega_n^{1/2})( \bOmega_n^{1/2} \bU_n^{-1}) =\bR^\top\bR,
$$
where $\bOmega_n$ is a diagonal matrix with positive values on the diagonal, and $\bU^{-1}$ is a unit upper triangular matrix.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:biconjugate-square-cholesky}]
Since the leading principal minors of positive definite matrices are positive, $w_i>0$ for all $i\in \{1, 2,\ldots, n\}$. It can be easily verified via the LDU from biconjugation decomposition and the symmetric property of $\bA$ that $\bA = \bU_n^{-\top} \bOmega_n \bU_n^{-1}$. And since $w_i$'s are positive, thus $\bOmega_n$ is positive definite and it can be factored as $\bOmega_n = \bOmega_n^{1/2}\bOmega_n^{1/2}$. This implies $\bOmega_n^{1/2} \bU_n^{-1}$ is the Cholesky factor.
\end{proof}

\subsection{QR Decomposition}

Without loss of generality, we  assume that $\bA\in \real^{n\times n}$ has full column rank so that the columns of $\bA$ can be factored as $\bA=\bQ\bR$ with $\bQ, \bR \in \real^{n\times n}$

\begin{theorem}[QR: Biconjugate Decomposition for Nonsingular Matrices]\label{theorem:biconjugate-square-qr}
For any nonsingular matrix $\bA\in \real^{n\times n}$, the QR decomposition of $\bA$ can be obtained from the Wedderburn sequence applied to $(\bI_n, \bA)$ as $(\bX_n, \bY_n)$.
In this case, $\bA$ can be factored as
$$
\bA = \bQ\bR,
$$
where $\bQ=\bV_n \bOmega_n^{-1/2}$ is an orthogonal matrix and $\bR = \bOmega_n^{1/2}\bR_n^{(x)}$ is an upper triangular matrix with \textcolor{blue}{Form 4} in Theorem~\ref{theorem:biconjugate-ldu} and let $\gamma=n$
$$
\bY_n^\top\bA\bX_n = \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)}=\bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}.
$$
where we set $\gamma=n$ since $\gamma$ is any value with $\gamma\leq r$ and the rank $r=n$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:biconjugate-square-qr}]
Since $(\bX_n, \bY_n) = (\bI_n, \bA)$. Then
By Theorem~\ref{theorem:biconjugate-ldu}, we have the decomposition
$$
\bY_n^\top\bA\bX_n = \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)}=\bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}.
$$
Substitute $(\bI_n, \bA)$ into the above decomposition, we have
\begin{equation}\label{equation:biconjugate-qr-ata1}
\begin{aligned}
	\bY_n^\top\bA\bX_n &= \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)} = \bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}\\
	\bA^\top \bA &= \bR_n^{(y)\top} \bOmega_n \bR_n^{(x)} \\
	\bA^\top \bA &= \bR_1^\top \bOmega_n \bR_1 \qquad (\text{$\bA^\top\bA$ is symmetric and let $\bR_1=\bR_n^{(x)}=\bR_n^{(y)}$})\\
	\bA^\top \bA &= (\bR_1^\top \bOmega_n^{1/2\top}) (\bOmega_n^{1/2}\bR_1) \\
	\bA^\top \bA &= \bR^\top\bR. \qquad (\text{Let $\bR = \bOmega_n^{1/2}\bR_1$})
\end{aligned}
\end{equation}
To see why $\bOmega_n$ can be factored as $\bOmega_n = \bOmega_n^{1/2\top}\bOmega_n^{1/2}$.
Suppose $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ is the column partition of $\bA$. We obtain $w_i = \ba_i^\top \ba_i>0$ since $\bA$ is nonsingular. Thus $\bOmega_n=diag(w_1, w_2, \ldots, w_n)$ is positive definite and it can be factored as
\begin{equation}\label{equation:omega-half-qr}
\bOmega_n = \bOmega_n^{1/2}\bOmega_n^{1/2}= \bOmega_n^{1/2\top}\bOmega_n^{1/2}.
\end{equation}
By $\bxgamma = \bugamma\brx$ in Theorem~\ref{theorem:biconjugate-ldu} for all $\gamma\in \{1, 2, \ldots, n\}$, we have
$$
\begin{aligned}
\bX_n &= \bU_n\bR_1 \\
\bI_n &= \bU_n\bR_1, \qquad (\text{Since $\bX_n = \bI_n$}) \\
\bU_n &= \bR_1^{-1}
\end{aligned}
$$
By $\bygamma = \bvgamma\bry$ in Theorem~\ref{theorem:biconjugate-ldu}  for all $\gamma\in \{1, 2, \ldots, n\}$, we have
\begin{equation}\label{equation:biconjugate-qr-ata2}
\begin{aligned}
	\bY_n &= \bV_n\bR_1\\
	\bA &= \bV_n\bR_1, \qquad &(\text{$\bA=\bY_n$}) \\
	\bA^\top \bA &= \bR_1^\top\bV_n^\top \bV_n\bR_1 \\
	\bR_1^\top \bOmega_n \bR_1&=\bR_1^\top\bV_n^\top \bV_n\bR_1, \qquad &(\text{Equation~\eqref{equation:biconjugate-qr-ata1}})\\
	(\bR_1^\top \bOmega_n^{1/2\top}) (\bOmega_n^{1/2}\bR_1) &= (\bR_1^\top \bOmega_n^{1/2\top} \bOmega_n^{-1/2\top})\bV_n^\top \bV_n (\bOmega_n^{-1/2}\bOmega_n^{1/2}\bR_1), \qquad &\text{(Equation~\eqref{equation:omega-half-qr})} \\
	\bR^\top\bR &= \bR^\top (\bOmega_n^{-1/2\top} \bV_n^\top) (\bV_n \bOmega_n^{-1/2})  \bR
\end{aligned}
\end{equation}
Thus, $\bQ=\bV_n \bOmega_n^{-1/2}$ is an orthogonal matrix.
\end{proof}

\subsection{SVD}
To differentiate the notation, let $\bA=\bU^\mathrm{svd} \bSigma^\mathrm{svd} \bV^{\mathrm{svd}\top}$ be the SVD of $\bA$ where $\bU^\mathrm{svd} = [\bu_1^\mathrm{svd}, \bu_2^\mathrm{svd}, \ldots, \bu_n^\mathrm{svd}]$, $\bV^\mathrm{svd} = [\bv_1^\mathrm{svd}, \bv_2^\mathrm{svd}, \ldots, \bv_n^\mathrm{svd}]$ and $\bSigma^\mathrm{svd} = diag(\sigma_1, \sigma_2, \ldots, \sigma_n)$. Without loss of generality, we assume $\bA\in \real^{n\times n}$ and $rank(\bA)=n$. Readers can prove the equivalence for general matrix $\bA\in \real^{m\times n}$.

If $\bX_n=\bV^\mathrm{svd}$, $\bY_n=\bU^\mathrm{svd}$ effect a rank-reducing process for $\bA$.
From the definition of $\bu_k$ and $\bv_k$ in Equation~\eqref{equation:properties-of-wedderburn-ukvk} or Equation~\eqref{equation:properties-of-wedderburn-ukvk2-inform2}, we have
$$
\bu_k = \bv_k^\mathrm{svd} \qquad \text{and} \qquad \bv_k = \bu_k^\mathrm{svd} \qquad \text{and} \qquad w_k = \by_k^\top \bA \bx_k=\sigma_k.
$$
That is $\bV_n = \bU^\mathrm{svd}$, $\bU_n = \bV^\mathrm{svd}$, and $\bOmega_n = \bSigma^\mathrm{svd}$, where we set $\gamma=n$ since $\gamma$ is any value that $\gamma\leq r$ and the rank $r=n$.

By $\bX_n= \bU_n\bR_n^{(x)}$ in Theorem~\ref{theorem:biconjugate-ldu}, we have
$$
\bX_n = \bU_n\bR_n^{(x)} \leadto
\bV^\mathrm{svd} = \bV^\mathrm{svd}\bR_n^{(x)}	 \leadto
\bI_n = \bR_n^{(x)}
$$
By $\bY_n = \bV_n\bR_n^{(y)}$ in Theorem~\ref{theorem:biconjugate-ldu}, we have
$$
\bY_n = \bV_n\bR_n^{(y)}\leadto
\bU^\mathrm{svd} = \bU^\mathrm{svd}\bR_n^{(y)} \leadto
\bI_n=\bR_n^{(y)}
$$
Again, from Theorem~\ref{theorem:biconjugate-ldu} and let $\gamma=n$, we have
$$
\bY_n^\top\bA\bX_n = \bR_n^{(y)\top} \bV_n^\top \bA \bU_n\bR_n^{(x)}=\bR_n^{(y)\top} \bOmega_n \bR_n^{(x)}.
$$
That is
$$
\bU^{\mathrm{svd}\top}\bA \bV^\mathrm{svd} = \bSigma^\mathrm{svd},
$$
which is exactly the form of SVD and we prove the equivalence between the SVD and the biconjugate decomposition when the Wedderburn sequence is applied to $(\bV^\mathrm{svd}, \bU^\mathrm{svd})$ as $(\bX_n, \bY_n)$.

\section{Proof General Term Formula of Wedderburn Sequence}\label{appendix:wedderburn-general-term}
We define the Wedderburn sequence of $\bA$ by $\bA_{k+1} = \bA_k -w_k^{-1} \bA_k\bx_k\by_k^\top \bA_k$ and $\bA_1 = \bA$. The proof of the general term formula of this sequence is then:
\begin{proof}[of Lemma~\ref{lemma:wedderburn-sequence-general}]
	For $\bA_2$, we have:
	$$
	\begin{aligned}
		\bA_2 &=\bA_1 -w_1^{-1} \bA_1\bx_1\by_1^\top \bA_1\\
		&=\bA -w_1^{-1} \bA\bu_1\bv_1^\top \bA, \text{ where } \bu_1=\bx_1 \text{, } \bv_1=\by_1.
	\end{aligned}
	$$
	For $\bA_3$, we can write out the equation:
	$$
	\begin{aligned}
		\bA_3 &= \bA_2 -w_2^{-1} \bA_2\bx_2\by_2^\top \bA_2 \\
		&=(\bA -w_1^{-1} \bA\bu_1\bv_1^\top \bA) \\
		&\gap - w_2^{-1}(\bA -w_1^{-1} \bA\bu_1\bv_1^\top \bA)\bx_2\by_2^\top(\bA -w_1^{-1} \bA\bu_1\bv_1^\top \bA) \qquad &\text{(substitute $\bA_2$)}\\
		&=(\bA -w_1^{-1} \bA\bu_1\bv_1^\top \bA) \\
		&\gap - w_2^{-1}\textcolor{blue}{\bA}(\textcolor{blue}{\bx_2 }-w_1^{-1} \bu_1\bv_1^\top \bA\textcolor{blue}{\bx_2 })(\textcolor{blue}{\by_2^\top} -w_1^{-1}\textcolor{blue}{\by_2^\top} \bA\bu_1\bv_1^\top )\textcolor{blue}{\bA} \qquad &\text{(take out $\bA$)}\\
		&=\bA -w_1^{-1} \bA\bu_1\bv_1^\top \bA - w_2^{-1}\bA\bu_2\bv_2^\top\bA\\
		&=\bA -\sum_{i=1}^{2}w_i^{-1} \bA\bu_i\bv_i^\top \bA,
	\end{aligned}
	$$
	where $\bu_2=\bx_2 -w_1^{-1} \bu_1\bv_1^\top \bA\bx_2=\bx_2 -\frac{\bv_1^\top \bA\bx_2}{w_1}\bu_1$, $\bv_2=\by_2 -w_1^{-1}\by_2^\top \bA\bu_1\bv_1=\by_2 -\frac{\by_2^\top \bA\bu_1}{w_1}\bv_1$.
	Similarly, we can find the expression of $\bA_4$ by $\bA$:
	$$
	\begin{aligned}
		\bA_4 &= \bA_3 -w_3^{-1} \bA_3\bx_3\by_3^\top \bA_3 \\
		&=\bA -\sum_{i=1}^{2}w_i^{-1} \bA\bu_i\bv_i^\top \bA \\
		&\gap - w_3^{-1} (\bA -\sum_{i=1}^{2}w_i^{-1} \bA\bu_i\bv_i^\top \bA)\bx_3\by_3^\top (\bA -\sum_{i=1}^{2}w_i^{-1} \bA\bu_i\bv_i^\top \bA) \qquad &\text{(substitute $\bA_3$)}\\
		&=\bA -\sum_{i=1}^{2}w_i^{-1} \bA\bu_i\bv_i^\top \bA \\
		&\gap - w_3^{-1}\textcolor{blue}{\bA} (\textcolor{blue}{\bx_3} -\sum_{i=1}^{2}w_i^{-1} \textcolor{blue}{\bx_3}\bu_i\bv_i^\top \bA) (\textcolor{blue}{\by_3^\top} -\sum_{i=1}^{2}w_i^{-1}\textcolor{blue}{\by_3^\top} \bA\bu_i\bv_i^\top )\textcolor{blue}{\bA} \qquad &\text{(take out $\bA$)}\\
		&=\bA -\sum_{i=1}^{2}w_i^{-1} \bA\bu_i\bv_i^\top \bA - w_3^{-1}\bA\bu_3\bv_3^\top\bA\\
		&=\bA -\sum_{i=1}^{3}w_i^{-1} \bA\bu_i\bv_i^\top \bA,
	\end{aligned}
	$$
	where $\bu_3=\bx_3 -\sum_{i=1}^{2}\frac{\bv_i^\top \bA\bx_3}{w_i}\bu_i$, $\bv_3=\by_3 -\sum_{i=1}^{2}\frac{\by_2^\top \bA\bu_i}{w_i}\bv_i$.

	Continue this process, we can define
	$$
	\bu_k=\bx_k -\sum_{i=1}^{k-1}\frac{\bv_i^\top \bA\bx_k}{w_i}\bu_i,
	\qquad \text{and} \qquad
	\bv_k=\by_k -\sum_{i=1}^{k-1}\frac{\by_k^\top \bA\bu_i}{w_i}\bv_i,
	$$
	and find the general term of Wedderburn sequence.
\end{proof}

\chapter*{Acknowledgments}
We thank Gilbert Strang for raising the question formulated in Corollary~\ref{corollary:invertible-intersection}, checking the writing of the book, for a stream of ideas and references about the three factorizations from the steps of elimination, and for the generous sharing of the manuscript of \citep{strang2021three}.

\part{Data Interpretation and Information Distillation}\label{part:data-interation}

\chapter{CR Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\index{Data interpretation}
\section{CR Decomposition}\label{section:cr-decomposition}
CR decomposition is proposed in \citet{strang2021every, stranglu}. As usual, we firstly give the result and we will discuss the existence and the origin of this decomposition in the following sections.
\begin{theoremHigh}[CR Decomposition]\label{theorem:cr-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA} = \underset{m\times r}{\bC} \gap \underset{r\times n}{\bR}
$$
where $\bC$ is the first $r$ linearly independent columns of $\bA$, and $\bR$ is an $r\times n$ matrix to reconstruct the columns of $\bA$ from columns of $\bC$. In particular, $\bR$ is the row reduced echelon form (RREF) of $\bA$ without the zero rows.

\item The storage for the decomposition is then reduced or potentially increased from $mn$ to $r(m+n)$.
\end{theoremHigh}

\section{Existence of the CR Decomposition}

Since matrix $\bA$ is of rank $r$, there are some $r$ linearly independent columns in $\bA$. We then choose linearly independent columns from $\bA$ and put them into $\bC$:

\begin{tcolorbox}[title={Find $r$ linearly Independent Columns From $\bA$}]
1. If column 1 of $\bA$ is not zero, put it into the column of $\bC$;

2. If column 2 of $\bA$ is not a multiple of column 1, put it into the column of $\bC$;

3. If column 3 of $\bA$ is not a combination of columns 1 and 2, put it into the column of $\bC$;

4. Continue this process until we find $r$ linearly independent columns (or all the linearly independent columns if we do not know the rank $r$ beforehand).
\end{tcolorbox}

When we have the $r$ linearly independent columns from $\bA$, we can prove the existence of CR decomposition by the column space view of matrix multiplication.

\paragraph{Column space view of matrix multiplication} A multiplication of two matrices $\bD\in \real^{m\times k}, \bE\in \real^{k\times n}$ is $\bA=\bD\bE=\bD[\be_1, \be_2, \ldots, \be_n] = [\bD\be_1, \bD\be_2, \ldots, \bD\be_n]$, i.e., each column of $\bA$ is a combination of columns from $\bD$.

\begin{proof}[of Theorem~\ref{theorem:cr-decomposition}]
As the rank of matrix $\bA$ is $r$ and $\bC$ contains $r$ linearly independent columns from $\bA$, the column space of $\bC$ is equivalent to the column space of $\bA$.
If we take any other column $\ba_i$ of $\bA$, $\ba_i$ can be represented as a linear combination of the columns of $\bC$, i.e., there exists a vector $\br_i$ such that $\ba_i = \bC \br_i$, $\forall i\in \{1, 2, \ldots, n\}$. Put these $\br_i$'s into the columns of matrix $\bR$, we obtain
$$
\bA = [\ba_1, \ba_2, \ldots, \ba_n] = [\bC \br_1, \bC \br_2, \ldots, \bC \br_n]= \bC \bR,
$$
from which the result follows.
\end{proof}

\index{Row echelon form}
\index{Reduced row echelon form}
\section{Reduced Row Echelon Form (RREF)}\label{section:rref-cr}
In Gaussian elimination Section~\ref{section:gaussian-elimination} (p.~\pageref{section:gaussian-elimination}), we introduced the elimination matrix (a lower triangular matrix) and permutation matrix to transform $\bA$ into an upper triangular form. We rewrite the Gaussian elimination for a $4\times 4$ square matrix, where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed:
\begin{tcolorbox}[title={Gaussian Elimination for a Square Matrix}]
	$$
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	\stackrel{\bE_1}{\longrightarrow}
	\begin{sbmatrix}{\bE_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\bP_1}{\longrightarrow}
	\begin{sbmatrix}{\bP_1\bE_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}} & \bm{\boxtimes} \\
		0 & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	\stackrel{\bE_2}{\longrightarrow}
	\begin{sbmatrix}{\bE_2\bP_1\bE_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 &  \textcolor{blue}{\boxtimes} & \boxtimes & \boxtimes \\
		0 & 0  & \textcolor{blue}{\boxtimes} & \boxtimes \\
		0 & \bm{0}  & \bm{0} & \textcolor{blue}{\bm{\boxtimes}}
	\end{sbmatrix}.
	$$
\end{tcolorbox}

\index{Gaussian elimination}
Furthermore, the Gaussian elimination can also be applied to a rectangular matrix, we give an example for a $4\times 5$ matrix as follows:
\begin{tcolorbox}[title={Gaussian Elimination for a Rectangular Matrix}]
	$$
	\begin{sbmatrix}{\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\end{sbmatrix}
	\stackrel{\bE_1}{\longrightarrow}
	\begin{sbmatrix}{\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		\bm{0} & \bm{0} & \textcolor{blue}{\bm{5}} & \bm{6} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \bm{2} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\end{sbmatrix}
	\stackrel{\bE_2}{\longrightarrow}
	\begin{sbmatrix}{\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		0 & 0 & \textcolor{blue}{5} & 6 & \boxtimes\\
		0 & 0 & \bm{0} & \textcolor{blue}{\bm{3}} & \bm{\boxtimes}\\
		0 & 0 & \bm{0} & \bm{0} & \bm{0}\\
	\end{sbmatrix},
	$$
\end{tcolorbox}
\noindent where the \textcolor{blue}{blue}-colored numbers are \textit{pivots} as we defined previously (Definition~\ref{definition:pivot}, p.~\pageref{definition:pivot}) and we call the last matrix above \textit{row echelon form} of matrix $\bA$. Note that we get the 4-th row as a zero row in this specific example. Going further, if we subtract each row by a multiple of the next row to make the entries above the pivots to be zero:
\begin{tcolorbox}[title={Reduced Row Echelon Form: Get Zero Above Pivots}]
	$$
	\begin{sbmatrix}{\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 10 & 9 & \boxtimes\\
		0 & 0 & \textcolor{blue}{5} & 6 & \boxtimes\\
		0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
		0 & 0 & 0 & 0 & 0\\
	\end{sbmatrix}
	\stackrel{\bE_3}{\longrightarrow}
	\begin{sbmatrix}{\bE_3\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & \bm{0}  & \bm{-3}  & \bm{\boxtimes} \\
		0 & 0 & \textcolor{blue}{5} & 6 & \boxtimes\\
		0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
		0 & 0 & 0 & 0 & 0\\
	\end{sbmatrix}
	\stackrel{\bE_4}{\longrightarrow}
	\begin{sbmatrix}{\bE_4\bE_3\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 0 & \bm{0} & \bm{\boxtimes}\\
		0 & 0 & \textcolor{blue}{5} & \bm{0} & \bm{\boxtimes}\\
		0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
		0 & 0 & 0 & 0 & 0\\
	\end{sbmatrix},
	$$
\end{tcolorbox}
\noindent where $\bE_3$ subtracts 2 times the $2$-nd row from the $1$-st row, and $\bE_4$ adds the $3$-rd row to the $1$-st row and subtracts 2 times the $3$-rd row from the $2$-nd row. Finally, we get the full \textit{row reduced echelon form} by making the pivots to be 1:
\begin{tcolorbox}[title={Reduced Row Echelon Form: Make The Pivots To Be 1}]
	$$
	\begin{sbmatrix}{\bE_4\bE_3\bE_2\bE_1\bA}
		\textcolor{blue}{2} & \boxtimes & 0 & 0 & \boxtimes\\
		0 & 0 & \textcolor{blue}{5} & 0 & \boxtimes\\
		0 & 0 & 0 & \textcolor{blue}{3} & \boxtimes\\
		0 & 0 & 0 & 0 & 0\\
	\end{sbmatrix}
	\stackrel{\bE_5}{\longrightarrow}
	\begin{sbmatrix}{\bE_5\bE_4\bE_3\bE_2\bE_1\bA}
		\textcolor{blue}{\bm{1}} & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \textcolor{blue}{\bm{1}} & \bm{0} & \bm{\boxtimes}\\
		\bm{0} & \bm{0} & \bm{0} & \textcolor{blue}{\bm{1}} & \bm{\boxtimes}\\
		0 & 0 & 0 & 0 & 0\\
	\end{sbmatrix},
	$$
\end{tcolorbox}
\noindent where $\bE_5$ makes the pivots to be 1. Note here,
it is not necessary for
the transformation matrices $\bE_1, \bE_2, \ldots, \bE_5$ to be lower triangular matrices as they are in LU decomposition.
They can also be permutation matrices or other matrices. We call this final matrix the \textbf{reduced row echelon form} of $\bA$ where it has 1's as pivots and zeros above the pivots.

\index{Rank}
\index{Pivot}
\begin{lemma}[Rank and Pivots]\label{lemma:rank-is-pivots}
	The rank of $\bA$ is equal to the number of pivots.
\end{lemma}

\index{Reduced row echelon form}
\begin{lemma}[RREF in CR]\label{lemma:r-in-cr-decomposition}
	The reduced row echelon form of the matrix $\bA$ without zero rows is the matrix $\bR$ in the CR decomposition.
\end{lemma}

In short, we first compute the reduced row echelon form of matrix $\bA$ by $rref(\bA)$, Then $\bC$ is obtained by removing from $\bA$ all the non-pivot columns (this can be determined by looking for columns in $rref(\bA)$ which do not contain a pivot). And $\bR$ is obtained by eliminating zero rows of $rref(\bA)$. And this is actually a special case of \textit{rank decomposition} (Definition~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}) of matrix $\bA$. However, CR decomposition is so special that it involves the reduced row echelon form so that we introduce it here particularly.

$\bR$ has a remarkable form whose $r$ columns containing the pivots form an $r\times r$ identity matrix. Note again that we can just remove the zero rows from the row reduced echelon form to obtain this matrix $\bR$. In \citet{strang2021every}, the author gives a specific notation for the row reduced echelon form without removing the zero rows as $\bR_0$:
$$
\bR_0 = rref(\bA)=
\begin{bmatrix}
	\bR \\
	\bzero
\end{bmatrix}=
\begin{bmatrix}
	\bI_r & \bF \\
	\bzero & \bzero
\end{bmatrix}\bP, \footnote{Permutation matrix $\bP$ on the right side of a matrix is to permute the column of that matrix.
}
$$
where the $n\times n$ permutation matrix $\bP$ puts the columns of $r\times r$ identity matrix $\bI_r$ into the correct positions, matching the first $r$ linearly independent columns of the original matrix $\bA$.

The CR decomposition reveals a great theorem of linear algebra that the row rank equals the column rank of any matrix.

\begin{proof}[\textbf{of Theorem~\ref{lemma:equal-dimension-rank}, A Third Way}]
For CR decomposition of matrix $\bA=\bC\bR$, we have $\bR = [\bI_r, \bF ]\bP$, where $\bP$ is an $n\times n$ permutation to put the columns of the $r\times r$ identity matrix $\bI_r$ into the correct positions as shown above. It can be easily verified that the $r$ rows of $\bR$ are linearly independent of the submatrix of $\bI_r$ (since $\bI_r$ is nonsingular) such that the row rank of $\bR$ is $r$.

Firstly, from the definition of the CR decomposition, the $r$ columns of $\bC$ are from $r$ linearly independent columns of $\bA$, and the column rank of $\bA$ is $r$. Further,

$\bullet$ Since $\bA=\bC\bR$, all rows of $\bA$ are combinations of the rows of $\bR$. That is, the row rank of $\bA$ is no larger than the row rank of $\bR$;

$\bullet$ From $\bA=\bC\bR$, we also have $(\bC^\top\bC)^{-1}\bC^\top\bC\bR = (\bC^\top\bC)^{-1}\bC^\top\bA$, that is $\bR = (\bC^\top\bC)^{-1}\bC^\top\bA$. $\bC^\top\bC$ is nonsingular since it has full column rank $r$. Then all rows of $\bR$ are also combinations of the rows of $\bA$. That is, the row rank of $\bR$ is no larger than the row rank of $\bA$;

$\bullet$ By ``sandwiching", the row rank of $\bA$ is equal to the row rank of $\bR$ which is $r$.

Therefore, both the row rank and column rank of $\bA$ are equal to $r$ from which the result follows.
\end{proof}

\index{Rank decomposition}
\section{Rank Decomposition}
We previously mentioned that CR decomposition is a special case of rank decomposition. Formally, we prove the existence of the rank decomposition rigorously in the following theorem.

\begin{theoremHigh}[Rank Decomposition]\label{theorem:rank-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA }= \underset{m\times r}{\bD}\gap  \underset{r\times n}{\bF},
$$
where $\bD \in \real^{m\times r}$ has rank $r$, and $\bF \in \real^{r\times n}$ also has rank $r$, i.e., $\bD,\bF$ have full rank $r$.

The storage for the decomposition is then reduced or potentially increased from $mn$ to $r(m+n)$.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:rank-decomposition}]
By ULV decomposition in Theorem~\ref{theorem:ulv-decomposition} (p.~\pageref{theorem:ulv-decomposition}), we can decompose $\bA$ by
$$
\bA = \bU \begin{bmatrix}
	\bL & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV.
$$
Let $\bU_0 = \bU_{:,1:r}$ and $\bV_0 = \bV_{1:r,:}$, i.e., $\bU_0$ contains only the first $r$ columns of $\bU$, and $\bV_0$ contains only the first $r$ rows of $\bV$. Then, we still have $\bA = \bU_0 \bL\bV_0$ where $\bU_0 \in \real^{m\times r}$ and $\bV_0\in \real^{r\times n}$. This is also known as the reduced ULV decomposition. Let \{$\bD = \bU_0\bL$ and $\bF =\bV_0$\}, or \{$\bD = \bU_0$ and $\bF =\bL\bV_0$\}, we find such rank decomposition.
\end{proof}
The rank decomposition is not unique. Even by elementary transformations, we have
$$
\bA =
\bE_1
\begin{bmatrix}
	\bZ & \bzero \\
	\bzero & \bzero
\end{bmatrix}
\bE_2,
$$
where $\bE_1 \in \real^{m\times m}, \bE_2\in \real^{n\times n}$ represent elementary row and column operations, $\bZ\in \real^{r\times r}$. The transformation is rather general, and there are dozens of these $\bE_1,\bE_2,\bZ$. Similar construction on this decomposition as shown in the above proof, we can recover another rank decomposition.

Analogously, we can find such $\bD,\bF$ by SVD, URV, CR, CUR, and many other decompositional algorithms. However, we may connect the different rank decompositions by the following lemma.
\begin{lemma}[Connection Between Rank Decompositions]\label{lemma:connection-rank-decom}
For any two rank decompositions of $\bA=\bD_1\bF_1=\bD_2\bF_2$, there exists a nonsingular matrix $\bP$ such that
$$
\bD_1 = \bD_2\bP
\qquad
\text{and}
\qquad
\bF_1 = \bP^{-1}\bF_2.
$$
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:connection-rank-decom}]
Since $\bD_1\bF_1=\bD_2\bF_2$, we have $\bD_1\bF_1\bF_1^\top=\bD_2\bF_2\bF_1^\top$. It is trivial that $rank(\bF_1\bF_1^\top)=rank(\bF_1)=r$ such that $\bF_1\bF_1^\top$ is a square matrix with full rank and thus is nonsingular. This implies $\bD_1=\bD_2\bF_2\bF_1^\top(\bF_1\bF_1^\top)^{-1}$. Let $\bP=\bF_2\bF_1^\top(\bF_1\bF_1^\top)^{-1}$, we have $\bD_1=\bD_2\bP$ and $\bF_1 = \bP^{-1}\bF_2$.
\end{proof}

\index{Idempotent matrix}
\section{Application: Rank and Trace of an Idempotent Matrix}
The CR decomposition is quite useful to prove the rank of an idempotent matrix. See also how it works in the orthogonal projection in \citet{lu2021numerical, lu2021rigorous}.
\begin{lemma}[Rank and Trace of an Idempotent Matrix\index{Trace}]\label{lemma:rank-of-symmetric-idempotent2_tmp}
	For any $n\times n$ idempotent matrix $\bA$ (i.e., $\bA^2 = \bA$), the rank of $\bA$ equals the trace of $\bA$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-of-symmetric-idempotent2_tmp}]
Any $n\times n$  rank-$r$ matrix $\bA$ has CR decomposition $\bA = \bC\bR$, where $\bC\in\real^{n\times r}$ and $\bR\in \real^{r\times n}$ with $\bC, \bR$ having full rank $r$.
Then,
$$
\begin{aligned}
	\bA^2 &= \bA, \\
	\bC\bR\bC\bR &= \bC\bR, \\
	\bR\bC\bR &=\bR, \\
	\bR\bC &=\bI_r,
\end{aligned}
$$
where $\bI_r$ is an $r\times r$ identity matrix. Thus
$$
trace(\bA) = trace(\bC\bR) =trace(\bR\bC) = trace(\bI_r) = r,
$$
which equals the rank of $\bA$. The equality above is from the invariant of cyclic permutation of trace.
\end{proof}

\part{Eigenvalue Problem}

\chapter{Eigenvalue and Jordan Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Eigenvalue and Jordan Decomposition}
\begin{theoremHigh}[Eigenvalue Decomposition]\label{theorem:eigenvalue-decomposition}
Any square matrix $\bA\in \real^{n\times n}$ with linearly independent eigenvectors can be factored as
$$
\bA = \bX\bLambda\bX^{-1},
$$
where $\bX$ contains the eigenvectors of $\bA$ as columns, and $\bLambda$ is a diagonal matrix $diag(\lambda_1, \lambda_2,$ $\ldots, \lambda_n)$ and $\lambda_1, \lambda_2, \ldots, \lambda_n$ are eigenvalues of $\bA$.
\end{theoremHigh}

Eigenvalue decomposition is also known as to diagonalize the matrix $\bA$. When no eigenvalues of $\bA$ are repeated, the eigenvectors are sure to be linearly independent. Then $\bA$ can be diagonalized. Note here without $n$ linearly independent eigenvectors, we cannot diagonalize. In Section~\ref{section:otherform-spectral} (p.~\pageref{section:otherform-spectral}), we will further discuss conditions under which the matrix has linearly independent eigenvectors.

\section{Existence of the Eigenvalue Decomposition}
\begin{proof}[of Theorem~\ref{theorem:eigenvalue-decomposition}]
Let $\bX=[\bx_1, \bx_2, \ldots, \bx_n]$ as the linearly independent eigenvectors of $\bA$. Clearly, we have
$$
\bA\bx_1=\lambda_1\bx_1,\qquad \bA\bx_2=\lambda_2\bx_2, \qquad \ldots, \qquad\bA\bx_n=\lambda_n\bx_n.
$$
In the matrix form,
$$
\bA\bX = [\bA\bx_1, \bA\bx_2, \ldots, \bA\bx_n] = [\lambda_1\bx_1, \lambda_2\bx_2, \ldots, \lambda_n\bx_n] = \bX\bLambda.
$$
Since we assume the eigenvectors are linearly independent, then $\bX$ has full rank and is invertible. We obtain
$$
\bA = \bX\bLambda \bX^{-1}.
$$
This completes the proof.
\end{proof}

We will discuss some similar forms of eigenvalue decomposition in the spectral decomposition section (Section~\ref{section:spectral-decomposition}, p.~\pageref{section:spectral-decomposition}), where the matrix $\bA$ is required to be symmetric, and the $\bX$ is not only nonsingular but also orthogonal. Or, the matrix $\bA$ is required to be a \textit{simple matrix}, that is, the algebraic multiplicity and geometric multiplicity are the same for $\bA$, and $\bX$ will be a trivial nonsingular matrix that may not contain the eigenvectors of $\bA$. The decomposition also has a geometric meaning, which we will discuss in Section~\ref{section:coordinate-transformation} (p.~\pageref{section:coordinate-transformation}).

A matrix decomposition in the form of $\bA =\bX\bLambda\bX^{-1}$ has a nice property that we can compute the $m$-th power efficiently.

\begin{remark}[$m$-th Power]\label{remark:power-eigenvalue-decom}
The $m$-th power of $\bA$ is $\bA^m = \bX\bLambda^m\bX^{-1}$ if the matrix $\bA$ can be factored as $\bA=\bX\bLambda\bX^{-1}$.
\end{remark}

We notice that we require $\bA$ have linearly independent eigenvectors to prove the existence of the eigenvalue decomposition. Under specific conditions, the requirement is intrinsically satisfied.
\begin{lemma}[Different Eigenvalues]\label{lemma:diff-eigenvec-decompo}
Suppose the eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_n$ of $\bA\in \real^{n\times n}$ are all different. Then the corresponding eigenvectors are automatically independent. In another word, any square matrix with different eigenvalues can be diagonalized.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:diff-eigenvec-decompo}]
Suppose the eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_n$ are all different, and the eigenvectors $\bx_1,\bx_2, \ldots, \bx_n$ are linearly dependent. That is, there exists a nonzero vector $\bc = [c_1,c_2,\ldots,c_{n-1}]^\top$ such that
$$
\bx_n = \sum_{i=1}^{n-1} c_i\bx_{i}.
$$
Then we have
$$
\begin{aligned}
\bA \bx_n &= \bA (\sum_{i=1}^{n-1} c_i\bx_{i}) \\
&=c_1\lambda_1 \bx_1 + c_2\lambda_2 \bx_2 + \ldots + c_{n-1}\lambda_{n-1}\bx_{n-1}.
\end{aligned}
$$
and
$$
\begin{aligned}
	\bA \bx_n &= \lambda_n\bx_n\\
	&=\lambda_n (c_1\bx_1 +c_2\bx_2+\ldots +c_{n-1} \bx_{n-1}).
\end{aligned}
$$
Combining the above two equations, we have
$$
\sum_{i=1}^{n-1} (\lambda_n - \lambda_i)c_i \bx_i = \bzero .
$$
This leads to a contradiction since $\lambda_n \neq \lambda_i$ for all $i\in \{1,2,\ldots,n-1\}$, from which the result follows.
\end{proof}

\begin{remark}[Limitation of Eigenvalue Decomposition]
The limitation of eigenvalue decomposition is that:

$\bullet$ The eigenvectors in $\bX$ are usually not orthogonal and there are not always enough eigenvectors (i.e., some eigenvalues are equal).

$\bullet$ To compute the eigenvalues and eigenvectors $\bA\bx = \lambda\bx$ requires $\bA$ to be square. Rectangular matrices cannot be diagonalized by eigenvalue decomposition.
\end{remark}

\section{Jordan Decomposition}
In eigenvalue decomposition, we suppose matrix $\bA$ has $n$ linearly independent eigenvectors. However, this is not necessarily true for all square matrices. We introduce further a generalized version of eigenvalue decomposition which is called the Jordan decomposition named after Camille Jordan \citep{jordan1870traite}.

We first introduce the definition of Jordan blocks and Jordan form for the further description of Jordan decomposition.
\begin{definition}[Jordan Block]
	An $m\times m$ upper triangular matrix $B(\lambda, m)$ is called a Jordan block provided all $m$ diagonal elements are the same eigenvalue $\lambda$ and all upper subdigonal elements are ones:
	$$
	B(\lambda, m)=
	\begin{bmatrix}
		\lambda & 1        & 0 & \ldots & 0 & 0 & 0 \\
		0       & \lambda  & 1 & \ldots & 0 & 0 & 0 \\
		0       & 0  & \lambda & \ldots & 0 & 0 & 0 \\
		\vdots  & \vdots   & \vdots & \vdots & \vdots & \vdots\\
		0 & 0 & 0 & \ldots & \lambda & 1& 0\\
		0 & 0 & 0 & \ldots & 0 &\lambda & 1\\
		0 & 0 & 0 & \ldots & 0 & 0 & \lambda
	\end{bmatrix}_{m\times m}
	$$
\end{definition}
\begin{definition}[Jordan Form\index{Jordan block}]
Given an $n\times n$ matrix $\bA$, a Jordan form $\bJ$ for $\bA$ is a block diagonal matrix defined as
$$
\bJ=diag(B(\lambda_1, m_1), B(\lambda_2, m_2), \ldots B(\lambda_k, m_k))
$$
where $\lambda_1, \lambda_2, \ldots, \lambda_k$ are eigenvalues of $\bA$ (duplicates possible) and $m_1+m_2+\ldots+m_k=n$.
\end{definition}

Then, the Jordan decomposition follows:
\begin{theoremHigh}[Jordan Decomposition]\label{theorem:jordan-decomposition}
Any square matrix $\bA\in \real^{n\times n}$ can be factored as
$$
\bA = \bX\bJ\bX^{-1},
$$
where $\bX$ is a nonsingular matrix containing the generalized eigenvectors of $\bA$ as columns, and $\bJ$ is a Jordan form matrix $diag(\bJ_1, \bJ_2, \ldots, \bJ_k)$ where
$$
\bJ_i =
\begin{bmatrix}
	\lambda_i & 1        & 0 & \ldots & 0 & 0 & 0 \\
	0       & \lambda_i  & 1 & \ldots & 0 & 0 & 0 \\
	0       & 0  & \lambda_i & \ldots & 0 & 0 & 0 \\
	\vdots  & \vdots   & \vdots & \vdots & \vdots & \vdots\\
	0 & 0 & 0 & \ldots & \lambda_i & 1& 0\\
	0 & 0 & 0 & \ldots & 0 &\lambda_i & 1\\
	0 & 0 & 0 & \ldots & 0 & 0 & \lambda_i
\end{bmatrix}_{m_i\times m_i}
$$
is an $m_i\times m_i$ square matrix with $m_i$ being the number of repetitions of eigenvalue $\lambda_i$ and $m_1+m_2+\ldots +m_k = n$. $\bJ_i$'s are referred to as Jordan blocks.

Further, nonsingular matrix $\bX$ is called the \textit{matrix of generalized eigenvectors} of $\bA$.
\end{theoremHigh}

As an example, a Jordan form can have the following structure:
$$
\begin{aligned}
	\bJ&=diag(B(\lambda_1, m_1), B(\lambda_2, m_2), \ldots, B(\lambda_k, m_k))\\
	&=
	\begin{bmatrix}
		\begin{bmatrix}
			\lambda_1 & 1        & 0 \\
			0       & \lambda_1  & 1 \\
			0       & 0  & \lambda_1
		\end{bmatrix} &  & & &\\
		& \begin{bmatrix}
			\lambda_2
		\end{bmatrix} & & &\\
		&  &\begin{bmatrix}
			\lambda_3 & 1 \\
			0 & \lambda_3
		\end{bmatrix} & &\\
		&  &   & \ddots & &\\
		&  &   &  & &\begin{bmatrix}
			\lambda_k & 1 \\
			0 & \lambda_k
		\end{bmatrix}\\
	\end{bmatrix}.
\end{aligned}
$$
\textbf{Decoding a Jordan Decomposition}: Note that zeros can appear on the upper sub-diagonal of $\bJ$ and the first column is always a diagonal containing only eigenvalues of $\bA$ in each block. Take out one block to decode, without loss of generality, we take out the first block $\bJ_1$. We shall show the columns $1, 2, \ldots, m_1$ of $\bA\bX = \bX\bJ$ with $\bX=[\bx_1, \bx_2, \ldots, \bx_n]$:
$$
\begin{aligned}
\bA\bx_1 &= \lambda_1 \bx_1 \\
\bA\bx_2 &= \lambda_1 \bx_2 + \bx_1	\\
\vdots &= \vdots \\
\bA\bx_{m_1} &= \lambda_1 \bx_{m_1} + \bx_{m_1-1}.
\end{aligned}
$$
For more details about Jordan decomposition, please refer to \citet{gohberg1996simple, hales1999jordan}.

The Jordan decomposition is not particularly interesting in practice as it is extremely sensitive to perturbation. Even with the smallest random change to a matrix, the matrix can be made diagonalizable \citep{van2020advanced}. As a result, there is no practical mathematical software library or tool that computes it. And the proof takes dozens of pages to discuss. For this reason, we leave the proof to interesting readers.

\section{Application: Computing Fibonacci Numbers}
We use eigenvalue decomposition to compute the Fibonacci number. This example is drawn from \citet{strang1993introduction}. Every new Fibonacci number $F_{k+2}$ is the sum of the two previous Fibonacci numbers $F_{k+1}+F_{k}$. The sequence is $0, 1, 1, 2, 3, 5, 8, \ldots$. Now, the problem is what is the number of $F_{100}$?
The eigenvalue decomposition can help find the general formula of the sequence.
\index{Fibonacci number}
\index{General formula of a sequence}

Let $\bu_{k}=\begin{bmatrix}
	F_{k+1}\\
	F_k
\end{bmatrix}$.
Then $\bu_{k+1}=\begin{bmatrix}
	F_{k+2}\\
	F_{k+1}
\end{bmatrix}=
\begin{bmatrix}
	1&1\\
	1&0
\end{bmatrix}
\bu_k
$ by the rule that $F_{k+2}=F_{k+1}+F_k$ and $F_{k+1}=F_{k+1}$.

Let $\bA=\begin{bmatrix}
	1&1\\
	1&0
\end{bmatrix}$
, we then have the general formula $\bu_{100} = \bA^{100}\bu_0$ where $\bu_0=
\begin{bmatrix}
	1\\
	0
\end{bmatrix}
$.

We will see in Lemma~\ref{lemma:determinant-intermezzo} (p.~\pageref{lemma:determinant-intermezzo}) that $\det(\bA-\lambda\bI)=0$ where $\lambda$ is the eigenvalue of $\bA$. Simple calculation shows that $\det(\bA-\lambda\bI) = \lambda^2-\lambda+1=0$ and
$$
\lambda_1 = \frac{1+\sqrt{5}}{2}, \qquad \lambda_2 = \frac{1-\sqrt{5}}{2}.
$$
The corresponding eigenvectors are
$$
\bx_1 =
\begin{bmatrix}
	\lambda_1\\
	1
\end{bmatrix}, \qquad
\bx_2 =
\begin{bmatrix}
	\lambda_2\\
	1
\end{bmatrix}.
$$
By Remark~\ref{remark:power-eigenvalue-decom} (p.~\pageref{remark:power-eigenvalue-decom}), $\bA^{100} = \bX\bLambda^{100}\bX^{-1} = \bX
\begin{bmatrix}
	\lambda_1^{100}&0\\
	0&\lambda_2^{100}
\end{bmatrix}\bX^{-1}$ where $\bX^{-1}$ can be easily calculated as $\bX^{-1} =
\begin{bmatrix}
	\frac{1}{\lambda_1-\lambda_2} & \frac{-\lambda_2}{\lambda_1-\lambda_2} \\
	-\frac{1}{\lambda_1-\lambda_2} & \frac{\lambda_1}{\lambda_1-\lambda_2}
\end{bmatrix}
=\begin{bmatrix}
	\frac{\sqrt{5}}{5} & \frac{5-\sqrt{5}}{10} \\
	-\frac{\sqrt{5}}{5} & \frac{5+\sqrt{5}}{10}
\end{bmatrix}$. We notice that $\bu_{100} = \bA^{100}\bu_0$ is just the first column of $\bA^{100}$ which is
$$
\bu_{100} =
\begin{bmatrix}
	F_{101}\\
	F_{100}
\end{bmatrix}=
\begin{bmatrix}
	\frac{\lambda_1^{101}-\lambda_2^{101}}{\lambda_1-\lambda_2}\\
	\frac{\lambda_1^{100}-\lambda_2^{100}}{\lambda_1-\lambda_2}
\end{bmatrix}.
$$
A simple check on the calculation, we have $F_{100}=3.542248481792631e+20$. Or more generally,
$$
\bu_{K} =
\begin{bmatrix}
	F_{K+1}\\
	F_{K}
\end{bmatrix}=
\begin{bmatrix}
	\frac{\lambda_1^{K+1}-\lambda_2^{K+1}}{\lambda_1-\lambda_2}\\
	\frac{\lambda_1^{K}-\lambda_2^{K}}{\lambda_1-\lambda_2}
\end{bmatrix},
$$
where the general form of $F_K$ is given by $F_K=\frac{\lambda_1^{K}-\lambda_2^{K}}{\lambda_1-\lambda_2}$.

\paragraph{Extension to general 2 by 2 matrices}
The above matrix $\begin{bmatrix}
	1 & 1 \\ 1 & 0
\end{bmatrix}$ is special in that it has two independent eigenvectors so that it admits eigenvalue decomposition. However, it is also interesting to compute the $n$-th power of general 2 by 2 matrices, say
$$
\bA=
\begin{bmatrix}
	a & b \\ c & d
\end{bmatrix}.
$$
Suppose $\alpha, \beta$ are eigenvalues of $\bA$ so that $\alpha, \beta$ are two roots of the characteristic polynomial
$$
-\det(\bA-\lambda\bI)= \lambda^2 + a_1 \lambda + a_0 = 0
\leadto  \alpha, \beta = \frac{-a_1 \pm \sqrt{a_1^2 - 4a_0}}{2}.
$$
This reveals that $a_1 = -(\alpha+\beta)$ and $a_0=\alpha\beta$. By the \textit{Cayley-Hamilton theorem}, a matrix satisfies its own characteristic equation:
$$
\bA^2 + a_1 \bA + a_0 \bI = \bzero,
$$
so that
\begin{equation}\label{eqiation:2by2-cayley}
	\bA^2 - (\alpha+\beta)\bA + \alpha\beta\bI = \bzero.
\end{equation}
Following the tricks in \citet{Williams1992npower}, we define matrices $\bX,\bY,\bZ$ such that
$$
\left\{
\begin{aligned}
	\bX &= \frac{\bA-\beta\bI}{\alpha-\beta}, \gap \bY = \frac{\bA-\alpha\bI}{\beta-\alpha}, \gap &\text{if $\alpha\neq \beta$;}\\
	\bZ &= \bA-\alpha\bI, &\text{if $\alpha=\beta$,}
\end{aligned}
\right.
$$
where $\bA = \alpha\bX + \beta\bY$, and
we have properties from Equation~\eqref{eqiation:2by2-cayley}, for any $k\geq 2$:
$$
\left\{
\begin{aligned}
	\bX^k &= \bX, \gap \bX\bY=\bY\bX=\bzero, \gap \bY^k=\bY, \gap  &\text{if $\alpha\neq \beta$;}\\
	\bZ^k &= \bzero, &\text{if $\alpha=\beta$.}
\end{aligned}
\right.
$$
Therefore,
\begin{equation}\label{equation:genera2by2power}
	\bA^n=
	\left\{
	\begin{aligned}
		(\alpha\bX+\beta\bY)^n &= \alpha^n\bX+ \beta^n\bY= \alpha^n \left(\frac{\bA-\beta\bI}{\alpha-\beta}\right)+\beta^n\left(\frac{\bA-\alpha\bI}{\beta-\alpha}\right), \gap  &\text{if $\alpha\neq \beta$;}\\
		(\bZ+\alpha\bI)^n &= n\alpha^{n-1}\bZ + \alpha^n\bI=\alpha^{n-1}\left( n\bA - \alpha(n-1)\bI \right), &\text{if $\alpha=\beta$.}
	\end{aligned}
	\right.
\end{equation}
Hence, if matrix $\bA$ is nonsingular, $\alpha\neq0, \beta\neq 0$, Equation~\eqref{equation:genera2by2power} holds for all integer values of $n$; if matrix $\bA$ is real, but the eigenvalues are complex with some power of them real, e.g., $\alpha^m=\beta^m=f$ is real, then we have $\bA^m = f\bI$.

The $n$-th power for general matrices results in the general form of any sequence, e.g.,
$$
\bu_{k+1} =
\begin{bmatrix}
	2 & 3 \\2 & 0
\end{bmatrix}
\bu_k.
$$
And we shall repeat the details.

\chapter{Schur Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup
\section{Schur Decomposition}
\begin{theoremHigh}[Schur Decomposition]\label{theorem:schur-decomposition}
Any square matrix $\bA\in \real^{n\times n}$ with real eigenvalues can be factored as
$$
\bA = \bQ\bU\bQ^\top,
$$
where $\bQ$ is an orthogonal matrix, and $\bU$ is an upper triangular matrix. That is, all square matrix $\bA$ with real eigenvalues can be triangularized.
\end{theoremHigh}

\paragraph{A close look at Schur decomposition} The first column of $\bA\bQ$ and $\bQ\bU$ are $\bA\bq_1$ and  $\bU_{11}\bq_1$. Then, $\bU_{11}, \bq_1$ are eigenvalue and eigenvector of $\bA$. But other columns of $\bQ$ need not be eigenvectors of $\bA$.

\paragraph{Schur decomposition for symmetric matrices} Symmetric matrix $\bA=\bA^\top$ leads to $\bQ\bU\bQ^\top = \bQ\bU^\top\bQ^\top$. Then $\bU$ is a diagonal matrix. And this diagonal matrix actually contains eigenvalues of $\bA$. All the columns of $\bQ$ are eigenvectors of $\bA$. We conclude that all symmetric matrices are diagonalizable even with repeated eigenvalues.

\index{Determinant}
\section{Existence of the Schur Decomposition}
To prove Theorem~\ref{theorem:schur-decomposition}, we need to use the following lemmas.
\begin{lemma}[Determinant Intermezzo]\label{lemma:determinant-intermezzo}
We have the following properties for determinant of matrices:

$\bullet$ The determinant of multiplication of two matrices is $\det(\bA\bB)=\det (\bA)\det(\bB)$;

$\bullet$ The determinant of the transpose is $\det(\bA^\top) = \det(\bA)$;

$\bullet$ Suppose matrix $\bA$ has eigenvalue $\lambda$, then $\det(\bA-\lambda\bI) =0$;

$\bullet$ Determinant of any identity matrix is $1$;

$\bullet$ Determinant of an orthogonal matrix $\bQ$:
$$
\det(\bQ) = \det(\bQ^\top) = \pm 1, \qquad \text{since  } \det(\bQ^\top)\det(\bQ)=\det(\bQ^\top\bQ)=\det(\bI)=1;
$$

$\bullet$ Any square matrix $\bA$, we then have an orthogonal matrix $\bQ$:

$$
\det(\bA) = \det(\bQ^\top) \det(\bA)\det(\bQ) =\det(\bQ^\top\bA\bQ);
$$
\end{lemma}

\begin{lemma}[Submatrix with Same Eigenvalue]\label{lemma:submatrix-same-eigenvalue}
Suppose square matrix $\bA_{k+1}\in \real^{(k+1)\times (k+1)}$ has real eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_{k+1}$. Then we can construct a $k\times k$ matrix $\bA_{k}$ with eigenvalues $\lambda_2, \lambda_3, \ldots, \lambda_{k+1}$ by
$$
\bA_{k} =
\begin{bmatrix}
-\bp_2^\top- \\
-\bp_3^\top- \\
\vdots \\
-\bp_{k+1}^\top-
\end{bmatrix}
\bA_{k+1}
\begin{bmatrix}
\bp_2 & \bp_3 &\ldots &\bp_{k+1}
\end{bmatrix},
$$
where $\bp_1$ is a eigenvector of $\bA_{k+1}$ with norm 1 corresponding to eigenvalue $\lambda_1$, and $\bp_2, \bp_3, \ldots, \bp_{k+1}$ are any orthonormal vectors orthogonal to $\bp_1$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:submatrix-same-eigenvalue}]
Let $\bP_{k+1} = [\bp_1, \bp_2, \ldots, \bp_{k+1}]$. Then $\bP_{k+1}^\top\bP_{k+1}=\bI$, and
$$
\bP_{k+1}^\top \bA_{k+1} \bP_{k+1} =
\begin{bmatrix}
	\lambda_1 & \bzero \\
	\bzero  & \bA_{k}
\end{bmatrix}.
$$
For any eigenvalue $\lambda = \{\lambda_2, \lambda_3, \ldots, \lambda_{k+1}\}$, by Lemma~\ref{lemma:determinant-intermezzo}, we have
$$
\begin{aligned}
\det(\bA_{k+1} -\lambda\bI) &= \det(\bP_{k+1}^\top (\bA_{k+1}-\lambda\bI)  \bP_{k+1}) \\
&=\det(\bP_{k+1}^\top \bA_{k+1}\bP_{k+1} - \lambda\bP_{k+1}^\top\bP_{k+1}) \\
&= \det\left(
\begin{bmatrix}
	\lambda_1-\lambda & \bzero \\
	\bzero & \bA_k - \lambda\bI
\end{bmatrix}
\right)\\
&=(\lambda_1-\lambda)\det(\bA_k-\lambda\bI).
\end{aligned}
$$
Where the last equality is from the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bE & \bF \\
		\bG & \bH
	\end{bmatrix}$, then $\det(\bM) = \det(\bE)\det(\bH-\bG\bE^{-1}\bF)$.
Since $\lambda$ is an eigenvalue of $\bA$ and $\lambda \neq \lambda_1$, then $\det(\bA_{k+1} -\lambda\bI) = (\lambda_1-\lambda)\det(\bA_{k}-\lambda\bI)=0$ means $\lambda$ is also an eigenvalue of $\bA_{k}$.
\end{proof}

We then prove the existence of the Schur decomposition by induction.
\begin{proof}[\textbf{of Theorem~\ref{theorem:schur-decomposition}: Existence of Schur Decomposition}]
We note that the theorem is trivial when $n=1$ by setting $Q=1$ and $U=A$. Suppose the theorem is true for $n=k$ for some $k\geq 1$. If we prove the theorem is also true for $n=k+1$, then we complete the proof.

Suppose for $n=k$, the theorem is true for $\bA_k =\bQ_k \bU_k \bQ_k^\top$.

Suppose further \textit{$\bP_{k+1}$ contains orthogonal vectors $\bP_{k+1} = [\bp_1, \bp_2, \ldots, \bp_{k+1}]$ as constructed in Lemma~\ref{lemma:submatrix-same-eigenvalue} where $\bp_1$ is an eigenvector of $\bA_{k+1}$ corresponding to eigenvalue $\lambda_1$ and its norm is 1, $\bp_2, \ldots, \bp_{k+1}$ are orthonormal to $\bp_1$}. Let the other $k$ eigenvalues of $\bA_{k+1}$ be $\lambda_2, \lambda_3, \ldots, \lambda_{k+1}$. Since we suppose for $n=k$, the theorem is true, we can find a matrix $\bA_{k}$ with eigenvalues $\lambda_2, \lambda_3, \ldots, \lambda_{k+1}$. So we have the following property by Lemma~\ref{lemma:submatrix-same-eigenvalue}:
$$
\bP_{k+1}^\top \bA_{k+1} \bP_{k+1} =
\begin{bmatrix}
	\lambda &\bzero \\
	\bzero & \bA_k
\end{bmatrix} \qquad
\text{and} \qquad
\bA_{k+1} \bP_{k+1} =
\bP_{k+1}
\begin{bmatrix}
	\lambda_1 &\bzero \\
	\bzero & \bA_k
\end{bmatrix}.
$$
Let
$
\bQ_{k+1} = \bP_{k+1}
\begin{bmatrix}
1 &\bzero \\
\bzero & \bQ_k
\end{bmatrix}.
$
Then, it follows that
$$
\begin{aligned}
\bA_{k+1} \bQ_{k+1} &=
\bA_{k+1}
\bP_{k+1}
\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix}\\
&=
\bP_{k+1}
\begin{bmatrix}
	\lambda_1 &\bzero \\
	\bzero & \bA_k
\end{bmatrix}
\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix} \\
&=
\bP_{k+1}
\begin{bmatrix}
\lambda_1 & \bzero \\
\bzero & \bA_k\bQ_k
\end{bmatrix}\\
&=
\bP_{k+1}
\begin{bmatrix}
	\lambda_1 & \bzero \\
	\bzero & \bQ_k \bU_k
\end{bmatrix}\qquad &\text{(By the assumption for $n=k$)} \\
&=\bP_{k+1}
\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix}
\begin{bmatrix}
	\lambda_1 &\bzero \\
	\bzero & \bU_k
\end{bmatrix}\\
&=\bQ_{k+1}\bU_{k+1}. \qquad  &\text{($\bU_{k+1} = \begin{bmatrix}
		\lambda_1 &\bzero \\
		\bzero & \bU_k
	\end{bmatrix}$)}
\end{aligned}
$$
We then have $\bA_{k+1} = \bQ_{k+1}\bU_{k+1}\bQ_{k+1}^\top$, where $\bU_{k+1}$ is an upper triangular matrix, and $\bQ_{k+1}$ is an orthogonal matrix since $\bP_{k+1}$ and
$\begin{bmatrix}
	1 &\bzero \\
	\bzero & \bQ_k
\end{bmatrix}$ are both orthogonal matrices.
\end{proof}

\section{Other Forms of the Schur Decomposition}\label{section:other-form-schur-decom}

From the proof of the Schur decomposition, we obtain the upper triangular matrix $\bU_{k+1}$ by appending the eigenvalue $\lambda_1$ to $\bU_k$. From this process, the values on the diagonal are always eigenvalues. Therefore, we can decompose the upper triangular into two parts.

\begin{corollaryHigh}[Form 2 of Schur Decomposition]
Any square matrix $\bA\in \real^{n\times n}$ with real eigenvalues can be factored as
$$
\bQ^\top\bA\bQ = \bLambda +\bT, \qquad \textbf{or} \qquad \bA = \bQ(\bLambda +\bT)\bQ^\top,
$$
where $\bQ$ is an orthogonal matrix, $\bLambda=diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$ is a diagonal matrix containing the eigenvalues of $\bA$, and $\bT$ is a \textit{strictly upper triangular} matrix.
\end{corollaryHigh}
A strictly upper triangular matrix is an upper triangular matrix having 0's along the diagonal as well as the lower portion. Another proof of this decomposition is that $\bA$ and $\bU$ (where $\bU = \bQ^\top\bA\bQ$) are similar matrices so that they have the same eigenvalues (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). And the eigenvalues of any upper triangular matrices are on the diagonal.
To see this, for any upper triangular matrix $\bR \in \real^{n\times n}$ where the diagonal values are $r_{ii}$ for all $i\in \{1,2,\ldots,n\}$. We have
$$
\bR \be_i = r_{ii}\be_i,
$$
where $\be_i$ is the $i$-th basis vector in $\real^n$, i.e., $\be_i$ is the $i$-th column of the $n\times n$ identity matrix $\bI_n$.
So we can decompose $\bU$ into $\bLambda$ and $\bT$.

A final observation on the second form of the Schur decomposition is shown as follows. From $\bA\bQ = \bQ(\bLambda +\bT)$, it follows that
$$
\bA \bq_k = \lambda_k\bq_k + \sum_{i=1}^{k-1}t_{ik}\bq_i,
$$
where $t_{ik}$ is the ($i,k$)-th entry of $\bT$. The form is quite close to the eigenvalue decomposition. Nevertheless, the columns become orthonormal bases and these orthonormal bases are correlated.

\part{Reduction to Hessenberg, Tridiagonal, and Bidiagonal Form}

\chapter{Hessenberg Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section*{Preliminary}
In real applications, we often want to factor matrix $\bA$ into two orthogonal matrices $\bA =\bQ\bLambda\bQ^\top$ where $\bLambda$ is diagonal or upper triangular, e.g., eigen analysis via the Schur decomposition and principal component analysis (PCA) via the spectral decomposition. This can be computed via a sequence of \textit{orthogonal similarity transformations}:
$$
\underbrace{\bQ_k^\top\ldots \bQ_2^\top \bQ_1^\top}_{\bQ^\top} \bA \underbrace{\bQ_1\bQ_2\ldots\bQ_k}_{\bQ}
$$
which converges to $\bLambda$. However, this transformation will always be very hard to handle in practice, for example, via the Householder reflectors. Following the QR decomposition via the Householder example, \footnote{where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed. } the sequence of orthogonal similarity transformation can be constructed via the Householder reflectors:
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bH_1\times }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times \bH_1^\top }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix},
\end{aligned}
$$
where the left Householder introduces zeros in the first column below the main diagonal (see Section~\ref{section:qr-via-householder}, p.~\pageref{section:qr-via-householder}), and unfortunately the right Householder will destroy the zeros introduced by the left Householder.

However, if we are less ambitious to modify the algorithms into two phases, where the first phase transforms into a Hessenberg matrix (Definition~\ref{definition:upper-hessenbert}, p.~\pageref{definition:upper-hessenbert}) or a tridiagonal matrix (Definition~\ref{definition:tridiagonal-hessenbert}, p.~\pageref{definition:tridiagonal-hessenbert}). And if we find a second phase algorithm to transform the results from the first one to the goal we want to find, then we complete the algorithm:
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes
	\end{sbmatrix}
	&\stackrel{\bH_1\times }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times \bH_1^\top }{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix},
\end{aligned}
$$
where the left Householder will not influence the first row, and the right Householder will not influence the first column.
A phase 2 \footnote{which is usually an iterative algorithm.} algorithm to find the triangular matrix is shown as follows:
$$
\begin{aligned}
	\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top \bH_3^\top}
		\boxtimes & \boxtimes  & \boxtimes  & \boxtimes & \boxtimes  \\
		\boxtimes & \boxtimes  & \boxtimes  & \boxtimes & \boxtimes  \\
		0 & \boxtimes  & \boxtimes  & \boxtimes & \boxtimes  \\
		0 & 0 & \boxtimes  & \boxtimes & \boxtimes  \\
		0 & 0 & 0 & \boxtimes  & \boxtimes
	\end{sbmatrix}
	\stackrel{\text{Phase 2} }{\longrightarrow}
	\begin{sbmatrix}{\bLambda}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & \bm{0}  & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & \bm{0}  & \bm{\boxtimes}& \bm{\boxtimes} \\
		0 & 0 & 0 & \bm{0}  & \bm{\boxtimes}
	\end{sbmatrix}
\end{aligned}
$$

From the discussion above, to compute the spectral decomposition, Schur decomposition, or singular value decomposition (SVD), we usually reach a compromise to calculate the Hessenberg, tridiagonal, or bidiagonal form in the first phase and leave the second phase to finish the rest \citep{van2012families, van2014restructuring, trefethen1997numerical}.

\section{Hessenberg Decomposition}

We firstly give the rigorous definition of the upper Hessenberg matrix.
\begin{definition}[Upper Hessenberg Matrix\index{Hessenbert matrix}]\label{definition:upper-hessenbert}
An \textbf{upper Hessenberg matrix} is a square matrix where all the entries below the first diagonal (i.e., the ones below the \textbf{main diagonal}) (a.k.a., \textbf{lower subdiagonal}) are zeros. Similarly, a lower Hessenberg matrix is a square matrix where all the entries above the first diagonal (i.e., the ones above the main diagonal) are zeros.

 The definition of the upper Hessenberg can also be extended to rectangular matrices, and the form can be implied from the context.

 In matrix language, for any matrix $\bH\in \real^{n\times n}$, and the entry ($i,j$) denoted by $h_{ij}$ for all $i,j\in \{1,2,\ldots, n\}$. Then $\bH$ with $h_{ij}=0$ for all $i\geq j+2$ is known as a Hessenberg matrix.

 Let $i$ denote the smallest positive integer for which $h_{i+1, i}=0$ where $i\in \{1,2,\ldots, n-1\}$, then $\bH$ is \textbf{unreduced} if $i=n-1$.
\end{definition}

Take a $5\times 5$ matrix as an example, the lower triangular below the lower subdiagonal are zero in the upper Hessenberg matrix:
$$
\begin{sbmatrix}{possibly\,\, unreduced}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}
\qquad
\text{or}
\qquad
\begin{sbmatrix}{reduced}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	0 & 0 & \textcolor{blue}{0} & \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}.
$$
Then we have the following Hessenberg decomposition:
\begin{theoremHigh}[Hessenberg Decomposition]\label{theorem:hessenberg-decom}
Every $n\times n$ square matrix $\bA$ can be factored as
$$
\bA = \bQ\bH\bQ^\top \qquad \text{or} \qquad \bH = \bQ^\top \bA\bQ,
$$
where $\bH$ is an upper Hessenberg matrix, and $\bQ$ is an orthogonal matrix.
\end{theoremHigh}
It's not hard to find that a lower Hessenberg decomposition of $\bA^\top$ is given by $\bA^\top = \bQ\bH^\top\bQ^\top$ if $\bA$ admits the Hessenberg decomposition $\bA = \bQ\bH\bQ^\top$. The Hessenberg decomposition shares a similar form as the QR decomposition in that they both reduce a matrix into a sparse form where the lower parts of both are zero.
\begin{remark}[Why Hessenberg Decomposition]
We will see that the zeros introduced into $\bH$ from $\bA$ is accomplished by the left orthogonal matrix $\bQ$ (same as the QR decomposition) and the right orthogonal matrix $\bQ^\top$ here does not transform the matrix into any better or simple form. Then why do we want the Hessenberg decomposition rather than just a QR decomposition which has a simpler structure in that it even has zeros in the lower subdiagonal? As mentioned above, the answer is that the Hessenberg decomposition is usually used by other algorithms as a phase 1 step to find a decomposition that factor the matrix into two orthogonal matrices, e.g., SVD, UTV, and so on. And if we employ an aggressive algorithm that even favors zeros in the lower subdiagonal (again, as in the QR decomposition), the right orthogonal transform $\bQ^\top$ will destroy the zeros that can be seen very shortly.

On the other hand, the form $\bA = \bQ\bH\bQ^\top$ on $\bH$ is known as the \textit{orthogonal similarity transformation} (Definition~\ref{definition:similar-matrices}, p.~\pageref{definition:similar-matrices}) on $\bA$ such that the eigenvalues, rank and trace of $\bA$ and $\bH$ are the same (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). Then if we want to study the properties of $\bA$, exploration on $\bH$ can be a relatively simpler task.
\end{remark}

\section{Similarity Transformation and Orthogonal Similarity Transformation}
As mentioned previously, the Hessenberg decomposition introduced in this section, the tridiagonal decomposition in the next section, the Schur decomposition (Theorem~\ref{theorem:schur-decomposition}, p.~\pageref{theorem:schur-decomposition}), and the spectral decomposition (Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}) share a similar form that transforms the matrix into a similar matrix. We now give the rigorous definition of similar matrices and similarity transformations. \index{Similar matrices}\index{Similarity tansformation}
\begin{definition}[Similar Matrices and Similarity Transformation]\label{definition:similar-matrices}
	$\bA$ and $\bB$ are called \textbf{similar matrices} if there exists a nonsingular matrix $\bP$ such that $\bB = \bP\bA\bP^{-1}$.

	In words, for any nonsingular matrix $\bP$, the matrices $\bA$ and $\bP\bA\bP^{-1}$ are similar matrices.
	And in this sense, given the nonsingular matrix $\bP$, $\bP\bA\bP^{-1}$
	is called a \textbf{similarity transformation} applied to matrix $\bA$.

	Moreover, when $\bP$ is orthogonal, then $\bP\bA\bP^\top$ is also known as the \textbf{orthogonal similarity transformation} of $\bA$.
\end{definition}
The difference between the similarity transformation and orthogonal similarity transformation is partly explained in the sense of coordinate transformation (Section~\ref{section:coordinate-transformation}, p.~\pageref{section:coordinate-transformation}). Now we prove the important properties of similar matrices that will be shown very useful in the sequel.
\begin{lemma}[Eigenvalue, Trace and Rank of Similar Matrices\index{Trace}]\label{lemma:eigenvalue-similar-matrices}
	Any eigenvalue of $\bA$ is also an eigenvalue of $\bP\bA\bP^{-1}$. The converse is also true that any eigenvalue of $\bP\bA\bP^{-1}$ is also an eigenvalue of $\bA$. I.e., $\Lambda(\bA) = \Lambda(\bB)$, where $\Lambda(\bX)$ is the spectrum of matrix $\bX$ (Definition~\ref{definition:spectrum}, p.~\pageref{definition:spectrum}).

	And also the trace and rank of $\bA$ are equal to those of matrix $\bP\bA\bP^{-1}$ for any nonsingular matrix $\bP$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:eigenvalue-similar-matrices}]
	For any eigenvalue $\lambda$ of $\bA$, we have $\bA\bx =\lambda \bx$. Then $\lambda \bP\bx = \bP\bA\bP^{-1} \bP\bx$ such that $\bP\bx$ is an eigenvector of $\bP\bA\bP^{-1}$ corresponding to $\lambda$.

	Similarly, for any eigenvalue $\lambda$ of $\bP\bA\bP^{-1}$, we have $\bP\bA\bP^{-1} \bx = \lambda \bx$. Then $\bA\bP^{-1} \bx = \lambda \bP^{-1}\bx$ such that $\bP^{-1}\bx$ is an eigenvector of $\bA$ corresponding to $\lambda$.

	For the trace of $\bP\bA\bP^{-1}$, we have $trace(\bP\bA\bP^{-1}) = trace(\bA\bP^{-1}\bP) = trace(\bA)$, where the first equality comes from the fact that the
	trace of a product is invariant under cyclical permutations of the factors:
	\begin{equation}
		trace(\bA\bB\bC) = trace(\bB\bC\bA) = trace(\bC\bA\bB), \nonumber
	\end{equation}
	if all $\bA\bB\bC$, $\bB\bC\bA$, and $\bC\bA\bB$ exist.

	For the rank of $\bP\bA\bP^{-1}$, we separate it into two claims as follows.
	\paragraph{Rank claim 1: $rank(\bZ\bA)=rank(\bA)$ if $\bZ$ is nonsingular}
	We will first show that $rank(\bZ\bA)=rank(\bA)$ if $\bZ$ is nonsingular. For any vector $\bn$ in the null space of $\bA$, that is $\bA\bn = \bzero$. Thus, $\bZ\bA\bn = \bzero$, that is, $\bn$ is also in the null space of $\bZ\bA$. And this implies $\nspace(\bA)\subseteq \nspace(\bZ\bA)$.

	Conversely, for any vector $\bmm$ in the null space of $\bZ\bA$, that is $\bZ\bA\bmm = \bzero$, we have $\bA\bmm = \bZ^{-1} \bzero=\bzero$. That is, $\bmm$ is also in the null space of $\bA$. And this indicates $\nspace(\bZ\bA)\subseteq \nspace(\bA)$.

	By ``sandwiching", the above two arguments imply
	$$
	\nspace(\bA) = \nspace(\bZ\bA)\quad  \longrightarrow \quad rank(\bZ\bA)=rank(\bA).
	$$

	\paragraph{Rank claim 2: $rank(\bA\bZ)=rank(\bA)$ if $\bZ$ is nonsingular}
	We notice that the row rank is equal to the column rank of any matrix (Corollary~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}). Then $rank(\bA\bZ) = rank(\bZ^\top\bA^\top)$. Since $\bZ^\top$ is nonsingular, by claim 1, we have $rank(\bZ^\top\bA^\top) = rank(\bA^\top) = rank(\bA)$ where the last equality is again from the fact that the row rank is equal to the column rank of any matrix. This results in $rank(\bA\bZ)=rank(\bA)$ as claimed.

	Since $\bP, \bP^{-1}$ are nonsingular, we then have $rank(\bP\bA\bP^{-1}) = rank(\bA\bP^{-1}) = rank(\bA)$ where the first equality is from claim 1 and the second equality is from claim 2. We complete the proof.
\end{proof}

\section{Existence of the Hessenberg Decomposition}
We will prove that any $n\times n$ matrix can be reduced to Hessenberg form via a sequence of Householder transformations that are applied from the left and the right to the matrix  in an interleaved manner.
Previously, we utilized a Householder reflector to triangularize matrices and introduce zeros below the diagonal to obtain the QR decomposition. A similar approach can be applied to introduce zeros below the subdiagonal.

Before introducing the mathematical construction of such decomposition, we emphasize the following remark which will be very useful in the finding of the decomposition.
\begin{remark}[Left and Right Multiplied by a Matrix with Block Identity]\label{remark:left-right-identity}
For square matrix $\bA\in \real^{n\times n}$, and a matrix
$$
\bB = \begin{bmatrix}
	\bI_k &\bzero \\
	\bzero & \bB_{n-k}
\end{bmatrix},
$$
where $\bI_k$ is a $k\times k$ identity matrix. Then $\bB\bA$ will not change the first $k$ rows of $\bA$, and $\bA\bB$ will not change the first $k$ columns of $\bA$.
\end{remark}
The proof of this remark is trivial.

\subsection*{\textbf{First Step: Introduce Zeros for the First Column}}
Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ be the column partitions of $\bA$, and each $\ba_i \in \real^{n}$. Suppose $\bar{\ba}_1, \bar{\ba}_2, \ldots, \bar{\ba}_n \in \real^{n-1}$ are vectors removing the first component in $\ba_i$'s. Let
$$
 r_1 = ||\bar{\ba}_1||, \qquad \bu_1 = \frac{\bar{\ba}_1 - r_1 \be_1}{||\bar{\ba}_1 - r_1 \be_1||}, \qquad \text{and}\qquad \widetilde{\bH}_1 = \bI - 2\bu_1\bu_1^\top \in \real^{(n-1)\times (n-1)},
$$
where $\be_1$ here is the first basis for $\real^{n-1}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \real^{n-1}$. To introduce zeros below the subdiagonal and operate on the submatrix $\bA_{2:n,1:n}$, we append the Householder reflector into
$$
\bH_1 = \begin{bmatrix}
	1 &\bzero \\
	\bzero & \widetilde{\bH}_1
\end{bmatrix},
$$
in which case, $\bH_1\bA$ will introduce zeros in the first column of $\bA$ below entry (2,1). The first row of $\bA$ will not be affected at all and kept unchanged by Remark~\ref{remark:left-right-identity}. And we can easily verify that both $\bH_1$ and $\widetilde{\bH}_1$ are orthogonal matrices and they are symmetric (from the definition of the Householder reflector). To have the form in Theorem~\ref{theorem:hessenberg-decom}, we multiply $\bH_1\bA$ on the right by $\bH_1^\top$ which results in $\bH_1\bA\bH_1^\top$. The $\bH_1^\top$ on the right will not change the first column of $\bH_1\bA$ and thus keep the zeros introduced in the first column.

An example of a $5\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	\stackrel{\bH_1\times}{\rightarrow}
	&\begin{sbmatrix}{\bH_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_1^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}\\
\end{aligned}
$$

\subsection*{\textbf{Second Step: Introduce Zeros for the Second Column}}
Let $\bB = \bH_1\bA\bH_1^\top$, where the entries in the first column below entry (2,1) are all zeros. And the goal is to introduce zeros in the second column below entry (3,2). Let $\bB_2 = \bB_{2:n,2:n}=[\bb_1, \bb_2, \ldots, \bb_{n-1}]$. Suppose again $\bar{\bb}_1, \bar{\bb}_2, \ldots, \bar{\bb}_{n-1} \in \real^{n-2}$ are vectors removing the first component in $\bb_i$'s. We can again construct a Householder reflector
\begin{equation}\label{equation:householder-qr-lengthr}
r_1 = ||\bar{\bb}_1||,  \qquad \bu_2 = \frac{\bar{\bb}_1 - r_1 \be_1}{||\bar{\bb}_1 - r_1 \be_1||}, \qquad \text{and}\qquad \widetilde{\bH}_2 = \bI - 2\bu_2\bu_2^\top\in \real^{(n-2)\times (n-2)},
\end{equation}
where $\be_1$ now is the first basis for $\real^{n-2}$. To introduce zeros below the subdiagonal and operate on the submatrix $\bB_{3:n,1:n}$, we append the Householder reflector into
$$
\bH_2 = \begin{bmatrix}
	\bI_2 &\bzero \\
	\bzero & \widetilde{\bH}_2
\end{bmatrix},
$$
where $\bI_2$ is a $2\times 2$ identity matrix. We can see that $\bH_2\bH_1\bA\bH_1^\top$ will not change the first two rows of $\bH_1\bA\bH_1^\top$, and as the Householder cannot reflect a zero vector such that the zeros in the first column will be kept. Again, putting $\bH_2^\top$ on the right of $\bH_2\bH_1\bA\bH_1^\top$ will not change the first 2 columns so that the zeros will be kept.

Following the example of a $5\times 5$ matrix, the second step is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
	\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		0 & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		0 & \boxtimes & \boxtimes & \boxtimes &  \boxtimes  \\
		0 & \boxtimes & \boxtimes & \boxtimes &  \boxtimes
	\end{sbmatrix}
	\stackrel{\bH_2\times}{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\times\bH_2^\top}{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
		\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
\end{aligned}
$$

The same process can go on, and there are $n-2$ such steps. We will finally triangularize by
$$
\bH = \bH_{n-2} \bH_{n-3}\ldots\bH_1 \bA\bH_1^\top\bH_2^\top\ldots\bH_{n-2}^\top.
$$
And since $\bH_i$'s are symmetric and orthogonal, the above equation can be simply reduced to
$$
\bH =\bH_{n-2} \bH_{n-3}\ldots\bH_1 \bA\bH_1\bH_2\ldots\bH_{n-2}.
$$
Note here only $n-2$ such stages exist rather than $n-1$ or $n$. We will verify this number of steps by the example below.
The example of a $5\times 5$ matrix as a whole is shown as follows where again $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\begin{mdframed}[hidealllines=\mdframehidelineNote,backgroundcolor=\mdframecolorSkip, frametitle={A Complete Example of Hessenberg Decomposition}]
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bH_1\times}{\rightarrow}
&\begin{sbmatrix}{\bH_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bH_1^\top}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA\bH_1^\top}
	\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\end{aligned}
$$
$$
\begin{aligned}
\qquad \qquad\qquad \qquad \qquad \stackrel{\bH_2\times}{\rightarrow}
&\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bH_2^\top}{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
	\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\stackrel{\bH_3\times}{\rightarrow}
&\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bH_3^\top}{\rightarrow}
\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bH_1^\top\bH_2^\top\bH_3^\top}
	\boxtimes & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix},
\end{aligned}
$$
\end{mdframed}

\section{Properties of the Hessenberg Decomposition}\label{section:hessenberg-decomposition}
The Hessenberg decomposition is not unique in the different ways to construct the Householder reflectors (say Equation~\eqref{equation:householder-qr-lengthr}, p.~\pageref{equation:householder-qr-lengthr}). However, under mild conditions, we can claim a similar structure in different decompositions.

\begin{theorem}[Implicit Q Theorem for Hessenberg Decomposition\index{Implicit Q theorem}]\label{theorem:implicit-q-hessenberg}
Suppose two Hessenberg decompositions of matrix $\bA\in \real^{n\times n}$ are given by $\bA=\bU\bH\bU^\top=\bV\bG\bV^\top$ where $\bU=[\bu_1, \bu_2, \ldots, \bu_n]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU,\bV$. Suppose further that $k$ is the smallest positive integer for which $h_{k+1,k}=0$ where $h_{ij}$ is the entry $(i,j)$ of $\bH$. Then
\begin{itemize}
\item If $\bu_1=\bv_1$, then $\bu_i = \pm \bv_i$ and $|h_{i,i-1}| = |g_{i,i-1}|$ for $i\in \{2,3,\ldots,k\}$.
\item When $k=n-1$, the Hessenberg matrix $\bH$ is known as \textit{unreduced}. However, if $k<n-1$, then $g_{k+1,k}=0$.
\end{itemize}
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:implicit-q-hessenberg}]
Define the orthogonal matrix $\bQ=\bV^\top\bU$ and we have
$$
\left.
\begin{aligned}
\bG\bQ &= \bV^\top\bA\bV \bV^\top\bU = \bV^\top\bA\bU \\
\bQ\bH &= \bV^\top\bU \bU^\top\bA\bU = \bV^\top\bA\bU
\end{aligned}
\right\}
\leadto
\bG\bQ = \bQ\bH,
$$
the $(i-1)$-th column of each can be represented as
$$
\bG\bq_{i-1} = \bQ\bh_{i-1},
$$
where $\bq_{i-1}$ and $\bh_{i-1}$ are the $(i-1)$-th column of $\bQ$ and $\bH$ respectively. Since $h_{l,i-1}=0$ for $l\geq i+1$ (by the definition of upper Hessenberg matrices), $\bQ\bh_{i-1}$ can be represented as
$$
\bQ\bh_{i-1} = \sum_{j=1}^{i} h_{j,i-1} \bq_j = h_{i,i-1}\bq_i + \sum_{j=1}^{i-1} h_{j,i-1} \bq_j.
$$
Combine the two findings above, it follows that
$$
h_{i,i-1}\bq_i  = \bG\bq_{i-1} - \sum_{j=1}^{i-1} h_{j,i-1} \bq_j.
$$
A moment of reflexion reveals that $[\bq_1, \bq_2, \ldots,\bq_k]$ is upper triangular. And since $\bQ$ is orthogonal, it must be diagonal and each value on the diagonal is in $\{-1, 1\}$ for $i\in \{2,\ldots, k\}$. Then, $\bq_1=\be_1$ and $\bq_i = \pm \be_i$ for $i\in \{2,\ldots, k\}$. Further, since $\bq_i =\bV^\top\bu_i$ and $h_{i,i-1}=\bq_i^\top (\bG\bq_{i-1} - \sum_{j=1}^{i-1} h_{j,i-1} \bq_j)=\bq_i^\top \bG\bq_{i-1}$. For $i\in \{2,\ldots,k\}$, $\bq_i^\top \bG\bq_{i-1}$ is just $\pm g_{i,i-1}$. It follows that
$$
\begin{aligned}
|h_{i,i-1}| &= |g_{i,i-1}|, \qquad &\forall i\in \{2,\ldots,k\},\\
\bu_i &= \pm\bv_i, \qquad &\forall i\in \{2,\ldots,k\}.
\end{aligned}
$$
This proves the first part. For the second part, if $k<n-1$,
$$
\begin{aligned}
g_{k+1,k} &= \be_{k+1}^\top\bG\be_{k} = \pm \be_{k+1}^\top\underbrace{\bG\bQ}_{\bQ\bH} \be_{k} =  \pm \be_{k+1}^\top\underbrace{\bQ\bH \be_{k}}_{\text{$k$-th column of $\bQ\bH$}} \\
&= \pm \be_{k+1}^\top \bQ\bh_k = \pm \be_{k+1}^\top \sum_{j}^{k+1} h_{jk}\bq_j
=\pm \be_{k+1}^\top \sum_{j}^{\textcolor{blue}{k}} h_{jk}\bq_j=0,
\end{aligned}
$$
where the penultimate equality is from the assumption that $h_{k+1,k}=0$. This completes the proof.
\end{proof}
We observe from the above theorem, when two Hessenberg decompositions of matrix $\bA$ are both unreduced and have the same first column in the orthogonal matrices, then the Hessenberg matrices $\bH, \bG$ are similar matrices such that $\bH = \bD\bG\bD^{-1}$ where $\bD=\diag(\pm 1, \pm 1, \ldots, \pm 1)$. \textit{Moreover, and most importantly, if we restrict the elements in the lower subdiagonal of the Hessenberg matrix $\bH$ to be positive (if possible), then the Hessenberg decomposition $\bA=\bQ\bH\bQ^\top$ is uniquely determined by $\bA$ and the first column of $\bQ$.} This is similar to what we have claimed on the uniqueness of the QR decomposition (Corollary~\ref{corollary:unique-qr}, p.~\pageref{corollary:unique-qr}) and it is important to reduce the complexity of the QR algorithm for computing the singular value decomposition or eigenvalues of a matrix in general \citep{lu2021numerical}.

The next finding involves a Krylov matrix defined as follows:
\begin{definition}[Krylov Matrix\index{Krylov matrix}]\label{definition:krylov-matrix}
	Given a matrix $\bA\in \real^{n\times n}$, a vector $\bq\in \real^n$, and a scalar $k$, the \textit{Krylov matrix} is defined to be
	$$
	\bK(\bA, \bq, k) =
	\begin{bmatrix}
		\bq, & \bA\bq, & \ldots, & \bA^{k-1}\bq
	\end{bmatrix}
	\in \real^{n\times n}.
	$$
\end{definition}

\begin{theorem}[Reduced Hessenberg]\label{theorem:implicit-q-hessenberg-v2}
	Suppose there exists an orthogonal matrix $\bQ$ such that $\bA\in \real^{n\times n}$ can be factored as $\bA = \bQ\bH\bQ^\top$. Then $\bQ^\top\bA\bQ=\bH$ is an unreduced upper Hessenberg matrix if and only if $\bR=\bQ^\top \bK(\bA, \bq_1, n)$ is nonsingular and upper triangular where $\bq_1$ is the first column of $\bQ$.

		\item If $\bR$ is singular and $k$ is the smallest index so that $r_{kk}=0$, then $k$ is also the smallest index that $h_{k,k-1}=0$.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:implicit-q-hessenberg-v2}] We prove by forward implication and converse implication separately as follows:
\paragraph{Forward implication}
Suppose $\bH$ is unreduced, write out the following matrix
$$
\bR = \bQ^\top \bK(\bA, \bq_1, n) = [\be_1, \bH\be_1, \ldots, \bH^{n-1}\be_1],
$$
where $\bR$ is upper triangular with $r_{11}=1$ obviously. Observe that $r_{ii} = h_{21}h_{32}\ldots h_{i,i-1}$ for $i\in \{2,3,\ldots, n\}$. When $\bH$ is unreduced, $\bR$ is nonsingular as well.
\paragraph{Converse implication}
Now suppose $\bR$ is upper triangular and nonsingular, we observe that $\br_{k+1} = \bH\br_{k}$ such that the $(k+2:n)$-th rows of $\bH$ are zero and $h_{k+1,k}\neq 0$ for $k\in \{1,2,\ldots, n-1\}$. Then $\bH$ is unreduced.

If $\bR$ is singular and $k$ is the smallest index so that $r_{kk}=0$, then
$$
\left.
\begin{aligned}
r_{k-1,k-1}&=h_{21}h_{32}\ldots h_{k-1,k-2}&\neq 0 \\
r_{kk}&=h_{21}h_{32}\ldots h_{k-1,k-2} h_{k,k-1}&= 0
\end{aligned}
\right\}
\leadto
h_{k,k-1} =0,
$$
from which the result follows.
\end{proof}

\chapter{Tridiagonal Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Tridiagonal Decomposition: Hessenberg in Symmetric Matrices}

We firstly give the formal definition of the tridiagonal matrix.
\begin{definition}[Tridiagonal Matrix\index{Tridiagonal matrix}]\label{definition:tridiagonal-hessenbert}
	A tridiagonal matrix is a square matrix where all the entries below the lower subdiagonal and the entries above the upper subdiagonal are zeros. I.e., the tridiagonal matrix is a \textbf{band matrix}.

	 The definition of the tridiagonal matrix can also be extended to rectangular matrices, and the form can be implied from the context.

	 In matrix language, for any matrix $\bT\in \real^{n\times n}$, and the entry ($i,j$) denoted by $t_{ij}$ for all $i,j\in \{1,2,\ldots, n\}$. Then $\bT$ with $t_{ij}=0$ for all $i\geq j+2$ and $i \leq j-2$ is known as a tridiagonal matrix.

	 Let $i$ denote the smallest positive integer for which $h_{i+1, i}=0$ where $i\in \{1,2,\ldots, n-1\}$, then $\bT$ is \textbf{unreduced} if $i=n-1$.
\end{definition}

Take a $5\times 5$ matrix as an example, the lower triangular below the lower subdiagonal and upper triangular above the upper subdiagonal are zero in the tridiagonal matrix:
$$
\begin{sbmatrix}{possibly\,\, unreduced}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
	0 & \boxtimes & \boxtimes & \boxtimes & 0\\
	0 & 0 & \boxtimes & \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}
\qquad
\begin{sbmatrix}{reduced}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
	0 & \boxtimes & \boxtimes & \boxtimes & 0\\
	0 & 0 & \textcolor{blue}{0} & \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes
\end{sbmatrix}.
$$
Obviously, a tridiagonal matrix is a special case of an upper Hessenberg matrix.
Then we have the following tridiagonal decomposition:
\begin{theoremHigh}[Tridiagonal Decomposition]\label{theorem:tridiagonal-decom}
	Every $n\times n$ symmetric matrix $\bA$ can be factored as
	$$
	\bA = \bQ\bT\bQ^\top \qquad \text{or} \qquad \bT = \bQ^\top \bA\bQ,
	$$
	where $\bT$ is a \textit{symmetric} tridiagonal matrix, and $\bQ$ is an orthogonal matrix.
\end{theoremHigh}
The existence of the tridiagonal matrix is trivial by applying the Hessenberg decomposition to symmetric matrix $\bA$.

\section{Properties of the Tridiagonal Decomposition}\label{section:tridiagonal-decomposition}
Similarly, the tridiagonal decomposition is not unique. However, and most importantly, if we restrict the elements in the lower subdiagonal of the tridiagonal matrix $\bT$ to be positive (if possible), then the tridiagonal decomposition $\bA=\bQ\bT\bQ^\top$ is uniquely determined by $\bA$ and the first column of $\bQ$.
\begin{theorem}[Implicit Q Theorem for Tridiagonal\index{Implicit Q theorem}]\label{theorem:implicit-q-tridiagonal}
	Suppose two Tridiagonal decompositions of symmetric matrix $\bA\in \real^{n\times n}$ are given by $\bA=\bU\bT\bU^\top=\bV\bG\bV^\top$ where $\bU=[\bu_1, \bu_2, \ldots, \bu_n]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU,\bV$. Suppose further that $k$ is the smallest positive integer for which $t_{k+1,k}=0$ where $t_{ij}$ is the entry $(i,j)$ of $\bT$. Then
	\begin{itemize}
		\item If $\bu_1=\bv_1$, then $\bu_i = \pm \bv_i$ and $|t_{i,i-1}| = |g_{i,i-1}|$ for $i\in \{2,3,\ldots,k\}$.
		\item When $k=n-1$, the tridiagonal matrix $\bT$ is known as unreduced. However, if $k<n-1$, then $g_{k+1,k}=0$.
	\end{itemize}
\end{theorem}
From the above theorem, we observe that  if we restrict the elements in the lower sub-diagonal of the tridiagonal matrix $\bT$ to be positive (if possible), i.e., \textit{unreduced}, then the tridiagonal decomposition $\bA=\bQ\bT\bQ^\top$ is uniquely determined by $\bA$ and the first column of $\bQ$. This again is similar to what we have claimed on the uniqueness of the QR decomposition (Corollary~\ref{corollary:unique-qr}, p.~\pageref{corollary:unique-qr}).

Similarly, a reduced tridiagonal decomposition can be obtained from the implication of the Krylov matrix (Definition~\ref{definition:krylov-matrix}, p.~\pageref{definition:krylov-matrix}).
\begin{theorem}[Reduced Tridiagonal]\label{theorem:implicit-q-tridiagonal-v2}
	Suppose there exists an orthogonal matrix $\bQ$ such that $\bA\in \real^{n\times n}$ can be factored as $\bA = \bQ\bT\bQ^\top$. Then $\bQ^\top\bA\bQ=\bT$ is an unreduced tridiagonal matrix if and only if $\bR=\bQ^\top \bK(\bA, \bq_1, n)$ is nonsingular and upper triangular where $\bq_1$ is the first column of $\bQ$.

	\item If $\bR$ is singular and $k$ is the smallest index so that $r_{kk}=0$, then $k$ is also the smallest index that $t_{k,k-1}=0$.
\end{theorem}
The proofs of the above two theorems are the same as those in Theorem~\ref{theorem:implicit-q-hessenberg} and~\ref{theorem:implicit-q-hessenberg-v2}.

\chapter{Bidiagonal Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup
\section{Bidiagonal Decomposition}\label{section:bidiagonal-decompo}
We firstly give the rigorous definition of the upper Bidiagonal matrix as follows:
\begin{definition}[Upper Bidiagonal Matrix\index{Bidiagonal matrix}]\label{definition:bidiagonal-matrix}
	An upper bidiagonal matrix is a square matrix which  is a banded matrix with non-zero entries along the \textbf{main diagonal} and the \textbf{upper subdiagonal} (i.e., the ones above the main diagonal). This means there are exactly two nonzero diagonals in the matrix.

	 Furthermore, when the diagonal below the main diagonal has the non-zero entries, the matrix is lower bidiagonal.

	 The definition of bidigonal matrices can also be extended to rectangular matrices, and the form can be implied from the context.
\end{definition}

Take a $7\times 5$ matrix as an example, the lower triangular below the main diagonal and the upper triangular above the upper subdiagonal are zero in the upper bidiagonal matrix:
$$
\begin{bmatrix}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	0 & \boxtimes & \boxtimes & 0 & 0\\
	0 & 0 & \boxtimes & \boxtimes & 0\\
	0 & 0 & 0 & \boxtimes & \boxtimes\\
	0 & 0 & 0 & 0 & \boxtimes\\
	0 & 0 & 0 & 0 & 0\\
	0 & 0 & 0 & 0 & 0
\end{bmatrix}.
$$
Then we have the following bidiagonal decomposition:
\begin{theoremHigh}[Bidiagonal Decomposition]\label{theorem:Golub-Kahan-Bidiagonalization-decom}
	Every $m\times n$ matrix $\bA$ can be factored as
	$$
	\bA = \bU\bB\bV^\top \qquad \text{or} \qquad \bB= \bU^\top \bA\bV,
	$$
	where $\bB$ is an upper bidiagonal matrix, and $\bU, \bV$ are orthogonal matrices.
\end{theoremHigh}
We will see the bidiagonalization resembles the form of a singular value decomposition where the only difference is the values of $\bB$ in bidiagonal form has nonzero entries on the upper subdiagonal such that it will be shown to play an important role in the calculation of the singular value decomposition.

\section{Existence of the Bidiagonal Decomposition: Golub-Kahan Bidiagonalization}
Previously, we utilized a Householder reflector to triangularize matrices and introduce zeros below the diagonal to obtain the QR decomposition, and introduce zeros below the sub-diagonal to obtain the Hessenberg decomposition. A similar approach can be employed to find the bidiagonal decomposition.

\subsection*{\textbf{First Step 1.1: Introduce Zeros for the First Column}}
Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ be the column partitions of $\bA$, and each $\ba_i \in \real^{m}$.
We can construct the Householder reflector as follows:
$$
r_1 = ||\ba_1||, \qquad \bu_1 = \frac{\ba_1 - r_1 \be_1}{||\ba_1 - r_1 \be_1||} ,\qquad \text{and}\qquad \bH_1 = \bI - 2\bu_1\bu_1^\top \in \textcolor{blue}{\real^{m\times m}},
$$
where $\be_1$ here is the first basis for $\textcolor{blue}{\real^{m}}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{m}}$.
In this case, $\bH_1\bA$ will introduce zeros in the first column of $\bA$ below entry (1,1), i.e., reflect $\ba_1$ to $r_1\be_1$.
We can easily verify that both $\bH_1$ is a symmetric and orthogonal matrix (from the definition of Householder reflector).

An example of a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bH_1\times}{\rightarrow}
&\begin{sbmatrix}{\bH_1\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}.
\end{aligned}
$$
Till now, this is exactly what we have done in the QR decomposition via the Householder reflector in Section~\ref{section:qr-via-householder} (p.~\pageref{section:qr-via-householder}).
Going further, to introduce zeros above the upper sub-diagonal of $\bH_1\bA$ is equivalent to introducing zeros below the lower subdiagonal of $(\bH_1\bA)^\top$.

\subsection*{\textbf{First Step 1.2: Introduce Zeros for the First Row}}

Now suppose we are looking at the \textit{transpose} of $\bH_1\bA$, that is $(\bH_1\bA)^\top =\bA^\top\bH_1^\top \in \real^{n\times m}$ and the column partition is given by $\bA^\top\bH_1^\top = [\bz_1, \bz_2, \ldots, \bz_m]$ where each $\bz_i \in \real^n$.
Suppose $\bar{\bz}_1, \bar{\bz}_2, \ldots, \bar{\bz}_m \in \real^{n-1}$ are vectors removing the first component in $\bz_i$'s.
Let
$$
r_1 = ||\bar{\bz}_1||, \qquad \bv_1 = \frac{\bar{\bz}_1 - r_1 \be_1}{||\bar{\bz}_1 - r_1 \be_1||},  \qquad \text{and}\qquad \widetilde{\bL}_1 = \bI - 2\bv_1\bv_1^\top \in\textcolor{blue}{\real^{(n-1)\times (n-1)}},
$$
where $\be_1$ now is the first basis for $\textcolor{blue}{\real^{n-1}}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{n-1}}$. To introduce zeros below the subdiagonal and operate on the submatrix $(\bA^\top\bH_1^\top)_{2:n,1:m}$, we append the Householder reflector into
$$
\bL_1 = \begin{bmatrix}
	1 &\bzero \\
	\bzero & \widetilde{\bL}_1
\end{bmatrix},
$$
in which case, $\bL_1(\bA^\top\bH_1^\top)$ will introduce zeros in the first column of $(\bA^\top\bH_1^\top)$ below entry (2,1), i.e., reflect $\bar{\bz}_1$ to $r_1\be_1$. The first row of $(\bA^\top\bH_1^\top)$ will not be affected at all and kept unchanged by Remark~\ref{remark:left-right-identity} (p.~\pageref{remark:left-right-identity}) such that the zeros introduced in Step 1.1) will be kept. And we can easily verify that both $\bL_1$ and $\widetilde{\bL}_1$ are orthogonal matrices and they are symmetric (from the definition of the Householder reflector).

Come back to the original \textit{untransposed} matrix $\bH_1\bA$, multiply on the right by $\bL_1^\top$ is to introduce zeros in the first row to the right of entry (1,2).
Again, following the example above, a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bA^\top\bH_1^\top}
	\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{sbmatrix}
\stackrel{\bL_1\times}{\rightarrow}
\begin{sbmatrix}{\bL_1 \bA^\top\bH_1^\top}
\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}
\stackrel{(\cdot)^\top}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA\bL_1^\top }
	\boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0} \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}.
\end{aligned}
$$

In short, $\bH_1\bA\bL_1^\top$ finishes the first step to introduce zeros for the first column and the first row of $\bA$.

\subsection*{\textbf{Second Step 2.1: Introduce Zeros for the Second Column}}
Let $\bB = \bH_1\bA\bL_1^\top$, where the entries in the first column below entry (1,1) are all zeros and the entries in the first row to the right of entry (1,2) are all zeros as well.
And the goal is to introduce zeros in the second column below entry (2,2).
Let $\bB_2 = \bB_{2:m,2:n}=[\bb_1, \bb_2, \ldots, \bb_{n-1}] \in \real^{(m-1)\times (n-1)}$.
We can again construct a Householder reflector
$$
r_1 = ||\bb_1||,\qquad	\bu_2 = \frac{\bb_1 - r_1 \be_1}{||\bb_1 - r_1 \be_1||}, \qquad  \text{and}\qquad \widetilde{\bH}_2 = \bI - 2\bu_2\bu_2^\top\in \textcolor{blue}{\real^{(m-1)\times (m-1)}},
$$
where $\be_1$ now is the first basis for $\textcolor{blue}{\real^{m-1}}$ i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{m-1}}$. To introduce zeros below the main diagonal and operate on the submatrix $\bB_{2:m,2:n}$, we append the Householder reflector into
$$
\bH_2 = \begin{bmatrix}
	1 &\bzero \\
	\bzero & \widetilde{\bH}_2
\end{bmatrix},
$$
in which case, we can see that $\bH_2(\bH_1\bA\bL_1^\top)$ will not change the first  row of $(\bH_1\bA\bL_1^\top)$ by Remark~\ref{remark:left-right-identity} (p.~\pageref{remark:left-right-identity}), and as the Householder cannot reflect a zero vector such that the zeros in the first column will be kept as well.

Following the example above, a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bH_1\bA\bL_1^\top }
\boxtimes & \boxtimes & 0 & 0 & 0 \\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{sbmatrix}
\stackrel{\bH_2\times }{\rightarrow}
	\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top }
	\boxtimes & \boxtimes & 0& 0 & 0 \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}.
\end{aligned}
$$

\subsection*{\textbf{Second Step 2.2: Introduce Zeros for the Second Row}}
Same as step 1.2), now suppose we are looking at the \textit{transpose} of $\bH_2\bH_1\bA\bL_1^\top$, that is $(\bH_2\bH_1\bA\bL_1^\top)^\top =\bL_1\bA^\top\bH_1^\top\bH_2^\top \in \real^{n\times m}$ and the column partition is given by $\bL_1\bA^\top\bH_1^\top\bH_2^\top = [\bx_1, \bx_2, \ldots, \bx_m]$ where each $\bx_i \in \real^n$.
Suppose $\bar{\bx}_1, \bar{\bx}_2, \ldots, \bar{\bx}_m \in \real^{n-2}$ are vectors removing the first two components in $\bx_i$'s.
Construct the Householder reflector as follows:
$$
 r_1 = ||\bar{\bx}_1||,\qquad \bv_2 = \frac{\bar{\bx}_1 - r_1 \be_1}{||\bar{\bx}_1 - r_1 \be_1||}, \qquad \text{and}\qquad \widetilde{\bL}_2 = \bI - 2\bv_2\bv_2^\top \in \textcolor{blue}{\real^{(n-2)\times (n-2)}},
$$
where $\be_1$ now is the first basis for $\textcolor{blue}{\real^{n-2}}$, i.e., $\be_1=[1;0;0;\ldots;0]\in \textcolor{blue}{\real^{n-2}}$. To introduce zeros below the subdiagonal and operate on the submatrix $(\bL_1\bA^\top\bH_1\bH_2)_{3:n,1:m}$, we append the Householder reflector into
$$
\bL_1 = \begin{bmatrix}
	\bI_2 &\bzero \\
	\bzero & \widetilde{\bL}_2
\end{bmatrix},
$$
where $\bI_2$ is a $2\times 2$ identity matrix.
In this case, $\bL_2(\bL_1\bA^\top\bH_1^\top\bH_2^\top)$ will introduce zeros in the second column of $(\bL_1\bA^\top\bH_1^\top\bH_2^\top)$ below entry (3,2). The first two rows of $(\bL_1\bA^\top\bH_1^\top\bH_2^\top)$ will not be affected at all and kept unchanged by Remark~\ref{remark:left-right-identity} (p.~\pageref{remark:left-right-identity}). \textbf{Further, the first column of it will be kept unchanged as well}. And we can easily verify that both $\bL_1$ and $\widetilde{\bL}_1$ are orthogonal matrices and they are symmetric (from the definition of the Householder reflector).

Come back to the original \textit{untransposed} matrix $\bH_2\bH_1\bA\bL_1^\top$, multiply on the right by $\bL_2^\top$ is to introduce zeros in the second row to the right of entry (2,3). Following the example above, a $7\times 5$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bL_1\bA^\top\bH_1^\top\bH_2^\top }
	\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\boxtimes & \boxtimes & 0 & 0 & 0 & 0 & 0\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\end{sbmatrix}
	\stackrel{\bL_2\times }{\rightarrow}
	\begin{sbmatrix}{\bL_2\bL_1\bA^\top\bH_1^\top\bH_2^\top }
	\boxtimes & 0 & 0 & 0 & 0 & 0 & 0 \\
\boxtimes & \boxtimes & 0 & 0 & 0 & 0 & 0\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	\end{sbmatrix}
\stackrel{(\cdot)^\top }{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top\bL_2^\top}
	\boxtimes & \boxtimes & 0 & 0 & 0\\
	0 & \boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} \\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}.
\end{aligned}
$$

In short, $\bH_2(\bH_1\bA\bL_1^\top)\bL_2^\top$ finishes the second step to introduce zeros for the second column and the second row of $\bA$.
The same process can go on, and we shall notice that there are $n$ such $\bH_i$ Householder reflectors on the left and $n-2$ such $\bL_i$ Householder reflectors on the right (suppose $m>n$ for simplicity). The interleaved Householder factorization is known as the \textit{Golub-Kahan Bidiagonalization} \citep{golub1965calculating}. We will finally bidiagonalize
$$
\bB = \bH_{n} \bH_{n-1}\ldots\bH_1 \bA\bL_1^\top\bL_2^\top\ldots\bL_{n-2}^\top.
$$
And since the $\bH_i$'s and $\bL_i$'s are symmetric and orthogonal, we have
$$
\bB =\bH_{n} \bH_{n-1}\ldots\bH_1 \bA\bL_1\bL_2\ldots\bL_{n-2}.
$$
A full example of a $7\times 5$ matrix is shown as follows where again $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\begin{mdframed}[hidealllines=\mdframehidelineNote,backgroundcolor=\mdframecolorSkip,frametitle={A Complete Example of Golub-Kahan Bidiagonalization}]
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes\\
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
\stackrel{\bH_1\times}{\rightarrow}
&\begin{sbmatrix}{\bH_1\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}
\stackrel{\times\bL_1^\top}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA\bL_1^\top}
\boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} & \bm{0} \\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
\end{sbmatrix}\\
\end{aligned}
$$

$$
\begin{aligned}
\qquad \qquad \qquad \qquad \gap \stackrel{\bH_2\times}{\rightarrow}
&\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bL_2^\top}{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA\bL_1^\top\bL_2^\top}
\boxtimes & \boxtimes & 0 & 0 & 0\\
0 & \boxtimes & \bm{\boxtimes} & \bm{0} & \bm{0} \\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\end{aligned}
$$

$$
\begin{aligned}
\qquad \qquad \qquad \qquad \qquad \qquad \stackrel{\bH_3\times}{\rightarrow}
&\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2^\top}
\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\times\bL_3^\top}{\rightarrow}
\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2^\top\bL_3^\top}
\boxtimes & \boxtimes & 0 & 0 & 0\\
\boxtimes & \boxtimes & \boxtimes & 0 & 0\\
0 & \boxtimes & \boxtimes & \bm{\boxtimes} & \bm{0}\\
0 & 0 & \boxtimes & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}\\
0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
\stackrel{\bH_4\times}{\rightarrow}
&\begin{sbmatrix}{\bH_4\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2\bL_3^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes& \boxtimes & \boxtimes\\
	0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{0} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{0} & \bm{\boxtimes}\\
	0 & 0 & 0 & \bm{0} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bH_5\times}{\rightarrow}
\begin{sbmatrix}{\bH_5\bH_4\bH_3\bH_2\bH_1\bA\bL_1^\top\bL_2\bL_3^\top}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes& \boxtimes & \boxtimes\\
	0 & 0 & 0 & \boxtimes & \boxtimes\\
	0 & 0 & 0 & 0 & \bm{\boxtimes}\\
	0 & 0 & 0 & 0 & \bm{0}\\
	0 & 0 & 0 & 0 & \bm{0}
\end{sbmatrix}.
\end{aligned}
$$
\end{mdframed}
We present in a way where a right Householder reflector $\bL_i$ follows from a left one $\bH_i$. However, a trivial error that might be employed is that we do the left ones altogether, and the right ones follow. That is, a bidiagonal decomposition is a combination of a QR decomposition and a Hessenberg decomposition. Nevertheless, this is problematic, the right Householder reflector $\bL_1$ will destroy the zeros introduced by the left ones. Therefore, the left and right reflectors need to be employed in an interleaved manner to introduce back the zeros.

The Golub-Kahan bidiagonalization is not the most efficient way to calculate the bidiagonal decomposition. It requires $\sim 4mn^2-\frac{4}{3}n^3$ flops to compute a bidiagonal decomposition of an $m\times n$ matrix with $m>n$. Further, if $\bU, \bV$ are needed explicitly, additional $\sim 4m^2n-2mn^2 + 2n^3$ flops are required \citep{lu2021numerical}.

\paragraph{LHC Bidiagonalization}
Nevertheless, when $m\gg n$, we can extract the square triangular matrix (i.e., the QR decomposition) and apply the Golub-Kahan diagonalization on the square $n\times n$ matrix. This is known as the \textit{Lawson-Hanson-Chan (LHC) bidiagonalization} \citep{lawson1995solving, chan1982improved} and the procedure is shown in Figure~\ref{fig:lhc-bidiagonal}.\index{LHC bidiagonalization}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.9\textwidth]{imgs/LHC-bidiagonal.pdf}
	\caption{Demonstration of LHC-bidiagonalization of a matrix}
	\label{fig:lhc-bidiagonal}
\end{figure}
The LHC bidiagonalization starts by computing the QR decomposition $\bA = \bQ\bR$. Then follows by applying the Golub-Kahan process such that $\widetilde{\bR} = \widetilde{\bU} \widetilde{\bB} \bV^\top$ where $\widetilde{\bR}$ is the square $n\times n$ triangular submatrix inside $\bR$. Append $\widetilde{\bU}$ into
$$
\bU_0 =
\begin{bmatrix}
	\widetilde{\bU} & \\
	& \bI_{m-n}
\end{bmatrix},
$$
which results in $\bR=\bU_0\bB \bV^\top$ and $\bA = \bQ\bU_0\bB \bV^\top$. Let $\bU=\bQ\bU_0$, we obtain the bidiagonal decomposition. The QR decomposition requires $2mn^2-\frac{2}{3}n^3$ flops and the Golub-Kahan process now requires $\frac{8}{3}n^3$ (operating on an $n\times n$ submatrix) \citep{lu2021numerical}. Thus the total complexity to obtain the bidiagonal matrix $\bB$ is
then reduced to
$$
\text{LHC bidiagonalization:   } \sim 2mn^2 + 2n^3 \text{  flops}.
$$
The LHC process creates zeros and then destroys them again in the lower triangle of the upper $n\times n $ square of $\bR$, but the zeros in the lower $(m-n)\times n$ rectangular matrix of $\bR$ will be kept. Thus when $m-n$ is large enough (or $m\gg n$), there is a net gain. Simple calculations will show the LHC bidiagonalization costs less when $m>\frac{5}{3}n$ compared to the Golub-Kahan bidiagonalization.

\index{Golub-Kahan process}
\index{LHC process}
\paragraph{Three-Step Bidiagonalization}
The LHC procedure is advantageous only when $m>\frac{5}{3}n$. A further trick is to apply the QR decomposition not at the beginning of the computation, but at a suitable point in the middle \citep{trefethen1997numerical}. In particular, the procedure is shown in Figure~\ref{fig:lhc-bidiagonal2} where we apply the first $k$ steps of left and right Householder reflectors as in the Golub-Kahan process leaving the bottom-right $(m-k)\times(n-k)$ submatrix ``unreflected". Then following the same LHC process on the submatrix to obtain the final bidiagonal decomposition. By doing so, the complexity reduces when $n<m<2n$.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.9\textwidth]{imgs/LHC-bidiagonal2.pdf}
	\caption{Demonstration of Three-Step bidiagonalization of a matrix}
	\label{fig:lhc-bidiagonal2}
\end{figure}

To conclude, the costs of the three methods are shown as follows:
$$
\left\{
\begin{aligned}
	&\text{Golub-Kahan: }  \sim 4mn^2-\frac{4}{3}n^3 \,\, \text{ flops},   \\
	&\text{LHC: } \sim 2mn^2 + 2n^3  \,\, \text{ flops},  \\
	&\text{Three-Step: } \sim 2mn^2 + 2m^2n -\frac{2}{3}m^3 -\frac{2}{3}n^3 \,\, \text{ flops} .
\end{aligned}
\right.
$$
When $m>2n$, LHC is preferred; when $n<m<2n$, the Three-Step method is preferred though the improvement is small enough as shown in Figure~\ref{fig:bidiagonal-loss-compare} where the operation counts for the three methods are plotted as a function of $\frac{m}{n}$. Notice that the complexity discussed here does not involve the extra computation of $\bU, \bV$. We shall not discuss the issue for simplicity.

\begin{SCfigure
	\centering
	\includegraphics[width=0.6\textwidth]{imgs/bidiagonal-loss.pdf}
	\caption{Comparison of the complexity among the three bidiagonal methods. When $m>2n$, LHC is preferred; when $n<m<2n$, the Three-Step method is preferred though the improvement is small enough.}
	\label{fig:bidiagonal-loss-compare}
\end{SCfigure}

\section{Connection to Tridiagonal Decomposition}

We fist illustrate the connection by the following lemma that reveals how to construct a tridiagonal matrix from a bidiagonal one.
\begin{lemma}[Construct Tridiagonal From Bidiagonal]\label{lemma:construct-triangular-from-bidia}
Suppose $\bB\in \real^{n\times n}$ is upper bidiagonal, then $\bT_1=\bB^\top\bB$ and $\bT_2=\bB\bB^\top$ are \textit{symmetric} triangular matrices.
\end{lemma}
The lemma above reveals an important property. Suppose $\bA=\bU\bB\bV^\top$ is the bidiagonal decomposition of $\bA$, then the symmetric matrix $\bA\bA^\top$ has a tridiagonal decomposition
$$
\bA\bA^\top=\bU\bB\bV^\top \bV\bB^\top\bU^\top = \bU\bB\bB^\top\bU^\top.
$$
And the symmetric matrix $\bA^\top\bA$ has a tridiagonal decomposition
$$
\bA^\top\bA=\bV\bB^\top\bU^\top \bU\bB\bV^\top=\bV\bB^\top\bB\bV^\top.
$$
As a final result in this section, we state a theorem giving the tridiagonal decomposition of a symmetric matrix with special eigenvalues.
\begin{theoremHigh}[Tridiagonal Decomposition for Nonnegative Eigenvalues]\label{theorem:tri-nonnegative-eigen}
Suppose $n\times n$ symmetric matrix $\bA$ has nonnegative eigenvalues, then there exists a matrix $\bZ$ such that
$$
\bA=\bZ\bZ^\top.
$$
Moreover, the tridiagonal decomposition of $\bA$ can be reduced to a problem to find the bidiagonal decomposition of $\bZ =\bU\bB\bV^\top$ such that the tridiagonal decomposition of $\bA$ is given by
$$
\bA = \bZ\bZ^\top = \bU\bB\bB^\top\bU^\top.
$$
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:tri-nonnegative-eigen}]
The eigenvectors of symmetric matrices can be chosen to be orthogonal (Lemma~\ref{lemma:orthogonal-eigenvectors}, p.~\pageref{lemma:orthogonal-eigenvectors}) such that symmetric matrix $\bA$ can be decomposed into $\bA=\bQ\bLambda\bQ^\top$ (spectral theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}) where $\bLambda$ is a diagonal matrix containing the eigenvalues of $\bA$. When eigenvalues are nonnegative, $\bLambda$ can be factored as $\bLambda=\bLambda^{1/2} \bLambda^{1/2}$. Let $\bZ = \bQ\bLambda^{1/2}$, $\bA$ can be factored as $\bA=\bZ\bZ^\top$. Thus, combining our findings yields the result.
\end{proof}

\section*{Introduction and Background}
\addcontentsline{toc}{section}{Introduction and Background}

Matrix decomposition has become a core technology in statistics \citep{banerjee2014linear, gentle1998numerical}, optimization \citep{gill2021numerical}, machine learning \citep{goodfellow2016deep, bishop2006pattern}, and deep learning largely due to the development of back propagation algorithm in fitting a neural network and the low-rank neural networks in efficient deep learning.
The sole aim of this book is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, and Hilbert space. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields. Some excellent examples include \citet{householder2006principles, trefethen1997numerical, strang1993introduction, stewart2000decompositional, gentle2007matrix, higham2002accuracy, quarteroni2010numerical, golub2013matrix, beck2017first, gallier2017fundamentals, boyd2018introduction, strang2019linear, van2020advanced, strang2021every}. Most importantly, this book will only cover the compact proofs of the existence of the matrix decomposition methods. For more details on how to reduce the calculation complexity, rigorous discussion in various applications and examples, why each matrix decomposition method is important in practice, and preliminaries on tensor decomposition, one can refer to \citet{lu2021numerical}.

A matrix decomposition is a way of reducing a complex matrix into its constituent parts which are in simpler forms.
The
 underlying principle of the decompositional approach to matrix computation is that it is not the
business of the matrix algorithmists to solve particular problems, but
it is an approach that can simplify more complex matrix operations which can be performed on the decomposed parts rather than on the original matrix itself. At a general level, a matrix decomposition task on matrix $\bA$ can be cast as
\begin{itemize}
\item $\bA=\bQ\bU$: where $\bQ$ is an orthogonal matrix that contains the same column space as $\bA$ and $\bU$ is a relatively simple and sparse matrix to reconstruct $\bA$.
\item $\bA=\bQ\bT\bQ^\top$: where $\bQ$ is orthogonal such that $\bA$ and $\bT$ are \textit{similar matrices} \footnote{See Definition~\ref{definition:similar-matrices} (p.~\pageref{definition:similar-matrices}) for a rigorous definition.} that share the same properties such as same eigenvalues, and sparsity. Moreover, working on $\bT$ is an easier task compared to that of $\bA$.
\item $\bA=\bU\bT\bV$: where $\bU, \bV$ are orthogonal matrices such that the columns of $\bU$ and the rows of $\bV$ constitute an orthonormal basis of the column space and row space of $\bA$ respectively.
\item $\underset{m\times n}{\bA}=\underset{m\times r}{\bB}\gap \underset{r\times n}{\bC}$: where $\bB,\bC$ are full rank matrices that can reduce the memory storage of $\bA$. In practice, a low-rank approximation $\underset{m\times n}{\bA}\approx \underset{m\times k}{\bD}\gap \underset{k\times n}{\bF}$ can be employed where $k<r$ is called the \textit{numerical rank} of the matrix such that  the matrix can be stored much more inexpensively and can be multiplied rapidly with
vectors or other matrices.
An approximation of the form $\bA=\bD\bF$ is useful for storing the matrix $\bA$ more frugally (we can store $\bD$ and $\bF$ using $k(m+n)$ floats, as opposed to $mn$ numbers for storing $\bA$), for efficiently computing a matrix-vector product $\bb = \bA\bx$ (via $\bc = \bF\bx$ and $\bb = \bD\bc$), for data interpretation, and much more.

\item A matrix decomposition, which though is usually expensive to
compute, can be reused to solve new problems involving the
original matrix in different scenarios, e.g., as long as the factorization of $\bA$ is obtained, it can be reused to solve the set of linear systems $\{\bb_1=\bA\bx_1, \bb_2=\bA\bx_2, \ldots, \bb_k=\bA\bx_k\}$.
\item More generally, a matrix decomposition can help to understand the internal meaning and inherent logic of  what happens when multiplied by the matrix such that each constituent has a geometrical transformation (see Section~\ref{section:coordinate-transformation}, p.~\pageref{section:coordinate-transformation}).
\end{itemize}

The matrix decomposition algorithms can fall into numerous categories. Nevertheless, we outline the fundamental six types here:
\begin{enumerate}
\item Factorizations arise from Gaussian elimination including the LU decomposition and its positive definite alternative - Cholesky decomposition;
\item Factorizations obtained when orthogonalizing the columns or the rows of a matrix such that the data can be explained well on an orthonormal basis;
\item Factorizations where the matrices are skeletoned such that a subset of the columns or the rows can represent the whole data in a small reconstruction error, whilst, the sparsity and nonnegativity of the matrices are kept as they are;
\item Reduction to Hessenberg, tridiagonal, or bidiagonal form, as a result, the properties of the matrices can be explored in these reduced matrices such as rank, eigenvalues, and so on;
\item Factorizations result from the computation of the eigenvalues of matrices;
\item In particular, the rest can be cast as a special kind of decompositions that involve optimization methods, and high-level ideas where the category may not be straightforward to determine.
\end{enumerate}

The world pictures for decomposition in Figure~\ref{fig:matrix-decom-world-picture} and \ref{fig:matrix-decom-world-picture2} connect each decomposition method by their internal relationships and also separate different methods by the criteria or prerequisites of them. Readers will get more information about the two pictures after reading the text.

\paragraph{Notation and preliminaries} In the rest of this section we will introduce and recap some basic knowledge about linear algebra. For the rest of the important concepts, we define and discuss them as per need for clarity.
Readers with enough background in matrix analysis can skip this section.
In the text, we simplify matters by considering only matrices that are real. Without special consideration, the eigenvalues of the discussed matrices are also real. We also assume throughout that $||\cdot||=||\cdot||_2$.

In all cases, scalars will be denoted in a non-bold font possibly with subscripts (e.g., $a$, $\alpha$, $\alpha_i$). We will use \textbf{boldface} lower case letters possibly with subscripts to denote vectors (e.g., $\bmu$, $\bx$, $\bx_n$, $\bz$) and
\textbf{boldface} upper case letters possibly with subscripts to denote matrices (e.g., $\bA$, $\bL_j$). The $i$-th element of a vector $\bz$ will be denoted by $\bz_i$ in bold font (or $z_i$ in the non-bold font). \textit{The $i$-th row and $j$-th column value of matrix $\bA$ will be denoted by $\bA_{ij}$ if block submatrices are involved, or by $a_{ij}$ if block submatrices are not involved}. Furthermore, it will be helpful to utilize the \textbf{Matlab-style notation}, the $i$-th row to $j$-th row and $k$-th column to $m$-th column submatrix of matrix $\bA$ will be denoted by $\bA_{i:j,k:m}$. When the index is not continuous, given ordered subindex sets $I$ and $J$, $\bA[I, J]$ denotes the submatrix of $\bA$ obtained by extracting the rows and columns of $\bA$ indexed by $I$ and $J$, respectively; and $\bA[:, J]$ denotes the submatrix of $\bA$ obtained by extracting the columns of $\bA$ indexed by $J$.
\index{Matlab-style notation}

 And in all cases, vectors are formulated in a column rather than in a row. A row vector will be denoted by a transpose of a column vector such as $\ba^\top$. A specific column vector with values is split by the symbol $``;"$, e.g., $\bx=[1;2;3]$ is a column vector in $\real^3$. Similarly, a specific row vector with values is split by the symbol $``,"$, e.g., $\by=[1,2,3]$ is a row vector with 3 values. Further, a column vector can be denoted by the transpose of a row vector e.g., $\by=[1,2,3]^\top$ is a column vector.

The transpose of a matrix $\bA$ will be denoted by $\bA^\top$ and its inverse will be denoted by $\bA^{-1}$ . We will denote the $p \times p$ identity matrix by $\bI_p$. A vector or matrix of all zeros will be denoted by a \textbf{boldface} zero $\bzero$ whose size should be clear from context, or we denote $\bzero_p$ to be the vector of all zeros with $p$ entries.

\index{Eigenvalue}
\begin{definition}[Eigenvalue]
Given any vector space $E$ and any linear map $A: E \rightarrow E$, a scalar $\lambda \in K$ is called an eigenvalue, or proper value, or characteristic value of $\bA$ if there is some nonzero vector $\bu \in E$ such that
\begin{equation*}
\bA \bu = \lambda \bu.
\end{equation*}
\end{definition}

\index{Spectrum}
\index{Spectral radius}
\begin{definition}[Spectrum and Spectral Radius]\label{definition:spectrum}
The set of all eigenvalues of $\bA$ is called the spectrum of $\bA$ and is denoted by $\Lambda(\bA)$. The largest magnitude of the eigenvalues is known as the spectral radius $\rho(\bA)$:
$$
\rho(\bA) = \mathop{\max}_{\lambda\in \Lambda(\bA)}  |\lambda|.
$$
\end{definition}

\index{Eigenvector}
\begin{definition}[Eigenvector]
A vector $\bu \in E$ is called an eigenvector, or proper vector, or characteristic vector of $\bA$ if $\bu \neq 0$ and if there is some $\lambda \in K$ such that
\begin{equation*}
\bA \bu  = \lambda \bu,
\end{equation*}
where the scalar $\lambda$ is then an eigenvalue. And we say that $\bu$ is an eigenvector associated with $\lambda$.
\end{definition}

Moreover, the tuple $(\lambda, \bu)$ above is said to be an \textbf{eigenpair}. Intuitively, the above definitions mean that multiplying matrix $\bA$ by the vector $\bu$ results in a new vector that is in the same direction as $\bu$, but only scaled by a factor $\lambda$. For any eigenvector $\bu$, we can scale it by a scalar $s$ such that $s\bu$ is still an eigenvector of $\bA$. That's why we call the eigenvector an eigenvector of $\bA$ associated with eigenvalue $\lambda$. To avoid ambiguity, we usually assume that the eigenvector is normalized to have length $1$ and the first entry is positive (or negative) since both $\bu$ and $-\bu$ are eigenvectors.

\index{Linearly independent}
In this context, we will highly use the idea of the linear independence of a set of vectors. Two equivalent definitions are given as follows.
\begin{definition}[Linearly Independent]
A set of vectors $\{\ba_1, \ba_2, \ldots, \ba_m\}$ is called linearly independent if there is no combination can get $x_1\ba_1+x_2\ba_2+\ldots+x_m\ba_m=0$ except all $x_i$'s are zero. An equivalent definition is that $\ba_1\neq \bzero$, and for every $k>1$, the vector $\ba_k$ does not belong to the span of $\{\ba_1, \ba_2, \ldots, \ba_{k-1}\}$.
\end{definition}

In the study of linear algebra, every vector space has a basis and every vector is a linear combination of members of the basis. We then define the span and dimension of a subspace via the basis.

\index{Span}
\begin{definition}[Span]
	If every vector $\bv$ in subspace $\mathcal{V}$ can be expressed as a linear combination of $\{\ba_1, \ba_2, \ldots,$ $\ba_m\}$, then $\{\ba_1, \ba_2, \ldots, \ba_m\}$ is said to span $\mathcal{V}$.
\end{definition}

\index{Subspace}
\begin{definition}[Subspace]
	A nonempty subset $\mathcal{V}$ of $\real^n$ is called a subspace if $x\ba+y\ba\in \mathcal{V}$ for every $\ba,\bb\in \mathcal{V}$ and every $x,y\in \real$.
\end{definition}

\index{Basis}
\index{Dimension}
\begin{definition}[Basis and Dimension]
A set of vectors $\{\ba_1, \ba_2, \ldots, \ba_m\}$ is called a basis of $\mathcal{V}$ if they are linearly independent and span $\mathcal{V}$. Every basis of a given subspace has the same number of vectors, and the number of vectors in any basis is called the dimension of the subspace $\mathcal{V}$. By convention, the subspace $\{\bzero\}$ is said to  have dimension zero. Furthermore, every subspace of nonzero dimension has a basis that is orthogonal, i.e., the basis of a subspace can be chosen orthogonal.
\end{definition}

\index{Column space}
\index{Range}
\begin{definition}[Column Space (Range)]
	If $\bA$ is an $m \times n$ real matrix, we define the column space (or range) of $\bA$ to be the set spanned by its columns:
	\begin{equation*}
		\mathcal{C} (\bA) = \{ \by\in \mathbb{R}^m: \exists \bx \in \mathbb{R}^n, \, \by = \bA \bx \}.
	\end{equation*}
	And the row space of $\bA$ is the set spanned by its rows, which is equal to the column space of $\bA^\top$:
	\begin{equation*}
		\mathcal{C} (\bA^\top) = \{ \bx\in \mathbb{R}^n: \exists \by \in \mathbb{R}^m, \, \bx = \bA^\top \by \}.
	\end{equation*}
\end{definition}

\index{Null space}
\index{Nullspace}
\index{Kernel}
\begin{definition}[Null Space (Nullspace, Kernel)]
	If $\bA$ is an $m \times n$ real matrix, we define the null space (or kernel, or nullspace) of $\bA$ to be the set:
	\begin{equation*}
		\nspace (\bA) = \{\by \in \mathbb{R}^n:  \, \bA \by = \bzero \}.
	\end{equation*}
	And the null space of $\bA^\top$ is defined as 	\begin{equation*}
		\nspace (\bA^\top) = \{\bx \in \mathbb{R}^m:  \, \bA^\top \bx = \bzero \}.
	\end{equation*}
\end{definition}

Both the column space of $\bA$ and the null space of $\bA^\top$ are subspaces of $\real^n$. In fact, every vector in $\nspace(\bA^\top)$ is perpendicular to $\cspace(\bA)$ and vice versa.\footnote{Every vector in $\nspace(\bA)$ is also perpendicular to $\cspace(\bA^\top)$ and vice versa.}

\index{Rank}
\begin{definition}[Rank]
The $rank$ of a matrix $\bA\in \real^{m\times n}$ is the dimension of the column space of $\bA$. That is, the rank of $\bA$ is equal to the maximal number of linearly independent columns of $\bA$, and is also the maximal number of linearly independent rows of $\bA$. The matrix $\bA$ and its transpose $\bA^\top$ have the same rank. We say that $\bA$ has full rank if its rank is equal to $min\{m,n\}$.
Specifically, given a vector $\bu \in \real^m$ and a vector $\bv \in \real^n$, then the $m\times n$ matrix $\bu\bv^\top$ is of rank 1. In short, the rank of a matrix is equal to:
\begin{itemize}
\item number of linearly independent columns;
\item number of linearly independent rows;
\item and remarkably, these are always the same (see Theorem~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}).
\end{itemize}
\end{definition}

\index{Orthogonal complement}
\begin{definition}[Orthogonal Complement in General]
The orthogonal complement $\mathcalV^\perp$ of a subspace $\mathcalV$ contains every vector that is perpendicular to $\mathcalV$. That is,
$$
\mathcalV^\perp = \{\bv | \bv^\top\bu=0, \,\,\, \forall \bu\in \mathcalV  \}.
$$
The two subspaces are disjoint that span the entire space. The dimensions of $\mathcalV$ and $\mathcalV^\perp$ add to the dimension of the whole space. Furthermore, it satisfies that  $(\mathcalV^\perp)^\perp=\mathcalV$.
\end{definition}

\begin{definition}[Orthogonal Complement of Column Space]
	If $\bA$ is an $m \times n$ real matrix, the orthogonal complement of $\mathcal{C}(\bA)$, $\mathcal{C}^{\bot}(\bA)$ is the subspace defined as:
	\begin{equation*}
		\begin{aligned}
			\mathcal{C}^{\bot}(\bA) &= \{\by\in \mathbb{R}^m: \, \by^\top \bA \bx=\bzero, \, \forall \bx \in \mathbb{R}^n \} \\
			&=\{\by\in \mathbb{R}^m: \, \by^\top \bv = \bzero, \, \forall \bv \in \mathcal{C}(\bA) \}.
		\end{aligned}
	\end{equation*}
\end{definition}
Then we have the four fundamental spaces for any matrix $\bA\in \real^{m\times n}$ with rank $r$ as shown in Theorem~\ref{theorem:fundamental-linear-algebra}. To see the fundamental theorem of linear algebra, we first prove the following result, the row rank of a matrix equals the column rank,  whose proof is essential in the sequel.

\index{Rank}
\index{Matrix rank}
\begin{theorem}[Row Rank Equals Column Rank]\label{lemma:equal-dimension-rank}
	The dimension of the column space of a matrix $\bA\in \real^{m\times n}$ is equal to the dimension of its
	row space, i.e., the row rank and the column rank of a matrix $\bA$ are equal.
\end{theorem}

\begin{proof}[{of Theorem~\ref{lemma:equal-dimension-rank}}]
	We first notice that the null space of $\bA$ is orthogonal complementary to the row space of $\bA$: $\nspace(\bA) \bot \cspace(\bA^\top)$ (where the row space of $\bA$ is exactly the column space of $\bA^\top$), that is, vectors in the null space of $\bA$ are orthogonal to vectors in the row space of $\bA$. To see this, suppose $\bA$ has rows $\ba_1^\top, \ba_2^\top, \ldots, \ba_m^\top$ and $\bA=[\ba_1^\top; \ba_2^\top; \ldots; \ba_m^\top]$. For any vector $\bx\in \nspace(\bA)$, we have $\bA\bx = \bzero$, that is, $[\ba_1^\top\bx; \ba_2^\top\bx; \ldots; \ba_m^\top\bx]=\bzero$. And since the row space of $\bA$ is spanned by $\ba_1^\top, \ba_2^\top, \ldots, \ba_m^\top$. Then $\bx$ is perpendicular to any vectors from $\cspace(\bA^\top)$ which means $\nspace(\bA) \bot \cspace(\bA^\top)$.

	Now suppose, the dimension of row space of $\bA$ is $r$. \textcolor{blue}{Let $\br_1, \br_2, \ldots, \br_r$ be a set of vectors in $\real^n$ and form a basis for the row space}. Then the $r$ vectors $\bA\br_1, \bA\br_2, \ldots, \bA\br_r$ are in the column space of $\bA$, which are linearly independent. To see this, suppose we have a linear combination of the $r$ vectors: $x_1\bA\br_1 + x_2\bA\br_2+ \ldots+ x_r\bA\br_r=\bzero$, that is, $\bA(x_1\br_1 + x_2\br_2+ \ldots+ x_r\br_r)=\bzero$ and the vector $\bv=x_1\br_1 + x_2\br_2+ \ldots+ x_r\br_r$ is in null space of $\bA$. But since $\br_1, \br_2, \ldots, \br_r$ is a basis for the row space of $\bA$, $\bv$ is thus also in the row space of $\bA$. We have shown that vectors from null space of $\bA$ is perpendicular to vectors from row space of $\bA$, thus $\bv^\top\bv=0$ and $x_1=x_2=\ldots=x_r=0$. Then \textcolor{blue}{$\bA\br_1, \bA\br_2, \ldots, \bA\br_r$ are in the column space of $\bA$, and they are linearly independent} which means the dimension of the column space of $\bA$ is larger than $r$. This result shows that \textbf{row rank of $\bA\leq $ column rank of $\bA$}.

	If we apply this process again for $\bA^\top$. We will have \textbf{column rank of $\bA\leq $ row rank of $\bA$}. This completes the proof.
\end{proof}

Further information can be drawn from this proof is that if $\{\br_1, \br_2, \ldots, \br_r\}$ is a set of vectors in $\real^n$ that forms a basis for the row space, then \textcolor{blue}{$\{\bA\br_1, \bA\br_2, \ldots, \bA\br_r\}$ is a basis for the column space of $\bA$}. We formulate this finding into the following lemma.

\begin{lemma}[Column Basis from Row Basis]\label{lemma:column-basis-from-row-basis}
	For any matrix $\bA\in \real^{m\times n}$, suppose that $\{\br_1, \br_2, \ldots, \br_r\}$ is a set of vectors in $\real^n$ which forms a basis for the row space, then $\{\bA\br_1, \bA\br_2, \ldots, \bA\br_r\}$ is a basis for the column space of $\bA$.
\end{lemma}

For any matrix $\bA\in \real^{m\times n}$, it can be easily verified that any vector in the row space of $\bA$ is perpendicular to any vector in the null space of $\bA$. Suppose $\bx_n \in \nspace(\bA)$, then $\bA\bx_n = \bzero$ such that $\bx_n$ is perpendicular to every row of $\bA$ which agrees with our claim.

Similarly, we can also show that any vector in the column space of $\bA$ is perpendicular to any vector in the null space of $\bA^\top$. Further, the column space of $\bA$ together with the null space of $\bA^\top$ span the whole $\real^m$ which is known as the fundamental theorem of linear algebra.

The fundamental theorem contains two parts, the dimension of the subspaces and the orthogonality of the subspaces. The orthogonality can be easily verified as shown above. Moreover, when the row space has dimension $r$, the null space has dimension $n-r$. This cannot be easily stated and we prove it in the following theorem.

\index{Fundamental spaces}
\index{Fundamental theorem of linear algebra}
\begin{figure}[h!]
	\centering
	\includegraphics[width=0.98\textwidth]{imgs/lafundamental.pdf}
	\caption{Two pairs of orthogonal subspaces in $\real^n$ and $\real^m$. $\dim(\cspace(\bA^\top)) + \dim(\nspace(\bA))=n$ and $\dim(\nspace(\bA^\top)) + \dim(\cspace(\bA))=m$. The null space component goes to zero as $\bA\bx_n = \bzero \in \real^m$. The row space component goes to column space as $\bA\bx_r = \bA(\bx_r+\bx_n)=\bb \in \cspace(\bA)$.}
	\label{fig:lafundamental}
\end{figure}
\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolor]
	\begin{theorem}[The Fundamental Theorem of Linear Algebra]\label{theorem:fundamental-linear-algebra}

	\textit{Orthogonal Complement} and \textit{Rank-Nullity Theorem}: for any matrix $\bA\in \real^{m\times n}$, we have

		$\bullet$ $\nspace(\bA)$ is orthogonal complement to the row space $\cspace(\bA^\top)$ in $\real^n$: $dim(\nspace(\bA))+dim(\cspace(\bA^\top))=n$;

		$\bullet$ $\nspace(\bA^\top)$ is orthogonal complement to the column space $\cspace(\bA)$ in $\real^m$: $dim(\nspace(\bA^\top))+dim(\cspace(\bA))=m$;

		$\bullet$ For rank-$r$ matrix $\bA$, $dim(\cspace(\bA^\top)) = dim(\cspace(\bA)) = r$, that is, $dim(\nspace(\bA)) = n-r$ and $dim(\nspace(\bA^\top))=m-r$.
	\end{theorem}
\end{mdframed}

\begin{proof}[of Theorem~\ref{theorem:fundamental-linear-algebra}]
	Following from the proof of Theorem~\ref{lemma:equal-dimension-rank}. Let $\br_1, \br_2, \ldots, \br_r$ be a set of vectors in $\real^n$ that form a basis for the row space, then \textcolor{blue}{$\bA\br_1, \bA\br_2, \ldots, \bA\br_r$ is a basis for the column space of $\bA$}. Let $\bn_1, \bn_2, \ldots, \bn_k \in \real^n$ form a basis for the null space of $\bA$. Following again from the proof of Theorem~\ref{lemma:equal-dimension-rank}, $\nspace(\bA) \bot \cspace(\bA^\top)$, thus, $\br_1, \br_2, \ldots, \br_r$ are perpendicular to $\bn_1, \bn_2, \ldots, \bn_k$. Then, $\{\br_1, \br_2, \ldots, \br_r, \bn_1, \bn_2, \ldots, \bn_k\}$ is linearly independent in $\real^n$.

	For any vector $\bx\in \real^n $, $\bA\bx$ is in the column space of $\bA$. Then it can be expressed by a combination of $\bA\br_1, \bA\br_2, \ldots, \bA\br_r$: $\bA\bx = \sum_{i=1}^{r}a_i\bA\br_i$ which states that $\bA(\bx-\sum_{i=1}^{r}a_i\br_i) = \bzero$ and $\bx-\sum_{i=1}^{r}a_i\br_i$ is thus in $\nspace(\bA)$. Since $\{\bn_1, \bn_2, \ldots, \bn_k\}$ is a basis for the null space of $\bA$, $\bx-\sum_{i=1}^{r}a_i\br_i$ can be expressed by a combination of $\bn_1, \bn_2, \ldots, \bn_k$: $\bx-\sum_{i=1}^{r}a_i\br_i = \sum_{j=1}^{k}b_j \bn_j$, i.e., $\bx=\sum_{i=1}^{r}a_i\br_i + \sum_{j=1}^{k}b_j \bn_j$. That is, any vector $\bx\in \real^n$ can be expressed by $\{\br_1, \br_2, \ldots, \br_r, \bn_1, \bn_2, \ldots, \bn_k\}$ and the set forms a basis for $\real^n$. Thus the dimension sum to $n$: $r+k=n$, i.e., $dim(\nspace(\bA))+dim(\cspace(\bA^\top))=n$. Similarly, we can prove $dim(\nspace(\bA^\top))+dim(\cspace(\bA))=m$.
\end{proof}

Figure~\ref{fig:lafundamental} demonstrates two pairs of such orthogonal subspaces and shows how $\bA$ takes $\bx$ into the column space. The dimensions of the row space of $\bA$ and the null space of $\bA$ add to $n$. And the dimensions of the column space of $\bA$ and the null space of $\bA^\top$ add to $m$. The null space component goes to zero as $\bA\bx_{\bn} = \bzero \in \real^m$ which is the intersection of column space of $\bA$ and null space of $\bA^\top$. The row space component goes to column space as $\bA\bx_{\br} = \bA(\bx_{\br} + \bx_{\bn})=\bb\in \real^m$.

\index{Orthogonal matrix}
\begin{definition}[Orthogonal Matrix]
A real square matrix $\bQ$ is an orthogonal matrix if the inverse of $\bQ$ equals the transpose of $\bQ$, that is $\bQ^{-1}=\bQ^\top$ and $\bQ\bQ^\top = \bQ^\top\bQ = \bI$. In another word, suppose $\bQ=[\bq_1, \bq_2, \ldots, \bq_n]$ where $\bq_i \in \real^n$ for all $i \in \{1, 2, \ldots, n\}$, then $\bq_i^\top \bq_j = \delta(i,j)$ with $\delta(i,j)$ being the Kronecker delta function. If $\bQ$ contains only $\gamma$ of these columns with $\gamma<n$, then $\bQ^\top\bQ = \bI_\gamma$ stills holds with $\bI_\gamma$ being the $\gamma\times \gamma$ identity matrix. But $\bQ\bQ^\top=\bI$ will not be true. For any vector $\bx$, the orthogonal matrix will preserve the length: $||\bQ\bx|| = ||\bx||$.
\end{definition}

\index{Permutaiton matrix}
\begin{definition}[Permutation Matrix]\label{definition:permutation-matrix}
A permutation matrix $\bP$ is a square binary matrix that has exactly one entry of 1 in each row and each column and 0's elsewhere.
\paragraph{Row Point} That is, the permutation matrix $\bP$ has the rows of the identity $\bI$ in any order and the order decides the sequence of the row permutation. Suppose we want to permute the rows of matrix $\bA$, we just multiply on the left by $\bP\bA$.
\paragraph{Column Point} Or, equivalently, the permutation matrix $\bP$ has the columns of the identity $\bI$ in any order and the order decides the sequence of the column permutation. And now, the column permutation of $\bA$ is to multiply on the right by $\bA\bP$.
\end{definition}

The permutation matrix $\bP$ can be more efficiently represented via a vector $J \in \integer_+^n$ of indices such that $\bP = \bI[:, J]$ where $\bI$ is the $n\times n$ identity matrix and notably, the elements in vector $J$ sum to $1+2+\ldots+n= \frac{n^2+n}{2}$.

\begin{example}[Permutation]
	Suppose,
	$$\bA=\begin{bmatrix}
		1 & 2&3\\
		4&5&6\\
		7&8&9
	\end{bmatrix}
	,\qquad \text{and} \qquad
	\bP=\begin{bmatrix}
		&1&\\
		&&1\\
		1&&
	\end{bmatrix}.
	$$
	The row permutation is given by
	$$
	\bP\bA = \begin{bmatrix}
		4&5&6\\
		7&8&9\\
		1 & 2&3\\
	\end{bmatrix},
	$$
	where the order of the rows of $\bA$ appearing in $\bP\bA$ matches the order of the rows of $\bI$ in $\bP$. And the column permutation is given by
	$$
	\bA\bP = \begin{bmatrix}
		3 & 1 & 2 \\
		6 & 4 & 5\\
		9 & 7 & 8
	\end{bmatrix},
	$$
	where the order of the columns of $\bA$ appearing in $\bA\bP$ matches the order of the columns of $\bI$ in $\bP$. \exampbar
\end{example}

\index{Nonsingular matrix}
From an introductory course on linear algebra, we have the following remark on the equivalent claims of nonsingular matrices.
\begin{remark}[List of Equivalence of Nonsingularity for a Matrix]
For a square matrix $\bA\in \real^{n\times n}$, the following claims are equivalent:
\begin{itemize}
\item $\bA$ is nonsingular;
\item $\bA$ is invertible, i.e., $\bA^{-1}$ exists;
\item $\bA\bx=\bb$ has a unique solution $\bx = \bA^{-1}\bb$;
\item $\bA\bx = \bzero$ has a unique, trivial solution: $\bx=\bzero$;
\item Columns of $\bA$ are linearly independent;
\item Rows of $\bA$ are linearly independent;
\item $\det(\bA) \neq 0$;
\item $\dim(\nspace(\bA))=0$;
\item $\nspace(\bA) = \{\bzero\}$, i.e., the null space is trivial;
\item $\cspace(\bA)=\cspace(\bA^\top) = \real^n$, i.e., the column space or row space span the whole $\real^n$;
\item $\bA$ has full rank $r=n$;
\item The reduced row echelon form is $\bR=\bI$;
\item $\bA^\top\bA$ is symmetric positive definite;
\item $\bA$ has $n$ nonzero (positive) singular values;
\item All eigenvalues are nonzero.
\end{itemize}
\end{remark}

\index{Singular matrix}
It will be shown important to take the above equivalence into mind, otherwise, we will easily get lost. On the other hand, the following remark also shows the equivalent claims for singular matrices.
\begin{remark}[List of Equivalence of Singularity for a Matrix]
For a square matrix $\bA\in \real^{n\times n}$ with eigenpair $(\lambda, \bu)$, the following claims are equivalent:
\begin{itemize}
\item $(\bA-\lambda\bI)$ is singular;
\item $(\bA-\lambda\bI)$ is not invertible;
\item $(\bA-\lambda\bI)\bx = \bzero$ has nonzero $\bx\neq \bzero$ solutions, and $\bx=\bu$ is one of such solutions;
\item $(\bA-\lambda\bI)$ has linearly dependent columns;
\item $\det(\bA-\lambda\bI) = 0$;
\item $\dim(\nspace(\bA-\lambda\bI))>0$;
\item Null space of $(\bA-\lambda\bI)$ is nontrivial;
\item Columns of $\bA$ are linearly dependent;
\item Rows of $\bA$ are linearly dependent;
\item $\bA$ has rank $r<n$;
\item Dimension of column space = dimension of row space = $r<n$;
\item $\bA^\top\bA$ is symmetric semidefinite;
\item $\bA$ has $r<n$ nonzero (positive) singular values;
\item Zero is an eigenvalue of $\bA$.
\end{itemize}
\end{remark}

\part{Special Topics}

\chapter{Transformation in Matrix Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Coordinate Transformation in Matrix Decomposition}\label{section:coordinate-transformation}
Suppose a vector $\bv\in \real^3$ and it has elements $\bv = [3;7; 2]$. But what do these values 3, 7, and 2 mean? In the Cartesian coordinate system, it means it has a component of 3 on the $x$-axis, a component of 7 on the $y$-axis, and a component of 2 on the $z$-axis.\index{Coordinate transformation}

\section{An Overview of Matrix Multiplication}
\paragraph{Coordinate defined by a nonsingular matrix} Suppose further a $3\times 3$ nonsingular matrix $\bB$ such that $\bB$ is invertible and columns of $\bB$ are linearly independent. Thus the 3 columns of $\bB$ form a basis for the space $\real^{3}$. One step forward, we can take the 3 columns of $\bB$ as a basis for a \textcolor{blue}{new coordinate system}, which we call the \textcolor{blue}{$B$ coordinate system}. Going back to the Cartesian coordinate system, we also have three vectors as a basis, $\be_1, \be_2, \be_3$. If we put the three vectors into columns of a matrix, the matrix will be an identity matrix. So $\bI\bv = \bv$ means \textcolor{blue}{transferring $\bv$ from the Cartesian coordinate system into the Cartesian coordinate system}, the same coordinate. Similarly, \textcolor{blue}{$\bB\bv=\bu$ is to transfer $\bv$ from the Cartesian coordinate system into the $B$ system}. Specifically, for $\bv = [3;7; 2]$ and $\bB=[\bb_1, \bb_2, \bb_3]$, we have $\bu=\bB\bv = 3\bb_1+7\bb_2+2\bb_3$, i.e., $\bu$ contains 3 of the first basis $\bb_1$ of $\bB$, 7 of the second basis $\bb_2$ of $\bB$, and 2 of the third basis $\bb_3$ of $\bB$.
If again, we want to transfer the vector $\bu$ from $B$ coordinate system back to the Cartesian coordinate system, we just need to multiply by $\bB^{-1}\bu = \bv$.

\paragraph{Coordinate defined by an orthogonal matrix} A $3\times 3$ orthogonal matrix $\bQ$ defines a ``better" coordinate system since the three columns (i.e., basis) are orthonormal to each other. $\bQ\bv$ is to transfer $\bv$ from the Cartesian to the coordinate system defined by the orthogonal matrix. Since the basis vectors from the orthogonal matrix are orthonormal, just like the three vectors $\be_1, \be_2, \be_3$ in the Cartesian coordinate system, the transformation defined by the orthogonal matrix just rotates or reflects the Cartesian system.
$\bQ^\top$ can help transfer back to the Cartesian coordinate system.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/eigenRotate.pdf}
	\caption{Eigenvalue Decomposition: $\bX^{-1}$ transforms to a different coordinate system. $\bLambda$ stretches and $\bX$ transforms back. $\bX^{-1}$ and $\bX$ are nonsingular, which will change the basis of the system, and the angle between the vectors $\bv_1$ and $\bv_2$ will \textbf{not} be preserved, that is, the angle between $\bv_1$ and $\bv_2$ is \textbf{different} from the angle between $\bv_1^\prime$ and $\bv_2^\prime$. The length of $\bv_1$ and $\bv_2$ are also \textbf{not} preserved, that is, $||\bv_1|| \neq ||\bv_1^\prime||$ and $||\bv_2|| \neq ||\bv_2^\prime||$.}
	\label{fig:eigen-rotate}
\end{figure}

\section{Eigenvalue Decomposition}
A square matrix $\bA$ with linearly independent eigenvectors can be factored as $\bA = \bX\bLambda\bX^{-1}$ where $\bX$ and $\bX^{-1}$ are nonsingular so that they define a system transformation intrinsically. $\bA\bu = \bX\bLambda\bX^{-1}\bu$ firstly transfers $\bu$ into the system defined by $\bX^{-1}$. Let's call this system the \textbf{eigen coordinate system}. $\bLambda$ is to stretch each component of the vector in the eigen system by the length of the eigenvalue. And then $\bX$ helps to transfer the resulting vector back to the Cartesian coordinate system. A demonstration of how the eigenvalue decomposition transforms between coordinate systems is shown in Figure~\ref{fig:eigen-rotate} where $\bv_1, \bv_2$ are two linearly independent eigenvectors of $\bA$ such that they form a basis for $\real^2$.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/spectralrotate.pdf}
	\caption{Spectral Decomposition $\bQ\bLambda \bQ^\top$: $\bQ^\top$ rotates or reflects, $\bLambda$ stretches cycle to ellipse, and $\bQ$ rotates or reflects back. Orthogonal matrices $\bQ^\top$ and $\bQ$ only change the basis of the system. However, they preserve the angle between the vectors $\bq_1$ and $\bq_2$, and the lengths of them.}
	\label{fig:spectral-rotate}
\end{figure}

\section{Spectral Decomposition}
A symmetric matrix $\bA$ can be factored as $\bA = \bQ\bLambda\bQ^\top$ where $\bQ$ and $\bQ^\top$ are orthogonal so that they define a system transformation intrinsically as well. $\bA\bu = \bQ\bLambda\bQ^\top\bu$ firstly rotates or reflects $\bu$ into the system defined by $\bQ^\top$. Let's call this system the \textbf{spectral coordinate system}. $\bLambda$ is to stretch each component of the vector in the spectral system by the length of eigenvalue. And then $\bQ$ helps to rotate or reflect the resulting vector back to the original coordinate system. A demonstration of how the spectral decomposition transforms between coordinate systems is shown in Figure~\ref{fig:spectral-rotate} where $\bq_1, \bq_2$ are two linearly independent eigenvectors of $\bA$ such that they form a basis for $\real^2$. The coordinate transformation in the spectral decomposition is similar to that of the eigenvalue decomposition. Except that in the spectral decomposition, the orthogonal vectors transferred by $\bQ^\top$ are still orthogonal. This is also a property of orthogonal matrices. That is, orthogonal matrices can be viewed as matrices which change the basis of other matrices. Hence they preserve the angle (inner product) between the vectors
$$
\bu^\top \bv = (\bQ\bu)^\top(\bQ\bv).
$$
The above invariance of the angle between vectors also relies on the invariance of their lengths:
$$
 ||\bQ\bu|| = ||\bu||.
$$

\section{SVD}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/svdrotate.pdf}
	\caption{SVD: $\bV^\top$ and $\bU$ rotate or reflect, $\bSigma$ stretches the circle to an ellipse. Orthogonal matrices $\bV^\top$ and $\bU$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$, and the lengths of them.}
	\label{fig:svd-rotate}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/polarrotate.pdf}
	\caption{$\bV\bSigma \bV^\top$ from SVD or Polar decomposition: $\bV^\top$ rotates or reflects, $\bSigma$ stretches cycle to ellipse, and $\bV$ rotates or reflects back. Orthogonal matrices $\bV^\top$ and $\bV$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$, and the lengths of them.}
	\label{fig:polar-rotate}
\end{figure}
Any $m\times n$ matrix can be factored as $\bA=\bU\bSigma\bV^\top$, the SVD. $\bA\bu=\bU\bSigma\bV^\top\bu$ then firstly rotates or reflects $\bu$ into the system defined by $\bV^\top$, which we call the \textbf{$V$ coordinate system}. $\bSigma$ stretches the first $r$ components of the resulted vector in the $V$ system by the lengths of the singular values. If $n\geq m$, then $\bSigma$ only keeps additional $m-r$ components which are stretched to zero while removing the final $n-m$ components. If $m>n$, the $\bSigma$ stretches $n-r$ components to zero, and also adds additional $m-n$ zero components. Finally, $\bU$ rotates or reflects the resulting vector into the \textbf{$U$ coordinate system} defined by $\bU$. A demonstration of how the SVD transforms in a $2\times 2$ example is shown in Figure~\ref{fig:svd-rotate}. Further, Figure~\ref{fig:polar-rotate} demonstrates the transformation of $\bV\bSigma \bV^\top$ in a $2\times 2$ example.
Similar to the spectral decomposition, orthogonal matrices $\bV^\top$ and $\bU$ only change the basis of the system. However, they preserve the angle between vectors $\bv_1$ and $\bv_2$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.99\textwidth]{imgs/polarrotate2.pdf}
	\caption{Polar decomposition: $\bV^\top$ rotates or reflects, $\bSigma$ stretches cycle to ellipse, and $\bV$ rotates or reflects back. Orthogonal matrices $\bV^\top$, $\bV$, $\bQ_l$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$, and the lengths of them.}
	\label{fig:polar-rotate2}
\end{figure}

\section{Polar Decomposition}
Any $n\times n$ square matrix $\bA$ can be factored as left polar decomposition $\bA = (\bU\bV^\top)( \bV\bSigma \bV^\top) = \bQ_l\bS$. Similarly, $\bA\bv = \bQ_l( \bV\bSigma \bV^\top)\bu$ is to transfer $\bu$ into the system defined by $\bV^\top$ and stretch each component by the lengths of the singular values. Then the resulted vector is transferred back into the Cartesian coordinate system by $\bV$. Finally, $\bQ_l$ will rotate or reflect the resulting vector from the Cartesian coordinate system into the $Q$ system defined by $\bQ_l$. The meaning of right polar decomposition shares a similar description. Similar to the spectral decomposition, orthogonal matrices $\bV^\top$ and $\bV$ only change the basis of the system. However, they preserve the angle between the vectors $\bv_1$ and $\bv_2$.

\part{Triangularization, Orthogonalization, and Gram-Schmidt Process}

\newpage
\chapter{QR Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{QR Decomposition}
In many applications, we are interested in the column space of a matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n] \in \real^{m\times n}$. The successive spaces spanned by the columns $\ba_1, \ba_2, \ldots$ of $\bA$ are
$$
\cspace([\ba_1])\,\,\,\, \subseteq\,\,\,\, \cspace([\ba_1, \ba_2]) \,\,\,\,\subseteq\,\,\,\, \cspace([\ba_1, \ba_2, \ba_3])\,\,\,\, \subseteq\,\,\,\, \ldots,
$$
where $\cspace([\ldots])$ is the subspace spanned by the vectors included in the brackets. The idea of QR decomposition is the construction of a sequence of orthonormal vectors $\bq_1, \bq_2, \ldots$ that span the same successive subspaces.
$$
\bigg\{\cspace([\bq_1])=\cspace([\ba_1])\bigg\}\subseteq
\bigg\{\cspace([\bq_1, \bq_2])=\cspace([\ba_1, \ba_2])\bigg\}\subseteq
\bigg\{\cspace([\bq_1, \bq_2, \bq_3])=\cspace([\ba_1, \ba_2, \ba_3])\bigg\}
\subseteq \ldots,
$$
We provide the result of QR decomposition in the following theorem and we delay the discussion of its existence in the next sections.

\begin{theoremHigh}[QR Decomposition]\label{theorem:qr-decomposition}
Every $m\times n$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$ (whether linearly independent or dependent columns) with $m\geq n$ can be factored as
$$
\bA = \bQ\bR,
$$
where

\begin{enumerate}
\item  \textbf{Reduced}: $\bQ$ is $m\times n$ with orthonormal columns and $\bR$ is an $n\times n$ upper triangular matrix which is known as the \textbf{reduced QR decomposition};
\item \textbf{Full}: $\bQ$ is $m\times m$ with orthonormal columns and $\bR$ is an $m\times n$ upper triangular matrix which is known as the \textbf{full QR decomposition}. If further restrict the upper triangular matrix to be a square matrix, the full QR decomposition can be denoted as
	$$
	\bA = \bQ\begin{bmatrix}
		\bR_0\\
		\bzero
	\end{bmatrix},
	$$
	where $\bR_0$ is an $n\times n$ upper triangular matrix.
\end{enumerate}

Specifically, when $\bA$ has full rank, i.e., $\bA$ has linearly independent columns, $\bR$ also has linearly independent columns, and $\bR$ is nonsingular for the \textit{reduced} case. This implies diagonals of $\bR$ are nonzero. Under this condition, when we further restrict elements on the diagonal of $\bR$ are positive, the \textit{reduced} QR decomposition is \textbf{unique}. The \textit{full} QR decomposition is normally not unique since the right-most $(m-n)$ columns in $\bQ$ can be in any order.
\end{theoremHigh}

\section{Project a Vector Onto Another Vector}\label{section:project-onto-a-vector}
Project a vector $\ba$ to a vector $\bb$ is to find the vector closest to $\ba$ on the line of $\bb$. The projection vector $\widehat{\ba}$ is some multiple of $\bb$. Let $\widehat{\ba} = \widehat{x} \bb$ and $\ba-\widehat{\ba}$ is perpendicular to $\bb$ as shown in Figure~\ref{fig:project-line}. We then get the following result:
\begin{tcolorbox}[title={Project Vector $\ba$ Onto Vector $\bb$}]
$\ba^\perp =\ba-\widehat{\ba}$ is perpendicular to $\bb$, so $(\ba-\widehat{x}\bb)^\top\bb=0$: $\widehat{x}$ = $\frac{\ba^\top\bb}{\bb^\top\bb}$ and $\widehat{\ba} = \frac{\ba^\top\bb}{\bb^\top\bb}\bb = \frac{\bb\bb^\top}{\bb^\top\bb}\ba$.
\end{tcolorbox}

\begin{figure}[h!]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Project onto a line]{\label{fig:project-line}
		\includegraphics[width=0.47\linewidth]{./imgs/projectline.pdf}}
	\quad
	\subfigure[Project onto a space]{\label{fig:project-space}
		\includegraphics[width=0.47\linewidth]{./imgs/projectspace.pdf}}
	\caption{Project a vector onto a line and a space.}
	\label{fig:projection-qr}
\end{figure}

\section{Project a Vector Onto a Plane}\label{section:project-onto-a-plane}
Project a vector $\ba$ to a space spanned by $\bb_1, \bb_2, \ldots, \bb_n$ is to find the vector closest to $\ba$ on the column space of $[\bb_1, \bb_2, \ldots, \bb_n]$. The projection vector $\widehat{\ba}$ is a combination of $\bb_1, \bb_2, \ldots, \bb_n$: $\widehat{\ba} = \widehat{x}_1\bb_1+ \widehat{x}_2\bb_2+\ldots+\widehat{x}_n\bb_n$. This is actually a least squares problem. To find the projection, we just solve the normal equation $\bB^\top\bB\widehat{\bx} = \bB^\top\ba$ where $\bB=[\bb_1, \bb_2, \ldots, \bb_n]$ and $\widehat{\bx}=[\widehat{x}_1, \widehat{x}_2, \ldots, \widehat{x}_n]$. We refer the details of this projection view in the least squares to \citet{strang1993introduction, trefethen1997numerical, yang2000matrix, golub2013matrix, lu2021rigorous} as it is not the main interest of this book. For each vector $\bb_i$, the projection of $\ba$ in the direction of $\bb_i$ can be analogously obtained by
$$
\widehat{\ba}_i = \frac{\bb_i\bb_i^\top}{\bb_i^\top\bb_i}\ba, \gap \forall i \in \{1,2,\ldots, n\}.
$$
Let $\widehat{\ba}=\sum_{i=1}^{n}\widehat{\ba_i}$, this results in
$$
\ba^\perp = (\ba-\widehat{\ba}) \perp \cspace(\bB),
$$
i.e., $(\ba-\widehat{\ba})$ is perpendicular to the column space of $\bB=[\bb_1, \bb_2, \ldots, \bb_n]$ as shown in Figure~\ref{fig:project-space}.

\section{Existence of the QR Decomposition via the Gram-Schmidt Process}\label{section:gram-schmidt-process}
For three linearly independent vectors $\{\ba_1, \ba_2, \ba_3\}$ and the space spanned by the three linearly independent vectors $\cspace{([\ba_1, \ba_2, \ba_3])}$, i.e., the column space of the matrix $[\ba_1, \ba_2, \ba_3]$. We intend to construct three orthogonal vectors $\{\bb_1, \bb_2, \bb_3\}$ in which case $\cspace{([\bb_1, \bb_2, \bb_3])}$ = $\cspace{([\ba_1, \ba_2, \ba_3])}$. Then we divide the orthogonal vectors by their length to normalize. This process produces three mutually orthonormal vectors $\bq_1 = \frac{\bb_1}{||\bb_1||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$.

For the first vector, we choose $\bb_1 = \ba_1$ directly. The second vector $\bb_2$ must be perpendicular to the first one. This is actually the vector $\ba_2$ subtracting its projection along $\bb_1$:
\begin{equation}
\begin{aligned}
\bb_2 &= \ba_2- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} \ba_2 = (\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} )\ba_2   \qquad &(\text{Projection view})\\
 &= \ba_2-  \underbrace{\frac{ \bb_1^\top \ba_2}{\bb_1^\top\bb_1} \bb_1}_{\widehat{\ba}_2}, \qquad &(\text{Combination view}) \nonumber
\end{aligned}
\end{equation}
where the first equation shows $\bb_2$ is a multiplication of the matrix $(\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} )$ and the vector $\ba_2$, i.e., project $\ba_2$ onto the orthogonal complement space of $\cspace{([\bb_1])}$. The second equality in the above equation shows $\ba_2$ is a combination of $\bb_1$ and $\bb_2$.
Clearly, the space spanned by $\bb_1, \bb_2$ is the same space spanned by $\ba_1, \ba_2$. The situation is shown in Figure~\ref{fig:gram-schmidt1} in which we choose \textbf{the direction of $\bb_1$ as the $x$-axis in the Cartesian coordinate system}. $\widehat{\ba}_2$ is the projection of $\ba_2$ onto line $\bb_1$. It can be clearly shown that the part of $\ba_2$ perpendicular to $\bb_1$ is $\bb_2 = \ba_2 - \widehat{\ba}_2$ from the figure.

For the third vector $\bb_3$, it must be perpendicular to both the $\bb_1$ and $\bb_2$ which is actually the vector $\ba_3$ subtracting its projection along the plane spanned by $\bb_1$ and $\bb_2$
\begin{equation}\label{equation:gram-schdt-eq2}
\begin{aligned}
\bb_3 &= \ba_3- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1} \ba_3 - \frac{\bb_2 \bb_2^\top}{\bb_2^\top\bb_2} \ba_3 = (\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1}  - \frac{\bb_2 \bb_2^\top}{\bb_2^\top\bb_2} )\ba_3   \qquad &(\text{Projection view})\\
&= \ba_3- \underbrace{\frac{ \bb_1^\top\ba_3}{\bb_1^\top\bb_1} \bb_1}_{\widehat{\ba}_3} - \underbrace{\frac{ \bb_2^\top\ba_3}{\bb_2^\top\bb_2}  \bb_2}_{\bar{\ba}_3},    \qquad &(\text{Combination view})
\end{aligned}
\end{equation}
where the first equation shows $\bb_3$ is a multiplication of the matrix $(\bI- \frac{\bb_1 \bb_1^\top}{\bb_1^\top\bb_1}  - \frac{\bb_2 \bb_2^\top}{\bb_2^\top\bb_2} )$ and the vector $\ba_3$, i.e., project $\ba_3$ onto the orthogonal complement space of $\cspace{([\bb_1, \bb_2])}$. The second equality in the above equation shows $\ba_3$ is a combination of $\bb_1, \bb_2, \bb_3$. We will see this property is essential in the idea of the QR decomposition.
Again, it can be shown that the space spanned by $\bb_1, \bb_2, \bb_3$ is the same space spanned by $\ba_1, \ba_2, \ba_3$. The situation is shown in Figure~\ref{fig:gram-schmidt2}, in which we choose \textbf{the direction of $\bb_2$ as the $y$-axis of the Cartesian coordinate system}. $\widehat{\ba}_3$ is the projection of $\ba_3$ onto line $\bb_1$, $\bar{\ba}_3$ is the projection of $\ba_3$ onto line $\bb_2$. It can be shown that the part of $\ba_3$ perpendicular to both $\bb_1$ and $\bb_2$ is $\bb_3=\ba_3-\widehat{\ba}_3-\bar{\ba}_3$ from the figure.

Finally, we normalize each vector by dividing their length which produces three orthonormal vectors $\bq_1 = \frac{\bb_1}{||\bb_1||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$, $\bq_2 = \frac{\bb_2}{||\bb_2||}$.

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Project $\ba_2$ onto the space perpendicular to $\bb_1$.]{\label{fig:gram-schmidt1}
		\includegraphics[width=0.47\linewidth]{./imgs/gram-schmidt1.pdf}}
	\quad
	\subfigure[Project $\ba_3$ onto the space perpendicular to $\bb_1, \bb_2$.]{\label{fig:gram-schmidt2}
		\includegraphics[width=0.47\linewidth]{./imgs/gram-schmidt2.pdf}}
	\caption{The Gram-Schmidt process.}
	\label{fig:gram-schmidt-12}
\end{figure}

This idea can be extended to a set of vectors rather than only three. And we call this process as \textit{Gram-Schmidt process}. After this process, matrix $\bA$ will be triangularized. The method is named after Jørgen Pedersen Gram and Erhard Schmidt, but it appeared earlier in the work of Pierre-Simon Laplace in the theory of Lie group decomposition.

As we mentioned previously, the idea of the QR decomposition is the construction of a sequence of orthonormal vectors $\bq_1, \bq_2, \ldots$ that span the same successive subspaces.
$$
\bigg\{\cspace([\bq_1])=\cspace([\ba_1])\bigg\} \subseteq
\bigg\{\cspace([\bq_1, \bq_2])=\cspace([\ba_1, \ba_2])\bigg\} \subseteq
\bigg\{\cspace([\bq_1, \bq_2, \bq_3])=\cspace([\ba_1, \ba_2, \ba_3])\bigg\}
\subseteq \ldots,
$$
This implies any $\ba_k$ is in the space spanned by $\cspace([\bq_1, \bq_2, \ldots, \bq_k])$. \footnote{And also, any $\bq_k$ is in the space spanned by $\cspace([\ba_1, \ba_2, \ldots, \ba_k])$.} As long as we have found these orthonormal vectors, to reconstruct $\ba_i$'s from the orthonormal matrix $\bQ=[\bq_1, \bq_2, \ldots, \bq_n]$, an upper triangular matrix $\bR$ is needed such that $\bA = \bQ\bR$.

The Gram–Schmidt process is not the only algorithm for finding the QR decomposition.
Several other QR decomposition algorithms exist such as \textit{Householder reflections} and \textit{Givens rotations} which are more reliable in the presence of round-off errors.  These QR decomposition methods may also change the
order in which the columns of $\bA$ are processed.

\index{Orthogonal matrix}
\index{Orthogonal}
\index{Orthonormal}
\section{Orthogonal vs Orthonormal}\label{section:orthogonal-orthonormal-qr}
The vectors $\bq_1, \bq_2, \ldots, \bq_n\in \real^m$ are \textit{mutually orthogonal} when their dot products $\bq_i^\top\bq_j$ are zero whenever $i \neq j$. When each vector is divided by its length, the vectors become orthogonal unit vectors. Then the vectors $\bq_1, \bq_2, \ldots, \bq_n$ are \textit{mutually orthonormal}. We put the orthonormal vectors into a matrix $\bQ$.

When $m\neq n$: the matrix $\bQ$ is easy to work with because $\bQ^\top\bQ=\bI \in \real^{n\times n}$. Such $\bQ$ with $m\neq n$ is sometimes referred to as a \textbf{semi-orthogonal} matrix.

When $m= n$: the matrix $\bQ$ is square, $\bQ^\top\bQ=\bI$ means that $\bQ^\top=\bQ^{-1}$, i.e., the transpose of $\bQ$ is also the inverse of $\bQ$. Then we also have $\bQ\bQ^\top=\bI$, i.e., $\bQ^\top$ is the \textbf{two-sided inverse} of $\bQ$. We call this $\bQ$ an \textbf{orthogonal matrix}. \footnote{Note here we use the term \textit{orthogonal matrix} to mean the matrix $\bQ$ has orthonormal columns. The term \textit{orthonormal matrix} is \textbf{not} used for historical reasons.} To see this, we have
$$
\begin{bmatrix}
	\bq_1^\top \\
	\bq_2^\top\\
	\vdots \\
	\bq_n^\top
\end{bmatrix}
\begin{bmatrix}
	\bq_1 &\bq_2 & \ldots & \bq_n
\end{bmatrix}
=
\begin{bmatrix}
	1 & & & \\
	& 1 & & \\
	& & \ddots & \\
	& & & 1
\end{bmatrix}.
$$
In other words, $\bq_i^\top \bq_j = \delta_{ij}$ where $\delta_{ij}$ is the \textit{Kronecker delta}. The columns of an orthogonal matrix $\bQ\in \real^{n\times n}$ form an \textbf{orthonormal basis} of $\real^n$.

Orthogonal matrices can be viewed as matrices that change the basis of other matrices. Hence they preserve the angle (inner product) between the vectors:
$\qquad \bu^\top \bv = (\bQ\bu)^\top(\bQ\bv)$.
This invariance of the inner products of angles between the vectors is preserved, which also relies on the invariance of their lengths: $\qquad ||\bQ\bu|| = ||\bu||$.
In real cases, multiplied by a orthogonal matrix $\bQ$ will rotate (if $\det(\bQ)=1$) or reflect (if $\det(\bQ)=-1$) the original vector space. Many decomposition algorithms will result in two orthogonal matrices, thus such rotations or reflections will happen twice.

\index{Classical Gram-Schmidt process}
\index{Modified Gram-Schimidt process}
\index{CGS}
\index{MGS}
\index{Numerical stability}
\section{Computing the Reduced QR Decomposition via CGS and MGS}\label{section:qr-gram-compute}

We write out this form of the reduced QR Decomposition such that $\bA=\bQ\bR$ where $\bQ\in \real^{m\times n}$ and $\bR\in \real^{n\times n}$:
\begin{equation}
\bA=\left[
\begin{matrix}
	\ba_1 & \ba_2 & ... & \ba_n
\end{matrix}
\right]
=\left[
\begin{matrix}
	\bq_1 & \bq_2 & ... & \bq_n
\end{matrix}
\right]
\begin{bmatrix}
	r_{11} & r_{12}& \dots & r_{1n}\\
	       & r_{22}& \dots & r_{2n}\\
	       &       &    \ddots  & \vdots \\
	\multicolumn{2}{c}{\raisebox{1.3ex}[0pt]{\Huge0}} & & r_{nn} \nonumber
\end{bmatrix}.
\end{equation}
The orthogonal matrix $\bQ$ can be easily calculated by the Gram-Schmidt process. To see why we have the upper triangular matrix $\bR$, we write out these equations
\begin{equation*}
\begin{aligned}
\ba_1 & = r_{11}\bq_1 &= \sum_{i=1}^{1} r_{i1}\bq_1, \\
&\vdots& \\
\ba_k &= r_{1k}\bq_1 + r_{2k}\bq_2 + \ldots + r_{kk}\bq_k  &= \sum_{i=1}^{k} r_{ik} \bq_k,\\
&\vdots& \\
\end{aligned}
\end{equation*}
which coincides with the second equation of Equation~\eqref{equation:gram-schdt-eq2} and conforms to the form of an upper triangular matrix $\bR$. And if we extend the idea of Equation~\eqref{equation:gram-schdt-eq2} into the $k$-th term, we will get
$$
\begin{aligned}
\ba_k &= \sum_{i=1}^{k-1}(\bq_i^\top\ba_k)\bq_i + \ba_k^\perp = \sum_{i=1}^{k-1}(\bq_i^\top\ba_k)\bq_i + ||\ba_k^\perp||\cdot \bq_k,
\end{aligned}
$$
where $\ba_k^\perp$ is such $\bb_k$ in Equation~\eqref{equation:gram-schdt-eq2} that we emphasize the ``perpendicular" property here.
This implies we can gradually orthonormalize $\bA$ to an orthonormal set $\bQ=[\bq_1, \bq_2, \ldots, \bq_n]$ by
\begin{equation}\label{equation:qr-gsp-equation}
\left\{
\begin{aligned}
	r_{ik} &= \bq_i^\top\ba_k, \,\,\,\,\forall i \in \{1,2,\ldots, k-1\};\\
	\ba_k^\perp&= \ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i;\\
	r_{kk} &= ||\ba_k^\perp||;\\
	\bq_k &= \ba_k^\perp/r_{kk}.
\end{aligned}
\right.
\end{equation}
The procedure is formulated in Algorithm~\ref{alg:reduced-qr}.
\begin{algorithm}[h]
\caption{Reduced QR Decomposition via Gram-Schmidt Process}
\label{alg:reduced-qr}
\begin{algorithmic}[1]
\Require Matrix $\bA$ has linearly independent columns with size $m\times n $ and $m\geq n$;
\For{$k=1$ to $n$} \Comment{compute $k$-th column of $\bQ,\bR$}
\For{$i=1$ to $k-1$}
\State $r_{ik} =\bq_i^\top\ba_k$; \Comment{entry ($i,k$) of $\bR$, $2m-1$ flops}
\EndFor  \Comment{all $k-1$ iterations: $(k-1)(2m-1)$ flops}
\State $\ba_k^\perp= \ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i$;\Comment{$2m(k-1)$ flops}
\State $r_{kk} = ||\ba_k^\perp||$; \Comment{main diagonal of $\bR$, $2m$ flops}
\State $\bq_k = \ba_k^\perp/r_{kk}$; \Comment{$m$ flops}
\EndFor
\State Output $\bQ=[\bq_1, \ldots, \bQ_n]$ and $\bR$ with entry $(i,k)$ being $r_{ik}$;

\end{algorithmic}
\end{algorithm}

\index{Orthogonal rojection}
\subsection*{\textbf{Orthogonal Projection}}
We notice again from Equation~\eqref{equation:qr-gsp-equation}, i.e., step 2 to step 6 in Algorithm~\ref{alg:reduced-qr}, that the first two equalities imply that
\begin{equation}\label{equation:qr-gsp-equation2}
\left.
\begin{aligned}
	r_{ik} &= \bq_i^\top\ba_k, \,\,\,\,\forall i \in \{1,2,\ldots, k-1\}\\
	\ba_k^\perp&= \ba_k-\sum_{i=1}^{k-1}r_{ik}\bq_i\\
\end{aligned}
\right\}
\rightarrow
\ba_k^\perp= \ba_k- \bQ_{k-1}\bQ_{k-1}^\top \ba_k=(\bI-\bQ_{k-1}\bQ_{k-1}^\top )\ba_k,
\end{equation}
where $\bQ_{k-1}=[\bq_1,\bq_2,\ldots, \bq_{k-1}]$. This implies $\bq_k$ can be obtained by
$$
\bq_k = \frac{\ba_k^\perp}{||\ba_k^\perp||} = \frac{(\bI-\bQ_{k-1}\bQ_{k-1}^\top )\ba_k}{||(\bI-\bQ_{k-1}\bQ_{k-1}^\top )\ba_k||}.
$$
The matrix $(\bI-\bQ_{k-1}\bQ_{k-1}^\top )$ in the above equation is known as an \textit{orthogonal projection matrix} that will project $\ba_k$ along the column space of $\bQ_{k-1}$, i.e., project a vector so that the vector is perpendicular to the column space of $\bQ_{k-1}$. The net result is that the $\ba_k^\perp$ or $\bq_k$ calculated in this way will be orthogonal to the $\cspace(\bQ_{k-1})$, i.e., in the null space of $\bQ_{k-1}^\top$: $\nspace(\bQ_{k-1}^\top)$ by the fundamental theorem of linear algebra (Theorem~\ref{theorem:fundamental-linear-algebra}, p.~\pageref{theorem:fundamental-linear-algebra}).

Let $\bP_1=(\bI-\bQ_{k-1}\bQ_{k-1}^\top )$ and we claimed above $\bP_1=(\bI-\bQ_{k-1}\bQ_{k-1}^\top )$ is an orthogonal projection matrix such that $\bP_1\bv$ will project the $\bv$ onto the null space of $\bQ_{k-1}$. And actually, let $\bP_2=\bQ_{k-1}\bQ_{k-1}^\top$, then $\bP_2$ is also an orthogonal projection matrix such that $\bP_2\bv$ will project the $\bv$ onto the column space of $\bQ_{k-1}$.

But why do the matrix $\bP_1, \bP_2$ can magically project a vector onto the corresponding subspaces? It can be easily shown that the column space of $\bQ_{k-1}$ is equal to the column space of $\bQ_{k-1}\bQ_{k-1}^\top$:
$$
\cspace(\bQ_{k-1})=\cspace(\bQ_{k-1}\bQ_{k-1}^\top)=\cspace(\bP_2).
$$
Therefore, the result of $\bP_2\bv$ is a linear combination of the columns of $\bP_2$, which is in the column space of $\bP_2$ or the column space of $\bQ_{k-1}$. The formal definition of a \textit{projection matrix} $\bP$ is that it is idempotent $\bP^2=\bP$ such that projecting twice is equal to projecting once. What makes the above $\bP_2=\bQ_{k-1}\bQ_{k-1}^\top $ different is that the projection $\widehat{\bv}$ of any vector $\bv$ is perpendicular to $\bv-\widehat{\bv}$:
$$
(\widehat{\bv}=\bP_2\bv) \perp (\bv-\widehat{\bv}).
$$
This goes to the original definition we gave above: the \textit{orthogonal projection matrix}. To
avoid confusion, one may use the term \textit{oblique projection matrix} in the nonorthogonal
case. When $\bP_2$ is an orthogonal projection matrix, $\bP_1=\bI-\bP_2$ is also an orthogonal projection matrix that will project any vector onto the space perpendicular to the $\cspace(\bQ_{k-1})$, i.e., $\nspace(\bQ_{k-1}^\top)$. Therefore, we conclude the two orthogonal projections:
$$
\left\{
\begin{aligned}
	\bP_1: &\gap \text{project onto $\nspace(\bQ_{k-1}^\top)$;} \\
	\bP_2: &\gap \text{project onto $\cspace(\bQ_{k-1})$} .
\end{aligned}
\right.
$$

The further result that is important to notice is when the columns of $\bQ_{k-1}$ are mutually orthonormal, we have the following decomposition:
\begin{equation}\label{equation:qr-orthogonal-equality}
\boxed{\bP_1 = \bI - \bQ_{k-1}\bQ_{k-1}^\top = (\bI-\bq_1\bq_1^\top)(\bI-\bq_2\bq_2^\top)\ldots (\bI-\bq_{k-1}\bq_{k-1}^\top),}
\end{equation}
where $\bQ_{k-1}=[\bq_1,\bq_2,\ldots, \bq_{k-1}]$ and each $(\bI-\bq_i\bq_i^\top)$ is to project a vector into the perpendicular space of $\bq_i$.
This finding is important to make a step further toward a \textit{modified Gram-Schmidt process (MGS)} where we project and subtract on the fly. To avoid confusion, the previous Gram-Schmidt is called the \textit{classical Gram-Schmidt process (CGS)}. The difference between the CGS and MGS is, in the CGS, we project the same vector onto the orthogonormal ones and subtract afterward. However, in the MGS, the projection and subtraction are done in an interleaved manner. A three-column example $\bA=[\ba_1,\ba_2,\ba_3]$ is shown in Figure~\ref{fig:projection-mgs-demons-3d} where each step is denoted in a different color. We summarize the difference between the CGS and MGS processes for obtaining $\bq_k$ via the $k$-th column $\ba_k$ of $\bA$ and the orthonormalized vectors $\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$:
$$
\begin{aligned}
	\text{(CGS)}: &\, \text{obtain $\bq_k$ by normalizing $\ba_k^\perp=(\bI-\bQ_{k-1}\bQ_{k-1}^\top)\ba_k$;} \\
	\text{(MGS)}: &\, \text{obtain $\bq_k$ by normalizing  $\ba_k^\perp=\left\{(\bI-\bq_{k-1}\bq_{k-1}^\top)\ldots\left[(\bI-\bq_2\bq_2^\top)\left((\bI-\bq_1\bq_1^\top) \ba_k\right)\right]\right\}$,}
\end{aligned}
$$
where the parentheses of the MGS indicate the order of the computation.
\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[CGS, step 1: \textcolor{blue}{blue} vector; step 2: \textcolor{green}{green} vector; step 3: \textcolor{purple}{purple} vector.]{\label{fig:projection-mgs-demons-cgs}
		\includegraphics[width=0.47\linewidth]{./imgs/projectqr-cgs.pdf}}
	\quad
	\subfigure[MGS, step 1: \textcolor{blue}{blue} vector; step 2: \textcolor{purple}{purple} vector.]{\label{fig:projection-mgs-demons-mgs}
		\includegraphics[width=0.47\linewidth]{./imgs/projectqr-mgs.pdf}}
	\caption{CGS vs MGS in 3-dimensional space where $\bq_2^\prime$ is parallel to $\bq_2$ so that projecting on $\bq_2$ is equivalent to projecting on $\bq_2^\prime$.}
	\label{fig:projection-mgs-demons-3d}
\end{figure}

\paragraph{What's the difference?}
Taking the three-column matrix $\bA=[\ba_1, \ba_2, \ba_3]$ as an example. Suppose we have computed $\{\bq_1, \bq_2\}$ such that $span\{\bq_1, \bq_2\}=span\{\ba_1, \ba_2\}$. And we want to proceed to compute the $\bq_3$.

In the CGS, the orthogonalization of column $\ba_n$ against column $\{\bq_1, \bq_2\}$ is performed by projecting the original column $\ba_3$ of $\bA$ onto $\{\bq_1, \bq_2\}$ respectively and subtracting at once:
\begin{equation}\label{equation:cgs-3d-exmp}
	\left\{
	\begin{aligned}
		\ba_3^\perp &= \ba_3 - (\bq_1^\top\ba_3)\bq_1 - (\bq_2^\top\ba_3)\bq_2\\
		&= \ba_3 - (\bq_1\bq_1^\top)\ba_3 - \boxed{(\bq_2\bq_2^\top)\ba_3}\\
		\bq_3 &=  \frac{\ba_3^\perp}{||\ba_3^\perp||},
	\end{aligned}
	\right.
\end{equation}
as shown in Figure~\ref{fig:projection-mgs-demons-cgs}.

In the MGS, on the other hand, the components along each $\{\bq_1, \bq_2\}$ are immediately subtracted out of the rest of the column $\ba_3$ as soon as the $\{\bq_1, \bq_2\}$ are computed. Therefore the orthogonalization of column $\ba_3$ against $\{\bq_1, \bq_2\}$ is not performed by projecting the original column $\ba_3$ against $\{\bq_1, \bq_2\}$ as it is in the CGS, but rather against a vector obtained by subtracting from that column $\ba_3$ of $\bA$ the components in the direction of $\bq_1, \bq_2$ successively. This is important because the error components of $\bq_i$ in $span\{\bq_1, \bq_2\}$ will be smaller (we will further discuss this in the following paragraphs).

More precisely, in the MGS the orthogonalization of column $\ba_3$ against $\bq_1$ is performed by subtracting the component of $\bq_1$ from the vector $\ba_3$:
$$
\ba_3^{(1) }=  (\bI-\bq_1\bq_1^\top)\ba_3 = \ba_3 - (\bq_1\bq_1^\top)\ba_3,
$$
where $\ba_3^{(1) }$ is the component of $\ba_3$ that lies in a space perpendicular to $\bq_1$. And the further step is performed by
\begin{equation}\label{equation:mgs-3d-exmp}
	\begin{aligned}
		\ba_3^{(2) }=  (\bI-\bq_2\bq_2^\top)\ba_3^{(1) }&=\ba_3^{(1) }-(\bq_2\bq_2^\top)\ba_3^{(1) }\\
		&=\ba_3 - (\bq_1\bq_1^\top)\ba_3-\boxed{(\bq_2\bq_2^\top)\textcolor{blue}{\ba_3^{(1) }}}
	\end{aligned}
\end{equation}
where $\ba_3^{(2) }$ is the component of $\ba_3^{(1) }$ that lies in a space perpendicular to $\bq_2$  and we highlight the difference to the CGS in Equation~\eqref{equation:cgs-3d-exmp} by \textcolor{blue}{blue} text. This net result is that $\ba_3^{(2) }$ is the component of $\ba_3$ that lies in the space perpendicular to $\{\bq_1, \bq_2\}$ as shown in Figure~\ref{fig:projection-mgs-demons-mgs}.

\paragraph{Main difference and catastrophic cancellation} The key difference is that the $\ba_3$ can in general have large components in $span\{\bq_1, \bq_2\}$ in which case one starts with large
values and ends up with small values with large relative errors in them. This is known as the problem of \textit{catastrophic cancellation}. Whereas $\ba_3^{(1) }$ is in the direction perpendicular to $\bq_1$ and has only a small ``error" component in the direction of $\bq_1$. Compare the \fbox{boxed} text in Equation~\eqref{equation:cgs-3d-exmp} and \eqref{equation:mgs-3d-exmp}, it is not hard to see $(\bq_2\bq_2^\top)\ba_3^{(1) }$ in Equation~\eqref{equation:mgs-3d-exmp} is more accurate by the above argument. And thus, because of the much smaller error in this projection factor, the MGS introduces less orthogonalization error at each subtraction step than that in the CGS. In fact, it can be shown that the final $\bQ$ obtained in the CGS satisfies
$$
||\bI-\bQ\bQ^\top|| \leq O(\epsilon \kappa^2(\bA)),
$$
where $\kappa(\bA)$ is a value larger than 1 determined by $\bA$.
Whereas, in the MGS, the error satisfies
$$
||\bI-\bQ\bQ^\top|| \leq O(\epsilon \kappa(\bA)).
$$
That is, the $\bQ$ obtained in the MGS is more orthogonal.

\paragraph{More to go, preliminaries for Householder and Givens methods} Although, we claimed here the MGS usually works better than the CGS in practice. The MGS can still fall victim to the \textit{catastrophic cancellation} problem. Suppose in iteration $k$ of the MGS algorithm, $\ba_k$ is almost in the span of $\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$. This will result in that $\ba_k^\perp$ has only a small component that is perpendicular to $span\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$, whereas the ``error" component in the $span\{\bq_1, \bq_2, \ldots, \bq_{k-1}\}$ will be amplified and the net result is $\bQ$ will be less orthonormal. In this case, if we can find a successive set of orthogonal matrices $\{\bQ_1, \bQ_2, \ldots, \bQ_l\}$ such that $\bQ_l\ldots\bQ_2\bQ_1\bA$ is triangularized, then $\bQ=(\bQ_l\ldots\bQ_2\bQ_1)^\top$ will be ``more" orthogonal than the CGS or the MGS. We will discuss this method in Section~\ref{section:qr-via-householder} (p.~\pageref{section:qr-via-householder}) and \ref{section:qr-givens}  (p.~\pageref{section:qr-givens}) via Householder reflectors and Givens rotations.

\section{Computing the Full QR Decomposition via the Gram-Schmidt Process}\label{section:silentcolu_qrdecomp}
A full QR decomposition of an $m\times n$ matrix with linearly independent columns goes further by appending additional $m-n$ orthonormal columns to $\bQ$ so that it becomes an $m\times m$ orthogonal matrix. In addition, rows of zeros are appended to $\bR$ so that it becomes an $m\times n$ upper triangular matrix. We call the additional columns in $\bQ$ \textbf{silent columns} and additional rows in $\bR$ \textbf{silent rows}. The comparison between the reduced QR decomposition and the full QR decomposition is shown in Figure~\ref{fig:qr-comparison} where silent columns in $\bQ$ are denoted in \textcolor{gray}{gray}, blank entries are zero and \textcolor{blue}{blue} entries are elements that are not necessarily zero.

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Reduced QR decomposition]{\label{fig:gphalf}
		\includegraphics[width=0.47\linewidth]{./imgs/qrreduced.pdf}}
	\quad
	\subfigure[Full QR decomposition]{\label{fig:gpall}
		\includegraphics[width=0.47\linewidth]{./imgs/qrfull.pdf}}
	\caption{Comparison between the reduced and full QR decompositions.}
	\label{fig:qr-comparison}
\end{figure}

\index{Independence check}
\section{Dependent Columns}\label{section:dependent-gram-schmidt-process}
Previously, we assumed matrix $\bA$ has linearly independent columns. However, this is not always necessary. Suppose in step $k$ of CGS or MGS, $\ba_k$ is in the plane spanned by $\bq_1, \bq_2, \ldots, \bq_{k-1}$ which is equivalent to the space spanned by $\ba_1, \ba_2, \ldots, \ba_{k-1}$, i.e., vectors $\ba_1, \ba_2, \ldots, \ba_k$ are dependent. Then $r_{kk}$ will be zero and $\bq_k$ does not exist because of the zero division. At this moment, we simply pick $\bq_k$ arbitrarily to be any normalized vector that is orthogonal to $\cspace([\bq_1, \bq_2, \ldots, \bq_{k-1}])$ and continue the Gram-Schmidt process. Again, for matrix $\bA$ with dependent columns, we have both reduced and full QR decomposition algorithms.

This idea can be further extended that when $\bq_k$ does not exist, we just skip the current steps. And add the silent columns in the end. In this sense, QR decomposition for a matrix with dependent columns is not unique. However, as long as you stick to a systematic process or a methodical procedure, QR decomposition for any matrix is unique.

This finding can also help to decide whether a set of vectors are linearly independent or not. Whenever $r_{kk}$ in CGS or MGS is zero, we report the vectors $\ba_1, \ba_2, \ldots, \ba_k$ are dependent and stop the algorithm for ``independence checking".

\section{QR with Column Pivoting: Column-Pivoted QR (CPQR)}\label{section:cpqr}

Suppose $\bA$ has dependent columns, a column-pivoted QR (CPQR) decomposition can be found as follows.
\begin{theoremHigh}[Column-Pivoted QR Decomposition\index{Column-pivoted QR (CPQR)}]\label{theorem:rank-revealing-qr-general}
Every $m\times n$ matrix $\bA=[\ba_1, \ba_2, ..., \ba_n]$ with $m\geq n$ and rank $r$ can be factored as
$$
\bA\bP = \bQ
\begin{bmatrix}
\bR_{11} & \bR_{12} \\
\bzero   & \bzero
\end{bmatrix},
$$
where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ\in \real^{m\times m}$ is an orthogonal matrix, and $\bP$ is a permutation matrix. This is also known as the \textbf{full} CPQR decomposition. Similarly, the \textbf{reduced} version is given by
$$
\bA\bP = \bQ_r
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
\end{bmatrix},
$$
where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ_r\in \real^{m\times r}$ contains orthonormal columns, and $\bP$ is a permutation matrix.
\end{theoremHigh}
\index{CPQR}
\index{Column pivoting}
\subsection{A Simple CPQR via CGS}
\paragraph{A Simple CPQR via CGS}
The classical Gram-Schmidt process can compute this CPQR decomposition.
Following the QR decomposition for dependent columns, when $r_{kk}=0$, the column $k$ of $\bA$ is dependent on the previous $k-1$ columns. Whenever this happens, we permute this column into the last column and continue the Gram-Schmidt process. We notice that $\bP$ is the permutation matrix that interchanges the dependent columns into the last $n-r$ columns. Suppose the first $r$ columns of $\bA\bP$ are $[\widehat{\ba}_1, \widehat{\ba}_2, \ldots, \widehat{\ba}_r]$, and the span of them is just the same as the span of $\bQ_r$ (in the reduced version), or the span of $\bQ_{:,:r}$ (in the full version)
$$
\cspace([\widehat{\ba}_1, \widehat{\ba}_2, \ldots, \widehat{\ba}_r]) = \cspace(\bQ_r) = \cspace(\bQ_{:,:r}).
$$
And $\bR_{12}$ is a matrix that recovers the dependent $n-r$ columns from the column space of $\bQ_r$ or column space of $\bQ_{:,:r}$. The comparison of reduced and full CPQR decomposition is shown in Figure~\ref{fig:qr-comparison-rank-reveal} where silent columns in $\bQ$ are denoted in \textcolor{gray}{grey}, blank entries are zero and \textcolor{blue}{blue}/\textcolor{orange}{orange} entries are elements that are not necessarily zero.

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Reduced CPQR decomposition]{\label{fig:gphalf-rank-reveal}
		\includegraphics[width=0.47\linewidth]{./imgs/qrreduced-revealing.pdf}}
	\quad
	\subfigure[Full CPQR decomposition]{\label{fig:gpall-rank-reveal}
		\includegraphics[width=0.47\linewidth]{./imgs/qrfull-revealing.pdf}}
	\caption{Comparison between the reduced and full CPQR decompositions.}
	\label{fig:qr-comparison-rank-reveal}
\end{figure}

\subsection{A Practical CPQR via CGS}\label{section:practical-cpqr-cgs}
\paragraph{A Practical CPQR via CGS}
We notice that the simple CPQR algorithm pivot the first $r$ independent columns into the first $r$ columns of $\bA\bP$. Let $\bA_1$ be the first $r$ columns of $\bA\bP$, and $\bA_2$ be the rest. Then, from the full CPQR, we have
$$
[\bA_1, \bA_2] =
\bQ \begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix}
=\left[
\bQ \begin{bmatrix}
	\bR_{11}  \\
	\bzero
\end{bmatrix}
,
\bQ \begin{bmatrix}
	\bR_{12}  \\
	\bzero
\end{bmatrix}
\right]
.
$$
It is not hard to see that
$$
||\bA_2|| = \left\Vert\bQ \begin{bmatrix}
	\bR_{12}  \\
	\bzero
\end{bmatrix}\right\Vert
=
\left\Vert\begin{bmatrix}
	\bR_{12}  \\
	\bzero
\end{bmatrix}\right\Vert
=
\left\Vert \bR_{12}  \right\Vert,
$$
where the penultimate equality comes from the orthogonal equivalence under the matrix norm. Therefore, the norm of $\bR_{12}$ is decided by the norm of $\bA_2$. When favoring well-conditioned CPQR, $\bR_{12}$ should be small in norm. And a practical CPQR decomposition is to permute columns of the matrix $\bA$ firstly such that the columns are ordered decreasingly in vector norm:
$$
\widetildebA = \bA\bP_0 = [\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_n}],
$$
where $\{j_1, j_2, \ldots, j_n\}$ is a permuted index set of $\{1,2,\ldots, n\}$ and
$$
||\ba_{j_1}||\geq  ||\ba_{j_2}||\geq  \ldots\geq  ||\ba_{j_n}||.
$$
Then apply the ``simple" reduced CPQR decomposition on $\widetildebA$ such that $\widetildebA \bP_1= \bQ_r[\bR_{11}, \bR_{12}]$. The ``practical" reduced CPQR of $\bA$ is then recovered as
$$
\bA\underbrace{\bP_0\bP_1}_{\bP} =\bQ_r[\bR_{11}, \bR_{12}].
$$
The further optimization of the CPQR algorithm is via the MGS where the extra bonus is to stop at a point when the factorization works on a rank deficient submatrix and the CPQR via this MGS can find the numerical rank \citep{lu2021numerical}. This is known as the \textit{partial factorization} and we shall not give the details.

\index{Column pivoting}
\index{Revealing rank one deficiency}
\section{QR with Column Pivoting: Revealing Rank One Deficiency}\index{Rank-revealing}
We notice that column-pivoted QR is just one method to find the column permutation where $\bA$ is rank deficient and we interchange the first linearly independent $r$ columns of $\bA$ into the first $r$ columns of the $\bA\bP$. If $\bA$ is nearly rank-one deficient and we would like to find a column permutation of $\bA$ such that the resulting pivotal element $r_{nn}$ of the QR decomposition is small. This is known as the \textit{revealing rank-one deficiency} problem.
\begin{theoremHigh}[Revealing Rank One Deficiency, \citep{chan1987rank}]\label{theorem:finding-good-qr-ordering}
If $\bA\in \real^{m\times n}$ and $\bv\in \real^n$ is a unit 2-norm vector (i.e., $||\bv||=1$), then there exists a permutation $\bP$ such that the reduced QR decomposition
$$
\bA\bP = \bQ\bR
$$
satisfies $r_{nn} \leq \sqrt{n} \epsilon$ where $\epsilon = ||\bA\bv||$ and $r_{nn}$ is the $n$-th diagonal of $\bR$. Note that $\bQ\in \real^{m\times n}$ and $\bR\in \real^{n\times n}$ in the reduced QR decomposition.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:finding-good-qr-ordering}]
Suppose $\bP\in \real^{n\times n}$ is a permutation matrix such that if $\bw=\bP^\top\bv$ where
$$
|w_n| = \max |v_i|,  \,\,\,\, \forall i \in \{1,2,\ldots,n\}.
$$
That is, the last component of $\bw$ is equal to the max component of $\bv$ in absolute value. Then we have $|w_n| \geq 1/\sqrt{n}$. Suppose the QR decomposition of $\bA\bP$ is $\bA\bP = \bQ\bR$, then
$$
\epsilon = ||\bA\bv|| = ||(\bQ^\top\bA\bP) (\bP^\top\bv)|| = ||\bR\bw|| =
\begin{bmatrix}
	\vdots \\
	r_{nn} w_n
\end{bmatrix}
\geq |r_{nn} w_n| \geq |r_{nn}|/\sqrt{n},
$$
This completes the proof.
\end{proof}
The following discussion is based on the existence of the singular value decomposition (SVD) which will be introduced in Section~\ref{section:SVD} (p.~\pageref{section:SVD}). Feel free to skip at a first reading. Suppose the SVD of $\bA$ is given by $\bA = \sum_{i=1}^{n} \sigma_i \bu_i\bv_i^\top$, where $\sigma_i$'s are singular values with $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_n$, i.e., $\sigma_n$ is the smallest singular value, and $\bu_i$'s, $\bv_i$'s are the left and right singular vectors respectively. Then, if we let $\bv = \bv_n$ such that $\bA\bv_n = \sigma_n \bu_n$, \footnote{We will prove that the right singular vector of $\bA$ is equal to the right singular vector of $\bR$ if the $\bA$ has QR decomposition $\bA=\bQ\bR$ in Lemma~\ref{lemma:svd-for-qr} (p.~\pageref{lemma:svd-for-qr}). The claim can also be applied to the singular values. So $\bv_n$ here is also the right singular vector of $\bR$.} we have
$$
||\bA\bv|| = \sigma_n.
$$
By constructing a permutation matrix $\bP$ such that
$$
|\bP^\top \bv|_n = \max |\bv_i|,  \,\,\,\, \forall i \in \{1,2,\ldots,n\},
$$
we will find a QR decomposition of $\bA=\bQ\bR$ with a pivot $r_{nn}$ smaller than $\sqrt{n}\sigma_n$. If $\bA$ is rank-one deficient, then $\sigma_n$ will be close to 0 and $r_{nn}$ is thus bounded to a small value in magnitude which is close to 0.

\index{Revealing rank r deficiency}
\section{QR with Column Pivoting: Revealing Rank r Deficiency*}\label{section:rank-r-qr}\index{Rank-revealing QR}
Following from the last section, suppose now we want to compute the reduced QR decomposition where $\bA\in \real^{m\times n}$ is nearly rank $r$ deficient with $r>1$. Our goal now is to find a permutation $\bP$ such that
\begin{equation}\label{equation:rankr-reval-qr}
\bA\bP =
\bQ\bR=
\bQ
\begin{bmatrix}
	\bL & \bM \\
	\bzero & \bN
\end{bmatrix},
\end{equation}
where $\bN \in \real^{r\times r}$ and $||\bN||$ is small in some norm.
A recursive algorithm can be applied to do so. Suppose we have already isolated a small $k\times k$ block $\bN_k$, based on which, if we can isolate a small $(k+1)\times (k+1)$ block $\bN_{k+1}$, then we can find the permutation matrix recursively. To repeat, suppose we have the permutation $\bP_k$ such that the $\bN_k \in \real^{k\times k}$ has a small norm,
$$
\bA\bP_k = \bQ_k \bR_k=
\bQ_k
\begin{bmatrix}
\bL_k & \bM_k \\
\bzero & \bN_k
\end{bmatrix}.
$$
We want to find a permutation $\bP_{k+1}$, such that $\bN_{k+1} \in \real^{(k+1)\times (k+1)}$ also has a small norm,
$$
\boxed{
\bA\bP_{k+1} = \bQ_{k+1} \bR_{k+1}=
\bQ_{k+1}
\begin{bmatrix}
	\bL_{k+1} & \bM_{k+1} \\
	\bzero & \bN_{k+1}
\end{bmatrix}}.
$$
From the algorithm introduced in the last section, there is an $(n-k)\times (n-k)$ permutation matrix $\widetilde{\bP}_{k+1}$ such that $\bL_k \in \real^{(n-k)\times (n-k)}$ has the QR decomposition $\bL_k \widetilde{\bP}_{k+1} = \widetilde{\bQ}_{k+1}\widetilde{\bL}_k$ such that the entry $(n-k, n-k)$ of $\widetilde{\bL}_k$ is small. By constructing
$$
\bP_{k+1} = \bP_k
\begin{bmatrix}
\widetilde{\bP}_{k+1} & \bzero \\
\bzero & \bI
\end{bmatrix},
\qquad
\bQ_{k+1} = \bQ_k
\begin{bmatrix}
\widetilde{\bQ}_{k+1} & \bzero \\
\bzero & \bI
\end{bmatrix},
$$
we have
$$
\boxed{
\bA \bP_{k+1} = \bQ_{k+1}
\begin{bmatrix}
\widetilde{\bL}_k & \widetilde{\bQ}_{k+1}^\top  \bM_k \\
\bzero & \bN_k
\end{bmatrix}}.
$$
We know that entry $(n-k, n-k)$ of $\widetilde{\bL}_k$ is small, if we can prove the last row of $\widetilde{\bQ}_{k+1}^\top  \bM_k$ is small in norm, then we find the QR decomposition revealing rank $k+1$ deficiency (see \citep{chan1987rank} for a proof).

\index{Householder transformation}
\index{Householder reflector}
\section{Existence of the QR Decomposition via the Householder Reflector}\label{section:qr-via-householder}
We first give the formal definition of a Householder reflector and we will take a look at its properties.
\begin{definition}[Householder Reflector]\label{definition:householder-reflector}
Let $\bu \in \real^n$ be a vector of unit length (i.e., $||\bu||=1$). Then $\bH = \bI - 2\bu\bu^\top$ is said to be a \textit{Householder reflector}, a.k.a., a \textit{Householder transformation}. We call this $\bH$ the Householder reflector associated with the unit vector $\bu$ where the unit vector $\bu$ is also known as the \textit{Householder vector}. If a vector $\bx$ is multiplied by $\bH$, then it is reflected in the hyperplane $span\{\bu\}^\perp$.

\item Note that if $||\bu|| \neq 1$, we can define $\bH = \bI - 2  \frac{\bu\bu^\top}{\bu^\top\bu} $ as the Householder reflector.
\end{definition}
Then we have the following corollary from this definition.
\begin{corollary}[Unreflected by Householder]
Any vector $\bv$ that is perpendicular to $\bu$ is left unchanged by the Householder transformation, that is, $\bH\bv=\bv$ if $\bu^\top\bv=0$.
\end{corollary}
The proof is trivial that $(\bI - 2\bu\bu^\top)\bv = \bv - 2\bu\bu^\top\bv=\bv$.

Suppose $\bu$ is a unit vector with $||\bu||=1$, and a vector $\bv$ is perpendicular to $\bu$. Then any vector $\bx$ on the plane can be decomposed into two parts, i.e., $\bx = \bx_{\bv} + \bx_{\bu}$: the first one $\bx_{\bu}$ is parallel to $\bu$ and the second one $\bx_{\bv}$ is perpendicular to $\bu$ (i.e., parallel to $\bv$). From Section~\ref{section:project-onto-a-vector} (p.~\pageref{section:project-onto-a-vector}) on the projection of a vector onto another one, $\bx_{\bu}$ can be computed by $\bx_{\bu} = \frac{\bu\bu^\top}{\bu^\top\bu} \bx = \bu\bu^\top\bx$. We then transform this $\bx$ by the Householder reflector associated with $\bu$, $\bH\bx = (\bI - 2\bu\bu^\top)(\bx_{\bv} + \bx_{\bu}) = \bx_{\bv} -\bu\bu^\top \bx = \bx_{\bv} - \bx_{\bu}$. That is, the space perpendicular to $\bu$ acts as a mirror and any vector $\bx$ is reflected by the Householder reflector associated with $\bu$ (i.e., reflected in the hyperplane $span\{\bu\}^\perp$). The situation is shown in Figure~\ref{fig:householder}.

\begin{SCfigure
	\centering
	\includegraphics[width=0.5\textwidth]{imgs/householder.pdf}
	\caption{Demonstration of the Householder reflector. The Householder reflector obtained by $\bH=\bI-2\bu\bu^\top$ where $||\bu||=1$ will reflect vector $\bx$ along the plane perpendicular to $\bu$: $\bx=\bx_{\bv} + \bx_{\bu} \rightarrow \bx_{\bv} - \bx_{\bu}$.}
	\label{fig:householder}
\end{SCfigure}

If we know two vectors are reflected to each other, the next corollary tells us how to find the corresponding Householder reflector.
\begin{corollary}[Finding the Householder Reflector]\label{corollary:householder-reflect-finding}
Suppose $\bx$ is reflected to $\by$ by a Householder reflector with $||\bx|| = ||\by||$, then the Householder reflector is obtained by
$$
\bH = \bI - 2 \bu\bu^\top, \text{ where } \bu = \frac{\bx-\by}{||\bx-\by||}.
$$
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:householder-reflect-finding}]
Write out the equation, we have
$$
\begin{aligned}
\bH\bx &= \bx - 2 \bu\bu^\top\bx =\bx - 2\frac{(\bx-\by)(\bx^\top-\by^\top)}{(\bx-\by)^\top(\bx-\by)} \bx\\
&= \bx - (\bx-\by) = \by.
\end{aligned}
$$
Note that the condition $||\bx|| = ||\by||$ is required to prove the result.
\end{proof}
The Householder reflectors are useful to set a block of components of a given vector to zero. Particularly, we usually would like to set the vector $\ba\in \real^n$ to be zero except the $i$-th element. Then the Householder vector can be chosen to be
$$
\bu = \frac{\ba - r\be_i}{||\ba - r\be_i||}, \qquad \text{where } r = \pm||\ba||
$$
which is a reasonable Householder vector since $||\ba|| = ||r\be_i|| = |r|$. We carefully notice that when $r=||\ba||$, $\ba$ is reflected to $||\ba||\be_i$ via the Householder reflector $\bH = \bI - 2 \bu\bu^\top$; otherwise when $r=-||\ba||$, $||\ba||$ is reflected to $-||\ba||\be_i$ via the Householder reflector.
\begin{remark}[Householder Properties]
If $\bH$ is a Householder reflector, then it has the following properties:

$\bullet$ $\bH\bH = \bI$;

$\bullet$ $\bH = \bH^\top$;

$\bullet$ $\bH^\top\bH = \bH\bH^\top = \bI$ such that Householder reflector is an orthogonal matrix;

$\bullet$ $\bH\bu = -\bu$, if $\bH = \bI - 2 \bu\bu^\top$.
\end{remark}

We note in the Gram-Schmidt section that QR decomposition is to use a triangular matrix to orthogonalize a matrix $\bA$. The further idea is that, if we have a set of orthogonal matrices that can make $\bA$ to be triangular step by step, then we can also recover the QR decomposition.
Specifically, if we have an orthogonal matrix $\bQ_1$ that can introduce zeros to the $1$-st column of $\bA$ except the entry (1,1); and an orthogonal matrix $\bQ_2$ that can introduce zeros to the $2$-nd column except the entries (2,1), (2,2); $\ldots$. Then, we can also find the QR decomposition.
For the way to introduce zeros, we could reflect the columns of the matrix to a basis vector $\be_1$ whose entries are all zero except the first entry.

Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n]\in \real^{m\times n}$ be the column partition of $\bA$, and let
\begin{equation}\label{equation:qr-householder-to-chooose-r-numeraically}
r_1 = ||\ba_1||,\qquad  \bu_1 = \frac{\ba_1 - r_1 \be_1}{||\ba_1 - r_1 \be_1||}, \qquad \text{and}\qquad \bH_1 = \bI - 2\bu_1\bu_1^\top,
\end{equation}
where $\be_1$ here is the first basis for $\real^m$, i.e., $\be_1 = [1;0;0;\ldots;0]\in \real^m$.
Then
\begin{equation}\label{equation:householder-qr-projection-step1}
\bH_1\bA = [\bH_1\ba_1, \bH_1\ba_2, \ldots, \bH_1\ba_n] =
\begin{bmatrix}
r_1 & \bR_{1,2:n} \\
\bzero&  \bB_2
\end{bmatrix},
\end{equation}
which reflects $\ba_1$ to $r_1\be_1$ and introduces zeros below the diagonal in the $1$-st column. We observe that the entries below $r_1$ are all zero now under this specific reflection. Notice that we reflect $\ba_1$ to $||\ba_1||\be_1$ the two of which have the same length, rather than reflect $\ba_1$ to $\be_1$ directly. This is for the purpose of \textbf{numerical stability}.

\textbf{Choice of $r_1$:} moreover, the choice of $r_1$ is \textbf{not unique}. For \textbf{numerical stability}, it is also desirable to choose $r_1 =-\text{sign}(a_{11}) ||\ba_1||$, where $a_{11}$ is the first component of $\ba_{1}$. Or even, $r_1 =\text{sign}(a_{11}) ||\ba_1||$ is also possible as long as $||\ba_1||$ is equal to $||r_1\be_1||$. However, we will not cover this topic here.

We can then apply this process to $\bB_2$ in Equation~\eqref{equation:householder-qr-projection-step1} to make the entries  below the entry (2,2) to be all zeros. Note that, we do not apply this process to the entire $\bH_1\bA$ but rather the submatrix $\bB_2$ in it because we have already introduced zeros in the first column, and reflecting again will introduce nonzero values back.

Suppose $\bB_2 = [\bb_2, \bb_3, \ldots, \bb_n]$ is the column partition of $\bB_2$, and let
$$
r_2 = ||\bb_2||,\qquad \bu_2 = \frac{\bb_2 - r_2 \be_1}{||\bb_2 - r_2 \be_1||}, \qquad  \qquad \widetilde{\bH}_2 = \bI - 2\bu_2\bu_2^\top, \qquad \text{and}\qquad  \bH_2 =
 \begin{bmatrix}
	1 & \bzero \\
	\bzero & \widetilde{\bH}_2
\end{bmatrix},
$$
where now $\be_1$ here is the first basis for $\real^{m-1}$ and $\bH_2$ is also an orthogonal matrix since $\widetilde{\bH}_2$ is an orthogonal matrix. Then it follows that
$$
\bH_2\bH_1\bA = [\bH_2\bH_1\ba_1, \bH_2\bH_1\ba_2, \ldots, \bH_2\bH_1\ba_n] =
\begin{bmatrix}
	r_1 & \bR_{12} & \bR_{1,3:n} \\
	0 & r_2 & \bR_{2,3:n} \\
	\bzero &  \bzero &\bC_3
\end{bmatrix}.
$$

The same process can go on, and we will finally triangularize $\bA = (\bH_n \bH_{n-1}\ldots\bH_1)^{-1} \bR = \bQ\bR$. And since the $\bH_i$'s are symmetric and orthogonal, we have $\bQ=(\bH_n \bH_{n-1}\ldots\bH_1)^{-1} = \bH_1 \bH_2\ldots\bH_n$.

An example of a $5\times 4$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
$$
\begin{aligned}
\begin{sbmatrix}{\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
&\stackrel{\bH_1}{\rightarrow}
\begin{sbmatrix}{\bH_1\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bH_2}{\rightarrow}
\begin{sbmatrix}{\bH_2\bH_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
\end{sbmatrix}\\
&\stackrel{\bH_3}{\rightarrow}
\begin{sbmatrix}{\bH_3\bH_2\bH_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & 0 & \bm{0} & \bm{\boxtimes} \\
	0 & 0 & \bm{0} & \bm{\boxtimes}
\end{sbmatrix}
\stackrel{\bH_4}{\rightarrow}
\begin{sbmatrix}{\bH_4\bH_3\bH_2\bH_1\bA}
	\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
	0 & 0 & \boxtimes & \boxtimes \\
	0 & 0 & 0 & \bm{\boxtimes} \\
	0 & 0 & 0 & \bm{0}
\end{sbmatrix}
\end{aligned}
$$

\paragraph{A closer look at the QR factorization} The Householder algorithm is a process that makes a matrix triangular by a sequence of orthogonal matrix operations. In the Gram-Schmidt process (both CGS and MGS), we use a triangular matrix to orthogonalize the matrix. However, in the Householder algorithm, we use orthogonal matrices to triangularize.  The difference between the two approaches is summarized as follows:
\begin{itemize}
	\item Gram-Schmidt: triangular orthogonalization;
	\item Householder: orthogonal triangularization.
\end{itemize}

We further notice that, in the Householder algorithm or the Givens algorithm that we will shortly see, a set of orthogonal matrices are applied so that the QR decomposition obtained is a \textit{full} QR decomposition. Whereas, the direct QR decomposition obtained by CGS or MGS is a \textit{reduced} one (although the silent columns or rows can be further added to find the full one).

\index{Givens rotation}
\section{Existence of the QR Decomposition via the Givens Rotation}\label{section:qr-givens}
We have defined the Givens rotation in Definition~\ref{definition:givens-rotation-in-qr} (p.~\pageref{definition:givens-rotation-in-qr}) to find the rank-one update/downdate of the Cholesky decomposition. Now consider the following $2\times 2$ orthogonal matrices
$$
\bF =
\begin{bmatrix}
-c & s\\
s & c
\end{bmatrix},
\qquad
\bJ=
\begin{bmatrix}
c & -s \\
s & c
\end{bmatrix},
\qquad
\bG=
\begin{bmatrix}
	c & s \\
	-s & c
\end{bmatrix},
$$
where $s = \sin \theta$ and $c=\cos \theta$ for some $\theta$. The first matrix has $\det(\bF)=-1$ and is a special case of a Householder reflector in dimension 2 such that $\bF=\bI-2\bu\bu^\top$ where $\bu=\begin{bmatrix}
	\sqrt{\frac{1+c}{2}}, &\sqrt{\frac{1-c}{2}}
\end{bmatrix}^\top$ or $\bu=\begin{bmatrix}
-\sqrt{\frac{1+c}{2}}, &-\sqrt{\frac{1-c}{2}}
\end{bmatrix}^\top$. The latter two matrices have $\det(\bJ)=\det(\bG)=1$ and effects a rotation instead of a reflection. Such a matrix is called a \textbf{\textit{Givens rotation}}.

\begin{figure}[H]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[$\by = \bJ\bx$, counter-clockwise rotation.]{\label{fig:rotation1}
		\includegraphics[width=0.47\linewidth]{imgs/rotation.pdf}}
	\quad
	\subfigure[$\by = \bG\bx$, clockwise rotation.]{\label{fig:rotation2}
		\includegraphics[width=0.47\linewidth]{imgs/rotation2.pdf}}
	\caption{Demonstration of two Givens rotations.}
	\label{fig:rotation}
\end{figure}

Figure~\ref{fig:rotation} demonstrate a rotation of $\bx$ under $\bJ$, where $\by = \bJ\bx$ such that
$$
\left\{
\begin{aligned}
	&y_1 = c\cdot x_1 - s\cdot x_2,   \\
	&y_2 = s \cdot x_1 + c\cdot x_2.
\end{aligned}
\right.
$$
We want to verify the angle between $\bx$ and $\by $ is actually $\theta$ (and counter-clockwise rotation) after the Givens rotation $\bJ$ as shown in Figure~\ref{fig:rotation1}. Firstly, we have
$$
\left\{
\begin{aligned}
	&\cos(\alpha) =\frac{x_1}{\sqrt{x_1^2+x_2^2}},   \\
	&\sin (\alpha) =\frac{x_2}{\sqrt{x_1^2+x_2^2}}.
\end{aligned}
\right.
\qquad
\text{and }\qquad
\left\{
\begin{aligned}
	&\cos(\theta) =c,   \\
	&\sin (\theta) =s.
\end{aligned}
\right.
$$
This implies $\cos(\theta+\alpha) = \cos(\theta)\cos(\alpha)-\sin(\theta)\sin(\alpha)$.
If we can show $\cos(\theta+\alpha) = \cos(\theta)\cos(\alpha)-\sin(\theta)\sin(\alpha)$ is equal to $\frac{y_1}{\sqrt{y_1^2+y_2^2}}$, then we complete the proof.

For the former one, $\cos(\theta+\alpha) = \cos(\theta)\cos(\alpha)-\sin(\theta)\sin(\alpha)=\frac{c\cdot x_1 - s\cdot x_2}{\sqrt{x_1^2+x_2^2}}$. For the latter one, it can be verified that $\sqrt{y_1^2+y_2^2}=\sqrt{x_1^2+x_2^2}$, and $\frac{y_1}{\sqrt{y_1^2+y_2^2}} = \frac{c\cdot x_1 - s\cdot x_2}{\sqrt{x_1^2+x_2^2}}$. This completes the proof. Similarly, we can also show that the angle between $\by=\bG\bx$ and $\bx$ is also $\theta$ in Figure~\ref{fig:rotation2} and the rotation is clockwise.

It can be easily verified that the $n$-th order Givens rotation (Definition~\ref{definition:givens-rotation-in-qr}, p.~\pageref{definition:givens-rotation-in-qr}) is an orthogonal matrix and its determinant is 1. For any vector $\bx =[x_1, x_2, \ldots, x_n]^\top \in \real^n$, we have $\by = \bG_{kl}\bx$, where
$$
\left\{
\begin{aligned}
	&y_k = c \cdot x_k + s\cdot x_l,   \\
	&y_l = -s\cdot x_k +c\cdot x_l,  \\
	&y_j = x_j , &  (j\neq k,l)
\end{aligned}
\right.
$$
That is, a Givens rotation applied to $\bx$ rotates two components of $\bx$ by some angle $\theta$ and leaves all other components the same.
When $\sqrt{x_k^2 + x_l^2} \neq 0$,
let $c = \frac{x_k}{\sqrt{x_k^2 + x_l^2}}$, $s=\frac{x_l}{\sqrt{x_k^2 + x_l^2}}$. Then,
$$
\left\{
\begin{aligned}
	&y_k = \sqrt{x_k^2 + x_l^2},   \\
	&y_l = 0,  \\
	&y_j = x_j . &  (j\neq k,l)
\end{aligned}
\right.
$$
This finding above is essential for the QR decomposition via the Givens rotation.

\begin{corollary}[Basis From Givens Rotations Forwards]\label{corollary:basis-from-givens}
For any vector $\bx \in \real^n$, there exists a set of Givens rotations $\{\bG_{12}, \bG_{13}, \ldots, \bG_{1n}\}$ such that $\bG_{1n}\ldots \bG_{13}\bG_{12}\bx = ||\bx||\be_1$ where $\be_1\in \real^n$ is the first unit basis in $\real^n$.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:basis-from-givens}]
From the finding above, we can find a set of Givens rotations $\bG_{12}, \bG_{13}, \bG_{14}$ such that
$$
\bG_{12}\bx = \left[\sqrt{x_1^2 + x_2^2}, 0, x_3, \ldots, x_n \right]^\top,
$$
$$
\bG_{13}\bG_{12}\bx = \left[\sqrt{x_1^2 + x_2^2+x_3^2}, 0, 0, x_4, \ldots, x_n \right]^\top,
$$
and
$$
\bG_{14}\bG_{13}\bG_{12}\bx = \left[\sqrt{x_1^2 + x_2^2+x_3^2+x_4^2},0, 0, 0, x_5, \ldots, x_n \right]^\top.
$$
Continue this process, we will obtain $\bG_{1n}\ldots \bG_{13}\bG_{12} = ||\bx||\be_1$.
\end{proof}

\begin{remark}[Basis From Givens Rotations Backwards]\label{remark:basis-from-givens2}
In Corollary~\ref{corollary:basis-from-givens}, we find the Givens rotation that introduces zeros from the $2$-nd entry to th $n$-th entry (i.e., forward). Sometimes we want the reverse order, i.e., introduce zeros from the $n$-th entry to the $2$-nd entrysuch that $\bG_{12}\bG_{13}\ldots  \bG_{1n}\bx = ||\bx||\be_1$ where $\be_1\in \real^n$ is the first unit basis in $\real^n$.

The procedure is similar, we can find a $\bG_{1n},\bG_{1,(n-1)}, \bG_{1,(n-2)}$ such that
$$
\bG_{1n}\bx = \left[\sqrt{x_1^2 + x_n^2}, x_2, x_3, \ldots, x_{n-1}, 0 \right]^\top,
$$
$$
\bG_{1,(n-1)}\bG_{1n}\bx = \left[\sqrt{x_1^2 + x_{n-1}^2+x_n^2}, x_2, x_3, \ldots, x_{n-2}, 0, 0  \right]^\top,
$$
and
$$
\bG_{1,(n-2)}\bG_{1,(n-1)}\bG_{1n}\bx = \left[\sqrt{x_1^2 + x_{n-2}^2+x_{n-1}^2+x_n^2}, x_2, x_3,  \ldots,x_{n-3},0, 0, 0  \right]^\top.
$$
Continue this process, we will obtain $\bG_{12}\bG_{13}\ldots  \bG_{1n}\bx = ||\bx||\be_1$.

\paragraph{An alternative form} Alternatively, there are rotations $\{\bG_{12}, \bG_{23}, \ldots,\bG_{(n-1),n}\}$ such that $\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}\bx = ||\bx||\be_1$ where $\be_1\in \real^n$ is the first unit basis in $\real^n$ with
$$
\bG_{(n-1),n}\bx = \left[x_1, x_2, \ldots, x_{n-2},\sqrt{x_{n-1}^2 + x_n^2}, 0 \right]^\top,
$$
$$
\bG_{(n-2),(n-1)}\bG_{(n-1),n}\bx = \left[x_1, x_2, \ldots,x_{n-3}, \sqrt{x_{n-2}^2+x_{n-1}^2 + x_n^2}, 0, 0  \right]^\top,
$$
and
$$
\begin{aligned}
\bG_{(n-3),(n-2)}\bG_{(n-2),(n-1)}\bG_{(n-1),n}\bx &= \\
	&\,\left[x_1,  x_2, \ldots ,x_{n-4}, \sqrt{x_{n-3}^2+x_{n-2}^2+x_{n-1}^2 + x_n^2},0, 0, 0  \right]^\top.
\end{aligned}
$$
Continue this process, we will obtain $\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}\bx = ||\bx||\be_1$.
\end{remark}

From  Corollary~\ref{corollary:basis-from-givens} above, for the way to introduce zeros, we could \textbf{rotate} the columns of the matrix to a basis vector $\be_1$ whose entries are all zero except the first entry.
Let $\bA=[\ba_1, \ba_2, \ldots, \ba_n] \in \real^{m\times n}$ be the column partition of $\bA$, and let
\begin{equation}\label{equation:qr-rotate-to-chooose-r-numeraically}
 \bG_1 = \bG_{1m}\ldots \bG_{13}\bG_{12},
\end{equation}
where $\be_1$ here is the first basis for $\real^m$, i.e., $\be_1 = [1;0;0;\ldots;0]\in \real^m$.
Then
\begin{equation}\label{equation:rotate-qr-projection-step1}
\begin{aligned}
	\bG_1\bA &= [\bG_1\ba_1, \bG_1\ba_2, \ldots, \bG_1\ba_n] =
\begin{bmatrix}
	||\ba_1|| & \bR_{1,2:n} \\
	\bzero&  \bB_2
\end{bmatrix},
\end{aligned}
\end{equation}
which rotates $\ba_1$ to $||\ba_1||\be_1$ and introduces zeros below the diagonal in the $1$-st column.

We can then apply this process to $\bB_2$ in Equation~\eqref{equation:rotate-qr-projection-step1} to make the entries  below the (2,2)-th entry to be all zeros.
Suppose $\bB_2 = [\bb_2, \bb_3, \ldots, \bb_n]$, and let
$$
\bG_2 = \bG_{2m}\ldots\bG_{24}\bG_{23},
$$
where $\bG_{2n}, \ldots, \bG_{24}, \bG_{23}$ can be implied from context.
Then
$$
\begin{aligned}
\bG_2\bG_1\bA &= [\bG_2\bG_1\ba_1, \bG_2\bG_1\ba_2, \ldots, \bG_2\bG_1\ba_n] =
\begin{bmatrix}
	||\ba_1|| & \bR_{12} & \bR_{1,3:n} \\
	0 & ||\bb_2|| & \bR_{2,3:n} \\
	\bzero &  \bzero &\bC_3
\end{bmatrix}.
\end{aligned}
$$

The same process can go on, and we will finally triangularize $\bA = (\bG_n \bG_{n-1}\ldots\bG_1)^{-1} \bR = \bQ\bR$. And since $\bG_i$'s are orthogonal, we have $\bQ=(\bG_n \bG_{n-1}\ldots\bG_1)^{-1} = \bG_1^\top \bG_2^\top\ldots\bG_n^\top $, and
\begin{equation}\label{equation:givens-q}
\begin{aligned}
	\bG_1^\top \bG_2^\top\ldots\bG_n^\top &=(\bG_n \ldots \bG_2 \bG_1)^\top \\
	&=\left\{(\bG_{nm} \ldots \bG_{n,(n+1)}) \ldots (\bG_{2m}\ldots \bG_{23})  ( \bG_{1m} \ldots \bG_{12} )\right\}^\top .
\end{aligned}
\end{equation}

The Givens rotation algorithm works better when $\bA$ already has a lot of zeros below the main diagonal. An example of a $5\times 4$ matrix is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
\paragraph{Givens rotations in $\bG_1$} For a $5\times 4$ example, we realize that $\bG_1 = \bG_{15}\bG_{14}\bG_{13}\bG_{12}$. And the process is shown as follows:
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_{12}}{\rightarrow}
	\begin{sbmatrix}{\bG_{12}\bA}
\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	\stackrel{\bG_{13}}{\rightarrow}
	\begin{sbmatrix}{\bG_{13}\bG_{12}\bA}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \boxtimes & \boxtimes &\boxtimes \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
\boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}\\
	&\stackrel{\bG_{14}}{\rightarrow}
\begin{sbmatrix}{\bG_{14}\bG_{13}\bG_{12}\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes &\boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	\boxtimes & \boxtimes & \boxtimes & \boxtimes
\end{sbmatrix}
	\stackrel{\bG_{15}}{\rightarrow}
\begin{sbmatrix}{\bG_{15}\bG_{14}\bG_{13}\bG_{12}\bA}
	\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
	0 & \boxtimes & \boxtimes &\boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
	0 & \boxtimes & \boxtimes & \boxtimes \\
\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
\end{sbmatrix}
\end{aligned}
$$

\paragraph{Givens rotation as a big picture} Take $\bG_1, \bG_2, \bG_3, \bG_4$ as a single matrix, we have
$$
\begin{aligned}
	\begin{sbmatrix}{\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		\boxtimes & \boxtimes & \boxtimes & \boxtimes
	\end{sbmatrix}
	&\stackrel{\bG_1}{\rightarrow}
	\begin{sbmatrix}{\bG_1\bA}
		\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\bG_2}{\rightarrow}
	\begin{sbmatrix}{\bG_2\bG_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes}
	\end{sbmatrix}\\
	&\stackrel{\bG_3}{\rightarrow}
	\begin{sbmatrix}{\bG_3\bG_2\bG_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} \\
		0 & 0 & \bm{0} & \bm{\boxtimes} \\
		0 & 0 & \bm{0} & \bm{\boxtimes}
	\end{sbmatrix}
	\stackrel{\bG_4}{\rightarrow}
	\begin{sbmatrix}{\bG_4\bG_3\bG_2\bG_1\bA}
		\boxtimes & \boxtimes & \boxtimes & \boxtimes \\
		0 & \boxtimes & \boxtimes & \boxtimes \\
		0 & 0 & \boxtimes & \boxtimes \\
		0 & 0 & 0 & \bm{\boxtimes} \\
		0 & 0 & 0 & \bm{0}
	\end{sbmatrix}
\end{aligned}
$$

\paragraph{Orders to introduce the zeros} With the Givens rotations for the QR decomposition, it is flexible to choose different orders to introduce the zeros of $\bR$. In our case, we introduce zeros column by column. It is also possible to introduce zeros row by row.

\index{Uniqueness}
\section{Uniqueness of the QR Decomposition}\label{section:nonunique-qr}
The results of the QR decomposition from the Gram-Schmidt process, the Householder algorithm, and the Givens algorithms are different. Even in the Householder algorithm, we have different methods to choose the sign of $r_1$ in Equation~\eqref{equation:qr-householder-to-chooose-r-numeraically}. Thus, from this point, QR decomposition is not unique.

However, if we use just the procedure described in the Gram-Schmidt process, or systematically choose the sign in the Householder algorithm, then the decomposition is unique.
The uniqueness of the \textit{reduced} QR decomposition for full column rank matrix $\bA$ is assured when $\bR$ has positive diagonals by inductive analysis \citep{lu2021numerical}.
We here provide another proof for the uniqueness of the \textit{reduced} QR decomposition for matrices if the diagonal values of $\bR$ are positive which will shed light on the implicit Q theorem in Hessenberg decomposition (Section~\ref{section:hessenberg-decomposition}, p.~\pageref{section:hessenberg-decomposition}) or tridiagonal decomposition (Theorem~\ref{section:tridiagonal-decomposition}, p.~\pageref{section:tridiagonal-decomposition}).
	\begin{corollary}[Uniqueness of the reduced QR Decomposition]\label{corollary:unique-qr}
		Suppose matrix $\bA$ is an $m\times n$ matrix with full column rank $n$ and $m\geq n$. Then, the \textit{reduced} QR decomposition is unique if the main diagonal values of $\bR$ are positive.
	\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:unique-qr}]
Suppose the \textit{reduced} QR decomposition is not unique, we can complete it into a \textit{full} QR decomposition, then we can find two such full decompositions so that $\bA=\bQ_1\bR_1 = \bQ_2\bR_2$ which implies $\bR_1 = \bQ_1^{-1}\bQ_2\bR_2 = \bV \bR_2$ where $\bV= \bQ_1^{-1}\bQ_2$ is an orthogonal matrix. Write out the equation, we have
$$
\begin{aligned}
\bR_1 &=
\begin{bmatrix}
	r_{11} & r_{12}& \dots & r_{1n}\\
	& r_{22}& \dots & r_{2n}\\
	&       &    \ddots  & \vdots \\
	\multicolumn{2}{c}{\raisebox{1.3ex}[0pt]{\Huge0}} & & r_{nn} \\
	\bzero & \bzero &\ldots & \bzero
\end{bmatrix}=
\begin{bmatrix}
	v_{11}& v_{12} & \ldots & v_{1m}\\
	v_{21} & v_{22} & \ldots & v_{2m}\\
	\vdots & \vdots & \ddots & \vdots\\
	v_{m1} & v_{m2} & \ldots & v_{\textcolor{black}{mm}}
\end{bmatrix}
\begin{bmatrix}
	s_{11} & s_{12}& \dots & s_{1n}\\
	& s_{22}& \dots & s_{2n}\\
	&       &    \ddots  & \vdots \\
	\multicolumn{2}{c}{\raisebox{1.3ex}[0pt]{\Huge0}} & & s_{nn} \\
		\bzero & \bzero &\ldots & \bzero
\end{bmatrix}= \bV\bR_2,
\end{aligned}
$$
This implies
$$
r_{11} = v_{11} s_{11}, \qquad v_{21}=v_{31}=v_{41}=\ldots=v_{m1}=0.
$$
Since $\bV$ contains mutually orthonormal columns and the first column of $\bV$ is of norm 1. Thus, $v_{11} = \pm 1$.
We notice that $r_{ii}> 0$ and $s_{ii}> 0$ for $i\in \{1,2,\ldots,n\}$ by assumption such that $r_{11}> 0$ and $s_{11}> 0$ and $v_{11}$ can only be positive 1. Since $\bV$ is an orthogonal matrix, we also have
$$
v_{12}=v_{13}=v_{14}=\ldots=v_{1m}=0.
$$
Applying this process to the submatrices of $\bR_1, \bV, \bR_2$, we will find the upper-left submatrix of $\bV$ is an identity: $\bV[1:n,1:n]=\bI_n$ such that $\bR_1=\bR_2$. This implies $\bQ_1[:,1:n]=\bQ_2[:,1:n]$ and leads to a contradiction such that the reduced QR decomposition is unique.
\end{proof}

\index{Row space}
\section{LQ Decomposition}
We previously proved the existence of the QR decomposition via the Gram-Schmidt process in which case we are interested in the column space of a matrix $\bA=[\ba_1, \ba_2, ..., \ba_n] \in \real^{m\times n}$.
However, in many applications (see \citet{schilders2009solution}), we are also interested in the row space of a matrix
$\bB=[\bb_1^\top; \bb_2^\top; ...;\bb_m^\top] \in \real^{m\times n}$, where $\bb_i$ is the $i$-th row of $\bB$. The successive spaces spanned by the rows $\bb_1, \bb_2, \ldots$ of $\bB$ are
$$
\cspace([\bb_1])\,\,\,\, \subseteq\,\,\,\, \cspace([\bb_1, \bb_2]) \,\,\,\,\subseteq\,\,\,\, \cspace([\bb_1, \bb_2, \bb_3])\,\,\,\, \subseteq\,\,\,\, \ldots.
$$
The QR decomposition thus has its sibling which finds the orthogonal row space.
By applying QR decomposition on $\bB^\top = \bQ_0\bR$, we recover the LQ decomposition of the matrix $\bB = \bL \bQ$ where $\bQ = \bQ_0^\top$ and $\bL = \bR^\top$.

	\begin{theoremHigh}[LQ Decomposition]\label{theorem:lq-decomposition}
		Every $m\times n$ matrix $\bB$ (whether linearly independent or dependent rows) with $n\geq m$ can be factored as
		$$
		\bB = \bL\bQ,
		$$
		where

		1. \textbf{Reduced}: $\bL$ is an $m\times m$ lower triangular matrix and $\bQ$ is $m\times n$ with orthonormal rows which is known as the \textbf{reduced LQ decomposition};

		2. \textbf{Full}: $\bL$ is an $m\times n$ lower triangular matrix and $\bQ$ is $n\times n$ with orthonormal rows which is known as the \textbf{full LQ decomposition}. If further restrict the lower triangular matrix to be a square matrix, the full LQ decomposition can be denoted as
		$$
		\bB = \begin{bmatrix}
			\bL_0 & \bzero
		\end{bmatrix}\bQ,
		$$
		where $\bL_0$ is an $m\times m$ square lower triangular matrix.
	\end{theoremHigh}

\paragraph{Row-pivoted LQ (RPLQ)\index{Row-pivoted}\index{RPLQ}} Similar to the column-pivoted QR in Section~\ref{section:cpqr} (p.~\pageref{section:cpqr}), there exists a row-pivoted LQ decomposition:
$$
\left\{
\begin{aligned}
	\text{Reduced RPLQ: }&\qquad
	\bP\bB &=&
	\underbrace{\begin{bmatrix}
			\bL_{11} \\
			\bL_{21}
	\end{bmatrix}}_{m\times r}
	\underbrace{\bQ_r }_{r\times n};\\
	\text{Full RPLQ: }&\qquad
	\bP\bB &=&
	\underbrace{\begin{bmatrix}
			\bL_{11} & \bzero \\
			\bL_{21} & \bzero
	\end{bmatrix}}_{m\times m}
	\underbrace{\bQ }_{m\times n},\\
\end{aligned}
\right.
$$
where $\bL_{11}\in \real^{r\times r}$ is lower triangular, $\bQ_r$ or $\bQ_{1:r,:}$ spans the same row space as $\bB$, and $\bP$ is a permutation matrix that interchanges independent rows into the upper-most rows.

\section{Two-Sided Orthogonal Decomposition}
\begin{theoremHigh}[Two-Sided Orthogonal Decomposition]\label{theorem:two-sided-orthogonal}
When square matrix $\bA\in \real^{n\times n}$ with rank $r$, the full CPQR, RPLQ of $\bA$ are given by
$$\bA\bP_1=\bQ_1
\begin{bmatrix}
	\bR_{11} & \bR_{12}\\
	\bzero & \bzero
\end{bmatrix},\qquad
\bP_2\bA=
\begin{bmatrix}
	\bL_{11} & \bzero \\
	\bL_{21} & \bzero
\end{bmatrix}
\bQ_2
$$
respectively. Then we would find out
$$
\bA\bP\bA = \bQ_1
\underbrace{\begin{bmatrix}
		\bR_{11}\bL_{11}+\bR_{12}\bL_{21} & \bzero \\
		\bzero & \bzero
\end{bmatrix}}_{\text{rank $r$}}
\bQ_2,
$$
where the first $r$ columns of $\bQ_1$ span the same column space of $\bA$, first $r$ rows of $\bQ_2$ span the same row space of $\bA$, and $\bP$ is a permutation matrix. We name this decomposition as \textbf{two-sided orthogonal decomposition}.
\end{theoremHigh}

This decomposition is very similar to the property of SVD: $\bA=\bU\bSigma\bV^\top$ that the first $r$ columns of $\bU$ span the column space of $\bA$ and the first $r$ columns of $\bV$ span the row space of $\bA$ (we shall see in Lemma~\ref{lemma:svd-four-orthonormal-Basis}, p.~\pageref{lemma:svd-four-orthonormal-Basis}). Therefore, the two-sided orthogonal decomposition can be regarded as an inexpensive alternative in this sense.

\index{Fundamental spaces}
\index{Orthonormal basis}
\begin{lemma}[Four Orthonormal Basis]
	Given the two-sided orthogonal decomposition of matrix $\bA\in \real^{n\times n}$ with rank $r$: $\bA\bP\bA = \bU \bF\bV^\top$, where $\bU=[\bu_1, \bu_2, \ldots,\bu_n]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU$ and $\bV$. Then, we have the following property:

	$\bullet$ $\{\bv_1, \bv_2, \ldots, \bv_r\} $ is an orthonormal basis of $\cspace(\bA^\top)$;

	$\bullet$ $\{\bv_{r+1},\bv_{r+2}, \ldots, \bv_n\}$ is an orthonormal basis of $\nspace(\bA)$;

	$\bullet$ $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is an orthonormal basis of $\cspace(\bA)$;

	$\bullet$ $\{\bu_{r+1}, \bu_{r+2},\ldots,\bu_n\}$ is an orthonormal basis of $\nspace(\bA^\top)$.
\end{lemma}

	\section{Rank-One Changes}
	We previously discussed the rank-one update/downdate of the Cholesky decomposition in Section~\ref{section:cholesky-rank-one-update} (p.~\pageref{section:cholesky-rank-one-update}). The rank-one change $\bA^\prime$ of matrix $\bA$ in the QR decomposition is defined in a similar form:\index{Rank-one update}
	$$
	\begin{aligned}
		\bA^\prime &= \bA + \bu\bv^\top, \\
		\downarrow &\gap  \downarrow\\
		\bQ^\prime\bR^\prime &=\bQ\bR + \bu\bv^\top,
	\end{aligned}
	$$
	where if we set $\bA^\prime = \bA - (-\bu)\bv^\top$, we recover the downdate form such that the update and downdate in the QR decomposition are the same. Let $\bw = \bQ^\top\bu$, we have
	$$
	\bA^\prime = \bQ(\bR + \bw\bv^\top).
	$$
	From the second form in Remark~\ref{remark:basis-from-givens2} (p.~\pageref{remark:basis-from-givens2}) on introducing zeros backwards, there exists a set of Givens rotations $\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}$ such that
	$$
	\bG_{12} \bG_{23} \ldots \bG_{(n-1),n} \bw = \pm ||\bw|| \be_1,
	$$
	where $\bG_{(k-1),k}$ is the Givens rotation in plane $k-1$ and $k$ that introduces zero in the $k$-th entry of $\bw$. Apply this rotation to $\bR$, we have
	$$
	\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}\bR = \bH_0 ,
	$$
	where the Givens rotations in this \textit{reverse order} are useful to transform the upper triangular $\bR$ into a ``simple" upper Hessenberg which is close to upper triangular matrices (see Definition~\ref{definition:upper-hessenbert}, p.~\pageref{definition:upper-hessenbert} that we will introduce in the Hessenberg decomposition). If the rotations are transforming $\bw$ into $\pm ||\bw||\be_1$ from \textit{forward order} as in Corollary~\ref{corollary:basis-from-givens} (p.~\pageref{corollary:basis-from-givens}), we will not have this upper Hessenberg $\bH_0$. To see this, suppose $\bR\in \real^{5\times 5}$, an example is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed. The backward rotations result in the upper Hessenberg $\bH_0$ which is relatively simple to handle:
	$$
	\begin{aligned}
		\text{Backward: }
		\begin{sbmatrix}{\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes \\
			0 & 0 & 0 & 0& \boxtimes
		\end{sbmatrix}
		&\stackrel{\bG_{45}}{\rightarrow}
		\begin{sbmatrix}{\bG_{45}\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & 0 & \bm{\boxtimes}& \bm{\boxtimes}
		\end{sbmatrix}
		\stackrel{\bG_{34}}{\rightarrow}
		\begin{sbmatrix}{\bG_{34}\bG_{45}\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & 0 & \boxtimes & \boxtimes
		\end{sbmatrix}\\
		&\stackrel{\bG_{23}}{\rightarrow}
		\begin{sbmatrix}{\bG_{23}\bG_{34}\bG_{45}\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes
		\end{sbmatrix}
		\stackrel{\bG_{12}}{\rightarrow}
		\begin{sbmatrix}{\bG_{12}\bG_{23}\bG_{34}\bG_{45}\bR}
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes
		\end{sbmatrix}.
	\end{aligned}
	$$
	And the forward rotations result in a \textbf{full matrix}:
	$$
	\begin{aligned}
		\text{Forward: }
		\begin{sbmatrix}{\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes \\
			0 & 0 & 0 & 0& \boxtimes
		\end{sbmatrix}
		&\stackrel{\bG_{12}}{\rightarrow}
		\begin{sbmatrix}{\bG_{12}\bR}
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes \\
			0 & 0 & 0 & 0& \boxtimes
		\end{sbmatrix}
		\stackrel{\bG_{23}}{\rightarrow}
		\begin{sbmatrix}{\bG_{23}\bG_{12}\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & 0 & \boxtimes& \boxtimes \\
			0 & 0 & 0 & 0& \boxtimes
		\end{sbmatrix}\\
		&\stackrel{\bG_{34}}{\rightarrow}
		\begin{sbmatrix}{\bG_{34}\bG_{23}\bG_{12}\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			0 & 0 & 0 & 0& \boxtimes
		\end{sbmatrix}
		\stackrel{\bG_{45}}{\rightarrow}
		\begin{sbmatrix}{\bG_{45}\bG_{34}\bG_{23}\bG_{12}\bR}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes}& \bm{\boxtimes} \\
		\end{sbmatrix}.
	\end{aligned}
	$$
	Generally, the backward rotation results in,
	$$
	\bG_{12} \bG_{23} \ldots \bG_{(n-1),n} (\bR+\bw\bv^\top) = \bH_0  \pm ||\bw|| \be_1 \bv^\top = \bH,
	$$
	which is also upper Hessenberg. Similar to triangularization via the Givens rotation in Section~\ref{section:qr-givens} (p.~\pageref{section:qr-givens}), there exists a set of rotations $\bJ_{12}, \bJ_{23}, \ldots, \bJ_{(n-1),n}$ such that
	$$
	\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}\bH = \bR^\prime,
	$$
	is upper triangular. Following the $5\times 5$ example, the triangularization is shown as follows
	$$
	\begin{aligned}
		\underbrace{\bH_0  \pm ||\bw|| \be_1 \bv^\top}_{\bH} =
		\begin{sbmatrix}{\bH}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes
		\end{sbmatrix}
		&\stackrel{\bJ_{12}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{12}\bH}
			\bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
			\bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & \boxtimes & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes
		\end{sbmatrix}
		\stackrel{\bJ_{23}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{23}\bJ_{12}\bH}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes
		\end{sbmatrix}\\
		&\stackrel{\bJ_{34}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{34}\bJ_{23}\bJ_{12}\bH}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & 0 & \bm{0} & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & 0 & 0 & \boxtimes& \boxtimes
		\end{sbmatrix}
		\stackrel{\bJ_{45}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{45}\bJ_{34}\bJ_{23}\bJ_{12}\bH}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \bm{\boxtimes} & \bm{\boxtimes} \\
			0 & 0 & 0 & \bm{0} & \bm{\boxtimes} \\
		\end{sbmatrix}.
	\end{aligned}
	$$
	And the QR decomposition of $\bA^\prime$ is thus given by
	$$
	\bA^\prime = \bQ^\prime \bR^\prime,
	$$
	where
\begin{equation}\label{equation:qr-rank-one-update}
	\left\{
	\begin{aligned}
		\bR^\prime &=(\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}) (\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}) (\bR+\bw\bv^\top);\\
		\bQ^\prime &= \bQ\left\{(\bJ_{(n-1),n} \ldots \bJ_{23}\bJ_{12}) (\bG_{12} \bG_{23} \ldots \bG_{(n-1),n}) \right\}^\top .\\
	\end{aligned}
	\right.
\end{equation}

	\section{Appending or Deleting a Column}\label{section:append-column-qr}
	\paragraph{Deleting a column}
	Suppose the QR decomposition of $\bA\in \real^{m\times n}$ is given by $\bA=\bQ\bR$ where the column partition of $\bA$ is $\bA=[\ba_1,\ba_2,\ldots,\ba_n]$. Now, if we delete the $k$-th column of $\bA$ such that $\bA^\prime = [\ba_1,\ldots,\ba_{k-1},\ba_{k+1},\ldots,\ba_n] \in \real^{m\times (n-1)}$. We want to find the QR decomposition of $\bA^\prime$ efficiently. Suppose further $\bR$ has the following form
	$$
	\begin{aligned}
		\begin{blockarray}{ccccc}
			\begin{block}{c[ccc]c}
				&	\bR_{11} & \ba & \bR_{12} & k-1  \\
				\bR = 	&  \bzero    &	r_{kk} & \bb^\top  & 1 \\
				&	\bzero  &\bzero& \bR_{22} & m-k  \\
			\end{block}
			& k-1 & 1 &  n-k & \\
		\end{blockarray}.\\
	\end{aligned}
	$$
	Apparently,
	$$
	\bQ^\top \bA^\prime =
	\begin{bmatrix}
		\bR_{11} &\bR_{12} \\
		\bzero & \bb^\top \\
		\bzero & \bR_{22}
	\end{bmatrix} = \bH
	$$
	is upper Hessenberg. A $6\times 5$ example is shown as follows where $k=3$:
	$$
	\begin{aligned}
		\begin{sbmatrix}{\bR = \bQ^\top\bA}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & 0 & \boxtimes& \boxtimes \\
			0 & 0 & 0 & 0& \boxtimes \\
			0 & 0 & 0 & 0& 0
		\end{sbmatrix}
		&\longrightarrow
		\begin{sbmatrix}{\bH = \bQ^\top\bA^\prime}
			\boxtimes & \boxtimes  & \boxtimes & \boxtimes \\
			0 & \boxtimes  & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes& \boxtimes \\
			0 & 0  & \boxtimes& \boxtimes \\
			0 & 0  & 0& \boxtimes \\
			0 & 0  & 0& 0
		\end{sbmatrix}.
	\end{aligned}
	$$
	Again, for columns $k$ to $n-1$ of $\bH$, there exists a set of rotations $\bG_{k,k+1}$, $\bG_{k+1,k+2}$, $\ldots$, $\bG_{n-1,n}$ that could introduce zeros for the elements $h_{k+1,k}$, $h_{k+2,k+1}$, $\ldots$, $h_{n,n-1}$ of $\bH$. The the triangular matrix $\bR^\prime$ is given by
	$$
	\bR^\prime = \bG_{n-1,n}\ldots \bG_{k+1,k+2}\bG_{k,k+1}\bQ^\top \bA^\prime.
	$$
	And the orthogonal matrix
	\begin{equation}\label{equation:qr-delete-column-finalq}
		\bQ^\prime = (\bG_{n-1,n}\ldots \bG_{k+1,k+2}\bG_{k,k+1}\bQ^\top )^\top = \bQ \bG_{k,k+1}^\top  \bG_{k+1,k+2}^\top \ldots \bG_{n-1,n}^\top,
	\end{equation}
	such that $\bA^\prime = \bQ^\prime\bR^\prime$.

	\paragraph{Appending a column}
	Similarly, suppose $\widetilde{\bA} = [\ba_1,\ba_k,\bw,\ba_{k+1},\ldots,\ba_n]$ where we append $\bw$ into the $(k+1)$-th column of $\bA$. We can obtain
	$$
	\bQ^\top \widetilde{\bA} = [\bQ^\top\ba_1,\ldots, \bQ^\top\ba_k, \bQ^\top\bw, \bQ^\top\ba_{k+1}, \ldots,\bQ^\top\ba_n] = \widetilde{\bH}.
	$$
	A set of Givens rotations $\bJ_{m-1,m}, \bJ_{m-2,m-1}, \ldots, \bJ_{k+1,k+2}$ can introduce zeros for the $\widetilde{h}_{m,k+1}$, $\widetilde{h}_{m-1,k+1}$, $\ldots$, $\widetilde{h}_{k+2,k+1}$ elements of $\widetilde{\bH}$ such that
	$$
	\widetilde{\bR} =\bJ_{k+1,k+2}\ldots \bJ_{m-2,m-1}\bJ_{m-1,m} \bQ^\top \widetilde{\bA}.
	$$
	Suppose $\widetilde{\bH}$ is of size $6\times 5$ and $k=2$, an example is shown as follows where $\boxtimes$ represents a value that is not necessarily zero, and \textbf{boldface} indicates the value has just been changed.
	$$
	\begin{aligned}
		\begin{sbmatrix}{\widetilde{\bH}}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & 0& \boxtimes \\
			0 & 0 & \boxtimes & 0& 0 \\
			0 & 0 & \boxtimes & 0& 0
		\end{sbmatrix}
		&\stackrel{\bJ_{56}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{56}\widetilde{\bH} \rightarrow \widetilde{h}_{63}=0}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \boxtimes & 0& \boxtimes \\
			0 & 0 & \bm{\boxtimes} & 0& 0 \\
			0 & 0 & \bm{0} & 0& 0
		\end{sbmatrix}
		\stackrel{\bJ_{45}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{45}\bJ_{56}\widetilde{\bH}\rightarrow \widetilde{h}_{53}=0}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \boxtimes & \boxtimes& \boxtimes \\
			0 & 0 & \bm{\boxtimes} & 0& \bm{\boxtimes} \\
			0 & 0 & \bm{0} & 0& \bm{\boxtimes}\\
			0 & 0 & 0 & 0& 0
		\end{sbmatrix}\stackrel{\bJ_{34}}{\rightarrow}
		\begin{sbmatrix}{\bJ_{34}\bJ_{45}\bJ_{56}\widetilde{\bH}\rightarrow \widetilde{h}_{43}=0}
			\boxtimes & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & \boxtimes & \boxtimes & \boxtimes & \boxtimes \\
			0 & 0 & \bm{\boxtimes}  & \bm{\boxtimes} & \bm{\boxtimes}  \\
			0 & 0 & \bm{0}  & \bm{\boxtimes} & \bm{\boxtimes}  \\
			0 & 0 & 0 & 0& \boxtimes\\
			0 & 0 & 0 & 0& 0
		\end{sbmatrix}.
	\end{aligned}
	$$
	And finally, the orthogonal matrix
	\begin{equation}\label{equation:qr-add-column-finalq}
		\widetilde{\bQ} = (\bJ_{k+1,k+2}\ldots \bJ_{m-2,m-1}\bJ_{m-1,m} \bQ^\top )^\top = \bQ \bJ_{m-1,m}^\top  \bJ_{m-2,m-1}^\top \ldots \bJ_{k+1,k+2}^\top,
	\end{equation}
	such that $\widetilde{\bA} = \widetilde{\bQ}\widetilde{\bR}$.

	\paragraph{Real world application} The method introduced above is useful for the efficient variable selection in the least squares problem via the QR decomposition. At each time we delete a column of the data matrix $\bA$, and apply an $F$-test to see if the variable is significant or not. If not, we will delete the variable and favor a simpler model \citep{lu2021rigorous}.

	\section{Appending or Deleting a Row}

	\paragraph{Appending a row}
	Suppose the full QR decomposition of $\bA\in \real^{m\times n}$ is given by $\bA= \begin{bmatrix}
		\bA_1 \\
		\bA_2
	\end{bmatrix}=\bQ\bR$ where $\bA_1\in \real^{k\times n}$ and $\bA_2 \in \real^{(m-k)\times n}$. Now, if we add a row such that $\bA^\prime = \begin{bmatrix}
		\bA_1 \\
		\bw^\top\\
		\bA_2
	\end{bmatrix} \in \real^{(m+1)\times n}$. We want to find the full QR decomposition of $\bA^\prime$ efficiently. Construct a permutation matrix
	$$
	\bP=
	\begin{bmatrix}
		\bzero & 1 & \bzero  \\
		\bI_k & \bzero & \bzero \\
		\bzero & \bzero & \bI_{m-k}
	\end{bmatrix}
	\longrightarrow
	\bP
	\begin{bmatrix}
		\bA_1 \\
		\bw^\top\\
		\bA_2
	\end{bmatrix}
	=
	\begin{bmatrix}
		\bw^\top\\
		\bA_1 \\
		\bA_2
	\end{bmatrix}
	.
	$$
	Then,
	$$
	\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bQ^\top
	\end{bmatrix}
	\bP
	\bA^\prime
	=
	\begin{bmatrix}
		\bw^\top \\
		\bR
	\end{bmatrix}=\bH
	$$
	is upper Hessenberg. Similarly, a set of rotations $\bG_{12}, \bG_{23}, \ldots, \bG_{n,n+1}$ can be applied to introduce zeros for the elements $h_{21}$, $h_{32}$, $\ldots$, $h_{n+1,n}$ of $\bH$. The triangular matrix $\bR^\prime$ is given by
	$$
	\bR^\prime = \bG_{n,n+1}\ldots \bG_{23}\bG_{12}\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bQ^\top
	\end{bmatrix}
	\bP   \bA^\prime.
	$$
	And the orthogonal matrix
	$$
	\bQ^\prime = \left(\bG_{n,n+1}\ldots \bG_{23}\bG_{12}\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bQ^\top
	\end{bmatrix}
	\bP \right)^\top
	=
	\bP^\top
	\begin{bmatrix}
		1 & \bzero \\
		\bzero & \bQ
	\end{bmatrix}
	\bG_{12}^\top  \bG_{23}^\top \ldots \bG_{n,n+1}^\top,
	$$
	such that $\bA^\prime = \bQ^\prime\bR^\prime$.

	\paragraph{Deleting a row} Suppose $\bA = \begin{bmatrix}
		\bA_1 \\
		\bw^\top\\
		\bA_2
	\end{bmatrix} \in \real^{m\times n}$ where $\bA_1\in\real^{k\times n}$, $\bA_2 \in \real^{(m-k-1)\times n}$ with the full QR decomposition given by $\bA=\bQ\bR$ where $\bQ\in \real^{m\times m}, \bR\in \real^{m\times n}$. We want to compute the full QR decomposition of $\widetilde{\bA} =  \begin{bmatrix}
		\bA_1 \\
		\bA_2
	\end{bmatrix}$ efficiently (assume $m-1\geq n$). Analogously, we can construct a permutation matrix
	$$
	\bP =
	\begin{bmatrix}
		\bzero & 1 & \bzero  \\
		\bI_k & \bzero & \bzero \\
		\bzero & \bzero & \bI_{m-k-1}
	\end{bmatrix}
	$$
	such that
	$$
	\bP\bA =
	\begin{bmatrix}
		\bzero & 1 & \bzero  \\
		\bI_k & \bzero & \bzero \\
		\bzero & \bzero & \bI_{m-k-1}
	\end{bmatrix}
	\begin{bmatrix}
		\bA_1 \\
		\bw^\top\\
		\bA_2
	\end{bmatrix}=
	\begin{bmatrix}
		\bw^\top \\
		\bA_1\\
		\bA_2
	\end{bmatrix} = \bP\bQ\bR =\bM\bR ,
	$$
	where $\bM = \bP\bQ$ is an orthogonal matrix. Let $\bmm^\top$ be the first row of $\bM$, and a set of givens rotations $\bG_{m-1,m}, \bG_{m-2,m-1}, \ldots, \bG_{1,2}$ introducing zeros for elements $m_m, m_{m-1}, \ldots, m_2$ of $\bmm$ respectively such that $\bG_{1,2}\ldots \bG_{m-2,m-1}\bG_{m-1,m}\bmm = \alpha \be_1$ where $\alpha = \pm 1$. Therefore, we have
	$$
	\bG_{1,2}\ldots \bG_{m-2,m-1}\bG_{m-1,m} \bR =
		\begin{blockarray}{cc}
		\begin{block}{[c]c}
			\bv^\top & 1 \\
			\bR_1& m-1  \\
		\end{block}
	\end{blockarray},
	$$
	which is upper Hessenberg with $\bR_1\in \real^{(m-1)\times n}$ being upper triangular. And
	$$
	\bM   \bG_{m-1,m}^\top \bG_{m-2,m-1}^\top \ldots  \bG_{1,2}^\top  =
	\begin{bmatrix}
		\alpha & \bzero \\
		\bzero & \bQ_1
	\end{bmatrix},
	$$
	where $\bQ_1\in \real^{(m-1)\times (m-1)}$ is an orthogonal matrix. The bottom-left block of the above matrix is a zero vector since $\alpha=\pm 1$ and $\bM$ is orthogonal. To see this, let $\bG=\bG_{m-1,m}^\top \bG_{m-2,m-1}^\top $ $\ldots  \bG_{1,2}^\top $ with the first column being $\bg$ and $\bM = [\bmm^\top; \bmm_2^\top; \bmm_3^\top; \ldots, \bmm_{m}^\top]$ being the row partition of $\bM$. We have
	$$
	\begin{aligned}
		\bmm^\top\bg &= \pm 1 \qquad  \rightarrow \qquad  \bg = \pm \bmm, \\
		\bmm_i^\top \bmm &=0,  \qquad \forall i \in \{2,3,\ldots,m\}.
	\end{aligned}
	$$
	This results in
	$$
	\begin{aligned}
		\bP\bA&=\bM\bR\\
		&=(\bM   \bG_{m-1,m}^\top \bG_{m-2,m-1}^\top \ldots  \bG_{1,2}\top ) (\bG_{1,2}\ldots \bG_{m-2,m-1}\bG_{m-1,m} \bR ) \\
		&=
		\begin{bmatrix}
			\alpha & \bzero \\
			\bzero & \bQ_1
		\end{bmatrix}
		\begin{bmatrix}
			\bv^\top \\
			\bR_1
		\end{bmatrix} =
		\begin{bmatrix}
			\alpha \bv^\top \\
			\bQ_1\bR_1
		\end{bmatrix}
		=
		\begin{bmatrix}
			\bw^\top \\
			\widetilde{\bA}
		\end{bmatrix}
		.
	\end{aligned}
	$$
	This implies $\bQ_1\bR_1$ is the full QR decomposition of $\widetilde{\bA}=\begin{bmatrix}
		\bA_1\\
		\bA_2
	\end{bmatrix}$.

\chapter{UTV Decomposition: ULV and URV Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{UTV Decomposition}\label{section:ulv-urv-decomposition}
The UTV decomposition goes further by factoring the matrix into two orthogonal matrices $\bA=\bU\bT\bV$, where $\bU, \bV$ are orthogonal, whilst $\bT$ is (upper/lower) triangular.\footnote{These decompositions fall into a category known as the \textit{double-sided orthogonal decomposition}. We will see  the UTV decomposition, complete orthogonal decomposition (Theorem~\ref{theorem:complete-orthogonal-decom}, p.~\pageref{theorem:complete-orthogonal-decom}), and singular value decomposition are all in this notion.}
The resulting $\bT$ supports rank estimation. The matrix $\bT$ can be lower triangular which results in the ULV decomposition, or it can be upper triangular which results in the URV decomposition. The UTV framework shares a similar form as the singular value decomposition (SVD, see Section~\ref{section:SVD}, p.~\pageref{section:SVD}) and can be regarded as an inexpensive alternative to the SVD.
\begin{theoremHigh}[Full ULV Decomposition]\label{theorem:ulv-decomposition}
Every $m\times n$ matrix $\bA$ with rank $r$ can be factored as
$$
\bA = \bU \begin{bmatrix}
\bL & \bzero \\
\bzero & \bzero
\end{bmatrix}\bV,
$$
where $\bU\in \real^{m\times m}$ and $\bV\in \real^{n\times n}$ are two orthogonal matrices, and $\bL\in \real^{r\times r}$ is a lower triangular matrix.
\end{theoremHigh}

The existence of the ULV decomposition is from the QR and LQ decomposition.
\begin{proof}[of Theorem~\ref{theorem:ulv-decomposition}]
For any rank $r$ matrix $\bA=[\ba_1, \ba_2, \ldots, \ba_n]$, we can use a column permutation matrix $\bP$ (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}) such that the linearly independent columns of $\bA$ appear in the first $r$ columns of $\bA\bP$. Without loss of generality, we assume $\bb_1, \bb_2, \ldots, \bb_r$ are the $r$ linearly independent columns of $\bA$ and
$$
\bA\bP = [\bb_1, \bb_2, \ldots, \bb_r, \bb_{r+1}, \ldots, \bb_n].
$$
Let $\bZ = [\bb_1, \bb_2, \ldots, \bb_r] \in \real^{m\times r}$. Since any $\bb_i$ is in the column space of $\bZ$, we can find a $\bE\in \real^{r\times (n-r)}$ such that
$$
[\bb_{r+1}, \bb_{r+2}, \ldots, \bb_n] = \bZ \bE.
$$
That is,
$$
\bA\bP = [\bb_1, \bb_2, \ldots, \bb_r, \bb_{r+1}, \ldots, \bb_n] = \bZ
\begin{bmatrix}
	\bI_r & \bE
\end{bmatrix},
$$
where $\bI_r$ is an $r\times r$ identity matrix. Moreover, $\bZ\in \real^{m\times r}$ has full column rank such that its full QR decomposition is given by $\bZ = \bU\begin{bmatrix}
	\bR \\
	\bzero
\end{bmatrix}$, where $\bR\in \real^{r\times r}$ is an upper triangular matrix with full rank and $\bU$ is an orthogonal matrix. This implies
\begin{equation}\label{equation:ulv-smpl}
\bA\bP = \bZ
\begin{bmatrix}
	\bI_r & \bE
\end{bmatrix}
=
\bU\begin{bmatrix}
	\bR \\
	\bzero
\end{bmatrix}
\begin{bmatrix}
	\bI_r & \bE
\end{bmatrix}
=
\bU\begin{bmatrix}
	\bR & \bR\bE \\
	\bzero & \bzero
\end{bmatrix}.
\end{equation}
Since $\bR$ has full rank, this means
$\begin{bmatrix}
	\bR & \bR\bE
\end{bmatrix}$ also has full rank such that its full LQ decomposition is given by
$\begin{bmatrix}
	\bL & \bzero
\end{bmatrix} \bV_0$ where $\bL\in \real^{r\times r}$ is a lower triangular matrix and $\bV_0$ is an orthogonal matrix. Substitute into Equation~\eqref{equation:ulv-smpl}, we have
$$
\bA = \bU \begin{bmatrix}
	\bL & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV_0 \bP^{-1}.
$$
Let $\bV =\bV_0 \bP^{-1}$ which is a product of two orthogonal matrices, and is also an orthogonal matrix. This completes the proof.
\end{proof}
A second way to see the proof of the ULV decomposition will be discussed in the proof of Theorem~\ref{theorem:complete-orthogonal-decom} shortly via the rank-revealing QR decomposition and trivial QR decomposition.
Now suppose the ULV decomposition of matrix $\bA$ is
$$
\bA = \bU \begin{bmatrix}
	\bL & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV.
$$
Let $\bU_0 = \bU_{:,1:r}$ and $\bV_0 = \bV_{1:r,:}$, i.e., $\bU_0$ contains only the first $r$ columns of $\bU$, and $\bV_0$ contains only the first $r$ rows of $\bV$. Then, we still have $\bA = \bU_0 \bL\bV_0$. This is known as the \textbf{reduced ULV decomposition}.
Similarly, we can also claim the URV decomposition as follows.
	\begin{theoremHigh}[Full URV Decomposition]\label{theorem:urv-decomposition}
		Every $m\times n$ matrix $\bA$ with rank $r$ can be factored as
		$$
		\bA = \bU \begin{bmatrix}
			\bR & \bzero \\
			\bzero & \bzero
		\end{bmatrix}\bV,
		$$
		where $\bU\in \real^{m\times m}$ and $\bV\in \real^{n\times n}$ are two orthogonal matrices, and $\bR\in \real^{r\times r}$ is an upper triangular matrix.
	\end{theoremHigh}
The proof is just similar to that of ULV decomposition and we shall not give the details.
Again, there is a version of reduced URV decomposition and the difference between the full and reduced URV can be implied from the context. The ULV and URV sometimes are referred to as the UTV decomposition framework \citep{fierro1997low, golub2013matrix}.

We will shortly see that the forms of ULV and URV are very close to the singular value decomposition (SVD). All of the three factor the matrix $\bA$ into two orthogonal matrices. Especially, there exists a set of bases for the four subspaces of $\bA$ in the fundamental theorem of linear algebra via the ULV and the URV. Taking ULV as an example, the first $r$ columns of $\bU$ form an orthonormal basis of $\cspace(\bA)$, and the last $(m-r)$ columns of $\bU$ form an orthonormal basis of $\nspace(\bA^\top)$. The first $r$ rows of $\bV$ form an orthonormal basis for the row space $\cspace(\bA^\top)$, and the last $(n-r)$ rows form an orthonormal basis for $\nspace(\bA)$ (similar to the two-sided orthogonal decomposition):
$$
\left\{
\begin{aligned}
	\cspace(\bA) &= span\{\bu_1, \bu_2, \ldots, \bu_r\}, \\
	\nspace(\bA) &= span\{\bv_{r+1}, \bv_{r+2}, \ldots, \bv_n\}, \\
	\cspace(\bA^\top) &= span\{ \bv_1, \bv_2, \ldots,\bv_r \} ,\\
	\nspace(\bA^\top) &=span\{\bu_{r+1}, \bu_{r+2}, \ldots, \bu_m\}.
\end{aligned}
\right.
$$
The SVD goes further that there is a connection between the two pairs of orthonormal basis, i.e., transforming from column basis into row basis, or left null space basis into right null space basis. We will get more details in the SVD section.

\section{Complete Orthogonal Decomposition}
What is related to the UTV decomposition is called the \textit{complete orthogonal decomposition} which factors into two orthogonal matrices as well.
\begin{theoremHigh}[Complete Orthogonal Decomposition]\label{theorem:complete-orthogonal-decom}
	Every $m\times n$ matrix $\bA$ with rank $r$ can be factored as
	$$
	\bA = \bU \begin{bmatrix}
		\bT & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV,
	$$
	where $\bU\in \real^{m\times m}$ and $\bV\in \real^{n\times n}$ are two orthogonal matrices, and $\bT\in \real^{r\times r}$ is an rank-$r$ matrix.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:complete-orthogonal-decom}]
By rank-revealing QR decomposition (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}), $\bA$ can be factored as
$$
\bQ_1^\top \bA\bP =
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix},
$$
where $\bR_{11} \in \real^{r\times r}$ is upper triangular, $\bR_{12} \in \real^{r\times (n-r)}$, $\bQ_1\in \real^{m\times m}$ is an orthogonal matrix, and $\bP$ is a permutation matrix.

\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolorSkip]
Then, it is not hard to find a decomposition such that
\begin{equation}\label{equation:orthogonal-complete-qr-or-not}
 \begin{bmatrix}
	\bR_{11}^\top \\
	\bR_{12}^\top
\end{bmatrix}
=
\bQ_2
\begin{bmatrix}
	\bS \\
	\bzero
\end{bmatrix},
\end{equation}
where $\bQ_2$ is an orthogonal matrix, $\bS$ is an rank-$r$ matrix. The decomposition is reasonable in the sense the matrix $ \begin{bmatrix}
	\bR_{11}^\top \\
	\bR_{12}^\top
\end{bmatrix} \in \real^{n\times r}$ has rank $r$ of which the columns stay in a subspace of $\real^{n}$. Nevertheless, the columns of $\bQ_2$ span the whole space of $\real^n$, where we can assume the first $r$ columns of $\bQ_2$ span the same space as that of  $ \begin{bmatrix}
\bR_{11}^\top \\
\bR_{12}^\top
\end{bmatrix}$. The matrix $\begin{bmatrix}
\bS \\
\bzero
\end{bmatrix}$ is to transform $\bQ_2$ into $ \begin{bmatrix}
\bR_{11}^\top \\
\bR_{12}^\top
\end{bmatrix}$.
\end{mdframed}
Then, it follows that
$$
\bQ_1^\top \bA\bP \bQ_2 =
\begin{bmatrix}
\bS^\top & \bzero \\
\bzero & \bzero
\end{bmatrix}.
$$
Let $\bU = \bQ_1$, $\bV=\bQ_2^\top\bP^\top$ and $\bT = \bS^\top$, we complete the proof.
\end{proof}
We can find that when Equation~\eqref{equation:orthogonal-complete-qr-or-not} is taken to be the reduced QR decomposition of $ \begin{bmatrix}
	\bR_{11}^\top \\
	\bR_{12}^\top
\end{bmatrix}$, then the complete orthogonal decomposition reduces to the ULV decomposition.

\section{Application: Row Rank equals Column Rank Again via UTV}
The UTV framework can prove the important theorem in linear algebra that the row rank and column rank of a matrix are equal.
Notice that to apply the UTV in the proof, a slight modification on the claim of the existence of the UTV decomposition needs to be taken care of. For example, in Theorem~\ref{theorem:ulv-decomposition}, the assumption of the matrix $\bA$ is to have rank $r$. Since rank $r$ already admits the fact that row rank equals column rank. A better claim here to this aim is to say matrix $\bA$ has column rank $r$ in Theorem~\ref{theorem:ulv-decomposition}. See
\citet{lu2021column} for a detailed discussion.

\begin{proof}[\textbf{of Theorem~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}, A Second Way}]
	Any $m\times n$ matrix $\bA$ with rank $r$ can be factored as
	$$
	\bA = \bU_0 \begin{bmatrix}
		\bL & \bzero \\
		\bzero & \bzero
	\end{bmatrix}\bV_0,
	$$
	where $\bU_0\in \real^{m\times m}$ and $\bV_0\in \real^{n\times n}$ are two orthogonal matrices, and $\bL\in \real^{r\times r}$ is a lower triangular matrix \footnote{Instead of using the ULV decomposition, in some texts, the authors use elementary transformations $\bE_1, \bE_2$ such that $
		\bA = \bE_1 \begin{bmatrix}
			\bI_r & \bzero \\
			\bzero & \bzero
		\end{bmatrix}\bE_2,
		$ to prove the result.}. Let $\bD = \begin{bmatrix}
		\bL & \bzero \\
		\bzero & \bzero
	\end{bmatrix}$, the row rank and column rank of $\bD$ are apparently the same. If we could prove the column rank of $\bA$ equals the column rank of $\bD$, and the row rank of $\bA$ equals the row rank of $\bD$, then we complete the proof.

	Let $\bU = \bU_0^\top$, $\bV=\bV_0^\top$, then $\bD = \bU\bA\bV$. Decompose the above idea into two steps, a moment of reflexion reveals that, if we could first prove the row rank and column rank of $\bA$ are equal to those of $\bU\bA$, and then, if we further prove the row rank and column rank of $\bU\bA $ are equal to those of $\bU\bA\bV$, we could also complete the proof.

	\paragraph{Row rank and column rank of $\bA$ are equal to those of $\bU\bA$} Let $\bB = \bU\bA$, and let further $\bA=[\ba_1,\ba_2,\ldots, \ba_n]$ and $\bB=[\bb_1,\bb_2,\ldots,\bb_n]$ be the column partitions of $\bA$ and $\bB$. Therefore, $[\bb_1,\bb_2,\ldots,\bb_n] = [\bU\ba_1,\bU\ba_2,\ldots, \bU\ba_n]$. If $x_1\ba_1+x_2\ba_2+\ldots+x_n\ba_n=0$, then we also have
	$$
	\bU(x_1\ba_1+x_2\ba_2+\ldots+x_n\ba_n) = x_1\bb_1+x_2\bb_2+\ldots+x_n\bb_n = 0.
	$$
	Let $j_1, j_2, \ldots, j_r$ be distinct indices between 1 and $n$, if the set $\{\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}\}$ is independent, the set $\{\bb_{j_1}, \bb_{j_2}, \ldots, \bb_{j_r}\}$ must also be linearly independent. This implies
	$$
	dim(\cspace(\bB)) \leq dim(\cspace(\bA)).
	$$
	Similarly, by $\bA = \bU^\top\bB$, it follows that
	$$
	dim(\cspace(\bA)) \leq dim(\cspace(\bB)).
	$$
	This implies
	$$
	dim(\cspace(\bB)) = dim(\cspace(\bA)).
	$$
	Apply the process to $\bB^\top$ and $\bA^\top$, we have
	$$
	dim(\cspace(\bB^\top)) = dim(\cspace(\bA^\top)).
	$$
	This implies the row rank and column rank of $\bA$ and $\bB=\bU\bA$ are the same.
	Similarly, we can also show that the row rank and column rank of $\bU\bA$ and $\bU\bA\bV$ are the same. This completes the proof.
\end{proof}

\chapter{Skeleton/CUR Decomposition}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\index{Skeleton}
\section{Skeleton/CUR Decomposition}
\begin{theoremHigh}[Skeleton Decomposition]\label{theorem:skeleton-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m\times n}{\bA }=
\underset{m\times r}{\bC} \gap \underset{r\times r}{\bU^{-1} }\gap \underset{r\times n}{\bR},
$$
where $\bC$ is some $r$ linearly independent columns of $\bA$, $\bR$ is some $r$ linearly independent rows of $\bA$, and $\bU$ is the nonsingular submatrix on the intersection.
\begin{itemize}
\item The storage for the decomposition is then reduced or potentially increased from $mn$ floats to $r(m+n)+r^2$ floats.
\item Or further, if we only record the position of the indices, it requires $mr$, $nr$ floats for storing $\bC, \bR$ respectively and extra $2r$ integers to remember the position of each column of $\bC$ in that of $\bA$ and each row of $\bR$ in that of $\bA$ (i.e., construct $\bU$ from $\bC,\bR$).
\end{itemize}
\end{theoremHigh}

Skeleton decomposition is also known as the \textit{CUR decomposition} following from the notation in the decomposition. The illustration of skeleton decomposition is shown in Figure~\ref{fig:skeleton} where the \textcolor{canaryyellow}{yellow} vectors denote the linearly independent columns of $\bA$ and \textcolor{green}{green} vectors denote the linearly independent rows of $\bA$.
In case when $\bA$ is square and invertible, we have skeleton decomposition $\bA=\bC\bU^{-1} \bR$ where $\bC=\bR=\bU=\bA$ such that the decomposition reduces to $\bA = \bA\bA^{-1}\bA$.
Specifically, given $I,J$ index vectors both with size $r$ that contain the indices of rows and columns selected from $\bA$ into $\bR$ and $\bC$ respectively, $\bU$ can be denoted as $\bU=\bA[I,J]$.
\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{imgs/skeleton.pdf}
\caption{Demonstration of skeleton decomposition of a matrix.}
\label{fig:skeleton}
\end{figure}

\section{Existence of the Skeleton Decomposition}
In Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}), we proved the row rank and the column rank of a matrix are equal. In another word, we can also claim that the dimension of the column space and the dimension of the row space are equal. This property is essential for the existence of the skeleton decomposition.
We are then ready to prove the existence of the skeleton decomposition. The proof is rather elementary.
\begin{proof}[of Theorem~\ref{theorem:skeleton-decomposition}]
The proof relies on the existence of such nonsingular matrix $\bU$ which is central to this decomposition method.

\paragraph{Existence of such nonsingular matrix $\bU$} Since matrix $\bA$ is rank-$r$, we can pick $r$ columns from $\bA$ so that they are linearly independent. Suppose we put the specific $r$ independent columns $\ba_{i1}, \ba_{i2}, \ldots, \ba_{ir}$ into the columns of an $m\times r$ matrix $\bN=[\ba_{i1}, \ba_{i2}, \ldots, \ba_{ir}] \in \real^{m\times r}$. The dimension of the column space of $\bN$ is $r$ so that the dimension of the row space of $\bN$ is also $r$ by Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}). Again, we can pick $r$ linearly independent rows $\bn_{j1}^\top,\bn_{j2}^\top, \ldots, \bn_{jr}^\top $ from $\bN$ and put the specific $r$ rows into rows of an $r\times r$ matrix $\bU = [\bn_{j1}^\top; \bn_{j2}^\top; \ldots; \bn_{jr}^\top]\in \real^{r\times r}$. Using Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}) again, the dimension of the column space of $\bU$ is also $r$ which means there are the $r$ linearly independent columns from $\bU$. So $\bU$ is such a nonsingular matrix with size $r\times r$.

\paragraph{Main proof}
As long as we find the nonsingular $r\times r$ matrix $\bU$ inside $\bA$, we can find the existence of the skeleton decomposition as follows.

Suppose $\bU=\bA[I,J]$ where $I,J$ are index vectors of size $r$. Since $\bU$ is a nonsingular matrix, the columns of $\bU$ are linearly independent. Thus the columns of matrix $\bC$ based on the columns of $\bU$ are also linearly independent (i.e., select the $r$ columns of $\bA$ with the same entries of the matrix $\bU$. Here $\bC$ is equal to the $\bN$ we construct above and $\bC=\bA[:,J]$).

As the rank of the matrix $\bA$ is $r$, if we take any other column $\ba_i$ of $\bA$, $\ba_i$ can be represented as a linear combination of the columns of $\bC$, i.e., there exists a vector $\bx$ such that $\ba_i = \bC \bx$, for all $ i\in \{1, 2, \ldots, n\}$. Let $r$ rows of $\ba_i$ corresponding to the row entries of $\bU$ be $\br_i \in \real^r$ for all $i\in \{1, 2, \ldots, n\}$ (i.e., $\br_i$ contains $r$ entries of $\ba_i$). That is, select the $r$ entries of $\ba_i$'s corresponding to the entries of $\bU$ as follows:
$$
\bA = [\ba_1,\ba_2, \ldots, \ba_n]\in \real^{m\times n} \qquad \longrightarrow \qquad
\bA[I,:]=[\br_1, \br_2, \ldots, \br_n] \in \real^{r\times n}.
$$
Since $\ba_i = \bC\bx$, $\bU$ is a submatrix inside $\bC$, and $\br_i$ is a subvector inside $\ba_i$, we have $\br_i = \bU \bx$ which is equivalent to $\bx = \bU^{-1} \br_i$. Thus for every $i$, we have $\ba_i = \bC \bU^{-1} \br_i$. Combine the $n$ columns of such $\br_i$ into $\bR=[\br_1, \br_2, \ldots, \br_n]$, we obtain
$$
\bA = [\ba_1, \ba_2, \ldots, \ba_n] = \bC \bU^{-1} \bR,
$$
from which the result follows.

In short, we first find $r$ linearly independent columns of $\bA$ into $\bC\in \real^{m\times r}$. From $\bC$, we find an $r\times r$ nonsingular submatrix $\bU$. The $r$ rows of $\bA$ corresponding to entries of $\bU$ can help to reconstruct the columns of $\bA$. Again, the situation is shown in Figure~\ref{fig:skeleton}.
\end{proof}

\paragraph{CR decomposition vs skeleton decomposition} We note that CR decomposition and skeleton decomposition share a similar form. Even for the symbols used $\bA=\bC\bR$ for the CR decomposition and $\bA=\bC\bU^{-1}\bR$ for the skeleton decomposition.

Both in the CR decomposition and the skeleton decomposition, we \textbf{can} select the first $r$ independent columns to obtain the matrix $\bC$ (the symbol for both the CR decomposition and the skeleton decomposition). So $\bC$'s in the CR decomposition and the skeleton decomposition are exactly the same. On the contrary, $\bR$ in the CR decomposition is the reduced row echelon form without the zero rows, whereas $\bR$ in the skeleton decomposition is exactly some rows from $\bA$ so that $\bR$'s have different meanings in the two decompositional methods.

\paragraph{A word on the uniqueness of CR decomposition and skeleton decomposition} As mentioned above, both in the CR decomposition and the skeleton decomposition, we select the first $r$ linearly independent columns to obtain the matrix $\bC$. In this sense, the CR and skeleton decompositions have a unique form.
However, if we select the last $r$ linearly independent columns, we will get a different CR decomposition or skeleton decomposition. We will not discuss this situation here as it is not the main interest of this text.

To repeat, in the above proof for the existence of the skeleton decomposition, we first find the $r$ linearly independent columns of $\bA$ into the matrix $\bC$. From $\bC$, we find an $r\times r$ nonsingular submatrix $\bU$. From the submatrix $\bU$, we finally find the final row submatrix $\bR\in \real^{r\times n}$. A further question can be posed that if matrix $\bA$ has rank $r$, matrix $\bC$ contains $r$ linearly independent columns, and matrix $\bR$ contains $r$ linearly independent rows, then whether the $r\times r$ ``intersection" of $\bC$ and $\bR$ is invertible or not \footnote{We thank Gilbert Strang for raising this interesting question.}.

\begin{corollary}[Nonsingular Intersection]\label{corollary:invertible-intersection}
If matrix $\bA \in \real^{m\times n}$ has rank $r$, matrix $\bC$ contains $r$ linearly independent columns, and matrix $\bR$ contains $r$ linearly independent rows, then the $r\times r$ ``intersection" matrix $\bU$ of $\bC$ and $\bR$ is invertible.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:invertible-intersection}]
If $I,J$ are the indices of rows and columns selected from $\bA$ into $\bR$ and $\bC$ respectively, then, $\bR$ can be denoted as $\bR=\bA[I, :]$, $\bC$ can be represented as $\bC = \bA[:,J]$, and $\bU$ can be denoted as $\bU=\bA[I,J]$.

Since $\bC$ contains $r$ linearly independent columns of $\bA$, any column $\ba_i$ of $\bA$ can be represented as $\ba_i = \bC\bx_i = \bA[:,J]\bx_i$ for all $i \in \{1,2,\ldots, n\}$. This implies the $r$ entries of $\ba_i$ corresponding to the $I$ indices can be represented by the columns of $\bU$ such that $\ba_i[I] = \bU\bx_i \in \real^{r}$ for all $i \in \{1,2,\ldots, n\}$, i.e.,
$$
\ba_i = \bC\bx_i = \bA[:,J]\bx_i \in \real^{m} \qquad \longrightarrow  \qquad
\ba_i[I] =\bA[I,J]\bx_i= \bU\bx_i \in \real^{r}.
$$
Since $\bR$ contains $r$ linearly independent rows of $\bA$, the row rank and column rank of $\bR$ are equal to $r$. Combining the facts above, the $r$ columns of $\bR$ corresponding to indices $J$ (i.e., the $r$ columns of $\bU$) are linearly independent.

Again, by applying Corollary~\ref{lemma:equal-dimension-rank} (p.~\pageref{lemma:equal-dimension-rank}), the dimension of the row space of $\bU$ is also equal to $r$ which means there are the $r$ linearly independent rows from $\bU$, and $\bU$ is invertible.
\end{proof}

\chapter{Interpolative Decomposition (ID)}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup
\section{Interpolative Decomposition (ID)}
Column interpolative decomposition (ID) factors a matrix as the product of two matrices, one of which contains selected columns from the original matrix, and the other of which has a subset of columns consisting of the identity matrix and all its values are no greater than 1 in absolute value. Formally, we have the following theorem describing the details of the column ID.
\begin{theoremHigh}[Column Interpolative Decomposition]\label{theorem:interpolative-decomposition}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\underset{m \times n}{\bA} = \underset{m\times r}{\bC} \gap  \underset{r\times n}{\bW},
$$
where $\bC\in \real^{m\times r}$ is some $r$ linearly independent columns of $\bA$, $\bW\in \real^{r\times n}$ is the matrix to reconstruct $\bA$ which contains an $r\times r$ identity submatrix (under a mild column permutation). Specifically entries in $\bW$ have values no larger than 1 in magnitude:
$$
\max |w_{ij}|\leq 1, \,\, \forall \,\, i\in [1,r], j\in [1,n].
$$
\item The storage for the decomposition is then reduced or potentially increased from $mn$ floats to $mr$, $(n-r)r$ floats for storing $\bC, \bW$ respectively and extra $r$ integers are required to remember the position of each column of $\bC$ in that of $\bA$.
\end{theoremHigh}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.7\textwidth]{imgs/id-column.pdf}
	\caption{Demonstration of the column ID of a matrix where the \textcolor{canaryyellow}{yellow} vector denotes the linearly independent columns of $\bA$, white entries denote zero, and \textcolor{canarypurple}{purple} entries denote one.}
	\label{fig:column-id}
\end{figure}

The illustration of the column ID is shown in Figure~\ref{fig:column-id} where the \textcolor{canaryyellow}{yellow} vectors denote the linearly independent columns of $\bA$ and the \textcolor{canarypurple}{purple} vectors in $\bW$ form an $r\times r$ identity submatrix. The positions of the \textcolor{canarypurple}{purple} vectors inside $\bW$ are exactly the same as the positions of the corresponding \textcolor{canaryyellow}{yellow} vectors inside $\bA$. The column ID is very similar to the CR decomposition (Theorem~\ref{theorem:cr-decomposition}, p.~\pageref{theorem:cr-decomposition}), both select $r$ linearly independent columns into the first factor and the second factor contains an $r\times r$ identity submatrix. The difference is in that the CR decomposition will exactly choose the first $r$ linearly independent columns into the first factor and the identity submatrix appears in the pivots (Definition~\ref{definition:pivot}, p.~\pageref{definition:pivot}). And more importantly, the second factor in the CR decomposition comes from the RREF (Lemma~\ref{lemma:r-in-cr-decomposition}, p.~\pageref{lemma:r-in-cr-decomposition}).
Therefore, the column ID can also be utilized in the applications of the CR decomposition as well, say proving the fact of rank equaling trace in idempotent matrices (Lemma~\ref{lemma:rank-of-symmetric-idempotent2_tmp}, p.~\pageref{lemma:rank-of-symmetric-idempotent2_tmp}), and proving the elementary theorem in linear algebra that column rank equals row rank of a matrix (Corollary~\ref{lemma:equal-dimension-rank}, p.~\pageref{lemma:equal-dimension-rank}).
Moreover, the column ID is also a special case of rank decomposition (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}) and is apparently not unique. The connection between different column IDs is given by Lemma~\ref{lemma:connection-rank-decom} (p.~\pageref{lemma:connection-rank-decom}).

\paragraph{Notations that will be extensively used in the sequel} Following again the Matlab-style notation, if $J_s$ is an index vector with size $r$ that contains the indices of columns selected from $\bA$ into $\bC$, then $\bC$ can be denoted as $\bC=\bA[:,J_s]$.
The matrix $\bC$ contains ``skeleton" columns of $\bA$, hence the subscript $s$ in $J_s$. From the ``skeleton" index vector $J_s$, the $r\times r$ identity matrix inside $\bW$ can be recovered by
$$
\bW[:,J_s] = \bI_r \in \real^{r\times r}.
$$
Suppose further we put the remaining indices of $\bA$ into an index vector $J_r$ where
$$
J_s\cap J_r=\varnothing \qquad \text{and}\qquad J_s\cup J_r = \{1,2,\ldots, n\}.
$$
The remaining $n-r$ columns in $\bW$ consist of an $r\times (n-r)$ \textit{expansion matrix} since the matrix contains \textit{expansion coefficients} to reconstruct the columns of $\bA$ from $\bC$:
$$
\bE = \bW[:,J_r] \in \real^{r\times (n-r)},
$$
where the entries of $\bE$ are known as the \textit{expansion coefficients}. Moreover, let $\bP\in \real^{n\times n}$ be a (column) permutation matrix (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}) defined by $\bP=\bI_n[:,(J_s, J_r)]$ so that
$$
\bA\bP = \bA[:,(J_s, J_r)] = \left[\bC, \bA[:,J_r]\right],
$$
and
\begin{equation}\label{equation:interpolatibve-w-ep}
\bW\bP = \bW[:,(J_s, J_r)] =\left[\bI_r, \bE \right] \leadto \bW = \left[\bI_r, \bE \right] \bP^\top.
\end{equation}
\section{Existence of the Column Interpolative Decomposition}\label{section:proof-column-id}
\paragraph{Cramer's rule}
The proof of the existence of the column ID relies on the Cramer's rule that we shall shortly discuss here. Consider a system of $n$ linear equations for $n$ unknowns, represented in matrix multiplication form as follows \index{Cramer's rule}:
$$
\bM \bx = \bl,
$$
where $\bM\in \real^{n\times n}$ is nonsingular and $\bx,\bl \in \real^n$. Then the theorem states that in this case, the system has a unique solution, whose individual values for the unknowns are given by:
$$
x_i = \frac{\det(\bM_i)}{\det(\bM)}, \qquad \text{for all}\gap i\in \{1,2,\ldots, n\},
$$
where $\bM_i$ is the matrix formed by replacing the $i$-th column of $\bM$ with the column vector $\bl$. In full generality, the Cramer's rule considers the matrix equation
$$
\bM\bX = \bL,
$$
where $\bM\in \real^{n\times n}$ is nonsingular and $\bX,\bL\in \real^{n\times m}$. Let $I=[i_1, i_2, \ldots, i_k]$ and $J=[j_1,j_2,\ldots, j_k]$ be two index vectors where $1\leq i_1\leq i_2\leq \ldots\leq i_k\leq n$ and $1\leq j_1\leq j_2\leq \ldots\leq j_k\leq n$. Then $\bX[I,J]$ is a $k\times k$ submatrix of $\bX$. Let further $\bM_{\bL}(I,J)$ be the $n\times n$ matrix formed by replacing the $i_s$ column of $\bM$ by $j_s$ column of $\bL$ for all $s\in \{1,2,\ldots, k\}$. Then
$$
\det(\bX[I,J]) = \frac{\det\left(\bM_{\bL}(I,J)\right)}{\det(\bM)}.
$$
When $I,J$ are of size 1, it follows that
\begin{equation}\label{equation:cramer-rule-general}
	x_{ij} = \frac{\det\left(\bM_{\bL}(i,j)\right)}{\det(\bM)}.
\end{equation}
Now we are ready to prove the existence of the column ID.
\begin{proof}[of Theorem~\ref{theorem:interpolative-decomposition}]
We have mentioned above the proof relies on the Cramer's rule. If we can show the entries of $\bW$ can be denoted by the Cramer's rule equality in Equation~\eqref{equation:cramer-rule-general} and the numerator is smaller than the denominator, then we can complete the proof. However, we notice that the matrix in the denominator of Equation~\eqref{equation:cramer-rule-general} is a square matrix. Here comes the trick.

\paragraph{Step 1: column ID for full row rank matrix}
For a start, we first consider the full row rank matrix $\bA$ (which implies $r=m$, $m\leq n$, and $\bA\in \real^{r\times n}$ such that the matrix $\bC\in \real^{r\times r}$ is a square matrix in the column ID $\bA=\bC\bW$ that we want). Determine the ``skeleton" index vector $J_s$ by
\begin{equation}\label{equation:interpolative-choose-js}
\boxed{	J_s = \mathop{\arg\max}_{J} \left\{|\det(\bA[:,J])|: \text{$J$ is a subset of $\{1,2,\ldots, n\}$ with size $r=m$} \right\},}
\end{equation}
i.e., $J_s$ is the index vector that is determined by maximizing the magnitude of the determinant of $\bA[:,J]$. As we have discussed in the last section, there exists a (column) permutation matrix such that
$$
\bA\bP =
\begin{bmatrix}
	\bA[:,J_s]&\bA[:,J_r]
\end{bmatrix}.
$$
Since $\bC=\bA[:,J_s]$ has full column rank $r=m$, it is then nonsingular. The above equation can be reformulated as
$$
\begin{aligned}
\bA
&=\begin{bmatrix}
	\bA[:,J_s] &\bA[:,J_r]
\end{bmatrix}\bP^\top\\
&=
\bA[:,J_s]
\bigg[
	\bI_r \gap \bA[:,J_s]^{-1}\bA[:,J_r]
\bigg]
\bP^\top\\
&= \bC
\underbrace{\begin{bmatrix}
		\bI_r & \bC^{-1}\bA[:,J_r]
	\end{bmatrix}
	\bP^\top}_{\bW}
\end{aligned},
$$
where the matrix $\bW$ is given by
$
\begin{bmatrix}
	\bI_r & \bC^{-1}\bA[:,J_r]
\end{bmatrix}\bP^\top
=
\begin{bmatrix}
	\bI_r & \bE
\end{bmatrix}\bP^\top
$ by Equation~\eqref{equation:interpolatibve-w-ep}. To prove the claim that the magnitude of $\bW$ is no larger than 1 is equivalent to proving that entries in $\bE=\bC^{-1}\bA[:,J_r]\in \real^{r\times (n-r)}$ are no greater than 1 in absolute value.

Define the index vector $[j_1,j_2,\ldots, j_n]$ as a permutation of $[1,2,\ldots, n]$ such that
$$
[j_1,j_2,\ldots, j_n] = [1,2,\ldots, n] \bP = [J_s, J_r].\footnote{Note here $[j_1,j_2,\ldots, j_n] $, $[1,2,\ldots, n]$, $J_s$, and $J_r$ are row vectors.}
$$
Thus, it follows from $\bC\bE=\bA[:,J_r]$ that
$$
\begin{aligned}
\underbrace{	[\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]}_{=\bC=\bA[:,J_s]} \bE &=
\underbrace{[\ba_{j_{r+1}}, \ba_{j_{r+2}}, \ldots, \ba_{j_n}]}_{=\bA[:,J_r]:=\bB},
\end{aligned}
$$
where $\ba_i$ is the $i$-th column of $\bA$ and let $\bB=\bA[:,J_r]$.
Therefore, by Cramer's rule in Equation~\eqref{equation:cramer-rule-general}, we have
\begin{equation}\label{equation:column-id-expansionmatrix}
\bE_{kl} =
\frac{\det\left(\bC_{\bB}(k,l)\right)}
{\det\left(\bC\right)},
\end{equation}
where $\bE_{kl}$ is the entry ($k,l$) of $\bE$ and $\bC_{\bB}(k,l)$ is the $r\times r$ matrix formed by replacing the $k$-th column of $\bC$ with the $l$-th column of $\bB$. For example,
$$
\begin{aligned}
\bE_{11} &=
\frac{\det\left([\textcolor{blue}{\ba_{j_{r+1}}}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)}
{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)},
\qquad
&\bE_{12} &=
\frac{\det\left([\textcolor{blue}{\ba_{j_{r+2}}}, \ba_{j_2},\ldots, \ba_{j_r}]\right)}
{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)},\\
\bE_{21} &=
\frac{\det\left([\ba_{j_1},\textcolor{blue}{\ba_{j_{r+1}}}, \ldots, \ba_{j_r}]\right)}
{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)},
\qquad
&\bE_{22} &=
\frac{\det\left([\ba_{j_1},\textcolor{blue}{\ba_{j_{r+2}}}, \ldots, \ba_{j_r}]\right)}
{\det\left([\ba_{j_1}, \ba_{j_2}, \ldots, \ba_{j_r}]\right)}.
\end{aligned}
$$
Since $J_s$ is chosen to maximize the magnitude of $\det(\bC)$ in Equation~\eqref{equation:interpolative-choose-js}, it follows that
$$
|\bE_{kl}|\leq 1, \qquad \text{for all}\gap k\in \{1,2,\ldots, r\}, l\in \{1,2,\ldots, n-r\}.
$$

\paragraph{Step 2: apply to general matrices}
To summarize what we have proved above and to abuse the notation. For any matrix $\bF\in \real^{r\times n}$ with \textbf{full} rank $r\leq n$, the column ID exists that $\bF=\bC_0\bW$ where the values in $\bW$ are no greater than 1 in absolute value.

Apply the finding to the full general matrix $\bA\in \real^{m\times n}$ with rank $r\leq \{m,n\}$, it is trivial that the matrix $\bA$ admits a rank decomposition (Theorem~\ref{theorem:rank-decomposition}, p.~\pageref{theorem:rank-decomposition}):
$$
\underset{m\times n}{\bA} = \underset{m\times r}{\bD}\gap \underset{r\times n}{\bF},
$$
where $\bD,\bF$ have full column rank $r$ and full row rank $r$ respectively. For the column ID of $\bF=\bC_0\bW$ where $\bC_0=\bF[:,J_s]$ contains $r$ linearly independent columns of $\bF$. We notice by $\bA=\bD\bF$ such that
$$
\bA[:,J_s]=\bD\bF[:,J_s],
$$
i.e., the columns indexed by $J_s$ of $(\bD\bF)$ can be obtained by $\bD\bF[:,J_s]$ which in turn are the columns of $\bA$ indexed by $J_s$. This makes
$$
\underbrace{\bA[:,J_s]}_{\bC}= \underbrace{\bD\bF[:,J_s]}_{\bD\bC_0},
$$
And
$$
\bA = \bD\bF =\bD\bC_0\bW = \underbrace{\bD\bF[:,J_s]}_{\bC}\bW=\bC\bW.
$$
This completes the proof.
\end{proof}

The above proof reveals an intuitive way to compute the optimal column ID of matrix $\bA$. However, any algorithm that is guaranteed to find such an optimally-conditioned factorization must have combinatorial complexity \citep{martinsson2019randomized, lu2022bayesian, lu2022comparative}. Therefore, randomized algorithms, approximation by column-pivoted QR (Section~\ref{section:cpqr}, p.~\pageref{section:cpqr}) and rank-revealing QR (Section~\ref{section:rank-r-qr}, p.~\pageref{section:rank-r-qr}) are applied to find a relatively well-conditioned decomposition for the column ID where $\bW$ is small in norm rather than having entries all smaller than 1 in magnitude. While, the Bayesian approach can constrain the components in $\bW$ in the range of -1 to 1 strictly \citep{lu2022bayesian, lu2022comparative}. And we shall not go into the details.

\section{Row ID and Two-Sided ID}
We term the decomposition above as column ID. This is no coincidence since it has its siblings:

\begin{theoremHigh}[The Whole Interpolative Decomposition]\label{theorem:interpolative-decomposition-row}
Any rank-$r$ matrix $\bA \in \real^{m \times n}$ can be factored as
$$
\begin{aligned}
\text{Column ID: }&\gap \underset{m \times n}{\bA} &=& \boxed{\underset{m\times r}{\bC}} \gap  \underset{r\times n}{\bW} ; \\
\text{Row ID: } &\gap &=&\underset{m\times r}{\bZ} \gap  \boxed{\underset{r\times n}{\bR}}; \\
\text{Two-Sided ID: } &\gap &=&\underset{m\times r}{\bZ} \gap \boxed{\underset{r\times r}{\bU}}  \gap  \underset{r\times n}{\bW}, \\
\end{aligned}
$$
where
\begin{itemize}
\item $\bC=\bA[:,J_s]\in \real^{m\times r}$ is some $r$ linearly independent columns of $\bA$, $\bW\in \real^{r\times n}$ is the matrix to reconstruct $\bA$ which contains an $r\times r$ identity submatrix (under a mild column permutation): $\bW[:,J_s]=\bI_r$;
\item $\bR=\bA[I_s,:]\in \real^{r\times n}$ is some $r$ linearly independent rows of $\bR$, $\bZ\in \real^{m\times r}$ is the matrix to reconstruct $\bA$ which contains an $r\times r$ identity submatrix (under a mild row permutation): $\bZ[I_s,:]=\bI_r$;
\item Entries in $\bW, \bZ$ have values no larger than 1 in magnitude: $\max |w_{ij}|\leq 1$ and $\max |z_{ij}|\leq 1$;
\item $\bU=\bA[I_s,J_s] \in \real^{r\times r}$ is the nonsingular submatrix on the intersection of $\bC,\bR$;
\item The three matrices $\bC,\bR,\bU$ in the $\boxed{\text{boxed}}$ texts share same notation as the skeleton decomposition (Theorem~\ref{theorem:skeleton-decomposition}, p.~\pageref{theorem:skeleton-decomposition}) where they even have same meanings such that the three matrices make the skeleton decomposition of $\bA$: $\bA=\bC\bU^{-1}\bR$.
\end{itemize}
\end{theoremHigh}
The proof of the row ID is just similar to that of the column ID. Suppose the column ID of $\bA^\top$ is given by $\bA^\top=\bC_0\bW_0$ where $\bC_0$ contains $r$ linearly independent columns of $\bA^\top$ (i.e., $r$ linearly independent rows of $\bA$). Let $\bR=\bC_0, \bZ=\bW_0$, the row ID is obtained by $\bA=\bZ\bR$.

For the two-sided ID, recall from the skeleton decomposition (Theorem~\ref{theorem:skeleton-decomposition}, p.~\pageref{theorem:skeleton-decomposition}). When $\bU$ is the intersection of $\bC, \bR$, it follows that $\bA=\bC\bU^{-1}\bR$. Thus $\bC\bU^{-1}=\bZ$ by the row ID. And this implies $\bC=\bZ\bU$. By column ID, it follows that $\bA=\bC\bW=\bZ\bU\bW$ which proves the existence of the two-sided ID.

\index{Data storage}
\paragraph{Data storage} For the data storage of each ID, we summarize as follows
\begin{itemize}
\item \textit{Column ID.} It requires $mr$ and $(n-r)r$ floats to store $\bC$ and $\bW$ respectively , and $r$ integers to store the indices of the selected columns in $\bA$;
\item \textit{Row ID.} It requires $nr$ and $(m-r)r$ floats to store $\bR$ and $\bZ$ respectively, and $r$ integers to store the indices of the selected rows in $\bA$;
\item \textit{Two-Sided ID.} It requires $(m-r)r$, $(n-r)r$, and $r^2$ floats to store $\bZ,\bW$, and $\bU$ respectively. And extra $2r$ integers are required to store the indices of the selected rows and columns in $\bA$.
\end{itemize}

\paragraph{Further reduction on the storage for two-sided ID for sparse matrix $\bA$}
Suppose the column ID of $\bA=\bC\bW$ where $\bC=\bA[:,J_s]$ and a good spanning rows index $I_s$ set of $\bC$ could be found:
$$
\bA[I_s,:] = \bC[I_s,:]\bW.
$$
We observe that $\bC[I_s,:] = \bA[I_s,J_s]\in \real^{r\times r}$ which is nonsingular (since full rank $r$ in the sense of both row rank and column rank). It follows that
$$
\bW = (\bA[I_s,J_s])^{-1} \bA[I_s,:].
$$
Therefore, there is no need to store the matrix $\bW$ explicitly. We only need to store $\bA[I_s,:]$ and $(\bA[I_s,J_s])^{-1}$. Or when we can compute the inverse of $\bA[I_s,J_s]$ on the fly, it only requires $r$ integers to store $J_s$ and recover $\bA[I_s,J_s]$ from $\bA[I_s,:]$. The storage of $\bA[I_s,:]$ is cheap if $\bA$ is sparse.

\chapter{Spectral Decomposition (Theorem)}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Spectral Decomposition (Theorem)}\label{section:spectral-decomposition}

\begin{theoremHigh}[Spectral Decomposition]\label{theorem:spectral_theorem}
A real matrix $\bA \in \real^{n\times n}$ is symmetric if and only if there exists an orthogonal matrix $\bQ$ and a diagonal matrix $\bLambda$ such that
\begin{equation*}
\bA = \bQ \bLambda \bQ^\top,
\end{equation*}
where the columns of $\bQ = [\bq_1, \bq_2, \ldots, \bq_n]$ are eigenvectors of $\bA$ and are mutually orthonormal, and the entries of $\bLambda=diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$ are the corresponding eigenvalues of $\bA$, which are real. And the rank of $\bA$ is the number of nonzero eigenvalues. This is known as the \textit{spectral decomposition} or \textit{spectral theorem} of real symmetric matrix $\bA$. Specifically, we have the following properties:

1. A symmetric matrix has only \textbf{real eigenvalues};

2. The eigenvectors are orthogonal such that they can be chosen \textbf{orthonormal} by normalization;

3. The rank of $\bA$ is the number of nonzero eigenvalues;

4. If the eigenvalues are distinct, the eigenvectors are unique as well.
\end{theoremHigh}

The above decomposition is called the spectral decomposition for real symmetric matrices and is often known as the \textit{spectral theorem}.
\paragraph{Spectral theorem vs eigenvalue decomposition} In the eigenvalue decomposition, we require the matrix $\bA$ to be square and the eigenvectors to be linearly independent. Whereas in the spectral theorem, any symmetric matrix can be diagonalized, and the eigenvectors are chosen to be orthonormal.

\paragraph{A word on the spectral decomposition} In Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}), we proved that the eigenvalues of similar matrices are the same. From the spectral decomposition, we notice that $\bA$ and $\bLambda$ are similar matrices such that their eigenvalues are the same. For any diagonal matrices, the eigenvalues are the diagonal components. \footnote{Actually, we have shown in the last section that the diagonal values for triangular matrices are the eigenvalues of it.} To see this, we realize that
$$
\bLambda \be_i = \lambda_i \be_i,
$$
where $\be_i$ is the $i$-th basis vector. Therefore, the matrix $\Lambda$ contains the eigenvalues of $\bA$.

\section{Existence of the Spectral Decomposition}\label{section:existence-of-spectral}

We prove the theorem in several steps.

\begin{tcolorbox}[title={Symmetric Matrix Property 1 of 4}]
\begin{lemma}[Real Eigenvalues]\label{lemma:real-eigenvalues-spectral}
The eigenvalues of any symmetric matrix are all real.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:real-eigenvalues-spectral}]
	Suppose eigenvalue $\lambda$ is a complex number $\lambda=a+ib$ where $a,b$ are real. Its complex conjugate is $\bar{\lambda}=a-ib$. Same for complex eigenvector $\bx = \bc+i\bd$ and its complex conjugate $\bar{\bx}=\bc-i\bd$ where $\bc, \bd$ are real vectors. We then have the following property
	$$
	\bA \bx = \lambda \bx\qquad   \underrightarrow{\text{ leads to }}\qquad  \bA \bar{\bx} = \bar{\lambda} \bar{\bx}\qquad   \underrightarrow{\text{ transpose to }}\qquad  \bar{\bx}^\top \bA =\bar{\lambda} \bar{\bx}^\top.
	$$
	We take the dot product of the first equation with $\bar{\bx}$ and the last equation with $\bx$:
	$$
	\bar{\bx}^\top \bA \bx = \lambda \bar{\bx}^\top \bx, \qquad \text{and } \qquad \bar{\bx}^\top \bA \bx = \bar{\lambda}\bar{\bx}^\top \bx.
	$$
	Then we have the equality $\lambda\bar{\bx}^\top \bx = \bar{\lambda} \bar{\bx}^\top\bx$. Since $\bar{\bx}^\top\bx = (\bc-i\bd)^\top(\bc+i\bd) = \bc^\top\bc+\bd^\top\bd$ is a real number. Therefore the imaginary part of $\lambda$ is zero and $\lambda$ is real.
\end{proof}

\begin{tcolorbox}[title={Symmetric Matrix Property 2 of 4}]
\begin{lemma}[Orthogonal Eigenvectors]\label{lemma:orthogonal-eigenvectors}
The eigenvectors  corresponding to distinct eigenvalues of any symmetric matrix are orthogonal so that we can normalize eigenvectors to make them orthonormal since $\bA\bx = \lambda \bx \underrightarrow{\text{ leads to } } \bA\frac{\bx}{||\bx||} = \lambda \frac{\bx}{||\bx||}$ which corresponds to the same eigenvalue.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:orthogonal-eigenvectors}]
	Suppose eigenvalues $\lambda_1, \lambda_2$ correspond to eigenvectors $\bx_1, \bx_2$ so that $\bA\bx_1=\lambda \bx_1$ and $\bA\bx_2 = \lambda_2\bx_2$. We have the following equality:
	$$
	\bA\bx_1=\lambda_1 \bx_1 \leadto \bx_1^\top \bA =\lambda_1 \bx_1^\top \leadto \bx_1^\top \bA \bx_2 =\lambda_1 \bx_1^\top\bx_2,
	$$
	and
	$$
	\bA\bx_2 = \lambda_2\bx_2 \leadto  \bx_1^\top\bA\bx_2 = \lambda_2\bx_1^\top\bx_2,
	$$
	which implies $\lambda_1 \bx_1^\top\bx_2=\lambda_2\bx_1^\top\bx_2$. Since eigenvalues $\lambda_1\neq \lambda_2$, the eigenvectors are orthogonal.
\end{proof}

In the above Lemma~\ref{lemma:orthogonal-eigenvectors}, we prove that the eigenvectors corresponding to distinct eigenvalues  of symmetric matrices are orthogonal. More generally, we prove the important theorem that eigenvectors corresponding to distinct eigenvalues of any matrix are linearly independent.
\begin{theorem}[Independent Eigenvector Theorem]\label{theorem:independent-eigenvector-theorem}
If a matrix $\bA\in \real^{n\times n}$ has $k$ distinct eigenvalues, then any set of $k$ corresponding eigenvectors are linearly independent.
\end{theorem}
\begin{proof}[of Theorem~\ref{theorem:independent-eigenvector-theorem}]
We will prove by induction. Firstly, we will prove that any two eigenvectors corresponding to distinct eigenvalues are linearly independent. Suppose $\bv_1,\bv_2$ correspond to distinct eigenvalues $\lambda_1$ and $\lambda_2$ respectively. Suppose further there exists a nonzero vector $\bx=[x_1,x_2] \neq \bzero $ that
\begin{equation}\label{equation:independent-eigenvector-eq1}
x_1\bv_1+x_2\bv_2=\bzero.
\end{equation}
That is, $\bv_1,\bv_2$ are linearly independent.
Multiply Equation~\eqref{equation:independent-eigenvector-eq1} on the left by $\bA$, we get
\begin{equation}\label{equation:independent-eigenvector-eq2}
x_1 \lambda_1\bv_1 + x_2\lambda_2\bv_2 = \bzero.
\end{equation}
Multiply Equation~\eqref{equation:independent-eigenvector-eq1} on the left by $\lambda_2$, we get
\begin{equation}\label{equation:independent-eigenvector-eq3}
x_1\lambda_2\bv_1 + x_2\lambda_2\bv_2 = \bzero.
\end{equation}
Subtract Equation~\eqref{equation:independent-eigenvector-eq2} from Equation~\eqref{equation:independent-eigenvector-eq3} to find
$$
x_1(\lambda_2-\lambda_1)\bv_1 = \bzero.
$$
Since $\lambda_2\neq \lambda_1$, $\bv_1\neq \bzero$, we must have $x_1=0$. From Equation~\eqref{equation:independent-eigenvector-eq1}, $\bv_2\neq \bzero$, we must also have $x_2=0$ which arrives at a contradiction. Thus $\bv_1,\bv_2$ are linearly independent.

Now, suppose any $j<k$ eigenvectors are linearly independent, if we could prove that any $j+1$ eigenvectors are also linearly independent, we finish the proof. Suppose $\bv_1, \bv_2, \ldots, \bv_j$ are linearly independent and $\bv_{j+1}$ is dependent on the first $j$ eigenvectors. That is, there exists a nonzero vector $\bx=[x_1,x_2,\ldots, x_{j}]\neq \bzero$ that
\begin{equation}\label{equation:independent-eigenvector-zero}
\bv_{j+1}=	x_1\bv_1+x_2\bv_2+\ldots+x_j\bv_j .
\end{equation}
Suppose the $j+1$ eigenvectors correspond to distinct eigenvalues $\lambda_1,\lambda_2,\ldots,\lambda_j,\lambda_{j+1}$.
Multiply Equation~\eqref{equation:independent-eigenvector-zero} on the left by $\bA$, we get
\begin{equation}\label{equation:independent-eigenvector-zero2}
\lambda_{j+1} \bv_{j+1} = x_1\lambda_1\bv_1+x_2\lambda_2\bv_2+\ldots+x_j \lambda_j\bv_j .
\end{equation}
Multiply Equation~\eqref{equation:independent-eigenvector-zero} on the left by $\lambda_{j+1}$, we get
\begin{equation}\label{equation:independent-eigenvector-zero3}
	\lambda_{j+1} \bv_{j+1} = x_1\lambda_{j+1}\bv_1+x_2\lambda_{j+1}\bv_2+\ldots+x_j \lambda_{j+1}\bv_j .
\end{equation}
Subtract Equation~\eqref{equation:independent-eigenvector-zero3} from Equation~\eqref{equation:independent-eigenvector-zero2}, we find
$$
x_1(\lambda_{j+1}-\lambda_1)\bv_1+x_2(\lambda_{j+1}-\lambda_2)\bv_2+\ldots+x_j (\lambda_{j+1}-\lambda_j)\bv_j = \bzero.
$$
From assumption, $\lambda_{j+1} \neq \lambda_i$ for all $i\in \{1,2,\ldots,j\}$, and $\bv_i\neq \bzero$ for all $i\in \{1,2,\ldots,j\}$. We must have $x_1=x_2=\ldots=x_j=0$ which leads to a contradiction. Then $\bv_1,\bv_2,\ldots,\bv_j,\bv_{j+1}$ are linearly independent. This completes the proof.
\end{proof}

\begin{corollary}[Independent Eigenvector Theorem, CNT.]\label{theorem:independent-eigenvector-theorem-basis}
If a matrix $\bA\in \real^{n\times n}$ has $n$ distinct eigenvalues, then any set of $n$ corresponding eigenvectors form a basis for $\real^n$.
\end{corollary}

\begin{tcolorbox}[title={Symmetric Matrix Property 3 of 4}]
\begin{lemma}[Orthonormal Eigenvectors for Duplicate Eigenvalue]\label{lemma:eigen-multiplicity}
If $\bA$ has a duplicate eigenvalue $\lambda_i$ with multiplicity $k\geq 2$, then there exist $k$ orthonormal eigenvectors corresponding to $\lambda_i$.
\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:eigen-multiplicity}]
We note that there is at least one eigenvector $\bx_{i1}$ corresponding to $\lambda_i$. And for such eigenvector $\bx_{i1}$, we can always find additional $n-1$ orthonormal vectors $\by_2, \by_3, \ldots, \by_n$ so that $\{\bx_{i1}, \by_2, \by_3, \ldots, \by_n\}$ forms an orthonormal basis in $\real^n$. Put the $\by_2, \by_3, \ldots, \by_n$ into matrix $\bY_1$ and $\{\bx_{i1}, \by_2, \by_3, \ldots, \by_n\}$ into matrix $\bP_1$
$$
\bY_1=[\by_2, \by_3, \ldots, \by_n] \qquad \text{and} \qquad \bP_1=[\bx_{i1}, \bY_1].
$$
We then have
$$
\bP_1^\top\bA\bP_1 = \begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bY_1^\top \bA\bY_1
\end{bmatrix}.
$$
As a result, $\bA$ and $\bP_1^\top\bA\bP_1$ are similar matrices such that they have the same eigenvalues since $\bP_1$ is nonsingular (even orthogonal here, see Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}). We obtain
$$
\det(\bP_1^\top\bA\bP_1 - \lambda\bI_n) =
\footnote{By the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
		\bA & \bB \\
		\bC & \bD
	\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$.
}
(\lambda_i - \lambda )\det(\bY_1^\top \bA\bY_1 - \lambda\bI_{n-1}).
$$
If $\lambda_i$ has multiplicity $k\geq 2$, then the term $(\lambda_i-\lambda)$ occurs $k$ times in the polynomial from the determinant $\det(\bP_1^\top\bA\bP_1 - \lambda\bI_n)$, i.e., the term occurs $k-1$ times in the polynomial from $\det(\bY_1^\top \bA\bY_1 - \lambda\bI_{n-1})$. In another word, $\det(\bY_1^\top \bA\bY_1 - \lambda_i\bI_{n-1})=0$ and $\lambda_i$ is an eigenvalue of $\bY_1^\top \bA\bY_1$.

Let $\bB=\bY_1^\top \bA\bY_1$. Since $\det(\bB-\lambda_i\bI_{n-1})=0$, the null space of $\bB-\lambda_i\bI_{n-1}$ is not none. Suppose $(\bB-\lambda_i\bI_{n-1})\bn = \bzero$, i.e., $\bB\bn=\lambda_i\bn$ and $\bn$ is an eigenvector of $\bB$.

From $
\bP_1^\top\bA\bP_1 = \begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bB
\end{bmatrix},
$
we have $
	\bA\bP_1
	\begin{bmatrix}
		z \\
		\bn
	\end{bmatrix}
	=
	\bP_1
	\begin{bmatrix}
		\lambda_i &\bzero \\
		\bzero & \bB
	\end{bmatrix}
	\begin{bmatrix}
		z \\
		\bn
	\end{bmatrix}$, where $z$ is any scalar. From the left side of this equation, we have
\begin{equation}\label{equation:spectral-pro4-right}
	\begin{aligned}
		\bA\bP_1
		\begin{bmatrix}
			z \\
			\bn
		\end{bmatrix}
		&=
		\begin{bmatrix}
			\lambda_i\bx_{i1}, \bA\bY_1
		\end{bmatrix}
		\begin{bmatrix}
			z \\
			\bn
		\end{bmatrix} \\
		&=\lambda_iz\bx_{i1} + \bA\bY_1\bn.
	\end{aligned}
\end{equation}
And from the right side of the equation, we have
\begin{equation}\label{equation:spectral-pro4-left}
\begin{aligned}
\bP_1
\begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bB
\end{bmatrix}
\begin{bmatrix}
	z \\
	\bn
\end{bmatrix}
&=
\begin{bmatrix}
	\bx_{i1} & \bY_1
\end{bmatrix}
\begin{bmatrix}
	\lambda_i &\bzero \\
	\bzero & \bB
\end{bmatrix}
\begin{bmatrix}
	z \\
	\bn
\end{bmatrix}\\
&=
\begin{bmatrix}
\lambda_i\bx_{i1} & \bY_1\bB
\end{bmatrix}
\begin{bmatrix}
	z \\
	\bn
\end{bmatrix}\\
&= \lambda_i z \bx_{i1} + \bY_1\bB \bn \\
&=\lambda_i z \bx_{i1} + \lambda_i \bY_1 \bn.  \qquad (\text{Since $\bB \bn=\lambda_i\bn$})\\
\end{aligned}
\end{equation}
Combine Equation~\eqref{equation:spectral-pro4-left} and Equation~\eqref{equation:spectral-pro4-right}, we obtain
$$
\bA\bY_1\bn = \lambda_i\bY_1 \bn,
$$
which means $\bY_1\bn$ is an eigenvector of $\bA$ corresponding to the eigenvalue $\lambda_i$ (same eigenvalue corresponding to $\bx_{i1}$). Since $\bY_1\bn$ is a combination of $\by_2, \by_3, \ldots, \by_n$ which are orthonormal to $\bx_{i1}$, the $\bY_1\bn$ can be chosen to be orthonormal to $\bx_{i1}$.

To conclude, if we have one eigenvector $\bx_{i1}$ corresponding to $\lambda_i$ whose multiplicity is $k\geq 2$, we could construct the second eigenvector by choosing one vector from the null space of $(\bB-\lambda_i\bI_{n-1})$ constructed above. Suppose now, we have constructed the second eigenvector $\bx_{i2}$ which is orthonormal to $\bx_{i1}$.
For such eigenvectors $\bx_{i1}, \bx_{i2}$, we can always find additional $n-2$ orthonormal vectors $\by_3, \by_4, \ldots, \by_n$ so that $\{\bx_{i1},\bx_{i2}, \by_3, \by_4, \ldots, \by_n\}$ forms an orthonormal basis in $\real^n$. Put the $\by_3, \by_4, \ldots, \by_n$ into matrix $\bY_2$ and $\{\bx_{i1},\bx_{i2},  \by_3, \by_4, \ldots, \by_n\}$ into matrix $\bP_2$:
$$
\bY_2=[\by_3, \by_4, \ldots, \by_n] \qquad \text{and} \qquad \bP_2=[\bx_{i1}, \bx_{i2},\bY_1].
$$
We then have
$$
\bP_2^\top\bA\bP_2 =
\begin{bmatrix}
	\lambda_i & 0 &\bzero \\
	 0& \lambda_i &\bzero \\
	\bzero & \bzero & \bY_2^\top \bA\bY_2
\end{bmatrix}
=
\begin{bmatrix}
	\lambda_i & 0 &\bzero \\
	0& \lambda_i &\bzero \\
	\bzero & \bzero & \bC
\end{bmatrix},
$$
where $\bC=\bY_2^\top \bA\bY_2$ such that $\det(\bP_2^\top\bA\bP_2 - \lambda\bI_n) = (\lambda_i-\lambda)^2 \det(\bC - \lambda\bI_{n-2})$. If the multiplicity of $\lambda_i$ is $k\geq 3$, $\det(\bC - \lambda_i\bI_{n-2})=0$ and the null space of $\bC - \lambda_i\bI_{n-2}$ is not none so that we can still find a vector from the null space of $\bC - \lambda_i\bI_{n-2}$ and $\bC\bn = \lambda_i \bn$. Now we can construct a vector $\begin{bmatrix}
	z_1 \\
	z_2\\
	\bn
\end{bmatrix}\in \real^n $, where $z_1, z_2$ are any scalar values, such that
$$
\bA\bP_2\begin{bmatrix}
	z_1 \\
	z_2\\
	\bn
\end{bmatrix} = \bP_2
\begin{bmatrix}
	\lambda_i & 0 &\bzero \\
	0& \lambda_i &\bzero \\
	\bzero & \bzero & \bC
\end{bmatrix}
\begin{bmatrix}
	z_1 \\
	z_2\\
	\bn
\end{bmatrix}.
$$
Similarly, from the left side of the above equation, we will get $\lambda_iz_1\bx_{i1} +\lambda_iz_2\bx_{i2}+\bA\bY_2\bn$. From the right side of the above equation, we will get $\lambda_iz_1\bx_{i1} +\lambda_i z_2\bx_{i2}+\lambda_i\bY_2\bn$. As a result,
$$
\bA\bY_2\bn = \lambda_i\bY_2\bn,
$$
where $\bY_2\bn$ is an eigenvector of $\bA$ and orthogonal to $\bx_{i1}, \bx_{i2}$. And it is easy to construct the eigenvector to be orthonormal to the first two.

The process can go on, and finally, we will find $k$ orthonormal eigenvectors corresponding to $\lambda_i$.

Actually, the dimension of the null space of $\bP_1^\top\bA\bP_1 -\lambda_i\bI_n$ is equal to the multiplicity $k$. It also follows that if the multiplicity of $\lambda_i$ is $k$, there cannot be more than $k$ orthogonal eigenvectors corresponding to $\lambda_i$. Otherwise, it will come to the conclusion that we could find more than $n$ orthogonal eigenvectors which leads to a contradiction.
\end{proof}

The proof of the existence of the spectral decomposition is trivial from the lemmas above. Also, we can use Schur decomposition to prove its existence.
\begin{proof}[\textbf{of Theorem~\ref{theorem:spectral_theorem}: Existence of Spectral Decomposition}]
From the Schur decomposition in Theorem~\ref{theorem:schur-decomposition} (p.~\pageref{theorem:schur-decomposition}), symmetric matrix $\bA=\bA^\top$ leads to $\bQ\bU\bQ^\top = \bQ\bU^\top\bQ^\top$. Then $\bU$ is a diagonal matrix. And this diagonal matrix actually contains eigenvalues of $\bA$. All the columns of $\bQ$ are eigenvectors of $\bA$. We conclude that all symmetric matrices are diagonalizable even with repeated eigenvalues.
\end{proof}

For any matrix multiplication, we have the rank of the multiplication result no larger than the rank of the inputs. However, the symmetric matrix $\bA^\top \bA$ is rather special in that the rank of $\bA^\top \bA$ is equal to that of $\bA$ which will be used in the proof of singular value decomposition in the next section.
	\begin{lemma}[Rank of $\bA\bB$]\label{lemma:rankAB}
		For any matrix $\bA\in \real^{m\times n}$, $\bB\in \real^{n\times k}$, then the matrix multiplication $\bA\bB\in \real^{m\times k}$ has $rank$($\bA\bB$)$\leq$min($rank$($\bA$), $rank$($\bB$)).
	\end{lemma}

\begin{proof}[of Lemma~\ref{lemma:rankAB}]
	For matrix multiplication $\bA\bB$, we have

	$\bullet$ All rows of $\bA\bB$ are combinations of rows of $\bB$, the row space of $\bA\bB$ is a subset of the row space of $\bB$. Thus $rank$($\bA\bB$)$\leq$$rank$($\bB$).

	$\bullet$ All columns of $\bA\bB$ are combinations of columns of $\bA$, the column space of $\bA\bB$ is a subset of the column space of $\bA$. Thus $rank$($\bA\bB$)$\leq$$rank$($\bA$).

	Therefore, $rank$($\bA\bB$)$\leq$min($rank$($\bA$), $rank$($\bB$)).
\end{proof}
\begin{tcolorbox}[title={Symmetric Matrix Property 4 of 4}]
	\begin{lemma}[Rank of Symmetric Matrices]\label{lemma:rank-of-symmetric}
		If $\bA$ is an $n\times n$ real symmetric matrix, then $rank(\bA)$ =
		the total number of nonzero eigenvalues of $\bA$.
		In particular, $\bA$ has full rank if and only if $\bA$ is nonsingular. Further, $\cspace(\bA)$ is the linear space spanned by the eigenvectors of $\bA$ that correspond to nonzero eigenvalues.
	\end{lemma}
\end{tcolorbox}
\begin{proof}[of Lemma~\ref{lemma:rank-of-symmetric}]
	For any symmetric matrix $\bA$, we have $\bA$, in spectral form, as $\bA = \bQ \bLambda\bQ^\top$ and also $\bLambda = \bQ^\top\bA\bQ$. Since we have shown in Lemma~\ref{lemma:rankAB} that the rank of the multiplication $rank$($\bA\bB$)$\leq$min($rank$($\bA$), $rank$($\bB$)).

	$\bullet$ From $\bA = \bQ \bLambda\bQ^\top$, we have $rank(\bA) \leq rank(\bQ \bLambda) \leq rank(\bLambda)$;

	$\bullet$ From $\bLambda = \bQ^\top\bA\bQ$, we have $rank(\bLambda) \leq rank(\bQ^\top\bA) \leq rank(\bA)$,

	The inequalities above give us a contradiction. And thus $rank(\bA) = rank(\bLambda)$ which is the total number of nonzero eigenvalues.

	Since $\bA$ is nonsingular if and only if all of its eigenvalues are nonzero, $\bA$ has full rank if and only if $\bA$ is nonsingular.
\end{proof}

Similar to the eigenvalue decomposition, we can compute the $m$-th power of matrix $\bA$ more efficiently via the spectral decomposition.
\begin{remark}[$m$-th Power]\label{remark:power-spectral}
The $m$-th power of $\bA$ is $\bA^m = \bQ\bLambda^m\bQ^\top$ if the matrix $\bA$ can be factored as $\bA=\bQ\bLambda\bQ^\top$.
\end{remark}

\section{Uniqueness of Spectral Decomposition}\label{section:uniqueness-spectral-decomposition}
Clearly, the spectral decomposition is not unique essentially because of the multiplicity of eigenvalues. One can imagine that eigenvalue $\lambda_i$ and $\lambda_j$ are the same for some $1\leq i,j\leq n$, and interchange the corresponding eigenvectors in $\bQ$ will have the same results but the decompositions are different.
But the \textit{eigenspaces} (i.e., the null space $\nspace(\bA - \lambda_i\bI)$ for eigenvalue $\lambda_i$) corresponding to each eigenvalue are fixed. So there is a unique decomposition in terms
of eigenspaces and then any orthonormal basis of these eigenspaces can be chosen.

\section{Other Forms, Connecting Eigenvalue Decomposition*}\label{section:otherform-spectral}
In this section, we discuss other forms of the spectral decomposition under different conditions. To see this, we provide a rigorous definition of the characteristic polynomial.
\begin{definition}[Characteristic Polynomial\index{Characteristic polynomial}]
For any square matrix $\bA \in \real^{n\times n}$, the \textbf{characteristic polynomial} $\det(\bA - \lambda \bI)$ is given by
$$
\begin{aligned}
\det(\lambda\bI-\bA ) &=\lambda^n - \gamma_{n-1} \lambda^{n-1} + \ldots + \gamma_1 \lambda  + \gamma_0\\
&=(\lambda-\lambda_1)^{k_1} (\lambda-\lambda_2)^{k_2} \ldots (\lambda-\lambda_m)^{k_m},
\end{aligned}
$$
where $\lambda_1, \lambda_2, \ldots, \lambda_m$ are the distinct roots of $\det( \lambda\bI-\bA)$ and also the eigenvalues of $\bA$, and $k_1+k_2+\ldots +k_m=n$, i.e., $\det(\lambda\bI-\bA)$ is a polynomial of degree $n$ for any matrix $\bA\in \real^{n\times n}$ (see proof of Lemma~\ref{lemma:eigen-multiplicity}, p.~\pageref{lemma:eigen-multiplicity}).
\end{definition}

An important multiplicity arises from the characteristic polynomial of a matrix is then defined as follows:
\begin{definition}[Algebraic Multiplicity and Geometric Multiplicity\index{Algebraic multiplicity}\index{Geometric multiplicity}]
Given the characteristic polynomial of matrix $\bA\in \real^{n\times n}$:
$$
\begin{aligned}
\det(\lambda\bI-\bA ) =(\lambda-\lambda_1)^{k_1} (\lambda-\lambda_2)^{k_2} \ldots (\lambda-\lambda_m)^{k_m}.
\end{aligned}
$$
The integer $k_i$ is called the \textbf{algebraic multiplicity} of the eigenvalue $\lambda_i$, i.e., the algebraic multiplicity of eigenvalue $\lambda_i$ is equal to the multiplicity of the corresponding root of the characteristic polynomial.

The \textbf{eigenspace associated to eigenvalue $\lambda_i$} is defined by the null space of $(\bA - \lambda_i\bI)$, i.e., $\nspace(\bA - \lambda_i\bI)$.

And the dimension of the eigenspace associated with $\lambda_i$, $\nspace(\bA - \lambda_i\bI)$, is called the \textbf{geometric multiplicity} of $\lambda_i$.

In short, we denote the algebraic multiplicity of $\lambda_i$ by $alg(\lambda_i)$, and its geometric multiplicity by $geo(\lambda_i)$.
\end{definition}

\begin{remark}[Geometric Multiplicity]\label{remark:geometric-mul-meaning}
Note that for matrix $\bA$ and the eigenspace $\nspace(\bA-\lambda_i\bI)$, the dimension of the eigenspace is also the number of linearly independent eigenvectors of $\bA$ associated to $\lambda_i$, namely a basis for the  eigenspace. This implies that
while there are an infinite number of eigenvectors associated with each eigenvalue $\lambda_i$, the fact that they form a subspace (provided the zero vector is added) means that they can be described by a finite number of vectors.
\end{remark}

By definition, the sum of the algebraic multiplicities is equal to $n$, but the sum of the geometric multiplicities can be strictly smaller.

\begin{corollary}[Multiplicity in Similar Matrices\index{Similar matrices}]\label{corollary:multipli-similar-matrix}
Similar matrices have same algebraic multiplicities and geometric multiplicities.
\end{corollary}
\begin{proof}[of Corollary~\ref{corollary:multipli-similar-matrix}]
In Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}), we proved that the eigenvalues of similar matrices are the same, therefore, the algebraic multiplicities of similar matrices are the same as well.

Suppose $\bA$ and $\bB= \bP\bA\bP^{-1}$ are similar matrices where $\bP$ is nonsingular. And the geometric multiplicity of an eigenvalue of $\bA$, say $\lambda$, is $k$. Then there exists a set of orthogonal vectors $\bv_1, \bv_2, \ldots, \bv_k$ that are the basis for the eigenspace $\nspace(\bA-\lambda\bI)$ such that $\bA\bv_i = \lambda \bv_i$ for all $i\in \{1, 2, \ldots, k\}$. Then, $\bw_i = \bP\bv_i$'s are the eigenvectors of $\bB$ associated with eigenvalue $\lambda$. Further, $\bw_i$'s are linearly independent since $\bP$ is nonsingular. Thus, the dimension of the eigenspace $\nspace(\bB-\lambda_i\bI)$ is at least $k$, that is, $dim(\nspace(\bA-\lambda\bI)) \leq dim(\nspace(\bB-\lambda\bI)) $.

Similarly, there exists a set of orthogonal vectors $\bw_1, \bw_2, \ldots, \bw_k$ that are the bases for the eigenspace $\nspace(\bB-\lambda\bI)$, then $\bv_i = \bP^{-1}\bw_i$ for all $i \in \{1, 2, \ldots, k\}$ are the eigenvectors of $\bA$ associated to $\lambda$. This will result in $dim(\nspace(\bB-\lambda\bI)) \leq dim(\nspace(\bA-\lambda\bI)) $.

Therefore, by ``sandwiching", we get $dim(\nspace(\bA-\lambda\bI)) = dim(\nspace(\bB-\lambda\bI)) $, which is the equality of the geometric multiplicities, and the claim follows.
\end{proof}

\begin{lemma}[Bounded Geometric Multiplicity]\label{lemma:bounded-geometri}
For any matrix $\bA\in \real^{n\times n}$, its geometric multiplicity is bounded by algebraic multiplicity for any eigenvalue $\lambda_i$:
$$
geo(\lambda_i) \leq alg(\lambda_i).
$$
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:bounded-geometri}]
If we can find a similar matrix $\bB$ of $\bA$ that has a specific form of the characteristic polynomial, then we complete the proof.

Suppose $\bP_1 = [\bv_1, \bv_2, \ldots, \bv_k]$ contains the eigenvectors of $\bA$ associated with $\lambda_i$ which are linearly independent. That is, the $k$ vectors are bases for the eigenspace $\nspace(\bA-\lambda\bI)$ and the geometric multiplicity associated with $\lambda_i$ is $k$. We can expand it to $n$ linearly independent vectors such that
$$
\bP = [\bP_1, \bP_2] = [\bv_1, \bv_2, \ldots, \bv_k, \bv_{k+1}, \ldots, \bv_n],
$$
where $\bP$ is nonsingular. Then $\bA\bP = [\lambda_i\bP_1, \bA\bP_2]$.

Construct a matrix $\bB = \begin{bmatrix}
\lambda_i \bI_k & \bC \\
\bzero  & \bD
\end{bmatrix}$ where $\bA\bP_2 = \bP_1\bC + \bP_2\bD$, then $\bP^{-1}\bA\bP = \bB$ such that $\bA$ and $\bB$ are similar matrices. We can always find such $\bC,\bD$ that satisfy the above condition, since $\bv_i$'s are linearly independent with spanning the whole space $\real^n$, and any column of $\bA\bP_2$ is in the column space of $\bP=[\bP_1,\bP_2]$.
Therefore,
$$
\begin{aligned}
\det(\bA-\lambda\bI) &= \det(\bP^{-1})\det(\bA-\lambda\bI)\det(\bP)  \qquad &(\text{$\det(\bP^{-1}) = 1/\det(\bP)$})\\
&= \det(\bP^{-1}(\bA-\lambda\bI)\bP)  \qquad &(\det(\bA)\det(\bB) = \det(\bA\bB))\\
&=  \det(\bB-\lambda\bI) \\
&= \det(\begin{bmatrix}
	(\lambda_i-\lambda) \bI_k  & \bC \\
	\bzero  & \bD - \lambda \bI
\end{bmatrix})\\
&= (\lambda_i-\lambda)^k \det(\bD-\lambda\bI),
\end{aligned}
$$
where the last equality is from the fact that if matrix $\bM$ has a block formulation: $\bM=\begin{bmatrix}
	\bA & \bB \\
	\bC & \bD
\end{bmatrix}$, then $\det(\bM) = \det(\bA)\det(\bD-\bC\bA^{-1}\bB)$.
This implies
$$
geo(\lambda_i) \leq alg(\lambda_i).
$$
And we complete the proof.
\end{proof}

Following from the proof of Lemma~\ref{lemma:eigen-multiplicity} (p.~\pageref{lemma:eigen-multiplicity}), we notice that the algebraic multiplicity and geometric multiplicity are the same for symmetric matrices. We call these matrices simple matrices.

\begin{definition}[Simple Matrix]
When the algebraic multiplicity and geometric multiplicity are the same for a matrix, we call it a simple matrix.
\end{definition}

\begin{definition}[Diagonalizable]
A matrix $\bA$ is diagonalizable if there exists a nonsingular matrix $\bP$ and a diagonal matrix $\bD$ such that $\bA = \bP\bD\bP^{-1}$.
\end{definition}
Eigenvalue decomposition in Theorem~\ref{theorem:eigenvalue-decomposition} (p.~\pageref{theorem:eigenvalue-decomposition}) and spectral decomposition in Theorem~\ref{theorem:spectral_theorem} (p.~\pageref{theorem:spectral_theorem}) are such kinds of matrices that are diagonalizable.
\begin{lemma}[Simple Matrices are Diagonalizable]\label{lemma:simple-diagonalizable}
A matrix is a simple matrix if and only if it is diagonalizable.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:simple-diagonalizable}]
We will show by forward implication and backward implication separately as follows.
\paragraph{Forward implication} Suppose that $\bA\in \real^{n\times n}$ is a simple matrix, such that the algebraic and geometric multiplicities for each eigenvalue are equal. For a specific eigenvalue $\lambda_i$, let $\{\bv_1^i, \bv_2^i, \ldots, \bv_{k_i}^i\}$ be a basis for the eigenspace $\nspace(\bA - \lambda_i\bI)$, that is, $\{\bv_1^i, \bv_2^i, \ldots, \bv_{k_i}^i\}$ is a set of linearly independent eigenvectors of $\bA$ associated to $\lambda_i$, where ${k_i}$ is the algebraic or geometric multiplicity associated to $\lambda_i$: $alg(\lambda_i)=geo(\lambda_i)=k_i$. Suppose there are $m$ distinct eigenvalues, since $k_1+k_2+\ldots +k_m = n$, the set of eigenvectors consists of the union of $n$ vectors. Suppose there is a set of $x_j$'s such that
\begin{equation}\label{equation:proof-simple-diagonalize}
	\bz = \sum_{j=1}^{k_1} x_j^1 \bv_j^1+ \sum_{j=1}^{k_2} x_j^2 \bv_j^2 + \ldots \sum_{j=1}^{k_m} x_j^m \bv_j^m = \bzero.
\end{equation}
Let $\bw^i = \sum_{j=1}^{k_i} x_j^i \bv_j^i$. Then $\bw^i$ is either an eigenvector associated with $\lambda_i$, or it is a zero vector. That is $\bz = \sum_{i=1}^{m} \bw^i$ is a sum of either zero vector or an eigenvector associated with different eigenvalues of $\bA$. Since eigenvectors associated with different eigenvalues are linearly independent. We must have $\bw^i =0$ for all $i\in \{1, 2, \ldots, m\}$. That is
$$
\bw^i = \sum_{j=1}^{k_i} x_j^i \bv_j^i = \bzero, \qquad \text{for all $i\in \{1, 2, \ldots, m\}$}.
$$
Since we assume the eigenvectors $\bv_j^i$'s associated with $\lambda_i$ are linearly independent, we must have $x_j^i=0$ for all $i \in \{1,2,\ldots, m\}, j\in \{1,2,\ldots,k_i\}$. Thus, the $n$ vectors are linearly independent:
$$
\{\bv_1^1, \bv_2^1, \ldots, \bv_{k_i}^1\},\{\bv_1^2, \bv_2^2, \ldots, \bv_{k_i}^2\},\ldots,\{\bv_1^m, \bv_2^m, \ldots, \bv_{k_i}^m\}.
$$
By eigenvalue decomposition in Theorem~\ref{theorem:eigenvalue-decomposition} (p.~\pageref{theorem:eigenvalue-decomposition}), matrix $\bA$ can be diagonalizable.

\paragraph{Backward implication} Suppose $\bA$ is diagonalizable. That is, there exists a nonsingular matrix $\bP$ and a diagonal matrix $\bD$ such that $\bA =\bP\bD\bP^{-1} $. $\bA$ and $\bD$ are similar matrices such that they have the same eigenvalues (Lemma~\ref{lemma:eigenvalue-similar-matrices}, p.~\pageref{lemma:eigenvalue-similar-matrices}), same algebraic multiplicities, and geometric multiplicities (Corollary~\ref{corollary:multipli-similar-matrix}, p.~\pageref{corollary:multipli-similar-matrix}). It can be easily verified that a diagonal matrix has equal algebraic multiplicity and geometric multiplicity such that $\bA$ is a simple matrix.
\end{proof}

	\begin{remark}[Equivalence on Diagonalization]
From Theorem~\ref{theorem:independent-eigenvector-theorem} (p.~\pageref{theorem:independent-eigenvector-theorem}) that any eigenvectors corresponding to different eigenvalues are linearly independent, and Remark~\ref{remark:geometric-mul-meaning} (p.~\pageref{remark:geometric-mul-meaning}) that the geometric multiplicity is the dimension of the eigenspace. We realize, if the geometric multiplicity is equal to the algebraic multiplicity, the eigenspace can span the whole space $\real^n$ if matrix $\bA\in \real^{n\times n}$. So the above Lemma is equivalent to claim that if the eigenspace can span the whole space $\real^n$, then $\bA$ can be diagonalizable.
	\end{remark}

\begin{corollary}
A square matrix $\bA$ with linearly independent eigenvectors is a simple matrix. Or if $\bA$ is symmetric, it is also a simple matrix.
\end{corollary}
From the eigenvalue decomposition in Theorem~\ref{theorem:eigenvalue-decomposition} (p.~\pageref{theorem:eigenvalue-decomposition}) and the spectral decomposition in Theorem~\ref{theorem:spectral_theorem} (p.~\pageref{theorem:spectral_theorem}), the proof is trivial for the above corollary.

Now we are ready to show the second form of the spectral decomposition.
\begin{theoremHigh}[Spectral Decomposition: The Second Form]\label{theorem:spectral_theorem_secondForm}
A \textbf{simple matrix} $\bA \in \real^{n\times n}$ can be factored as a sum of a set of idempotent matrices
$$
\bA = \sum_{i=1}^{n} \lambda_i \bA_i,
$$
where $\lambda_i$ for all $i\in \{1,2,\ldots, n\}$ are eigenvalues of $\bA$ (duplicate possible), and also known as the \textit{spectral values} of $\bA$. Specifically, we have the following properties:

1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, n\}$;

2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

3. Additivity: $\sum_{i=1}^{n} \bA_i = \bI_n$;

4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_n) = n$.

\end{theoremHigh}

\begin{proof}[of Theorem~\ref{theorem:spectral_theorem_secondForm}]
Since $\bA$ is a simple matrix, from Lemma~\ref{lemma:simple-diagonalizable} (p.~\pageref{lemma:simple-diagonalizable}), there exists a nonsingular matrix $\bP$ and a diagonal matrix $\bLambda$ such that $\bA=\bP\bLambda\bP^{-1}$ where $\bLambda=\diag(\lambda_1, \lambda_2, \ldots, \lambda_n)$, and $\lambda_i$'s are eigenvalues of $\bA$ and columns of $\bP$ are eigenvectors of $\bA$. Suppose
$$
\bP = \begin{bmatrix}
\bv_1 & \bv_2&\ldots & \bv_n
\end{bmatrix}
\qquad
\text{and }
\qquad
\bP^{-1} =
\begin{bmatrix}
\bw_1^\top \\
\bw_2^\top \\
\vdots \\
\bw_n^\top
\end{bmatrix}
$$
are the column and row partitions of $\bP$ and $\bP^{-1}$ respectively. Then, we have
$$
\bA= \bP\bLambda\bP^{-1} =
\begin{bmatrix}
	\bv_1 & \bv_2&\ldots & \bv_n
\end{bmatrix}
\bLambda
\begin{bmatrix}
	\bw_1^\top \\
	\bw_2^\top \\
	\vdots \\
	\bw_n^\top
\end{bmatrix}=
\sum_{i=1}^{n}\lambda_i \bv_i\bw_i^\top.
$$
Let $\bA_i = \bv_i\bw_i^\top$, we have $\bA = \sum_{i=1}^{n} \lambda_i \bA_i$.
We realize that $\bP^{-1}\bP = \bI$ such that
$$
\left\{
\begin{aligned}
	&\bw_i^\top\bv_j = 1 ,& \mathrm{\,\,if\,\,} i = j.  \\
	&\bw_i^\top\bv_j = 0 ,& \mathrm{\,\,if\,\,} i \neq j.
\end{aligned}
\right.
$$
Therefore,
$$
\bA_i\bA_j =\bv_i\bw_i^\top\bv_j\bw_j^\top = \left\{
\begin{aligned}
	&\bv_i\bw_i^\top = \bA_i ,& \mathrm{\,\,if\,\,} i = j.  \\
	& \bzero ,& \mathrm{\,\,if\,\,} i \neq j.
\end{aligned}
\right.
$$
This implies the idempotency and orthogonality of $\bA_i$'s. We also notice that $\sum_{i=1}^{n}\bA_i = \bP\bP^{-1}=\bI$, that is the additivity of $\bA_i$'s. The rank-additivity of the $\bA_i$'s is trivial since $rank(\bA_i)=1$ for all $i\in \{1,2,\ldots, n\}$.
\end{proof}
The decomposition is highly related to the Cochran's theorem and its application in the distribution theory of linear models \citep{lu2021numerical,lu2021rigorous}.

\begin{theoremHigh}[Spectral Decomposition: The Third Form]\label{Corollary:spectral_theorem_3Form}
A \textbf{simple matrix} $\bA \in \real^{n\times n}$ \textcolor{blue}{with $k$ distinct eigenvalues} can be factored as a sum of a set of idempotent matrices
$$
\bA = \sum_{i=1}^{\textcolor{blue}{k}} \lambda_i \bA_i,
$$
where $\lambda_i$ for all $i\in \{1,2,\ldots, \textcolor{blue}{k}\}$ are the distinct eigenvalues of $\bA$, and also known as the \textbf{spectral values} of $\bA$. Specifically, we have the following properties:

1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, \textcolor{blue}{k}\}$;

2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

3. Additivity: $\sum_{i=1}^{\textcolor{blue}{k}} \bA_i = \bI_n$;

4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{\textcolor{blue}{k}}) = n$.

\end{theoremHigh}
\begin{proof}[of Theorem~\ref{Corollary:spectral_theorem_3Form}]
From Theorem~\ref{theorem:spectral_theorem_secondForm}, we can decompose $\bA$ by $\bA =\sum_{j=1}^{n} \beta_i \bB_j$. Without loss of generality,
the eigenvalues $\beta_i$'s are ordered such that $\beta_1 \leq \beta_2 \leq \ldots \beta_n$ where duplicate is possible. Let $\lambda_i$'s be the distinct eigenvalues, and $\bA_i$ be the sum of the $\bB_j$'s associated with $\lambda_i$.
Suppose the multiplicity of $\lambda_i$ is $m_i$, and the $\bB_j$'s associated with $\lambda_i$ can be denoted as $\{\bB_{1}^i, \bB_{2}^i, \ldots, \bB_{m_i}^i\}$. Then $\bA_i$ can be denoted as $\bA_i = \sum_{j=1}^{m_i} \bB_{j}^i$. Apparently $\bA = \sum_{i=1}^{k} \lambda_i \bA_i$.

\paragraph{Idempotency} $\bA_i^2 = (\bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i)(\bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i)= \bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i = \bA_i$ from the idempotency and orthogonality of $\bB_j^i$'s.

\paragraph{Ortogonality} $\bA_i\bA_j = (\bB_1^i + \bB_2^i+\ldots \bB_{m_i}^i)(\bB_1^j + \bB_2^j+\ldots \bB_{m_j}^j)=\bzero$ from the orthogonality of the $\bB_j^i$'s.

\paragraph{Additivity} It is trivial that  $\sum_{i=1}^{k} \bA_i = \bI_n$.

\paragraph{Rank-Additivity} $rank(\bA_i ) = rank(\sum_{j=1}^{m_i} \bB_{j}^i) = m_i$ such that $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{k}) = m_1+m_2+\ldots+m_k=n$.
\end{proof}

\begin{theoremHigh}[Spectral Decomposition: Backward Implication]\label{Corollary:spectral_theorem_4Form}
If a matrix $\bA \in \real^{n\times n}$ with $k$ distinct eigenvalues can be factored as a sum of a set of idempotent matrices
$$
\bA = \sum_{i=1}^{k} \lambda_i \bA_i,
$$
where $\lambda_i$ for all $i\in \{1,2,\ldots, k\}$ are the distinct eigenvalues of $\bA$, and

1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, k\}$;

2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

3. Additivity: $\sum_{i=1}^{k} \bA_i = \bI_n$;

4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{k}) = n$.

\item Then, the matrix $\bA$ is a simple matrix.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{Corollary:spectral_theorem_4Form}]
Suppose $rank(\bA_i) = r_i$ for all $i \in \{1,2,\ldots, k\}$. By ULV decomposition in Theorem~\ref{theorem:ulv-decomposition} (p.~\pageref{theorem:ulv-decomposition}), $\bA_i$ can be factored as
$$
\bA_i = \bU_i \begin{bmatrix}
	\bL_i & \bzero \\
	\bzero & \bzero
\end{bmatrix}\bV_i,
$$
where $\bL_i \in \real^{r_i \times r_i}$, $\bU_i \in \real^{n \times n}$, and $\bV_i\in \real^{n \times n}$ are orthogonal matrices. Let
$$
\bX_i =
\bU_i \begin{bmatrix}
	\bL_i  \\
	\bzero
\end{bmatrix}
\qquad
\text{and}
\qquad
\bV_i =
\begin{bmatrix}
\bY_i \\
\bZ_i
\end{bmatrix},
$$
where $\bX_i \in \real^{n\times r_i}$, and $\bY_i \in \real^{r_i \times n}$ is the first $r_i$ rows of $\bV_i$. Then, we have
$$
\bA_i = \bX_i \bY_i.
$$
This can be seen as a \textbf{reduced} ULV decomposition of $\bA_i$. Appending the $\bX_i$'s and $\bY_i$'s into $\bX$ and $\bY$,
$$
\bX = [\bX_1, \bX_2, \ldots, \bX_k],
\qquad
\bY =
\begin{bmatrix}
\bY_1\\
\bY_2\\
\vdots \\
\bY_k
\end{bmatrix},
$$
where $\bX\in \real^{n\times n}$ and $\bY\in \real^{n\times n}$ (from rank-additivity). By block matrix multiplication and the additivity of $\bA_i$'s, we have
$$
\bX\bY = \sum_{i=1}^{k} \bX_i\bY_i = \sum_{i=1}^{k} \bA_i = \bI.
$$
Therefore $\bY$ is the inverse of $\bX$, and
$$
\bY\bX =
\begin{bmatrix}
	\bY_1\\
	\bY_2\\
	\vdots \\
	\bY_k
\end{bmatrix}
[\bX_1, \bX_2, \ldots, \bX_k]
=
\begin{bmatrix}
\bY_1\bX_1 & \bY_1\bX_2 & \ldots & \bY_1\bX_k\\
\bY_2\bX_1 & \bY_2\bX_2 & \ldots & \bY_2\bX_k\\
\vdots & \vdots & \ddots & \vdots\\
\bY_k\bX_1 & \bY_k\bX_2 & \ldots & \bY_k\bX_k\\
\end{bmatrix}
=\bI,
$$
such that
$$
\bY_i\bX_j = \left\{
\begin{aligned}
	&\bI_{r_i}  ,& \mathrm{\,\,if\,\,} i = j;  \\
	& \bzero ,& \mathrm{\,\,if\,\,} i \neq j.
\end{aligned}
\right.
$$
This implies
$$
\bA_i\bX_j = \left\{
\begin{aligned}
	&\bX_i  ,& \mathrm{\,\,if\,\,} i = j;  \\
	& \bzero ,& \mathrm{\,\,if\,\,} i \neq j,
\end{aligned}
\right.
\qquad
\text{and}
\qquad
\bA \bX_i = \lambda_i\bX_i.
$$
Finally, we have
$$
\begin{aligned}
\bA\bX &= \bA[\bX_1, \bX_2, \ldots, \bX_k] = [\lambda_1\bX_1, \lambda_2\bX_2, \ldots, \lambda_k\bX_k] = \bX\bLambda,
\end{aligned}
$$
where
$$\bLambda =
\begin{bmatrix}
\lambda_1 \bI_{r_1} & \bzero & \ldots & \bzero \\
\bzero &  \lambda_2 \bI_{r_2} & \ldots & \bzero \\
\vdots &  \vdots & \ddots & \vdots \\
\bzero & \bzero & \ldots &  \lambda_k \bI_{r_k} \\
\end{bmatrix}
$$
is a diagonal matrix. This implies $\bA$ can be diagonalized and from Lemma~\ref{lemma:simple-diagonalizable} (p.~\pageref{lemma:simple-diagonalizable}), $\bA$ is a simple matrix.
\end{proof}

	\begin{corollary}[Forward and Backward Spectral]
		Combine Theorem~\ref{Corollary:spectral_theorem_3Form} and Theorem~\ref{Corollary:spectral_theorem_4Form}, we can claim that matrix $\bA \in \real^{n\times n}$ is a simple matrix with $k$ distinct eigenvalues if and only if it can be factored as a sum of a set of idempotent matrices
		$$
		\bA = \sum_{i=1}^{k} \lambda_i \bA_i,
		$$
		where $\lambda_i$ for all $i\in \{1,2,\ldots, k\}$ are the distinct eigenvalues of $\bA$, and

		1. Idempotent: $\bA_i^2 = \bA_i$ for all $i\in \{1,2,\ldots, k\}$;

		2. Orthogonal: $\bA_i\bA_j = \bzero$ for all $i \neq j$;

		3. Additivity: $\sum_{i=1}^{k} \bA_i = \bI_n$;

		4. Rank-Additivity: $rank(\bA_1) + rank(\bA_2) + \ldots + rank(\bA_{k}) = n$.
	\end{corollary}

\section{Skew-Symmetric Matrix and its Properties*}
We have introduced the spectral decomposition for symmetric matrices. A special kind of matrices that's related to symmetric is called the \textit{skew-symmetric matrices}.
\begin{definition}[Skew-Symmetric Matrix\index{Skew-symmetric matrix}]
If matrix $\bA\in \real^{n\times n}$ has the following property, then it is known as a \textbf{skew-symmetric matrix}:
$$
\bA^\top = -\bA.
$$
Note that under this definition, for the diagonal values $a_{ii}$ for all $i \in \{1,2,\ldots, n\}$, we have $a_{ii} = -a_{ii}$ which implies all the diagonal components are 0.
\end{definition}

We have proved in Lemma~\ref{lemma:real-eigenvalues-spectral} (p.~\pageref{lemma:real-eigenvalues-spectral}) that all the eigenvalues of symmetric matrices are real. Similarly, we could show that all the eigenvalues of skew-symmetric matrices are imaginary.

\begin{lemma}[Imaginary Eigenvalues]\label{lemma:real-eigenvalues-spectral-skew}
The eigenvalues of any skew-symmetric matrix are all imaginary or zero.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:real-eigenvalues-spectral-skew}]
	Suppose eigenvalue $\lambda$ is a complex number $\lambda=a+ib$ where $a,b$ are real. Its complex conjugate is $\bar{\lambda}=a-ib$. Same for complex eigenvector $\bx = \bc+i\bd$ and its complex conjugate $\bar{\bx}=\bc-i\bd$ where $\bc, \bd$ are real vectors. We then have the following property
	$$
	\bA \bx = \lambda \bx\qquad   \underrightarrow{\text{ leads to }}\qquad  \bA \bar{\bx} = \bar{\lambda} \bar{\bx}\qquad   \underrightarrow{\text{ transpose to }}\qquad  \bar{\bx}^\top \bA^\top =\bar{\lambda} \bar{\bx}^\top.
	$$
	We take the dot product of the first equation with $\bar{\bx}$ and the last equation with $\bx$:
	$$
	\bar{\bx}^\top \bA \bx = \lambda \bar{\bx}^\top \bx, \qquad \text{and } \qquad \bar{\bx}^\top \bA^\top \bx = \bar{\lambda}\bar{\bx}^\top \bx.
	$$
	Then we have the equality $-\lambda\bar{\bx}^\top \bx = \bar{\lambda} \bar{\bx}^\top\bx$ (since $\bA^\top=-\bA$). Since $\bar{\bx}^\top\bx = (\bc-i\bd)^\top(\bc+i\bd) = \bc^\top\bc+\bd^\top\bd$ is a real number. Therefore the real part of $\lambda$ is zero and $\lambda$ is either imaginary or zero.
\end{proof}

\begin{lemma}[Odd Skew-Symmetric Determinant]\label{lemma:skew-symmetric-determinant}
Given skew-symmetric matrix $\bA\in \real^{n\times n}$, if $n$ is odd, then $\det(\bA)=0$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:skew-symmetric-determinant}]
When $n$ is odd, we have
$$
\det(\bA) = \det(\bA^\top) = \det(-\bA) = (-1)^n \det(\bA) = -\det(\bA).
$$
This implies $\det(\bA)=0$.
\end{proof}

\begin{theoremHigh}[Block-Diagonalization of Skew-Symmetric Matrices]\label{theorem:skew-block-diagonalization_theorem}
A real skew-symmetric matrix $\bA \in \real^{n\times n}$ can be factored as
\begin{equation*}
\bA = \bZ \bD \bZ^\top,
\end{equation*}
where $\bZ$ is an $n\times n$ nonsingular matrix, and $\bD$ is a block-diagonal matrix with the following form
$$
\bD =
\diag\left(\begin{bmatrix}
	0 & 1 \\
	-1 & 0
\end{bmatrix},
\ldots,
\begin{bmatrix}
	0 & 1 \\
	-1 & 0
\end{bmatrix},
0, \ldots, 0\right).
$$
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:skew-block-diagonalization_theorem}]
We will prove by recursive calculation.
As usual, we will denote the entry ($i,j$) of matrix $\bA$ by $\bA_{ij}$.

\paragraph{Case 1).} Suppose the first row of $\bA$ is nonzero, we notice that $\bE\bA\bE^\top$ is skew-symmetric if $\bA$ is skew-symmetric for any matrix $\bE$. This will make both the diagonals of $\bA$ and $\bE\bA\bE^\top$ zeros, and the upper-left $2\times 2$ submatrix of $\bE\bA\bE^\top$ has the following form
$$
(\bE\bA\bE^\top)_{1:2,1:2}=
\begin{bmatrix}
0 & x \\
-x & 0
\end{bmatrix}.
$$
Since we suppose the first row of $\bA$ is nonzero, there exists a permutation matrix $\bP$ (Definition~\ref{definition:permutation-matrix}, p.~\pageref{definition:permutation-matrix}), such that we will exchange the nonzero value, say $a$, in the first row to the second column of $\bP\bA\bP^\top$. And as discussed above, the upper-left $2\times 2$ submatrix of $\bP\bA\bP^\top$ has the following form
$$
(\bP\bA\bP^\top)_{1:2,1:2}=
\begin{bmatrix}
	0 & a \\
	-a & 0
\end{bmatrix}.
$$
Construct a nonsingular matrix $\bM = \begin{bmatrix}
1/a & \bzero \\
\bzero   & \bI_{n-1}
\end{bmatrix}$ such that the upper left $2\times 2$ submatrix of $\bM\bP\bA\bP^\top\bM^\top$ has the following form
$$
(\bM\bP\bA\bP^\top\bM^\top)_{1:2,1:2}=
\begin{bmatrix}
0 & 1 \\
-1 & 0
\end{bmatrix}.
$$
Now we finish diagonalizing the upper-left $2\times 2$ block. Suppose now $(\bM\bP\bA\bP^\top\bM^\top)$ above has a nonzero value, say $b$, in the first row with entry $(1,j)$ for some $j>2$, we can construct a nonsingular matrix $\bL = \bI - b\cdot\bE_{j2}$ where $\bE_{2j}$ is an all-zero matrix except the entry ($2,j$) is 1, such that $(\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top)$ will introduce 0 for the entry with value $b$.

\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolorSkip,frametitle={A Trivial Example}]
For example, suppose $\bM\bP\bA\bP^\top\bM^\top$ is a $3\times 3$ matrix with the following value
$$
\bM\bP\bA\bP^\top\bM^\top =
\begin{bmatrix}
0 & 1 & b \\
-1 & 0 & \times \\
\times & \times & 0
\end{bmatrix},
\qquad \text{and}\qquad
\bL =\bI - b\cdot\bE_{j2}=
\begin{bmatrix}
	1 & 0 & 0 \\
	0 & 1 & 0 \\
	0 &-b & 1
\end{bmatrix},
$$
where $j=3$ for this specific example. This results in
$$
\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top =
\begin{bmatrix}
	1 & 0 & 0 \\
	0 & 1 & 0 \\
	0 &-b & 1
\end{bmatrix}
\begin{bmatrix}
	0 & 1 & \textcolor{blue}{b} \\
	-1 & 0 & \times \\
	\times & \times & 0
\end{bmatrix}
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & -b \\
0 & 0 & 1
\end{bmatrix}
=
\begin{bmatrix}
	0 & 1 & \textcolor{blue}{0} \\
	-1 & 0 & \times \\
	\times & \times & 0
\end{bmatrix}.
$$
\end{mdframed}
Similarly, if the second row of $\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top$ contains a nonzero value, say $c$, we could construct a nonsingular matrix $\bK = \bI+c\cdot \bE_{j1}$ such that $\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top$ will introduce 0 for the entry with value $c$.
\begin{mdframed}[hidealllines=\mdframehideline,backgroundcolor=\mdframecolorSkip,frametitle={A Trivial Example}]
	For example, suppose $\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top$ is a $3\times 3$ matrix with the following value
	$$
	\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top =
	\begin{bmatrix}
		0 & 1 & 0 \\
		-1 & 0 & c \\
		\times & \times & 0
	\end{bmatrix},
	\qquad \text{and}\qquad
	\bK =\bI + c\cdot\bE_{j1}=
	\begin{bmatrix}
		1 & 0 & 0 \\
		0 & 1 & 0 \\
		c &0 & 1
	\end{bmatrix},
	$$
	where $j=3$ for this specific example. This results in
	$$
	\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top =
	\begin{bmatrix}
		1 & 0 & 0 \\
		0 & 1 & 0 \\
		c & 0 & 1
	\end{bmatrix}
	\begin{bmatrix}
		0 & 1 & 0 \\
		-1 & 0 & \textcolor{blue}{c} \\
		\times & \times & 0
	\end{bmatrix}
	\begin{bmatrix}
		1 & 0 & c \\
		0 & 1 & 0 \\
		0 & 0 & 1
	\end{bmatrix}
	=
	\begin{bmatrix}
		0 & 1 & 0 \\
		-1 & 0 & \textcolor{blue}{0} \\
		\times & \times & 0
	\end{bmatrix}.
	$$
Since we have shown that $\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top$ is also skew-symmetric, then, it is actually
$$
\bK\bL\bM\bP\bA\bP^\top\bM^\top\bL^\top\bK^\top=
\begin{bmatrix}
	0 & 1 & 0 \\
	-1 & 0 & \textcolor{blue}{0} \\
	\textcolor{red}{0} & \textcolor{red}{0} & 0
\end{bmatrix},
$$
so that we do not need to tackle the first 2 columns of the above equation.
\end{mdframed}
Apply this process to the bottom-right $(n-2)\times(n-2)$ submatrix, we will complete the proof.

\paragraph{Case 2).} Suppose the first row of $\bA$ is zero, a permutation matrix to put the first row into the last row and apply the process in case 1 to finish the proof.
\end{proof}

From the block-diagonalization of skew-symmetric matrices above, we could easily find that the rank of a skew-symmetric matrix is even.  And we could prove the determinant of skew-symmetric with even order is nonnegative as follows.
	\begin{lemma}[Even Skew-Symmetric Determinant]\label{lemma:skew-symmetric-determinant-even}
		For skew-symmetric matrix $\bA\in \real^{n\times n}$, if $n$ is even, then $\det(\bA)\geq 0$.
	\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:skew-symmetric-determinant-even}]
By Theorem~\ref{theorem:skew-block-diagonalization_theorem}, we could block-diagonalize $\bA = \bZ\bD\bZ^\top$ such that
$$
\det(\bA) = \det(\bZ\bD\bZ^\top) = \det(\bZ)^2 \det(\bD) \geq 0.
$$
This completes the proof.
\end{proof}

\section{Applications}
\subsection{Application: Eigenvalue of Projection Matrix}

In Section~\ref{section:application-ls-qr} (p.~\pageref{section:application-ls-qr}), we will show that the QR decomposition can be applied to  solve the least squares problem, where we consider the overdetermined system $\bA\bx = \bb$ with $\bA\in \real^{m\times n}$ being the data matrix, $\bb\in \real^m$ with $m\geq n$ being the observation matrix. Normally $\bA$ will have full column rank since the data from real work has a large chance to be unrelated. And the least squares solution is given by $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ for minimizing $||\bA\bx-\bb||^2$, where $\bA^\top\bA$ is invertible since $\bA$ has full column rank and $rank(\bA^T\bA)=rank(\bA)$. The recovered observation matrix is then $\hat{\bb} = \bA\bx_{LS} = \bA(\bA^\top\bA)^{-1}\bA^\top\bb$. $\bb$ may not be in the column space of $\bA$, but the recovered $\hat{\bb}$ is in this column space. We then define such a matrix $\bH=\bA(\bA^\top\bA)^{-1}\bA^\top$ to be a projection matrix, i.e., projecting $\bb$ onto the column space of $\bA$. Or, it is also known as hat matrix, since we put a hat on $\bb$. It can be easily verified the projection matrix is \textit{symmetric and idempotent} (i.e., $\bH=\bH^\top$ and  $\bH^2=\bH$).
\index{Projection matrix}
\index{Idempotent}
\index{Idempotency}
\begin{remark}[Column Space of Projection Matrix]
	We notice that the hat matrix $\bH = \bA(\bA^\top\bA)^{-1}\bA^\top$ is to project any vector in $\real^m$ into the column space of $\bA$. That is, $\bH\by \in \cspace(\bA)$. Notice again $\bH\by$ is the nothing but a combination of the columns of $\bH$, thus $\cspace(\bH) = \cspace(\bA)$.

	In general, for any projection matrix $\bH$ to project vector onto subspace $\mathcalV$, then $\cspace(\bH) = \mathcalV$. More formally, in a mathematical language, this property can be proved by singular value decomposition (Section~\ref{section:SVD}, p.~\pageref{section:SVD}).
\end{remark}

We now show that any projection matrix has specific eigenvalues.
\begin{proposition}[Eigenvalue of Projection Matrix]\label{proposition:eigen-of-projection-matrix}
The only possible eigenvalues of a projection matrix are 0 and 1.
\end{proposition}
\begin{proof}[of Proposition~\ref{proposition:eigen-of-projection-matrix}]
Since $\bH$ is symmetric, we have spectral decomposition $\bH =\bQ\bLambda\bQ^\top$. From the idempotent property, we have
$$
\begin{aligned}
(\bQ\bLambda\bQ^\top)^2 &= \bQ\bLambda\bQ^\top \\
\bQ\bLambda^2\bQ^\top &= \bQ\bLambda\bQ^\top \\
\bLambda^2 &=\bLambda \\
\lambda_i^2 &=\lambda_i,
\end{aligned}
$$
Therefore, the only possible eigenvalues for $\bH$ are 0 and 1.
\end{proof}

This property of the projection matrix is important for the analysis of distribution theory for linear models. See \citet{lu2021rigorous} for more details.
Following the eigenvalue of the projection matrix, it can also give rise to the perpendicular projection $\bI-\bH$.
\begin{proposition}[Project onto $\mathcalV^\perp$]\label{proposition:orthogonal-projection_tmp}
	Let $\mathcalV$ be a subspace and $\bH$ be a projection onto $\mathcalV$. Then $\bI-\bH$ is the projection matrix onto $\mathcalV^\perp$.
\end{proposition}

\begin{proof}[of Proposition~\ref{proposition:orthogonal-projection_tmp}]
First, $(\bI-\bH)$ is symmetric, $(\bI-\bH)^\top = \bI - \bH^\top = \bI-\bH$ since $\bH$ is symmetric. And
$$
(\bI-\bH)^2 = \bI^2 -\bI\bH -\bH\bI +\bH^2 = \bI-\bH,
$$
i.e., $\bI-\bH$ is idempotent.
Thus $\bI-\bH$ is a projection matrix. By spectral theorem again, let $\bH =\bQ\bLambda\bQ^\top$. Then $\bI-\bH = \bQ\bQ^\top - \bQ\bLambda\bQ^\top = \bQ(\bI-\bLambda)\bQ^\top$. Hence the column space of $\bI-\bH$ is spanned by the eigenvectors of $\bH$ corresponding to the zero eigenvalues of $\bH$ (by Proposition~\ref{proposition:eigen-of-projection-matrix}, p.~\pageref{proposition:eigen-of-projection-matrix}), which coincides with $\mathcalV^\perp$.
\end{proof}

Again, for a detailed analysis of the origin of the projection matrix and results behind the projection matrix, we highly recommend the readers refer to \citet{lu2021numerical} although it is not the main interest of matrix decomposition results.

\subsection{Application: An Alternative Definition of PD and PSD of Matrices}\label{section:equivalent-pd-psd}
In Definition~\ref{definition:psd-pd-defini} (p.~\pageref{definition:psd-pd-defini}), we defined the positive definite matrices and positive semidefinite matrices by the quadratic form of the matrices. We here prove that a symmetric matrix is positive definite if and only if all eigenvalues are positive.

\begin{lemma}[Eigenvalues of PD and PSD Matrices\index{Positive definite}\index{Positive semidefinite}]\label{lemma:eigens-of-PD-psd}
A matrix $\bA\in \real^{n\times n}$ is positive definite (PD) if and only if $\bA$ has only positive eigenvalues.
And a matrix $\bA\in \real^{n\times n}$ is positive semidefinite (PSD) if and only if $\bA$ has only nonnegative eigenvalues.
\end{lemma}

\begin{proof}[of Lemma~\ref{lemma:eigens-of-PD-psd}]
We will prove by forward implication and reverse implication separately as follows.
\paragraph{Forward implication:}
	Suppose $\bA$ is PD, then for any eigenvalue $\lambda$ and its corresponding eigenvector $\bv$ of $\bA$, we have $\bA\bv = \lambda\bv$. Thus
$$
\bv^\top \bA\bv = \lambda||\bv||^2 > 0.
$$
This implies $\lambda>0$.
\paragraph{Reverse implication:}
Conversely, suppose the eigenvalues are positive. By spectral decomposition of $\bA =\bQ\bLambda \bQ^\top$. If $\bx$ is a nonzero vector, let $\by=\bQ^\top\bx$, we have
$$
\bx^\top \bA \bx = \bx^\top (\bQ\bLambda \bQ^\top) \bx = (\bx^\top \bQ) \bLambda (\bQ^\top\bx) = \by^\top\bLambda\by = \sum_{i=1}^{n} \lambda_i y_i^2>0.
$$
That is, $\bA$ is PD.

Analogously, we can prove the second part of the claim.
\end{proof}

\begin{theoremHigh}[Nonsingular Factor of PSD and PD Matrices]\label{lemma:nonsingular-factor-of-PD}
A real symmetric matrix $\bA$ is PSD if and only if $\bA$ can be factored as $\bA=\bP^\top\bP$, and is PD if and only if $\bP$ is nonsingular.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{lemma:nonsingular-factor-of-PD}]
For the first part, we will prove by forward implication and reverse implication separately as follows.
\paragraph{Forward implication: }Suppose $\bA$ is PSD, its spectral decomposition is given by $\bA = \bQ\bLambda\bQ^\top$. Since eigenvalues of PSD matrices are nonnegative, we can decompose $\bLambda=\bLambda^{1/2}\bLambda^{1/2}$. Let $\bP = \bLambda^{1/2}\bQ^\top$, we can decompose $\bA$ by $\bA=\bP^\top\bP$.

\paragraph{Reverse implication: } If $\bA$ can be factored as $\bA=\bP^\top\bP$, then all eigenvalues of $\bA$ are nonnegative since for any eigenvalues $\lambda$ and its corresponding eigenvector $\bv$ of $\bA$, we have
$$
\lambda = \frac{\bv^\top\bA\bv}{\bv^\top\bv} = \frac{\bv^\top\bP^\top\bP\bv}{\bv^\top\bv}=\frac{||\bP\bv||^2}{||\bv||^2} \geq 0.
$$
This implies $\bA$ is PSD by Lemma~\ref{lemma:eigens-of-PD-psd}.

Similarly, we can prove the second part for PD matrices where the positive definiteness will result in the nonsingular $\bP$ and the nonsingular $\bP$ will result in the positiveness of the eigenvalues. \footnote{See also wiki page: https://en.wikipedia.org/wiki/Sylvester's\_criterion.}
\end{proof}

\subsection{Proof for Semidefinite Rank-Revealing Decomposition}\label{section:semi-rank-reveal-proof}

In this section, we provide a proof for Theorem~\ref{theorem:semidefinite-factor-rank-reveal} (p.~\pageref{theorem:semidefinite-factor-rank-reveal}), the existence of the rank-revealing decomposition for positive semidefinite matrices.\index{Rank-revealing}\index{Semidefinite rank-revealing}
\begin{proof}[of Theorem~\ref{theorem:semidefinite-factor-rank-reveal}]
The proof is a consequence of the nonsingular factor of PSD matrices (Theorem~\ref{lemma:nonsingular-factor-of-PD}, p.~\pageref{lemma:nonsingular-factor-of-PD}) and the existence of column-pivoted QR decomposition (Theorem~\ref{theorem:rank-revealing-qr-general}, p.~\pageref{theorem:rank-revealing-qr-general}).

By Theorem~\ref{lemma:nonsingular-factor-of-PD}, the nonsingular factor of PSD matrix $\bA$ is given by $\bA = \bZ^\top\bZ$, where $\bZ=\bLambda^{1/2}\bQ^\top$ and $\bA=\bQ\bLambda\bQ^\top$ is the spectral decomposition of $\bA$.

By Lemma~\ref{lemma:rank-of-symmetric} (p.~\pageref{lemma:rank-of-symmetric}), the rank of matrix $\bA$ is the number of nonzero eigenvalues (here the number of positive eigenvalues since $\bA$ is PSD). Therefore only $r$ components in the diagonal of $\bLambda^{1/2}$ are nonzero, and $\bZ=\bLambda^{1/2}\bQ^\top$ contains only $r$ independent columns, i.e., $\bZ$ is of rank $r$. By column-pivoted QR decomposition, we have
$$
\bZ\bP = \bQ
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix},
$$
where $\bP$ is a permutation matrix, $\bR_{11}\in \real^{r\times r}$ is upper triangular with positive diagonals, and $\bR_{12}\in \real^{r\times (n-r)}$. Therefore
$$
\bP^\top\bA\bP  =
\bP^\top\bZ^\top\bZ\bP =
\begin{bmatrix}
\bR_{11}^\top & \bzero \\
\bR_{12}^\top & \bzero
\end{bmatrix}
\begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix}.
$$
Let
$$
\bR = \begin{bmatrix}
	\bR_{11} & \bR_{12} \\
	\bzero   & \bzero
\end{bmatrix},
$$
we find the rank-revealing decomposition for semidefinite matrix $\bP^\top\bA\bP = \bR^\top\bR$.
\end{proof}

This decomposition is produced by using complete pivoting, which at each stage permutes the largest diagonal element
in the active submatrix into the pivot position. The procedure is similar to the partial pivoting discussed in Section~\ref{section:partial-pivot-lu} (p.~\pageref{section:partial-pivot-lu}).

\subsection{Application: Cholesky Decomposition via the QR Decomposition and the Spectral Decomposition}\label{section:cholesky-by-qr-spectral}
In this section, we provide another proof for the existence of Cholesky decomposition.

\begin{proof}[of Theorem~\ref{theorem:cholesky-factor-exist}]
From Theorem~\ref{lemma:nonsingular-factor-of-PD}, the PD matrix $\bA$ can be factored as $\bA=\bP^\top\bP$ where $\bP$ is a nonsingular matrix. Then, the QR decomposition of $\bP$ is given by $\bP = \bQ\bR$. This implies
$$
\bA = \bP^\top\bP = \bR^\top\bQ^\top\bQ\bR = \bR^\top\bR,
$$
where we notice that the form is very similar to the Cholesky decomposition except that we do not claim the $\bR$ has only positive diagonal values. From the CGS algorithm to compute the QR decomposition (Section~\ref{section:qr-gram-compute}, p.~\pageref{section:qr-gram-compute}), we realize that the diagonals of $\bR$ are nonnegative, and if $\bP$ is nonsingular, the diagonals of $\bR$ are also positive.
\end{proof}
The proof for the above theorem is a consequence of the existence of both the QR decomposition and the spectral decomposition. Thus, the existence of Cholesky decomposition can be shown via the QR decomposition and the spectral decomposition in this sense.

\subsection{Application: Unique Power Decomposition of Positive Definite Matrices}\label{section:unique-posere-pd}
\begin{theoremHigh}[Unique Power Decomposition of PD Matrices]\label{theorem:unique-factor-pd}
Any $n\times n$ positive definite matrix $\bA$ can be \textbf{uniquely} factored as a product of a positive definite matrix $\bB$ such that $\bA =\bB^2$.
\end{theoremHigh}
\begin{proof}[of Theorem~\ref{theorem:unique-factor-pd}]
We first prove that there exists such positive definite matrix $\bB$ so that $\bA = \bB^2$.
\paragraph{Existence} Since $\bA$ is PD which is also symmetric, the spectral decomposition of $\bA$ is given by $\bA = \bQ\bLambda\bQ^\top$. Since eigenvalues of PD matrices are positive by Lemma~\ref{lemma:eigens-of-PD-psd} (p.~\pageref{lemma:eigens-of-PD-psd}), the square root of $\bLambda$ exists. We can define $\bB = \bQ\bLambda^{1/2}\bQ^\top$ such that $\bA = \bB^2$ where $\bB$ is apparently PD.

\paragraph{Uniqueness} Suppose such factorization is not unique, then there exist two of this decomposition such that
$$
\bA = \bB_1^2 = \bB_2^2,
$$
where $\bB_1$ and $\bB_2$ are both PD. The spectral decompositions of them are given by
$$
\bB_1 = \bQ_1 \bLambda_1\bQ_1^\top, \qquad \text{and} \qquad \bB_2 = \bQ_2 \bLambda_2\bQ_2^\top.
$$
We notice that $\bLambda_1^2$ and $\bLambda_2^2$ contains the eigenvalues of $\bA$, and both eigenvalues of $\bB_1$ and $\bB_2$ contained in $\bLambda_1$ and $\bLambda_2$ are positive (since $\bB_1$ and $\bB_2$ are both PD). Without loss of generality, we suppose $\bLambda_1=\bLambda_2=\bLambda^{1/2}$, and $\bLambda=\diag(\lambda_1,\lambda_2, \ldots, \lambda_n)$ such that $\lambda_1\geq \lambda_2 \geq \ldots \geq \lambda_n$. By $\bB_1^2 = \bB_2^2$, we have
$$
\bQ_1 \bLambda \bQ_1^\top = \bQ_2 \bLambda \bQ_2^\top  \leadto \bQ_2^\top\bQ_1 \bLambda = \bLambda \bQ_2^\top\bQ_1.
$$
Let $\bZ = \bQ_2^\top\bQ_1 $, this implies $\bLambda$ and $\bZ$ commute, and $\bZ$ must be a block diagonal matrix whose partitioning conforms to the block structure of $\bLambda$. This results in $\bLambda^{1/2} = \bZ\bLambda^{1/2}\bZ^\top$ and
$$
\bB_2 = \bQ_2 \bLambda^{1/2}\bQ_2^\top = \bQ_2 \bQ_2^\top\bQ_1\bLambda^{1/2} \bQ_1^\top\bQ_2 \bQ_2^\top=\bB_1.
$$
This completes the proof.
\end{proof}
Similarly, we could prove the unique decomposition of PSD matrix $\bA =  \bB^2$ where $\bB$ is PSD. A more detailed discussion on this topic can be referred to \citet{koeber2006unique}.

\paragraph{Decomposition for PD matrices} To conclude, for PD matrix $\bA$, we can factor it into $\bA=\bR^\top\bR$ where $\bR$ is an upper triangular matrix with positive diagonals as shown in Theorem~\ref{theorem:cholesky-factor-exist} (p.~\pageref{theorem:cholesky-factor-exist}) by Cholesky decomposition, $\bA = \bP^\top\bP$ where $\bP$ is nonsingular in Theorem~\ref{lemma:nonsingular-factor-of-PD} (p.~\pageref{lemma:nonsingular-factor-of-PD}), and $\bA = \bB^2$ where $\bB$ is PD in Theorem~\ref{theorem:unique-factor-pd}.

\chapter{Singular Value Decomposition (SVD)}
\begingroup
\hypersetup{linkcolor=mydarkblue}
\minitoc \newpage
\endgroup

\section{Singular Value Decomposition (SVD)}\label{section:SVD}
In the eigenvalue decomposition, we factor the matrix into a diagonal matrix. However, this is not always true. If $\bA$ does not have linearly independent eigenvectors, such diagonalization does not exist. The singular value decomposition (SVD) fills this gap.
The matrix is split into two orthogonal matrices via SVD rather than being factored into an eigenvector matrix.
We provide the result of SVD in the following theorem and we will discuss the existence of SVD in the next sections.

\begin{theoremHigh}[Reduced SVD for Rectangular Matrices]\label{theorem:reduced_svd_rectangular}
For every real $m\times n$ matrix $\bA$ with rank $r$, then matrix $\bA$ can be factored as
$$
\bA = \bU \bSigma \bV^\top,
$$
where $\bSigma\in \real^{r\times r}$ is a diagonal matrix $\bSigma=diag(\sigma_1, \sigma_2 \ldots, \sigma_r)$ with $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r$ and
\begin{itemize}
\item $\sigma_i$'s are the nonzero \textit{singular values} of $\bA$, in the meantime, they are the (positive) square roots of the nonzero \textit{eigenvalues} of $\trans{\bA} \bA$ and $ \bA \trans{\bA}$.

\item Columns of $\bU\in \real^{m\times r}$ contain the $r$ eigenvectors of $\bA\bA^\top$ corresponding to the $r$ nonzero eigenvalues of $\bA\bA^\top$.

\item Columns of $\bV\in \real^{n\times r}$ contain the $r$ eigenvectors of $\bA^\top\bA$ corresponding to the $r$ nonzero eigenvalues of $\bA^\top\bA$.

\item Moreover, the columns of $\bU$ and $\bV$ are called the \textit{left and right singular vectors} of $\bA$, respectively.

\item Further, the columns of $\bU$ and $\bV$ are orthonormal (by Spectral Theorem~\ref{theorem:spectral_theorem}, p.~\pageref{theorem:spectral_theorem}).
\end{itemize}

In particular, we can write out the matrix decomposition by the sum of outer products of vectors $\bA = \bU \bSigma \bV^\top = \sum_{i=1}^r \sigma_i \bu_i \bv_i^\top$, which is a sum of $r$ rank-one matrices.
\end{theoremHigh}

If we append additional $m-r$ silent columns that are orthonormal to the $r$ eigenvectors of $\bA\bA^\top$, just like the silent columns in the QR decomposition (Section~\ref{section:silentcolu_qrdecomp}, p.~\pageref{section:silentcolu_qrdecomp}), we will have an orthogonal matrix $\bU\in \real^{m\times m}$. Similar situation for the columns of $\bV$.
The comparison between the reduced and the full SVD is shown in Figure~\ref{fig:svd-comparison} where white entries are zero and \textcolor{blue}{blue} entries are not necessarily zero.
\begin{figure}[h]
	\centering
	\vspace{-0.35cm}
	\subfigtopskip=2pt
	\subfigbottomskip=2pt
	\subfigcapskip=-5pt
	\subfigure[Reduced SVD decomposition]{\label{fig:svdhalf}
		\includegraphics[width=0.47\linewidth]{./imgs/svdreduced.pdf}}
	\quad
	\subfigure[Full SVD decomposition]{\label{fig:svdall}
		\includegraphics[width=0.47\linewidth]{./imgs/svdfull.pdf}}
	\caption{Comparison between the reduced and full SVD.}
	\label{fig:svd-comparison}
\end{figure}

\section{Existence of the SVD}
To prove the existence of the SVD, we need to use the following lemmas. We mentioned above that the singular values are the square roots of the eigenvalues of $\bA^\top\bA$. While negative values do not have square roots such that the eigenvalues must be nonnegative.
\begin{lemma}[Nonnegative Eigenvalues of $\bA^\top \bA$]\label{lemma:nonneg-eigen-ata}
Given any matrix $\bA\in \real^{m\times n}$, $\bA^\top \bA$ has nonnegative eigenvalues.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:nonneg-eigen-ata}]
For eigenvalue and its corresponding eigenvector $\lambda, \bx$ of $\bA^\top \bA$, we have
$$
\bA^\top \bA \bx = \lambda \bx \leadto \bx^\top \bA^\top \bA \bx = \lambda \bx^\top\bx.
$$
Since $\bx^\top \bA^\top \bA \bx  = ||\bA \bx||^2 \geq 0$ and $\bx^\top\bx \geq 0$. We then have $\lambda \geq 0$.
\end{proof}

Since $\bA^\top\bA$ has nonnegative eigenvalues, we then can define the \textit{singular value} $\sigma\geq 0$ of $\bA$ such that $\sigma^2$ is the eigenvalue of $\bA^\top\bA$, i.e., \fbox{$\bA^\top\bA \bv = \sigma^2 \bv$}. This is essential to the existence of the SVD.

We have shown in Lemma~\ref{lemma:rankAB} (p.~\pageref{lemma:rankAB}) that $rank$($\bA\bB$)$\leq$min$\{rank$($\bA$), $rank$($\bB$)\}.
However, the symmetric matrix $\bA^\top \bA$ is rather special in that the rank of $\bA^\top \bA$ is equal to $rank(\bA)$. And we now prove it.
\begin{lemma}[Rank of $\bA^\top \bA$]\label{lemma:rank-of-ata}
$\bA^\top \bA$ and $\bA$ have same rank.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-of-ata}]
Let $\bx\in \nspace(\bA)$, we have
$$
	\bA\bx  = \bzero \leadto \bA^\top\bA \bx =\bzero,
$$
i.e., $\bx\in \nspace(\bA) \leadtosmall \bx \in \nspace(\bA^\top \bA)$, therefore $\nspace(\bA) \subseteq \nspace(\bA^\top\bA)$.

Further, let $\bx \in \nspace(\bA^\top\bA)$, we have
$$
	\bA^\top \bA\bx = \bzero\leadtosmall \bx^\top \bA^\top \bA\bx = 0\leadtosmall ||\bA\bx||^2 = 0 \leadtosmall \bA\bx=\bzero,
$$
i.e., $\bx\in \nspace(\bA^\top \bA) \leadtosmall \bx\in \nspace(\bA)$, therefore $\nspace(\bA^\top\bA) \subseteq\nspace(\bA) $.

As a result, by ``sandwiching", it follows that
$$\nspace(\bA) = \nspace(\bA^\top\bA) \qquad
\text{and} \qquad
dim(\nspace(\bA)) = dim(\nspace(\bA^\top\bA)).
$$
By the fundamental theorem of linear algebra (Theorem~\ref{theorem:fundamental-linear-algebra}, p.~\pageref{theorem:fundamental-linear-algebra}), $\bA^\top \bA$ and $\bA$ have the same rank.
\end{proof}
Apply the observation to $\bA^\top$, we can also prove that $\bA\bA^\top$ and $\bA$ have the same rank:
$$
rank(\bA) = rank(\bA^\top \bA) = rank(\bA\bA^\top).
$$

In the form of the SVD, we claimed the matrix $\bA$ is a sum of $r$ rank-one matrices where $r$ is the number of nonzero singular values. And the number of nonzero singular values is actually the rank of the matrix.

\begin{lemma}[The Number of Nonzero Singular Values vs the Rank]\label{lemma:rank-equal-singular}
The number of nonzero singular values of matrix $\bA$ equals the rank of $\bA$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:rank-equal-singular}]
The rank of any symmetric matrix (here $\bA^\top\bA$) equals the number of nonzero eigenvalues (with repetitions) by Lemma~\ref{lemma:rank-of-symmetric} (p.~\pageref{lemma:rank-of-symmetric}). So the number of nonzero singular values equals the rank of $\bA^\top \bA$. By Lemma~\ref{lemma:rank-of-ata}, the number of nonzero singular values equals the rank of $\bA$.
\end{proof}

We are now ready to prove the existence of the SVD.
\begin{proof}[\textbf{of Theorem~\ref{theorem:reduced_svd_rectangular}: Existence of the Reduced SVD}]
Since $\bA^\top \bA$ is a symmetric matrix, by Spectral Theorem~\ref{theorem:spectral_theorem} (p.~\pageref{theorem:spectral_theorem}) and Lemma~\ref{lemma:nonneg-eigen-ata}, there exists an orthogonal matrix $\bV$ such that
$$
\boxed{\bA^\top \bA = \bV \bSigma^2 \bV^\top},
$$
where $\bSigma$ is a diagonal matrix containing the singular values of $\bA$, i.e., $\bSigma^2$ contains the eigenvalues of $\bA^\top \bA$.
Specifically, $\bSigma=diag(\sigma_1, \sigma_2, \ldots, \sigma_r)$ and $\{\sigma_1^2, \sigma_2^2, \ldots, \sigma_r^2\}$ are the nonzero eigenvalues of $\bA^\top \bA$ with $r$ being the rank of $\bA$. I.e., $\{\sigma_1, \ldots, \sigma_r\}$ are the singular values of $\bA$. In this case, $\bV\in \real^{n\times r}$.
Now we are into the central part.
\begin{mdframed}[hidealllines=\mdframehidelineNote,backgroundcolor=\mdframecolorSkip]
Start from \fbox{$\bA^\top\bA \bv_i = \sigma_i^2 \bv_i$}, $\forall i \in \{1, 2, \ldots, r\}$, i.e., the eigenvector $\bv_i$ of $\bA^\top\bA$ corresponding to $\sigma_i^2$:

1. Multiply both sides by $\bv_i^\top$:
$$
\bv_i^\top\bA^\top\bA \bv_i = \sigma_i^2 \bv_i^\top \bv_i \leadto ||\bA\bv_i||^2 = \sigma_i^2 \leadto ||\bA\bv_i||=\sigma_i
$$

2. Multiply both sides by $\bA$:
$$
\bA\bA^\top\bA \bv_i = \sigma_i^2 \bA \bv_i \leadto \bA\bA^\top \frac{\bA \bv_i }{\sigma_i}= \sigma_i^2 \frac{\bA \bv_i }{\sigma_i} \leadto \bA\bA^\top \bu_i = \sigma_i^2 \bu_i
$$
where we notice that this form can find the eigenvector of $\bA\bA^\top$ corresponding to $\sigma_i^2$ which is $\bA \bv_i$. Since the length of $\bA \bv_i$ is $\sigma_i$, we then define $\bu_i = \frac{\bA \bv_i }{\sigma_i}$ with norm 1.
\end{mdframed}
These $\bu_i$'s are orthogonal because $(\bA\bv_i)^\top(\bA\bv_j)=\bv_i^\top\bA^\top\bA\bv_j=\sigma_j^2 \bv_i^\top\bv_j=0$. That is
$$
\boxed{\bA \bA^\top = \bU \bSigma^2 \bU^\top}.
$$
Since \fbox{$\bA\bv_i = \sigma_i\bu_i$}, we have
$$
[\bA\bv_1, \bA\bv_2, \ldots, \bA\bv_r] = [ \sigma_1\bu_1,  \sigma_2\bu_2, \ldots,  \sigma_r\bu_r]\leadto
\bA\bV = \bU\bSigma,
$$
which completes the proof.
\end{proof}
By appending silent columns in $\bU$ and $\bV$, we can easily find the full SVD. A byproduct of the above proof is that the spectral decomposition of $\bA^\top\bA = \bV \bSigma^2 \bV^\top$ will result in the spectral decomposition of $\bA \bA^\top = \bU \bSigma^2 \bU^\top$ with the same eigenvalues.

\begin{corollary}[Eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$]
The nonzero eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$ are the same.
\end{corollary}
We have shown in Lemma~\ref{lemma:nonneg-eigen-ata} (p.~\pageref{lemma:nonneg-eigen-ata}) that the eigenvalues of $\bA^\top \bA$ are nonnegative, such that the eigenvalues of $\bA\bA^\top$ are nonnegative as well.
	\begin{corollary}[Nonnegative Eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$]
		The eigenvalues of $\bA^\top\bA$ and $\bA\bA^\top$ are nonnegative.
	\end{corollary}

The existence of the SVD is important for defining the effective rank of a matrix.
\begin{definition}[Effective Rank vs Exact Rank]
	\textit{Effective rank}, or also known as the \textit{numerical rank}.
	Following Lemma~\ref{lemma:rank-equal-singular}, the number of nonzero singular values is equal to the rank of a matrix.
	Assume the $i$-th largest singular value of $\bA$ is denoted as $\sigma_i(\bA)$. Then if $\sigma_r(\bA)\gg \sigma_{r+1}(\bA)\approx 0$, $r$ is known as the numerical rank of $\bA$. Whereas, when $\sigma_i(\bA)>\sigma_{r+1}(\bA)=0$, it is known as having \textit{exact rank} $r$ as we have used in most of our discussions.
\end{definition}
\section{Properties of the SVD}\label{section:property-svd}
\subsection{Four Subspaces in SVD}\label{section:four-space-svd}
For any matrix $\bA\in  \real^{m\times n}$, we have the following property:

$\bullet$ $\nspace(\bA)$ is the orthogonal complement of the row space $\cspace(\bA^\top)$ in $\real^n$: $dim(\nspace(\bA))+dim(\cspace(\bA^\top))=n$;

$\bullet$ $\nspace(\bA^\top)$ is the orthogonal complement of the column space $\cspace(\bA)$ in $\real^m$: $dim(\nspace(\bA^\top))+dim(\cspace(\bA))=m$;

This is called the fundamental theorem of linear algebra and is also known as the rank-nullity theorem (Theorem~\ref{theorem:fundamental-linear-algebra}, p.~\pageref{theorem:fundamental-linear-algebra}). From the SVD, we can find an orthonormal basis for each subspace.

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.98\textwidth]{imgs/lafundamental3-SVD.pdf}
	\caption{Orthonormal bases that diagonalize $\bA$ from SVD.}
	\label{fig:lafundamental3-SVD}
\end{figure}

\begin{lemma}[Four Orthonormal Basis]\label{lemma:svd-four-orthonormal-Basis}
Given the full SVD of matrix $\bA = \bU \bSigma \bV^\top$, where $\bU=[\bu_1, \bu_2, \ldots,\bu_m]$ and $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ are the column partitions of $\bU$ and $\bV$. Then, we have the following property:

$\bullet$ $\{\bv_1, \bv_2, \ldots, \bv_r\} $ is an orthonormal basis of $\cspace(\bA^\top)$;

$\bullet$ $\{\bv_{r+1},\bv_{r+2}, \ldots, \bv_n\}$ is an orthonormal basis of $\nspace(\bA)$;

$\bullet$ $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is an orthonormal basis of $\cspace(\bA)$;

$\bullet$ $\{\bu_{r+1}, \bu_{r+2},\ldots,\bu_m\}$ is an orthonormal basis of $\nspace(\bA^\top)$.

The relationship of the four subspaces is demonstrated in Figure~\ref{fig:lafundamental3-SVD} where $\bA$ transfer the row basis $\bv_i$ into column basis $\bu_i$ by $\sigma_i\bu_i=\bA\bv_i$ for all $i\in \{1, 2, \ldots, r\}$.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:svd-four-orthonormal-Basis}]
From Lemma~\ref{lemma:rank-of-symmetric} (p.~\pageref{lemma:rank-of-symmetric}), for symmetric matrix $\bA^\top\bA$, $\cspace(\bA^\top\bA)$ is spanned by the eigenvectors, thus $\{\bv_1,\bv_2, \ldots, \bv_r\}$ is an orthonormal basis of $\cspace(\bA^\top\bA)$.

Since,

1. $\bA^\top\bA$ is symmetric, then the row space of $\bA^\top\bA$ equals the column space of $\bA^\top\bA$.

2. All rows of $\bA^\top\bA$ are the combinations of the rows of $\bA$, so the row space of $\bA^\top\bA$ $\subseteq$ the row space of $\bA$, i.e., $\cspace(\bA^\top\bA) \subseteq \cspace(\bA^\top)$.

3. Since $rank(\bA^\top\bA) = rank(\bA)$ by Lemma~\ref{lemma:rank-of-ata} (p.~\pageref{lemma:rank-of-ata}), we then have

The row space of $\bA^\top\bA$ = the column space of $\bA^\top\bA$ =  the row space of $\bA$, i.e., $\cspace(\bA^\top\bA) = \cspace(\bA^\top)$. Thus $\{\bv_1, \bv_2,\ldots, \bv_r\}$ is an orthonormal basis of $\cspace(\bA^\top)$.

Further, the space spanned by $\{\bv_{r+1}, \bv_{r+2},\ldots, \bv_n\}$ is an orthogonal complement to the space spanned by $\{\bv_1,\bv_2, \ldots, \bv_r\}$, so $\{\bv_{r+1},\bv_{r+2}, \ldots, \bv_n\}$ is an orthonormal basis of $\nspace(\bA)$.

If we apply this process to $\bA\bA^\top$, we will prove the rest claims in the lemma. Also, we can see that $\{\bu_1,\bu_2, \ldots,\bu_r\}$ is a basis for the column space of $\bA$ by Lemma~\ref{lemma:column-basis-from-row-basis} (p.~\pageref{lemma:column-basis-from-row-basis}) \footnote{For any matrix $\bA$, let $\{\br_1, \br_2, \ldots, \br_r\}$ be a set of vectors in $\real^n$ which forms a basis for the row space, then $\{\bA\br_1, \bA\br_2, \ldots, \bA\br_r\}$ is a basis for the column space of $\bA$.}, since $\bu_i = \frac{\bA\bv_i}{\sigma_i},\, \forall i \in\{1, 2, \ldots, r\}$.
\end{proof}

\subsection{Relationship between Singular Values and Determinant}
Let $\bA\in \real^{n\times n}$ be a square matrix and the singular value decomposition of $\bA$ is given by $\bA = \bU\bSigma\bV^\top$, it follows that
$$
|\det(\bA)| = |\det(\bU\bSigma\bV^\top)| = |\det(\bSigma)| = \sigma_1 \sigma_2\ldots \sigma_n.
$$
If all the singular values $\sigma_i$ are nonzero, then $\det(\bA)\neq 0$. That is, $\bA$ is \textbf{nonsingular}. If there is at least one singular value such that $\sigma_i =0$, then $\det(\bA)=0$, and $\bA$ does not have full rank, and is not invertible. Then the matrix is called \textbf{singular}. This is why $\sigma_i$'s are known as the singular values.

\index{Orthogonal equivalence}
\subsection{Orthogonal Equivalence}
We have defined in Definition~\ref{definition:similar-matrices} (p.~\pageref{definition:similar-matrices}) that $\bA$ and $\bP\bA\bP^{-1}$ are similar matrices for any nonsingular matrix $\bP$. The \textit{orthogonal equivalence} is defined in a similar way.

\begin{definition}[Orthogonal Equivalent Matrices\index{Orthogonal equivalent matrices}]
Given any orthogonal matrices $\bU$ and $\bV$, the matrices $\bA$ and $\bU\bA\bV$ are called \textbf{orthogonal equivalent matrices}. Or \textbf{unitary equivalent} in the complex domain when $\bU$ and $\bV$ are unitary.
\end{definition}
Then, we have the following property for orthogonal equivalent matrices.

\begin{lemma}[Orthogonal Equivalent Matrices]\label{lemma:orthogonal-equivalent-matrix}
Given any orthogonal equivalent matrices $\bA$ and $\bB$, then singular values are the same.
\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:orthogonal-equivalent-matrix}]
Since $\bA$ and $\bB$ are orthogonal equivalent, there exist orthogonal matrices that $\bB = \bU\bA\bV$. We then have
$$
\bB\bB^\top = (\bU\bA\bV)(\bV^\top\bA^\top\bU^\top) = \bU\bA\bA^\top\bU^\top.
$$
This implies $\bB\bB^\top$ and $\bA\bA^\top$ are similar matrices. By Lemma~\ref{lemma:eigenvalue-similar-matrices} (p.~\pageref{lemma:eigenvalue-similar-matrices}), the eigenvalues of similar matrices are the same, which proves the singular values of $\bA$ and $\bB$ are the same.
\end{proof}

\subsection{SVD for QR}
	\begin{lemma}[Orthogonal Equivalent Matrices]\label{lemma:svd-for-qr}
	Suppose the full QR decomposition for matrix $\bA\in \real^{m\times n}$ with $m\geq n$ is given by $\bA=\bQ\bR$ where $\bQ\in \real^{m\times m}$ is orthogonal and $\bR\in \real^{m\times n}$ is upper triangular. Then $\bA$ and $\bR$ have the same singular values and right singular vectors.
	\end{lemma}
\begin{proof}[of Lemma~\ref{lemma:svd-for-qr}]
We notice that $\bA^\top\bA = \bR^\top\bR$ such that $\bA^\top\bA$ and $\bR^\top\bR$ have the same eigenvalues and eigenvectors, i.e., $\bA$ and $\bR$ have the same singular values and right singular vectors (i.e., the eigenvectors of $\bA^\top\bA$ or $\bR^\top\bR$).
\end{proof}

The above lemma implies that an SVD of a matrix can be constructed by the QR decomposition of itself. Suppose the QR decomposition of $\bA$ is given by $\bA=\bQ\bR$ and the SVD of $\bR$ is given by $\bR=\bU_0 \bSigma\bV^\top$. Therefore, the SVD of $\bA$ can be obtained by
$$
\bA =\underbrace{ \bQ\bU_0}_{\bU} \bSigma\bV^\top.
$$

\section{Polar Decomposition}

\begin{theoremHigh}[Polar Decomposition]\label{theorem:polar-decomposition}
Given real $n\times n$ square matrix $\bA$ with rank $r$, then matrix $\bA$ can be factored as
$$
\bA = \bQ_l \bS,
$$
where $\bQ_l$ is an orthogonal matrix, and $\bS$ is a positive semidefinite matrix. And this form is called the \textit{left polar decomposition}. Also matrix $\bA$ can be factored as
$$
\bA =  \bS\bQ_r,
$$
where $\bQ_r$ is an orthogonal matrix, and $\bS$ is a positive semidefinite matrix. And this form is called the \textit{right polar decomposition}.

Specially, the left and right polar decomposition of a square matrix $\bA$ is \textbf{unique}.
\end{theoremHigh}

Since every $n\times n$ square matrix $\bA$ has full SVD $\bA = \bU \bSigma \bV^\top$, where both $\bU$ and $\bV$ are $n\times n$ orthogonal matrix. We then have $\bA = (\bU\bV^\top)( \bV\bSigma \bV^\top) = \bQ_l\bS$ where it can be easily verified that $\bQ_l = \bU\bV^\top$ is an orthogonal matrix and $\bS = \bV\bSigma \bV^\top$ is a symmetric matrix. We notice that the singular values in $\bSigma$ are nonnegative, such that  $\bS=\bV\bSigma \bV^\top = \bV\bSigma^{1/2}\bSigma^{1/2\top} \bV^\top$ showing that $\bS$ is PSD.

Similarly, we have $\bA = \bU \bSigma \bU^\top \bU \bV^\top = (\bU \bSigma \bU^\top)( \bU \bV^\top)=\bS\bQ_r$. And $\bS=\bU \bSigma \bU^\top = \bU\bSigma^{1/2}\bSigma^{1/2\top} \bU^\top$ such that $\bS$ is PSD as well.

For the uniqueness of the right polar decomposition, we suppose the decomposition is not unique, and two of the decompositions are given by
$$
\bA = \bS_1\bQ_1 = \bS_2\bQ_2,
$$
such that
$$
\bS_1= \bS_2\bQ_2\bQ_1^\top.
$$
Since $\bS_1$ and $\bS_2$ are symmetric, we have
$$
\bS_1^2 = \bS_1\bS_1^\top =  \bS_2\bQ_2\bQ_1^\top\bQ_1 \bQ_2^\top\bS_2 = \bS_2^2.
$$
This implies $\bS_1=\bS_2$, and the decomposition is unique (Theorem~\ref{theorem:unique-factor-pd}, p.~\pageref{theorem:unique-factor-pd}). Similarly, the uniqueness of the left polar decomposition can be implied from the context.

\begin{corollary}[Full Rank Polar Decomposition]
When $\bA\in \real^{n\times n}$ has full rank, then the $\bS$ in both the left and right polar decomposition above is a symmetric positive definite matrix.
\end{corollary}

\section{Application: Least Squares via the Full QR Decomposition, UTV, SVD}\label{section:application-ls-qr}
\subsection*{\textbf{Least Squares via the Full QR Decomposition}}

Let's consider the overdetermined system $\bA\bx = \bb$, where $\bA\in \real^{m\times n}$ is the data matrix, $\bb\in \real^m$ with $m\geq n$ is the observation matrix. Normally $\bA$ will have full column rank since the data from real work has a large chance to be unrelated. And the least squares (LS) solution is given by $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$ for minimizing $||\bA\bx-\bb||^2$, where $\bA^\top\bA$ is invertible since $\bA$ has full column rank and $rank(\bA^T\bA)=rank(\bA)$.\index{Least squares}

However, the inverse of a matrix is not easy to compute, we can then use QR decomposition to find the least squares solution as illustrated in the following theorem.
\begin{theorem}[LS via QR for Full Column Rank Matrix]\label{theorem:qr-for-ls}
	Let $\bA\in \real^{m\times n}$ and $\bA=\bQ\bR$ is its full QR decomposition with $\bQ\in\real^{m\times m}$ being an orthogonal matrix, $\bR\in \real^{m\times n}$ being an upper triangular matrix appended by additional $m-n$ zero rows, and $\bA$ has full column rank with $m\geq n$. Suppose $\bR = \begin{bmatrix}
		\bR_1 \\
		\bzero
	\end{bmatrix}$, where $\bR_1 \in \real^{n\times n}$ is the square upper triangular in $\bR$, $\bb\in \real^m$, then the LS solution to $\bA\bx=\bb$ is given by
	$$
	\bx_{LS} = \bR_1^{-1}\bc,
	$$
	where $\bc$ is the first $n$ components of $\bQ^\top\bb$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:qr-for-ls}]
Since $\bA=\bQ\bR$ is the full QR decomposition of $\bA$ and $m\geq n$, the last $m-n$ rows of $\bR$ are zero as shown in Figure~\ref{fig:qr-comparison} (p.~\pageref{fig:qr-comparison}). Then $\bR_1 \in \real^{n\times n}$ is the square upper triangular in $\bR$ and
$
\bQ^\top \bA = \bR = \begin{bmatrix}
	\bR_1 \\
	\bzero
\end{bmatrix}.
$
Thus,
$$
\begin{aligned}
	||\bA\bx-\bb||^2 &= (\bA\bx-\bb)^\top(\bA\bx-\bb)\\
	&=(\bA\bx-\bb)^\top\bQ\bQ^\top (\bA\bx-\bb) \qquad &(\text{Since $\bQ$ is an orthogonal matrix})\\
	&=||\bQ^\top \bA \bx-\bQ^\top\bb||^2 \qquad &(\text{Invariant under orthogonal})\\
	&=\left\Vert\begin{bmatrix}
		\bR_1 \\
		\bzero
	\end{bmatrix} \bx-\bQ^\top\bb\right\Vert^2\\
	&=||\bR_1\bx - \bc||^2+||\bd||^2,
\end{aligned}
$$
where $\bc$ is the first $n$ components of $\bQ^\top\bb$ and $\bd$ is the last $m-n$ components of $\bQ^\top\bb$. And the LS solution can be calculated by back substitution of the upper triangular system $\bR_1\bx = \bc$, i.e., $\bx_{LS} = \bR_1^{-1}\bc$.
\end{proof}

To verify Theorem~\ref{theorem:qr-for-ls}, for the full QR decomposition of $\bA = \bQ\bR$ where $\bQ\in \real^{m\times m}$ and $\bR\in \real^{m\times n}$. Together with the LS solution $\bx_{LS} = (\bA^\top\bA)^{-1}\bA^\top\bb$, we obtain
\begin{equation}\label{equation:qr-for-ls-1}
	\begin{aligned}
		\bx_{LS} &= (\bA^\top\bA)^{-1}\bA^\top\bb \\
		&= (\bR^\top\bQ^\top\bQ\bR)^{-1} \bR^\top\bQ^\top \bb\\
		&= (\bR^\top\bR)^{-1} \bR^\top\bQ^\top \bb \\
		&= (\bR_1^\top\bR_1)^{-1} \bR^\top\bQ^\top \bb \\
		&=\bR_1^{-1} \bR_1^{-\top} \bR^\top\bQ^\top \bb\\
		&= \bR_1^{-1} \bR_1^{-\top} \bR_1^\top\bQ_1^\top \bb \\
		&=\bR_1^{-1} \bQ_1^\top \bb,
	\end{aligned}
\end{equation}
where $\bR = \begin{bmatrix}
	\bR_1 \\
	\bzero
\end{bmatrix}$ and $\bR_1\in \real^{n\times n}$  is an upper triangular matrix, and $\bQ_1 =\bQ_{1:m,1:n}\in \real^{m\times n}$ is the first $n$ columns of $\bQ$ (i.e., $\bQ_1\bR_1$ is the reduced QR decomposition of $\bA$). Then the result of Equation~\eqref{equation:qr-for-ls-1} agrees with Theorem~\ref{theorem:qr-for-ls}.

To conclude, using the QR decomposition, we first derive directly the least squares result which results in the argument in Theorem~\ref{theorem:qr-for-ls}. Moreover, we verify the result of LS from calculus indirectly by the QR decomposition as well. The two results coincide with each other. For readers who are interested in LS in linear algebra, a pictorial view of least squares for full column rank $\bA$ in the fundamental theorem of linear algebra is provided in \citet{lu2021revisit}.

\subsection*{\textbf{Least Squares via ULV/URV for Rank Deficient Matrices}}
In the above section, we introduced the LS via the full QR decomposition for full rank matrices.
However, if often happens that the matrix may be rank-deficient.  If $\bA$ does not have full column rank, $\bA^\top\bA$ is not invertible.
We can then use the ULV/URV decomposition to find the least squares solution as illustrated in the following theorem.\index{Least squares}\index{Rank deficient}
\begin{theorem}[LS via ULV/URV for Rank Definient Matrix]\label{theorem:qr-for-ls-urv}
	Let $\bA\in \real^{m\times n}$ with rank $r$ and $m\geq n$. Suppose $\bA=\bU\bT\bV$ is its full ULV/URV decomposition with $\bU\in\real^{m\times m}, \bV\in \real^{n\times n}$ being orthogonal matrix matrices, and
	$$
	\bT = \begin{bmatrix}
		\bT_{11} & \bzero \\
		\bzero & \bzero
	\end{bmatrix}
	$$
	where $\bT_{11} \in \real^{r\times r}$ is a lower triangular matrix or an upper triangular matrix.

		Suppose $\bb\in \real^m$, then the LS solution with the minimal 2-norm to $\bA\bx=\bb$ is given by
		$$
		\bx_{LS} = \bV^\top
		\begin{bmatrix}
			\bT_{11}^{-1}\bc\\
			\bzero
		\end{bmatrix},
		$$
		where $\bc$ is the first $r$ components of $\bU^\top\bb$.
	\end{theorem}

	\begin{proof}[of Theorem~\ref{theorem:qr-for-ls-urv}]
		Since $\bA=\bQ\bR$ is the full QR decomposition of $\bA$ and $m\geq n$, the last $m-n$ rows of $\bR$ are zero as shown in Figure~\ref{fig:qr-comparison} (p.~\pageref{fig:qr-comparison}). Then $\bR_1 \in \real^{n\times n}$ is the square upper triangular in $\bR$ and
		$
		\bQ^\top \bA = \bR = \begin{bmatrix}
			\bR_1 \\
			\bzero
		\end{bmatrix}.
		$
		Thus,
		$$
		\begin{aligned}
			||\bA\bx-\bb||^2 &= (\bA\bx-\bb)^\top(\bA\bx-\bb)\\
			&=(\bA\bx-\bb)^\top\bU\bU^\top (\bA\bx-\bb) \qquad &(\text{Since $\bU$ is an orthogonal matrix})\\
			&=||\bU^\top \bA \bx-\bU^\top\bb||^2 \qquad &(\text{Invariant under orthogonal})\\
			&=|| \bU^\top\bU\bT\bV \bx-\bU^\top\bb||^2\\
			&=|| \bT\bV \bx-\bU^\top\bb||^2\\
			&=||\bT_{11}\be - \bc||^2+||\bd||^2,
		\end{aligned}
		$$
		where $\bc$ is the first $r$ components of $\bU^\top\bb$ and $\bd$ is the last $m-r$ components of $\bU^\top\bb$, $\be$ is the first $r$ components of $\bV\bx$ and $\bff$ is the last $n-r$ components of $\bV\bx$:
		$$
		\bU^\top\bb
		= \begin{bmatrix}
			\bc \\
			\bd
		\end{bmatrix},
		\qquad
		\bV\bx
		= \begin{bmatrix}
			\be \\
			\bff
		\end{bmatrix}
		$$
		And the LS solution can be calculated by back/forward substitution of the upper/lower triangular system $\bT_{11}\be = \bc$, i.e., $\be = \bT_{11}^{-1}\bc$. For $\bx$ to have a minimal 2-norm, $\bff$ must be zero. That is
		$$
		\bx_{LS} = \bV^\top
		\begin{bmatrix}
			\bT_{11}^{-1}\bc\\
			\bzero
		\end{bmatrix}.
		$$
		This completes the proof.
	\end{proof}

	\paragraph{A word on the minimal 2-norm LS solution} For the least squares problem, the set of all minimizers
	$$
	\mathcal{X} = \{\bx\in \real^n: ||\bA\bx-\bb|| =\min \}
	$$
	is convex \citep{golub2013matrix}. And if $\bx_1, \bx_2 \in \mathcal{X}$ and $\lambda \in [0,1]$, then
	$$
	||\bA(\lambda\bx_1 + (1-\lambda)\bx_2) -\bb|| \leq \lambda||\bA\bx_1-\bb|| +(1-\lambda)||\bA\bx_2-\bb|| = \mathop{\min}_{\bx\in \real^n} ||\bA\bx-\bb||.
	$$
	Thus $\lambda\bx_1 + (1-\lambda)\bx_2 \in \mathcal{X}$. In the above proof, if we do not set $\bff=\bzero$, we will find more least squares solutions. However, the minimal 2-norm least squares solution is unique. For the full-rank case in the previous section, the least squares solution is unique and it must have a minimal 2-norm. See also \citet{foster2003solving, golub2013matrix} for a more detailed discussion on this topic.

\subsection*{\textbf{Least Squares via SVD for Rank Deficient Matrices}}
\index{Least squares}

Apart from the UTV decomposition for rank-deficient least squares solution, SVD serves as an alternative.
\begin{theorem}[LS via SVD for Rank Deficient Matrix]\label{theorem:svd-deficient-rank}
Let $\bA\in \real^{m\times n}$ and $\bA=\bU\bSigma\bV^\top$ is its full SVD decomposition with $\bU\in\real^{m\times m}$ and $\bV\in \real^{n\times n}$ being orthogonal matrices and $rank(\bA)=r$. Suppose $\bU=[\bu_1, \bu_2, \ldots, \bu_m]$, $\bV=[\bv_1, \bv_2, \ldots, \bv_n]$ and $\bb\in \real^m$, then the LS solution with minimal 2-norm to $\bA\bx=\bb$ is given by
\begin{equation}\label{equation:svd-ls-solution}
\bx_{LS} = \sum_{i=1}^{r}\frac{\bu_i^\top \bb}{\sigma_i}\bv_i = \bV\bSigma^+\bU^\top \bb,
\end{equation}
where the upper-left side of $\bSigma^+ \in \real^{n\times m}$ is a diagonal matrix, $\bSigma^+ = \begin{bmatrix}
	\bSigma_1^+ & \bzero \\
	\bzero & \bzero
\end{bmatrix}$ where $\bSigma_1^+=diag(\frac{1}{\sigma_1}, \frac{1}{\sigma_2}, \ldots, \frac{1}{\sigma_r})$.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:svd-deficient-rank}]
Write out the loss to be minimized
$$
\begin{aligned}
	||\bA\bx-\bb||^2 &= (\bA\bx-\bb)^\top(\bA\bx-\bb)\\
	&=(\bA\bx-\bb)^\top\bU\bU^\top (\bA\bx-\bb) \qquad &(\text{Since $\bU$ is an orthogonal matrix})\\
	&=||\bU^\top \bA \bx-\bU^\top\bb||^2 \qquad &(\text{Invariant under orthogonal})\\
	&=||\bU^\top \bA \bV\bV^\top \bx-\bU^\top\bb||^2 \qquad &(\text{Since $\bV$ is an orthogonal matrix})\\
	&=||\bSigma\balpha - \bU^\top\bb||^2  \qquad &(\text{Let $\balpha=\bV^\top \bx$})\\
	&=\sum_{i=1}^{r}(\sigma_i\balpha_i - \bu_i^\top\bb)^2 +\sum_{i=r+1}^{m}(\bu_i^\top \bb)^2. \qquad &(\text{Since $\sigma_{r+1}=\sigma_{r+2}= \ldots= \sigma_m=0$})
\end{aligned}
$$
Since $\bx$ only appears in $\balpha$, we just need to set $\balpha_i = \frac{\bu_i^\top\bb}{\sigma_i}$ for all $i\in \{1, 2, \ldots, r\}$ to minimize the above equation. For any value of $\balpha_{r+1}, \balpha_{r+2}, \ldots, \balpha_{n}$, it won't change the result. From the regularization point of view (or here, we want the minimal 2-norm) we can set them to be 0. This gives us the LS solution via SVD:
$$
\bx_{LS} = \sum_{i=1}^{r}\frac{\bu_i^\top \bb}{\sigma_i}\bv_i=\bV\bSigma^+\bU^\top \bb = \bA^+\bb,
$$
where $\bA^+=\bV\bSigma^+\bU^\top\in \real^{n\times m}$ is known as the \textit{pseudo-inverse} of $\bA$.\index{Pseudo-inverse}
\end{proof}

\section{Application: Principal Component Analysis (PCA) via the Spectral Decomposition and the SVD}
Given a data set of $n$ observations $\{\bx_1,\bx_2,\ldots,\bx_n\}$ where $\bx_i\in \real^p$ for all $i\in \{1,2,\ldots,n\}$. Our goal is to project the data onto a low-dimensional space, say $m<p$. Define the sample mean vector and sample covariance matrix\index{Principal component analysis}
$$
\overline{\bx} = \frac{1}{n}\sum_{i=1}^{n}\bx_i
\qquad
\text{and}
\qquad
\bS = \frac{1}{n-1}\sum_{i=1}^{n} (\bx_i - \overline{\bx})(\bx_i-\overline{\bx})^\top.
$$
where the $n-1$ term in the covariance matrix is to make it to be an unbiased consistent estimator of the covariance \citep{lu2021rigorous}. Or the covariance matrix can also be defined as $\bS = \frac{1}{\textcolor{blue}{n}}\sum_{i=1}^{n} (\bx_i - \overline{\bx})(\bx_i-\overline{\bx})^\top$ which is also a consistent estimator of covariance matrix \footnote{Consistency: An estimator $\theta_n $ of $\theta$ constructed on the basis of a sample of size $n$ is said to be consistent if $\theta_n\stackrel{p}{\rightarrow} \theta$   as $n \rightarrow \infty $.}.

Each data point $\bx_i$ is then projected onto a scalar value by $\bu_1$ such that $\bu_1^\top\bx_i$. The mean of the projected data is obtained by $\Exp[\bu_1^\top\bx_i] = \bu_1^\top \overline{\bx}$, and the variance of the projected data is given by
$$
\begin{aligned}
\Cov[\bu_1^\top\bx_i] &= \frac{1}{n-1} \sum_{i=1}^{n}( \bu_1^\top \bx_i - \bu_1^\top\overline{\bx})^2=
\frac{1}{n-1} \sum_{i=1}^{n}\bu_1^\top  ( \bx_i -\overline{\bx})( \bx_i -\overline{\bx})^\top\bu_1\\
&=\bu_1^\top\bS\bu_1.
\end{aligned}
$$
We want to maximize the projected variance $\bu_1^\top\bS\bu_1$ with respect to $\bu_1$ where we must constrain $||\bu_1||$ to prevent $||\bu_1|| \rightarrow \infty$ by setting it to be $\bu_1^\top\bu_1=1$. By Lagrange multiplier (see \citet{bishop2006pattern, boyd2004convex}), we have
$$
\bu_1^\top\bS\bu_1 + \lambda_1 (1 - \bu_1^\top\bu_1).
$$
A trivial calculation will lead to
$$
\bS\bu_1 = \lambda_1\bu_1 \leadto \bu_1^\top\bS\bu_1 = \lambda_1.
$$
That is, $\bu_1$ is an eigenvector of $\bS$ corresponding to eigenvalue $\lambda_1$. And the maximum variance projection $\bu_1$ is corresponding to the largest eigenvalues of $\bS$. The eigenvector is known as the \textit{first principal axis}.

Define the other principal axes by decremental eigenvalues until we have $m$ such principal components. The procedure brings about the dimension reduction. This is known as the \textit{maximum variance formulation} of PCA \citep{hotelling1933analysis, bishop2006pattern, shlens2014tutorial}. A \textit{minimum-error formulation} of PCA is discussed in \citet{pearson1901liii, bishop2006pattern}.

\paragraph{PCA via the spectral decomposition}
Now let's assume the data are centered such that $\overline{\bx}$ is zero, or we can set $\bx_i = \bx_i-\overline{\bx}$ to centralize the data. Let the data matrix $\bX \in \real^{n\times p}$ contain the data observations as rows. The covariance matrix is given by
$$
\bS= \frac{\bX^\top\bX}{n-1},
$$
which is a symmetric matrix, and its spectral decomposition is given by
\begin{equation}\label{equation:pca-equ1}
\bS = \bU\bLambda\bU^\top,
\end{equation}
where $\bU$ is an orthogonal matrix of eigenvectors (columns of $\bU$ are eigenvectors of $\bS$), and $\bLambda=diag(\lambda_1, \lambda_2,\ldots, \lambda_p)$ is a diagonal matrix with eigenvalues (ordered such that $\lambda_1 \geq \lambda_2 \geq \ldots \geq \lambda_p$). The eigenvectors are called \textit{principal axes} of the data, and they \textit{decorrelate} the the covariance matrix. Projections of the data on the principal axes are called the \textit{principal components}. The $i$-th principal component is given by the $i$-th column of $\bX\bU$. If we want to reduce the dimension from $p$ to $m$, we just select the first $m$ columns of $\bX\bU$.

\paragraph{PCA via the SVD}
If the SVD of $\bX$ is given by $\bX = \bP\bSigma\bQ^\top$, then the covariance matrix can be written as
\begin{equation}\label{equation:pca-equ2}
\bS= \frac{\bX^\top\bX}{n-1} = \bQ \frac{\bSigma^2}{n-1}\bQ^\top,
\end{equation}
where $\bQ\in \real^{p\times p}$ is an orthogonal matrix and contains the right singular vectors of $\bX$, and the upper left of $\bSigma$ is a diagonal matrix containing the singular values $diag(\sigma_1,\sigma_2,\ldots)$ with $\sigma_1\geq \sigma_2\geq \ldots$. The number of singular values is equal to $\min\{n,p\}$ which will not be larger than $p$ and some of which are zeros.

The above Equation~\eqref{equation:pca-equ2} compared with Equation~\eqref{equation:pca-equ1} implies Equation~\eqref{equation:pca-equ2} is also a spectral decomposition of $\bS$,
since the eigenvalues in $\bLambda$ and singular values in $\bSigma$ are ordered in a descending way, and the spectral decomposition in terms of the eigenspaces is unique (Section~\ref{section:uniqueness-spectral-decomposition}, p.~\pageref{section:uniqueness-spectral-decomposition}).

That is, the right singular vectors $\bQ$ are also the principal axes which decorrelate the covariance matrix,
and the singular values are related to the eigenvalues of the covariance matrix via $\lambda_i = \frac{\sigma_i^2 }{n-1}$. To reduce the dimensionality of the data from $p$ to $m$, we should select largest $m$ singular values and the corresponding right singular vectors. This is also related to the truncated SVD (TSVD) $\bX_m = \sum_{i=1}^{m}\sigma_i \bp_i\bq_i^\top$ that will be discussed in the next section, where $\bp_i$'s and $\bq_i$'s are the columns of $\bP$ and $\bQ$.

\paragraph{A byproduct of PCA via the SVD for high-dimensional data} For a principle axis $\bu_i$ of $\bS = \frac{\bX^\top\bX}{n-1}$, we have
$$
\frac{\bX^\top\bX}{n-1} \bu_i = \lambda_i \bu_i.
$$
Left multiply by $\bX$, we obtain
$$
\frac{\bX\bX^\top}{n-1} (\bX\bu_i) = \lambda_i (\bX\bu_i),
$$
which implies $\lambda_i$ is also an eigenvalue of $\frac{\bX\bX^\top}{n-1} \in \real^{n\times n}$, and the corresponding eigenvector is $\bX\bu_i$. This is also stated in the proof of Theorem~\ref{theorem:reduced_svd_rectangular} (p.~\pageref{theorem:reduced_svd_rectangular}), the existence of the SVD. If $p \gg n$, instead of finding the eigenvector of $\bS$, i.e., the principle axes of $\bS$, we can find the eigenvector of $\frac{\bX\bX^\top}{n-1}$. This reduces the complexity from $O(p^3)$ to $O(n^3)$. Suppose now, the eigenvector of $\frac{\bX\bX^\top}{n-1}$ is $\bv_i$ corresponding to nonzero eigenvalue $\lambda_i$,
$$
\frac{\bX\bX^\top}{n-1} \bv_i = \lambda_i \bv_i.
$$
Left multiply by $\bX^\top$, we obtain
$$
\frac{\bX^\top\bX}{n-1} (\bX^\top\bv_i) = \bS(\bX^\top\bv_i)   = \lambda_i (\bX^\top\bv_i),
$$
i.e., the eigenvector $\bu_i$ of $\bS$, is proportional to $\bX^\top\bv_i$, where $\bv_i$ is the eigenvector of $\frac{\bX\bX^\top}{n-1}$ corresponding to the same eigenvalue $\lambda_i$. A further normalization step is needed to make $||\bu_i||=1$.

\section{Application: Low-Rank Approximation}\label{section:svd-low-rank-approxi}
For a low-rank approximation problem, there are basically two types related due to the interplay of rank and error: \textit{fixed-precision approximation problem} and \textit{fixed-rank
 approximation problem}. In the fixed-precision approximation problem, for a given matrix $\bA$ and a given tolerance $\epsilon$, one wants to find a matrix $\bB$
with rank $r = r(\epsilon)$ such that $||\bA-\bB|| \leq \epsilon$ in an appropriate matrix norm. On the contrary, in the fixed-rank approximation problem, one looks for a matrix $\bB$ with fixed rank $k$ and an error $||\bA-\bB||$ as small as possible. In this section, we will consider the latter. Some excellent examples can also be found in \citet{kishore2017literature, martinsson2019randomized}.

Suppose we want to approximate matrix $\bA\in \real^{m\times n}$ with rank $r$ by a rank $k<r$ matrix $\bB$. The approximation is measured by spectral norm:\index{Low-rank approximation}
$$
\bB = \mathop{\arg\min}_{\bB} \ \ || \bA - \bB||_2,
$$
where the spectral norm is defined as follows:
\begin{definition}[Spectral Norm]\label{definition:spectral_norm}
The spectral norm of a matrix $\bA\in \real^{m\times n}$ is defined as
$$
||\bA||_2 = \mathop{\max}_{\bx\neq\bzero} \frac{||\bA\bx||_2}{||\bx||_2}  =\mathop{\max}_{\bu\in \real^n: ||\bu||_2=1}  ||\bA\bx||_2 ,
$$
which is also the maximal singular value of $\bA$, i.e., $||\bA||_2 = \sigma_1(\bA)$.
\end{definition}

Then, we can recover the best rank-$k$ approximation by the following theorem.
\begin{theorem}[Eckart-Young-Misky Theorem w.r.t. Spectral Norm\index{Eckart-Young-Misky theorem}]\label{theorem:young-theorem-spectral}
Given matrix $\bA\in \real^{m\times n}$ and $1\leq k\leq rank(\bA)=r$, and let $\bA_k$ be the truncated SVD (TSVD) of $\bA$ with the largest $k$ terms, i.e., $\bA_k = \sum_{i=1}^{k} \sigma_i\bu_i\bv_i^\top$ from SVD of $\bA=\sum_{i=1}^{r} \sigma_i\bu_i\bv_i^\top$ by zeroing out the $r-k$ trailing singular values of $\bA$. Then $\bA_k$ is the best rank-$k$ approximation to $\bA$ in terms of the spectral norm.
\end{theorem}

\begin{proof}[of Theorem~\ref{theorem:young-theorem-spectral}]
We need to show for any matrix $\bB$, if $rank(\bB)=k$, then $||\bA-\bB||_2 \geq ||\bA-\bA_k||_2$.

Since $rank(\bB)=k$, then $dim (\nspace(\bB))=n-k$. As a result, any $k+1$ basis in $\real^n$ intersects $\nspace(\bB)$. As shown in Lemma~\ref{lemma:svd-four-orthonormal-Basis} (p.~\pageref{lemma:svd-four-orthonormal-Basis}), $\{\bv_1,\bv_2, \ldots, \bv_r\}$ is an orthonormal basis of $\cspace(\bA^\top)\subset \real^n$, so that we can choose the first $k+1$ $\bv_i$'s as a $k+1$ basis for $\real^n$. Let $\bV_{k+1} = [\bv_1, \bv_2, \ldots, \bv_{k+1}]$, then there is a vector $\bx$ that
$$
\bx \in \nspace(\bB) \cap \cspace(\bV_{k+1}),\qquad s.t.\,\,\,\, ||\bx||_2=1.
$$
That is $\bx = \sum_{i=1}^{k+1} a_i \bv_i$, and $||\sum_{i=1}^{k+1} a_i \bv_i||_2 = \sum_{i=1}^{k+1}a_i^2=1$.
Thus,
$$
\begin{aligned}
	||\bA-\bB||_2^2 &\geq ||(\bA-\bB)\bx||_2^2\cdot ||\bx||_2^2, \qquad &(\text{From defintion of spectral norm}) \\
	&= ||\bA\bx||_2^2, \qquad &(\text{$\bx$ in null space of $\bB$}) \\
	&=\sum_{i=1}^{k+1} \sigma_i^2 (\bv_i^\top \bx)^2, \qquad &(\text{$\bx$ orthogonal to $\bv_{k+2}, \ldots, \bv_r$}) \\
	&\geq \sigma_{k+1}^2\sum_{i=1}^{k+1}  (\bv_i^\top \bx)^2, \qquad &(\sigma_{k+1}\leq \sigma_{k}\leq\ldots\leq \sigma_{1}) \\
	&\geq \sigma_{k+1}^2\sum_{i=1}^{k+1} a_i^2, \qquad &(\bv_i^\top \bx = a_i) \\
	&= \sigma_{k+1}^2.
\end{aligned}
$$
It is trivial that $||\bA-\bA_k||_2^2 = ||\sum_{i=k+1}^{r}\sigma_i\bu_i\bv_i^\top||_2^2=\sigma_{k+1}^2$. Thus, $||\bA-\bA_k||_2 \leq ||\bA-\bB||_2$, which completes the proof.
\end{proof}

Moreover, readers can prove that $\bA_k$ is the best rank-$k$ approximation to $\bA$ in terms of the Frobenius norm. The minimal error is given by the Euclidean norm of the singular values that have been zeroed out in the process $||\bA-\bA_k||_F =\sqrt{\sigma_{k+1}^2 +\sigma_{k+2}^2+\ldots +\sigma_{r}^2}$.

SVD thus gives the best approximation of a matrix. As mentioned in \citet{stewart1998matrix, kishore2017literature}, \emph{the singular value decomposition is the creme de la creme of rank-reducing decompositions — the decomposition that all others try to beat}. And also \emph{The SVD is the climax of this linear algebra course} in \citet{strang1993introduction}.

\section{Polar decomposition}

\chapter*{\centering \begin{normalsize}Preface\end{normalsize}}
In 1954, Alston S. Householder published \textit{Principles of Numerical Analysis}, one of the first modern treatments on matrix decomposition that favored a (block) LU decomposition-the factorization of a matrix into the product of lower and upper triangular matrices.
And now, matrix decomposition has become a core technology in machine learning, largely due to the development of the back propagation algorithm in fitting a neural network. The sole aim of this book is to give a self-contained introduction to concepts and mathematical tools in numerical linear algebra and matrix analysis in order to seamlessly introduce matrix decomposition techniques and their applications in subsequent sections. However, we clearly realize our inability to cover all the useful and interesting results concerning matrix decomposition and given the paucity of scope to present this discussion, e.g., the separated analysis of the Euclidean space, Hermitian space, Hilbert space, and things in the complex domain. We refer the reader to literature in the field of linear algebra for a more detailed introduction to the related fields.

This book is primarily a summary of purpose, significance of important matrix decomposition methods, e.g., LU, QR, and SVD, and the origin and complexity of the methods which shed light on their modern applications.
Most importantly, this article presents improved procedures for most of the calculations of the decomposition algorithms which potentially reduce the complexity they induce.
Again, this is a decomposition-based context, thus we will introduce the related background when it is needed and necessary. In many other textbooks on linear algebra, the principal ideas are discussed and the matrix decomposition methods serve as a ``byproduct". However, we focus on the decomposition methods instead and the principal ideas serve as fundamental tools for them. The mathematical prerequisite is a first course in linear algebra. Other than this modest background, the development is self-contained, with rigorous proof provided throughout.

\chapter*{\centering \begin{normalsize}Keywords\end{normalsize}}
Existence and computing of matrix decompositions, Complexity, Floating point operations (flops), Low-rank approximation, Pivot, LU decomposition for nonzero leading principal minors, Data distillation, CR decomposition, CUR/Skeleton decomposition, Interpolative decomposition, Biconjugate decomposition, Coordinate transformation, Hessenberg decomposition, ULV decomposition, URV decomposition, Rank decomposition, Gram-Schmidt process, Householder reflector, Givens rotation, Rank revealing decomposition, Cholesky decomposition and update/downdate, Eigenvalue problems, Alternating least squares, Randomized algorithm.

\input{chapter-worldmap.tex}
\input{chapter-intro.tex}
\input{chapter-LU.tex}
\input{chapter-qr.tex}
\input{chapter-cr.tex}
\input{chapter-skeleton.tex}
\input{chapter-hessenberg.tex}
\input{chapter-eigenvalue.tex}
\input{chapter-spectral.tex}
\input{chapter-svd.tex}
\input{chapter-polar.tex}
\input{chapter-als.tex}
\input{chapter-NMF.tex}
\input{chapter-biconjugate.tex}

\newpage
\vskip 0.2in

Numerical Analysis (math.NA); Machine Learning (cs.LG)

Matrix Decomposition and Applications