Unsupervised and Supervised Principal Component Analysis: Tutorial

Abstract: This is a detailed tutorial paper which explains the Principal Component Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start with projection, PCA with eigen-decomposition, PCA with one and multiple projection directions, properties of the projection matrix, reconstruction error minimization, and we connect to autoencoder. Then, PCA with singular value decomposition, dual PCA, and kernel PCA are covered. SPCA using both scoring and Hilbert-Schmidt independence criterion are explained. Kernel SPCA using both direct and dual approaches are then introduced. We cover all cases of projection and reconstruction of training and out-of-sample data. Finally, some simulations are provided on Frey and AT&T face datasets for verifying the theory in practice.

\section{Introduction}

Assume we have a dataset of \textit{instances} or \textit{data points} $\{(\ensuremath\boldsymbol{x}_i, \ensuremath\boldsymbol{y}_i)\}_{i=1}^n$ with sample size $n$ and dimensionality $\ensuremath\boldsymbol{x}_i  \in \mathbb{R}^d$ and $\ensuremath\boldsymbol{y}_i \in \mathbb{R}^\ell$.
The $\{\ensuremath\boldsymbol{x}_i\}_{i=1}^n$ are the input data to the model and the $\{\ensuremath\boldsymbol{y}_i\}_{i=1}^n$ are the observations (labels).
We define $\mathbb{R}^{d \times n} \ni \ensuremath\boldsymbol{X} := [\ensuremath\boldsymbol{x}_1, \dots, \ensuremath\boldsymbol{x}_n]$ and $\mathbb{R}^{\ell \times n} \ni \ensuremath\boldsymbol{Y} := [\ensuremath\boldsymbol{y}_1, \dots, \ensuremath\boldsymbol{y}_n]$.
We can also have an out-of-sample data point, $\ensuremath\boldsymbol{x}_t \in \mathbb{R}^d$, which is not in the training set. If there are $n_t$ out-of-sample data points, $\{\ensuremath\boldsymbol{x}_{t,i}\}_1^{n_t}$, we define $\mathbb{R}^{d \times n_t} \ni \ensuremath\boldsymbol{X}_t := [\ensuremath\boldsymbol{x}_{t,1}, \dots, \ensuremath\boldsymbol{x}_{t,n_t}]$.
Usually, the data points exist on a subspace or sub-manifold. Subspace or manifold learning tries to learn this sub-manifold \cite{ghojogh2019feature}.

Principal Component Analysis (PCA) \cite{jolliffe2011principal} is a very well-known and fundamental linear method for subspace and manifold learning \cite{friedman2001elements}. This method, which is also used for feature extraction \cite{ghojogh2019feature}, was first proposed by \cite{pearson1901liii}.
In order to learn a nonlinear sub-manifold, kernel PCA was proposed, first by \cite{scholkopf1997kernel,scholkopf1998nonlinear}, which maps the data to high dimensional feature space hoping to fall on a linear manifold in that space.

The PCA and kernel PCA are unsupervised methods for subspace learning. To use the class labels in PCA, supervised PCA was proposed \cite{bair2006prediction} which scores the features of the $\ensuremath\boldsymbol{X}$ and reduces the features before applying PCA. This type of SPCA were mostly used in bioinformatics \cite{ma2011principal}.

Afterwards, another type of SPCA \cite{barshan2011supervised} was proposed which has a very solid theory and PCA is really a special case of it when we the labels are not used.
This SPCA also has dual and kernel SPCA.

The PCA and SPCA have had many applications for example eigenfaces \cite{turk1991eigenfaces,turk1991face} and kernel eigenfaces \cite{yang2000face} for face recognition and detecting orientation of image using PCA \cite{mohammadzade2017critical}.
There exist many other applications of PCA and SPCA in the literature.
In this paper, we explain the theory of PCA, kernel SPCA, SPCA, and kernel SPCA and provide some simulations for verifying the theory in practice.

\section{Principal Component Analysis}

\subsection{Projection Formulation}

\subsubsection{A Projection Point of View}

Assume we have a data point $\ensuremath\boldsymbol{x} \in \mathbb{R}^d$. We want to project this data point onto the vector space spanned by $p$ vectors $\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$ where each vector is $d$-dimensional and usually $p \ll d$. We stack these vectors column-wise in matrix $\ensuremath\boldsymbol{U} = [\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p] \in \mathbb{R}^{d \times p}$. In other words, we want to project $\ensuremath\boldsymbol{x}$ onto the column space of $\ensuremath\boldsymbol{U}$, denoted by $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U})$.

The projection of $\ensuremath\boldsymbol{x} \in \mathbb{R}^d$ onto $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U}) \in \mathbb{R}^p$ and then its representation in the $\mathbb{R}^d$ (its reconstruction) can be seen as a linear system of equations:
\begin{align}\label{equation_projection}
\mathbb{R}^d \ni \widehat{\ensuremath\boldsymbol{x}} := \ensuremath\boldsymbol{U \beta},
\end{align}
where we should find the unknown coefficients $\ensuremath\boldsymbol{\beta} \in \mathbb{R}^p$.

If the $\ensuremath\boldsymbol{x}$ lies in the $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U})$ or $\textbf{span}\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$, this linear system has exact solution, so $\widehat{\ensuremath\boldsymbol{x}} = \ensuremath\boldsymbol{x} = \ensuremath\boldsymbol{U \beta}$. However, if $\ensuremath\boldsymbol{x}$ does not lie in this space, there is no any solution $\ensuremath\boldsymbol{\beta}$ for $\ensuremath\boldsymbol{x} = \ensuremath\boldsymbol{U \beta}$ and we should solve for projection of $\ensuremath\boldsymbol{x}$ onto $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U})$ or $\textbf{span}\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$ and then its reconstruction. In other words, we should solve for Eq. (\ref{equation_projection}). In this case, $\widehat{\ensuremath\boldsymbol{x}}$ and $\ensuremath\boldsymbol{x}$ are different and we have a residual:
\begin{align}\label{equation_residual_1}
\ensuremath\boldsymbol{r} = \ensuremath\boldsymbol{x} - \widehat{\ensuremath\boldsymbol{x}} = \ensuremath\boldsymbol{x} - \ensuremath\boldsymbol{U \beta},
\end{align}
which we want to be small. As can be seen in Fig. \ref{figure_residual_and_space}, the smallest residual vector is orthogonal to $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U})$; therefore:
\begin{align}
\ensuremath\boldsymbol{x} - \ensuremath\boldsymbol{U\beta} \perp \ensuremath\boldsymbol{U} &\implies \ensuremath\boldsymbol{U}^\top (\ensuremath\boldsymbol{x} - \ensuremath\boldsymbol{U \beta}) = 0, \nonumber \\
& \implies \ensuremath\boldsymbol{\beta} = (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U})^{-1} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x}. \label{equation_beta}
\end{align}
It is noteworthy that the Eq. (\ref{equation_beta}) is also the formula of coefficients in linear regression \cite{friedman2001elements} where the input data are the rows of $\ensuremath\boldsymbol{U}$ and the labels are $\ensuremath\boldsymbol{x}$; however, our goal here is different. Nevertheless, in Section \ref{section_PCA_reconstruction}, some similarities of PCA and regression will be introduced.

\begin{figure}[!t]
\centering
\includegraphics[width=2.2in]{./images/residual_and_space}
\caption{The residual and projection onto the column space of $\ensuremath\boldsymbol{U}$.}
\label{figure_residual_and_space}
\end{figure}

Plugging Eq. (\ref{equation_beta}) in Eq. (\ref{equation_projection}) gives us:
\begin{align*}
\widehat{\ensuremath\boldsymbol{x}} = \ensuremath\boldsymbol{U} (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U})^{-1} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x}.
\end{align*}
We define:
\begin{align}\label{equation_hat_matrix}
\mathbb{R}^{d \times d} \ni \ensuremath\boldsymbol{\Pi} := \ensuremath\boldsymbol{U} (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U})^{-1} \ensuremath\boldsymbol{U}^\top,
\end{align}
as ``projection matrix'' because it projects $\ensuremath\boldsymbol{x}$ onto $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U})$ (and reconstructs back).
Note that $\ensuremath\boldsymbol{\Pi}$ is also referred to as the ``hat matrix'' in the literature because it puts a hat on top of $\ensuremath\boldsymbol{x}$.

If the vectors $\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$ are orthonormal (the matrix $\ensuremath\boldsymbol{U}$ is orthogonal), we have $\ensuremath\boldsymbol{U}^\top = \ensuremath\boldsymbol{U}^{-1}$ and thus $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}$. Therefore, Eq. (\ref{equation_hat_matrix}) is simplified:
\begin{align}
& \ensuremath\boldsymbol{\Pi} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top.
\end{align}
So, we have:
\begin{align}\label{equation_x_hat}
\widehat{\ensuremath\boldsymbol{x}} = \ensuremath\boldsymbol{\Pi}\, \ensuremath\boldsymbol{x} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x}.
\end{align}

\begin{figure}[!t]
\centering
\includegraphics[width=2.8in]{./images/centered_data}
\caption{The principal directions P1 and P2 for (a) non-centered and (b) centered data. As can be seen, the data should be centered for PCA.}
\label{figure_centered_data}
\end{figure}

\subsubsection{Projection and Reconstruction in PCA}

The Eq. (\ref{equation_x_hat}) can be interpreted in this way: The $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x}$ projects $\ensuremath\boldsymbol{x}$ onto the row space of $\ensuremath\boldsymbol{U}$, i.e., $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U}^\top)$ (projection onto a space spanned by $d$ vectors which are $p$-dimensional). We call this projection, ``projection onto the PCA subspace''. It is ``subspace'' because we have $p \leq d$ where $p$ and $d$ are dimensionality of PCA subspace and the original $\ensuremath\boldsymbol{x}$, respectively. Afterwards, $\ensuremath\boldsymbol{U} (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x})$ projects the projected data back onto the column space of $\ensuremath\boldsymbol{U}$, i.e., $\mathbb{C}\text{ol}(\ensuremath\boldsymbol{U})$ (projection onto a space spanned by $p$ vectors which are $d$-dimensional). We call this step ``reconstruction from the PCA'' and we want the residual between $\ensuremath\boldsymbol{x}$ and its reconstruction $\widehat{\ensuremath\boldsymbol{x}}$ to be small.

If there exist $n$ training data points, i.e., $\{\ensuremath\boldsymbol{x}_i\}_{i=1}^n$, the projection of a training data point $\ensuremath\boldsymbol{x}$ is:
\begin{align}\label{equation_projection_onePoint}
\mathbb{R}^{p} \ni \widetilde{\ensuremath\boldsymbol{x}} := \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{x}},
\end{align}
where:
\begin{align}\label{equation_mean_removed_x}
&\mathbb{R}^{d} \ni \breve{\ensuremath\boldsymbol{x}} := \ensuremath\boldsymbol{x} - \ensuremath\boldsymbol{\mu}_x,
\end{align}
is the centered data point and:
\begin{align}
&\mathbb{R}^{d} \ni \ensuremath\boldsymbol{\mu}_x := \frac{1}{n} \sum_{i=1}^n \ensuremath\boldsymbol{x}_i,
\end{align}
is the mean of training data points.
The reconstruction of a training data point $\ensuremath\boldsymbol{x}$ after projection onto the PCA subspace is:
\begin{align}\label{equation_reconstruction_onePoint}
\mathbb{R}^{d} \ni \widehat{\ensuremath\boldsymbol{x}} := \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{x}} + \ensuremath\boldsymbol{\mu}_x = \ensuremath\boldsymbol{U}\widetilde{\ensuremath\boldsymbol{x}} + \ensuremath\boldsymbol{\mu}_x,
\end{align}
where the mean is added back because it was removed before projection.

Note that in PCA, all the data points should be centered, i.e., the mean should be removed first. The reason is shown in Fig. \ref{figure_centered_data}.
In some applications, centering the data does not make sense. For example, in natural language processing, the data are text and centering the data makes some negative measures which is non-sense for text. Therefore, data is not sometimes centered and PCA is applied on the non-centered data. This method is called Latent Semantic Indexing (LSI) or Latent Semantic Analysis (LSA) \cite{dumais2004latent}.

If we stack the $n$ data points column-wise in a matrix $\ensuremath\boldsymbol{X} = [\ensuremath\boldsymbol{x}_1, \dots, \ensuremath\boldsymbol{x}_n] \in \mathbb{R}^{d \times n}$, we first center them:
\begin{align}\label{equation_centered_training_data}
\mathbb{R}^{d \times n} \ni \breve{\ensuremath\boldsymbol{X}} := \ensuremath\boldsymbol{X} \ensuremath\boldsymbol{H} = \ensuremath\boldsymbol{X} - \ensuremath\boldsymbol{\mu}_x,
\end{align}
where $\breve{\ensuremath\boldsymbol{X}} = [\breve{\ensuremath\boldsymbol{x}}_1, \dots, \breve{\ensuremath\boldsymbol{x}}_n] = [\ensuremath\boldsymbol{x}_1 - \ensuremath\boldsymbol{\mu}_x, \dots, \ensuremath\boldsymbol{x}_n - \ensuremath\boldsymbol{\mu}_x]$ is the centered data and:
\begin{align}
\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{H} := \ensuremath\boldsymbol{I} - (1/n) \ensuremath\boldsymbol{1}\ensuremath\boldsymbol{1}^\top,
\end{align}
is the centering matrix.
See Appendix \ref{section_appendix_centering} for more details about the centering matrix.

The projection and reconstruction, Eqs. (\ref{equation_projection_onePoint}) and (\ref{equation_reconstruction_onePoint}), for the whole training data are:
\begin{align}
&\mathbb{R}^{p \times n} \ni \widetilde{\ensuremath\boldsymbol{X}} := \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}, \label{equation_projection_severalPoint} \\
&\mathbb{R}^{d \times n} \ni \widehat{\ensuremath\boldsymbol{X}} := \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}} + \ensuremath\boldsymbol{\mu}_x = \ensuremath\boldsymbol{U}\widetilde{\ensuremath\boldsymbol{X}} + \ensuremath\boldsymbol{\mu}_x, \label{equation_reconstruction_severalPoint}
\end{align}
where $\widetilde{\ensuremath\boldsymbol{X}} = [\widetilde{\ensuremath\boldsymbol{x}}_1, \dots, \widetilde{\ensuremath\boldsymbol{x}}_n]$ and $\widehat{\ensuremath\boldsymbol{X}} = [\widehat{\ensuremath\boldsymbol{x}}_1, \dots, \widehat{\ensuremath\boldsymbol{x}}_n]$ are the projected data onto PCA subspace and the reconstructed data, respectively.

We can also project a new data point onto the PCA subspace for $\ensuremath\boldsymbol{X}$ where the new data point is not a column of $\ensuremath\boldsymbol{X}$. In other words, the new data point has not had impact in constructing the PCA subspace. This new data point is also referred to as ``test data point'' or ``out-of-sample data'' in the literature.
The Eq. (\ref{equation_projection_severalPoint}) was for projection of $\ensuremath\boldsymbol{X}$ onto its PCA subspace. If $\ensuremath\boldsymbol{x}_t$ denotes an out-of-sample data point, its projection onto the PCA subspace ($\widetilde{\ensuremath\boldsymbol{x}}_t$) and its reconstruction ($\widehat{\ensuremath\boldsymbol{x}}_t$) are:
\begin{align}
&\mathbb{R}^{p} \ni \widetilde{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{x}}_t, \label{equation_outOfSample_projection_PCA} \\
&\mathbb{R}^{d} \ni \widehat{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{x}}_t + \ensuremath\boldsymbol{\mu}_x = \ensuremath\boldsymbol{U} \widetilde{\ensuremath\boldsymbol{x}}_t + \ensuremath\boldsymbol{\mu}_x, \label{equation_outOfSample_reconstruct_PCA}
\end{align}
where:
\begin{align}
\mathbb{R}^{d} \ni \breve{\ensuremath\boldsymbol{x}}_t := \ensuremath\boldsymbol{x}_t - \ensuremath\boldsymbol{\mu}_x,
\end{align}
is the centered out-of-sample data point which is centered using the mean of training data. Note that for centering the out-of-sample data point(s), we should use the mean of the training data and not the out-of-sample data.

If we consider the $n_t$ out-of-sample data points, $\mathbb{R}^{d \times n_t} \ni \ensuremath\boldsymbol{X}_t = [\ensuremath\boldsymbol{x}_{t,1}, \dots, \ensuremath\boldsymbol{x}_{t,n_t}]$, all together, the projection and reconstruction of them are:
\begin{align}
&\mathbb{R}^{p \times n_t} \ni \widetilde{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}_t, \\
&\mathbb{R}^{d \times n_t} \ni \widehat{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}_t + \ensuremath\boldsymbol{\mu}_x = \ensuremath\boldsymbol{U} \widetilde{\ensuremath\boldsymbol{X}}_t + \ensuremath\boldsymbol{\mu}_x,
\end{align}
respectively, where:
\begin{align}
\mathbb{R}^{d \times n_t} \ni \breve{\ensuremath\boldsymbol{X}}_t := \ensuremath\boldsymbol{X}_t - \ensuremath\boldsymbol{\mu}_x.
\end{align}

\subsection{PCA Using Eigen-Decomposition}

\subsubsection{Projection Onto One Direction}

In Eq. (\ref{equation_reconstruction_onePoint}), if $p=1$, we are projecting $\ensuremath\boldsymbol{x}$ onto only one vector $\ensuremath\boldsymbol{u}$ and reconstruct it. If we ignore adding the mean back, we have:
\begin{align*}
\widehat{\ensuremath\boldsymbol{x}} = \ensuremath\boldsymbol{u}\ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}}.
\end{align*}
The squared length (squared $\ell_2$-norm) of this reconstructed vector is:
\begin{align}
&||\widehat{\ensuremath\boldsymbol{x}}||_2^2 = ||\ensuremath\boldsymbol{u}\ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}}||_2^2 = (\ensuremath\boldsymbol{u}\ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}})^\top (\ensuremath\boldsymbol{u}\ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}}) \nonumber \\
& = \breve{\ensuremath\boldsymbol{x}}^\top \ensuremath\boldsymbol{u} \underbrace{\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{u}}_{1} \ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}} \overset{(a)}{=} \breve{\ensuremath\boldsymbol{x}}^\top \ensuremath\boldsymbol{u}\, \ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}} \overset{(b)}{=} \ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}}\, \breve{\ensuremath\boldsymbol{x}}^\top \ensuremath\boldsymbol{u}, \label{equation_x_hat_length_squared}
\end{align}
where $(a)$ is because $\ensuremath\boldsymbol{u}$ is a unit (normal) vector, i.e., $\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{u} = ||\ensuremath\boldsymbol{u}||_2^2 = 1$, and $(b)$ is because $\breve{\ensuremath\boldsymbol{x}}^\top \ensuremath\boldsymbol{u} = \ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}} \in \mathbb{R}$.

Suppose we have $n$ data points $\{\ensuremath\boldsymbol{x}_i\}_{i=1}^n$ where $\{\breve{\ensuremath\boldsymbol{x}}_i\}_{i=1}^n$ are the centered data.
The summation of the squared lengths of their projections $\{\widehat{\ensuremath\boldsymbol{x}}_i\}_{i=1}^n$ is:
\begin{align}\label{equation_sum_projected}
\sum_{i=1}^n ||\widehat{\ensuremath\boldsymbol{x}}_i||_2^2 \overset{(\ref{equation_x_hat_length_squared})}{=} \sum_{i=1}^n \ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{x}}_i\, \breve{\ensuremath\boldsymbol{x}}_i^\top \ensuremath\boldsymbol{u} = \ensuremath\boldsymbol{u}^\top \Big(\sum_{i=1}^n \breve{\ensuremath\boldsymbol{x}}_i\, \breve{\ensuremath\boldsymbol{x}}_i^\top\Big) \ensuremath\boldsymbol{u}.
\end{align}
Considering $\breve{\ensuremath\boldsymbol{X}} = [\breve{\ensuremath\boldsymbol{x}}_1, \dots, \breve{\ensuremath\boldsymbol{x}}_n] \in \mathbb{R}^{d \times n}$, we have:
\begin{align}\label{equation_covariance_matrix}
\mathbb{R}^{d \times d} \ni \ensuremath\boldsymbol{S} &:= \sum_{i=1}^n \breve{\ensuremath\boldsymbol{x}}_i\, \breve{\ensuremath\boldsymbol{x}}_i^\top = \breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top
\overset{(\ref{equation_centered_training_data})}{=} \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H} \ensuremath\boldsymbol{H}^\top\ensuremath\boldsymbol{X}^\top \nonumber \\
&\overset{(\ref{equation_centeringMatrix_is_symmetric})}{=} \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H} \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top \overset{(\ref{equation_centeringMatrix_is_idempotent})}{=} \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top,
\end{align}
where $\ensuremath\boldsymbol{S}$ is called the ``covariance matrix''. If the data were already centered, we would have $\ensuremath\boldsymbol{S} = \ensuremath\boldsymbol{X} \ensuremath\boldsymbol{X}^\top$.

Plugging Eq. (\ref{equation_covariance_matrix}) in Eq. (\ref{equation_sum_projected}) gives us:
\begin{align}
\sum_{i=1}^n ||\widehat{\ensuremath\boldsymbol{x}}_i||_2^2 = \ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u}.
\end{align}
Note that we can also say that $\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u}$ is the variance of the projected data onto PCA subspace. In other words, $\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u} = \mathbb{V}\text{ar}(\ensuremath\boldsymbol{u}^\top \breve{\ensuremath\boldsymbol{X}})$. This makes sense because when some non-random thing (here $\ensuremath\boldsymbol{u}$) is multiplied to the random data (here $\breve{\ensuremath\boldsymbol{X}}$), it will have squared (quadratic) effect on variance, and $\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u}$ is quadratic in $\ensuremath\boldsymbol{u}$.

Therefore, $\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u}$ can be interpreted in two ways: (I) the squared length of reconstruction and (II) the variance of projection.

We want to find a projection direction $\ensuremath\boldsymbol{u}$ which maximizes the squared length of reconstruction (or variance of projection):
\begin{equation}
\begin{aligned}
& \underset{\ensuremath\boldsymbol{u}}{\text{maximize}}
& & \ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u}, \\
& \text{subject to}
& & \ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{u} = 1,
\end{aligned}
\end{equation}
where the constraint ensures that the $\ensuremath\boldsymbol{u}$ is a unit (normal) vector as we assumed beforehand.

Using Lagrange multiplier \cite{boyd2004convex}, we have:
\begin{align*}
\mathcal{L} = \ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u} - \lambda (\ensuremath\boldsymbol{u}^\top \ensuremath\boldsymbol{u} - 1),
\end{align*}
Taking derivative of the Lagrangian and setting it to zero gives:
\begin{align}\label{equation_pca_eigendecomposition_1}
& \mathbb{R}^p \ni \frac{\partial \mathcal{L}}{\partial \ensuremath\boldsymbol{u}} = 2\ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u} - 2\lambda \ensuremath\boldsymbol{u} \overset{\text{set}}{=} 0 \implies \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{u} = \lambda \ensuremath\boldsymbol{u}.
\end{align}
The Eq. (\ref{equation_pca_eigendecomposition_1}) is the eigen-decomposition of $\ensuremath\boldsymbol{S}$ where $\ensuremath\boldsymbol{u}$ and $\lambda$ are the leading eigenvector and eigenvalue of $\ensuremath\boldsymbol{S}$, respectively \cite{ghojogh2019eigenvalue}. Note that the leading eigenvalue is the largest one. The reason of being leading is that we are maximizing in the optimization problem.
As a conclusion, if projecting onto one PCA direction, the PCA direction $\ensuremath\boldsymbol{u}$ is the leading eigenvector of the covariance matrix.
Note that the ``PCA direction'' is also called ``principal direction'' or ``principal axis'' in the literature. The dimensions (features) of the projected data onto PCA subspace are called ``principal components''.

\subsubsection{Projection Onto Span of Several Directions}

In Eq. (\ref{equation_reconstruction_onePoint}) or (\ref{equation_reconstruction_severalPoint}), if $p>1$, we are projecting $\breve{\ensuremath\boldsymbol{x}}$ or $\breve{\ensuremath\boldsymbol{X}}$ onto PCA subspace with dimensionality more than one and then reconstruct back. If we ignore adding the mean back, we have:
\begin{align*}
\widehat{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}.
\end{align*}
It means that we project every column of $\breve{\ensuremath\boldsymbol{X}}$, i.e., $\breve{\ensuremath\boldsymbol{x}}$, onto a space spanned by the $p$ vectors $\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$ each of which is $d$-dimensional. Therefore, the projected data are $p$-dimensional and the reconstructed data are $d$-dimensional.

The squared length (squared Frobenius Norm) of this reconstructed matrix is:
\begin{align*}
||\widehat{\ensuremath\boldsymbol{X}}||_F^2 &= ||\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}||_F^2 = \textbf{tr}\big((\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}})^\top (\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}})\big) \\
& = \textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} \underbrace{\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U}}_{\ensuremath\boldsymbol{I}} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}) \overset{(a)}{=} \textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}) \\
& \overset{(b)}{=} \textbf{tr}(\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U}),
\end{align*}
where $\textbf{tr}(.)$ denotes the trace of matrix, $(a)$ is because $\ensuremath\boldsymbol{U}$ is an orthogonal matrix (its columns are orthonormal), and $(b)$ is because $\textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}) = \textbf{tr}(\breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top) = \textbf{tr}(\ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U})$.
According to Eq. (\ref{equation_covariance_matrix}), the $\ensuremath\boldsymbol{S} = \breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top$ is the covariance matrix; therefore:
\begin{align}
||\widehat{\ensuremath\boldsymbol{X}}||_F^2 = \textbf{tr}(\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{S}\, \ensuremath\boldsymbol{U}).
\end{align}

We want to find several projection directions $\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$, as columns of $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$, which maximize the squared length of reconstruction (or variance of projection):
\begin{equation}
\begin{aligned}
& \underset{\ensuremath\boldsymbol{U}}{\text{maximize}}
& & \textbf{tr}(\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{S}\, \ensuremath\boldsymbol{U}), \\
& \text{subject to}
& & \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I},
\end{aligned}
\end{equation}
where the constraint ensures that the $\ensuremath\boldsymbol{U}$ is an orthogonal matrix as we assumed beforehand.

Using Lagrange multiplier \cite{boyd2004convex}, we have:
\begin{align*}
\mathcal{L} = \textbf{tr}(\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{S}\, \ensuremath\boldsymbol{U}) - \textbf{tr}\big(\ensuremath\boldsymbol{\Lambda}^\top (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} - \ensuremath\boldsymbol{I})\big),
\end{align*}
where $\ensuremath\boldsymbol{\Lambda} \in \mathbb{R}^{p \times p}$ is a diagonal matrix $\textbf{diag}([\lambda_1, \dots, \lambda_p]^\top)$ including the Lagrange multipliers.
\begin{align}
& \mathbb{R}^{d \times p} \ni \frac{\partial \mathcal{L}}{\partial \ensuremath\boldsymbol{U}} = 2\ensuremath\boldsymbol{S} \ensuremath\boldsymbol{U} - 2\ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda} \overset{\text{set}}{=} 0 \nonumber \\
& \implies \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda}. \label{equation_pca_eigendecomposition_2}
\end{align}
The Eq. (\ref{equation_pca_eigendecomposition_2}) is the eigen-decomposition of $\ensuremath\boldsymbol{S}$ where the columns of $\ensuremath\boldsymbol{U}$ and the diagonal of $\ensuremath\boldsymbol{\Lambda}$ are the eigenvectors and eigenvalues of $\ensuremath\boldsymbol{S}$, respectively \cite{ghojogh2019eigenvalue}. The eigenvectors and eigenvalues are sorted from the leading (largest eigenvalue) to the trailing (smallest eigenvalue) because we are maximizing in the optimization problem.
As a conclusion, if projecting onto the PCA subspace or $\textbf{span}\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$, the PCA directions $\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$ are the sorted eigenvectors of the covariance matrix of data $\ensuremath\boldsymbol{X}$.

\subsection{Properties of $\ensuremath\boldsymbol{U}$}

\subsubsection{Rank of the Covariance Matrix}

We consider two cases for $\breve{\ensuremath\boldsymbol{X}} \in \mathbb{R}^{d \times n}$:

\begin{enumerate}
\item If the original dimensionality of data is greater than the number of data points, i.e., $d \geq n$:
In this case, $\textbf{rank}(\breve{\ensuremath\boldsymbol{X}}) = \textbf{rank}(\breve{\ensuremath\boldsymbol{X}}^\top) \leq n$. Therefore, $\textbf{rank}(\ensuremath\boldsymbol{S}) = \textbf{rank}(\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top) \leq \min\big(\textbf{rank}(\breve{\ensuremath\boldsymbol{X}}), \textbf{rank}(\breve{\ensuremath\boldsymbol{X}}^\top)\big)-1 = n-1$. Note that $-1$ is because the data are centered. For example, if we only have one data point, it becomes zero after centering and the rank should be zero.
\item If the original dimensionality of data is less than the number of data points, i.e., $d \leq n-1$ (the $-1$ again is because of centering the data):
In this case, $\textbf{rank}(\breve{\ensuremath\boldsymbol{X}}) = \textbf{rank}(\breve{\ensuremath\boldsymbol{X}}^\top) \leq d$. Therefore, $\textbf{rank}(\ensuremath\boldsymbol{S}) = \textbf{rank}(\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top) \leq \min\big(\textbf{rank}(\breve{\ensuremath\boldsymbol{X}}), \textbf{rank}(\breve{\ensuremath\boldsymbol{X}}^\top)\big) = d$.
\end{enumerate}
So, we either have $\textbf{rank}(\ensuremath\boldsymbol{S}) \leq n-1$ or $\textbf{rank}(\ensuremath\boldsymbol{S}) \leq d$.

\subsubsection{Truncating $\ensuremath\boldsymbol{U}$}

Consider the following cases:

\begin{figure}[!t]
\centering
\includegraphics[width=3in]{./images/coordinate_rotation}
\caption{Rotation of coordinates because of PCA.}
\label{figure_coordinate_rotation}
\end{figure}

\begin{enumerate}
\item If $\textbf{rank}(\ensuremath\boldsymbol{S}) = d$:
we have $p=d$ (we have $d$ non-zero eigenvalues of $\ensuremath\boldsymbol{S}$), so that $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times d}$. It means that the dimensionality of the PCA subspace is $d$, equal to the dimensionality of the original space. Why does this happen? That is because $\textbf{rank}(\ensuremath\boldsymbol{S}) = d$ means that the data are spread wide enough in all dimensions of the original space up to a possible rotation (see Fig. \ref{figure_coordinate_rotation}). Therefore, the dimensionality of PCA subspace is equal to the original dimensionality; however, PCA might merely rotate the coordinate axes. In this case, $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times d}$ is a square orthogonal matrix so that $\mathbb{R}^{d \times d} \ni \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^{-1} = \ensuremath\boldsymbol{I}$ and $\mathbb{R}^{d \times d} \ni \ensuremath\boldsymbol{U}^\top\ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{U}^{-1}\ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}$ because $\textbf{rank}(\ensuremath\boldsymbol{U}) = d$, $\textbf{rank}(\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top) = d$, and $\textbf{rank}(\ensuremath\boldsymbol{U}^\top\ensuremath\boldsymbol{U}) = d$.
That is why in the literature, PCA is also referred to as coordinate rotation.
\item If $\textbf{rank}(\ensuremath\boldsymbol{S}) < d$ and $n > d$:
it means that we have enough data points but the data points exist on a subspace and do not fill the original space wide enough in every direction. In this case, $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$ is not square and $\textbf{rank}(\ensuremath\boldsymbol{U}) = p < d$ (we have $p$ non-zero eigenvalues of $\ensuremath\boldsymbol{S}$). Therefore, $\mathbb{R}^{d \times d} \ni \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \neq \ensuremath\boldsymbol{I}$ and $\mathbb{R}^{p \times p} \ni \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}$ because $\textbf{rank}(\ensuremath\boldsymbol{U}) = p$, $\textbf{rank}(\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top) = p < d$, and $\textbf{rank}(\ensuremath\boldsymbol{U}^\top\ensuremath\boldsymbol{U}) = p$.
\item If $\textbf{rank}(\ensuremath\boldsymbol{S}) \leq n-1 < d$:
it means that we do not have enough data points to properly represent the original space and the points have an ``intrinsic dimensionality''. For example, we have two three-dimensional points which are one a two-dimensional line (subspace). So, similar to previous case, the data points exist on a subspace and do not fill the original space wide enough in every direction. The discussions about $\ensuremath\boldsymbol{U}$, $\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top$, and $\ensuremath\boldsymbol{U}^\top\ensuremath\boldsymbol{U}$ are similar to previous case.
\end{enumerate}

Note that we might have $\textbf{rank}(\ensuremath\boldsymbol{S}) = d$ and thus $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times d}$ but want to ``truncate'' the matrix $\ensuremath\boldsymbol{U}$ to have $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$. Truncating $\ensuremath\boldsymbol{U}$ means that we take a subset of best (leading) eigenvectors rather than the whole $d$ eigenvectors with non-zero eigenvalues. In this case, again we have $\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \neq \ensuremath\boldsymbol{I}$ and $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}$.
The intuition of truncating is this: the variance of data might be noticeably smaller than another direction; in this case, we can only keep the $p<d$ top eigenvectors (PCA directions) and ``ignore'' the PCA directions with smaller eigenvalues to have $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$. Figure \ref{figure_PCA_oneDirection} illustrates this case for a 2D example.
Note that truncating can also be done when $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$ to have $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times q}$ where $p$ is the number of non-zero eigenvalues of $\ensuremath\boldsymbol{S}$ and $q < p$.

From all the above analyses, we conclude that as long as the columns of the matrix $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$ are orthonormal, we always have $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}$ regardless of the value $p$. If the orthogonal matrix $\ensuremath\boldsymbol{U}$ is not truncated and thus is a square matrix, we also have $\ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top = \ensuremath\boldsymbol{I}$.

\begin{figure}[!t]
\centering
\includegraphics[width=1.6in]{./images/PCA_oneDirection}
\caption{A 2D example where the data is almost on a line and the second principal direction is very small and can be ignored.}
\label{figure_PCA_oneDirection}
\end{figure}

\subsection{Reconstruction Error in PCA}\label{section_PCA_reconstruction}

\subsubsection{Reconstruction in Linear Projection}

If we center the data, the Eq. (\ref{equation_residual_1}) becomes $\ensuremath\boldsymbol{r} = \breve{\ensuremath\boldsymbol{x}} - \widehat{\ensuremath\boldsymbol{x}}$ because the reconstructed data will also be centered according to Eq. (\ref{equation_reconstruction_onePoint}).
According to Eqs. (\ref{equation_residual_1}), (\ref{equation_mean_removed_x}), and (\ref{equation_reconstruction_onePoint}), we have:
\begin{align}
\ensuremath\boldsymbol{r} = \ensuremath\boldsymbol{x} - \widehat{\ensuremath\boldsymbol{x}} = \breve{\ensuremath\boldsymbol{x}} + \ensuremath\boldsymbol{\mu}_x - \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{x}} - \ensuremath\boldsymbol{\mu}_x = \breve{\ensuremath\boldsymbol{x}} - \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{x}}.
\end{align}
Figure \ref{figure_reconstruction_error} shows the projection of a two-dimensional point (after the data being centered) onto the first principal direction, its reconstruction, and its reconstruction error. As can be seen in this figure, the reconstruction error is different from least square error in linear regression.

\begin{figure}[!t]
\centering
\includegraphics[width=3.1in]{./images/reconstruction_error}
\caption{(a) Projection of the black circle data points onto the principal direction where the green square data points are the projected data. (b) The reconstruction coordinate of the data points. (c) The reconstruction error in PCA. (d) The least square error in linear regression.}
\label{figure_reconstruction_error}
\end{figure}

For $n$ data points, we have:
\begin{align}
\ensuremath\boldsymbol{R} &:= \ensuremath\boldsymbol{X} - \widehat{\ensuremath\boldsymbol{X}} = \breve{\ensuremath\boldsymbol{X}} + \ensuremath\boldsymbol{\mu}_x - \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{\mu}_x \nonumber \\
&= \breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}},
\end{align}
where $\mathbb{R}^{d \times n} \ni \ensuremath\boldsymbol{R} = [\ensuremath\boldsymbol{r}_1, \dots, \ensuremath\boldsymbol{r}_n]$ is the matrix of residuals.

If we want to minimize the reconstruction error subject to the orthogonality of the projection matrix $\ensuremath\boldsymbol{U}$, we have:
\begin{equation}
\begin{aligned}
& \underset{\ensuremath\boldsymbol{U}}{\text{minimize}}
& & ||\breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}}||_F^2, \\
& \text{subject to}
& & \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}.
\end{aligned}
\end{equation}
The objective function can be simplified:
\begin{align*}
&||\breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}}||_F^2 \\
&= \textbf{tr}\big((\breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}})^\top (\breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}})\big) \\
&= \textbf{tr}\big((\breve{\ensuremath\boldsymbol{X}}^\top - \breve{\ensuremath\boldsymbol{X}}^\top\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top) (\breve{\ensuremath\boldsymbol{X}} - \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}})\big) \\
&= \textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top\breve{\ensuremath\boldsymbol{X}}-2\breve{\ensuremath\boldsymbol{X}}^\top\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}} + \breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} \underbrace{\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U}}_{\ensuremath\boldsymbol{I}} \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}) \\
&= \textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top\breve{\ensuremath\boldsymbol{X}}-\breve{\ensuremath\boldsymbol{X}}^\top\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}}) \\
&= \textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top\breve{\ensuremath\boldsymbol{X}})-\textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}}) \\
&= \textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top\breve{\ensuremath\boldsymbol{X}})-\textbf{tr}(\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top).
\end{align*}
Using Lagrange multiplier \cite{boyd2004convex}, we have:
\begin{align*}
\mathcal{L} = &\,\textbf{tr}(\breve{\ensuremath\boldsymbol{X}}^\top\breve{\ensuremath\boldsymbol{X}})-\textbf{tr}(\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top) \\
&- \textbf{tr}\big(\ensuremath\boldsymbol{\Lambda}^\top (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} - \ensuremath\boldsymbol{I})\big),
\end{align*}
where $\ensuremath\boldsymbol{\Lambda} \in \mathbb{R}^{p \times p}$ is a diagonal matrix $\textbf{diag}([\lambda_1, \dots, \lambda_p]^\top)$ containing the Lagrange multipliers.
Equating the derivative of Lagrangian to zero gives:
\begin{align}
& \mathbb{R}^{d \times p} \ni \frac{\partial \mathcal{L}}{\partial \ensuremath\boldsymbol{U}} = 2\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} - 2\ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda} \overset{\text{set}}{=} 0 \nonumber \\
& \implies \breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda}, \nonumber \\
& \overset{(\ref{equation_covariance_matrix})}{\implies} \ensuremath\boldsymbol{S} \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda},
\end{align}
which is again the eigenvalue problem \cite{ghojogh2019eigenvalue} for the covariance matrix $\ensuremath\boldsymbol{S}$. We had the same eigenvalue problem in PCA. Therefore, \textit{PCA subspace is the best linear projection in terms of reconstruction error. In other words, PCA has the least squared error in reconstruction.}

\subsubsection{Reconstruction in Autoencoder}

\begin{figure}[!t]
\centering
\includegraphics[width=2.3in]{./images/autoencoder_2}
\caption{(a) An example of autoencoder with five hidden layers and linear activation functions, and (b) its reduction to an autoencoder with one hidden layer.}
\label{figure_autoencoder}
\end{figure}

We saw that PCA is the best in reconstruction error for \textit{linear} projection. If we have $m>1$ successive linear projections, the reconstruction is:
\begin{align}
\widehat{\ensuremath\boldsymbol{X}} = \underbrace{\ensuremath\boldsymbol{U}_1 \cdots \ensuremath\boldsymbol{U}_m}_\text{reconstruct} \underbrace{\ensuremath\boldsymbol{U}_m^\top \cdots \ensuremath\boldsymbol{U}_1^\top}_\text{project} \breve{\ensuremath\boldsymbol{X}} + \ensuremath\boldsymbol{\mu}_x,
\end{align}
which can be seen as an undercomplete \textit{autoencoder} \cite{goodfellow2016deep} with $2m$ layers without activation function (or with identity activation functions $f(\ensuremath\boldsymbol{x}) = \ensuremath\boldsymbol{x}$). The $\ensuremath\boldsymbol{\mu}_x$ is modeled by the intercepts included as input to the neurons of auto-encoder layers. Figure \ref{figure_autoencoder} shows this autoencoder.
As we do not have any non-linearity between the projections, we can define:
\begin{align}
&\ddot{\ensuremath\boldsymbol{U}} := \ensuremath\boldsymbol{U}_1 \cdots \ensuremath\boldsymbol{U}_m \implies \ddot{\ensuremath\boldsymbol{U}}^\top = \ensuremath\boldsymbol{U}_m^\top \cdots \ensuremath\boldsymbol{U}_1^\top, \\
&\therefore ~~~~ \widehat{\ensuremath\boldsymbol{X}} = \ddot{\ensuremath\boldsymbol{U}} \ddot{\ensuremath\boldsymbol{U}}^\top \ensuremath\boldsymbol{X} + \ensuremath\boldsymbol{\mu}_x. \label{equation_linear_autoencoder_reconstruction}
\end{align}
The Eq. (\ref{equation_linear_autoencoder_reconstruction}) shows that the whole autoencoder can be reduced to an undercomplete autoencoder with one hidden layer where the weight matrix is $\ddot{\ensuremath\boldsymbol{U}}$ (see Fig. \ref{figure_autoencoder}).
In other words, in autoencoder neural network, every layer excluding the activation function behaves as a linear projection.

Comparing the Eqs. (\ref{equation_reconstruction_severalPoint}) and (\ref{equation_linear_autoencoder_reconstruction}) shows that the whole autoencoder is reduced to PCA.
Therefore, \textit{PCA is equivalent to an undercomplete autoencoder with one hidden layer without activation function}. Therefore, if we trained weights of such an autoencoder by back-propagation \cite{rumelhart1986learning} are roughly equal to the PCA directions.
Moreover, as PCA is the best linear projection in terms of reconstruction error, \textit{if we have an undercomplete autoencoder with ``one'' hidden layer, it is best not to use any activation function}; this is not noticed by some papers in the literature, unfortunately.

We saw that an autoencoder with $2m$ hidden layers without activation function reduces to linear PCA.
This explains why in autoencoders with more than one layer, we use non-linear activation function $f(.)$ as:
\begin{align}
&\widehat{\ensuremath\boldsymbol{X}} = f^{-1}(\ensuremath\boldsymbol{U}_1 \dots f^{-1}(\ensuremath\boldsymbol{U}_m f(\ensuremath\boldsymbol{U}_m^\top \nonumber \\
&~~~~~~~~~~~~~~~~~~~ \dots f(\ensuremath\boldsymbol{U}_1^\top \ensuremath\boldsymbol{X})\dots))\dots) + \ensuremath\boldsymbol{\mu}_x.
\end{align}

\subsection{PCA Using Singular Value Decomposition}

The PCA can be done using Singular Value Decomposition (SVD) of $\breve{\ensuremath\boldsymbol{X}}$, rather than eigen-decomposition of $\ensuremath\boldsymbol{S}$. Consider the complete SVD of $\breve{\ensuremath\boldsymbol{X}}$ (see Appendix \ref{section_appendix_SVD}):
\begin{align}
\mathbb{R}^{d \times n} \ni \breve{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top,
\end{align}
where the columns of $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times d}$ (called left singular vectors) are the eigenvectors of $\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top$, the columns of $\ensuremath\boldsymbol{V} \in \mathbb{R}^{n \times n}$ (called right singular vectors) are the eigenvectors of $\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{X}}$, and the $\ensuremath\boldsymbol{\Sigma} \in \mathbb{R}^{d \times n}$ is a rectangular diagonal matrix whose diagonal entries (called singular values) are the square root of eigenvalues of $\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top$ and/or $\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{X}}$.
See Proposition \ref{proposition_SVD} in Appendix \ref{section_appendix_SVD} for proof of this claim.

According to Eq. (\ref{equation_covariance_matrix}), the $\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top$ is the covariance matrix $\ensuremath\boldsymbol{S}$. In Eq. (\ref{equation_pca_eigendecomposition_2}), we saw that the eigenvectors of $\ensuremath\boldsymbol{S}$ are the principal directions. On the other hand, here, we saw that the columns of $\ensuremath\boldsymbol{U}$ are the eigenvectors of $\breve{\ensuremath\boldsymbol{X}}\breve{\ensuremath\boldsymbol{X}}^\top$.
Hence, we can apply SVD on $\breve{\ensuremath\boldsymbol{X}}$ and take the left singular vectors (columns of $\ensuremath\boldsymbol{U}$) as the principal directions.

An interesting thing is that in SVD of $\breve{\ensuremath\boldsymbol{X}}$, the columns of $\ensuremath\boldsymbol{U}$ are automatically sorted from largest to smallest singular values (eigenvalues) and we do not need to sort as we did in using eigenvalue decomposition for the covariance matrix.

\subsection{Determining the Number of Principal Directions}

Usually in PCA, the components with smallest eigenvalues are cut off to reduce the data. There are different methods for estimating the best number of components to keep (denoted by $p$), such as using Bayesian model selection \cite{minka2001automatic}, scree plot \cite{cattell1966scree}, and comparing the ratio $\lambda_j / \sum_{k=1}^d \lambda_k$ with a threshold \cite{abdi2010principal} where $\lambda_i$ denotes the eigenvalue related to the $j$-th principal component.
Here, we explain the two methods of scree plot and the ratio.

The scree plot \cite{cattell1966scree} is a plot of the eigenvalues versus sorted components from the leading (having largest eigenvalue) to trailing (having smallest eigenvalue). A threshold for the vertical (eigenvalue) axis chooses the components with the large enough eigenvalues and removes the rest of the components.
A good threshold is where the eigenvalue drops significantly. In most of the datasets, a significant drop of eigenvalue occurs.

Another way to choose the best components is the ratio \cite{abdi2010principal}:
\begin{align}
\frac{\lambda_j}{\sum_{k=1}^d \lambda_k},
\end{align}
for the $j$-th component. Then, we sort the features from the largest to smallest ratio and select the $p$ best components or up to the component where a significant drop of the ratio happens.

\section{Dual Principal Component Analysis}

Assume the case where the dimensionality of data is high and much greater than the sample size, i.e., $d \gg n$.
In this case, consider the incomplete SVD of $\breve{\ensuremath\boldsymbol{X}}$ (see Appendix \ref{section_appendix_SVD}):
\begin{align}\label{equation_SVD_dual}
\breve{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top,
\end{align}
where here, $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$ and $\ensuremath\boldsymbol{V} \in \mathbb{R}^{n \times p}$ contain the $p$ leading left and right singular vectors of $\breve{\ensuremath\boldsymbol{X}}$, respectively, where $p$ is the number of ``non-zero'' singular values of $\breve{\ensuremath\boldsymbol{X}}$ and usually $p \ll d$. Here, the $\ensuremath\boldsymbol{\Sigma} \in \mathbb{R}^{p \times p}$ is a square matrix having the $p$ largest non-zero singular values of $\breve{\ensuremath\boldsymbol{X}}$. As the $\ensuremath\boldsymbol{\Sigma}$ is a square diagonal matrix and its diagonal includes non-zero entries (is full-rank), it is invertible \cite{ghodsi2006dimensionality}.
Therefore, $\ensuremath\boldsymbol{\Sigma}^{-1} = \textbf{diag}([\frac{1}{\sigma_1}, \dots, \frac{1}{\sigma_p}]^\top)$ if we have $\ensuremath\boldsymbol{\Sigma} = \textbf{diag}([\sigma_1, \dots, \sigma_p]^\top)$.

\subsection{Projection}

Recall Eq. (\ref{equation_projection_severalPoint}) for projection onto PCA subspace: $\widetilde{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{X}}$.
On the other hand, according to Eq. (\ref{equation_SVD_dual}), we have:
\begin{align}\label{equation_dual_1}
\breve{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top \implies \ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{X}} = \underbrace{\ensuremath\boldsymbol{U}^\top\ensuremath\boldsymbol{U}}_{\ensuremath\boldsymbol{I}}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top = \ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top.
\end{align}
According to Eqs. (\ref{equation_projection_severalPoint}) and (\ref{equation_dual_1}), we have:
\begin{align}\label{equation_projected_dual}
\therefore ~~~~~~~ \widetilde{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top
\end{align}
The Eq. (\ref{equation_projected_dual}) can be used for projecting data onto PCA subspace instead of Eq. (\ref{equation_projection_severalPoint}). This is projection of training data in dual PCA.

\subsection{Reconstruction}

According to Eq. (\ref{equation_SVD_dual}), we have:
\begin{align}
\breve{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top &\implies \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\underbrace{\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{V}}_{\ensuremath\boldsymbol{I}} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma} \nonumber \\
&\implies \ensuremath\boldsymbol{U} = \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-1}. \label{equation_dual_U}
\end{align}

Plugging Eq. (\ref{equation_dual_U}) in Eq. (\ref{equation_reconstruction_severalPoint}) gives us:
\begin{align}
\widehat{\ensuremath\boldsymbol{X}} &= \ensuremath\boldsymbol{U} \widetilde{\ensuremath\boldsymbol{X}} + \ensuremath\boldsymbol{\mu}_x \overset{(\ref{equation_dual_U})}{=} \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-1}\widetilde{\ensuremath\boldsymbol{X}} + \ensuremath\boldsymbol{\mu}_x \nonumber \\
&\overset{(\ref{equation_projected_dual})}{=} \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\underbrace{\ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{\Sigma}}_{\ensuremath\boldsymbol{I}}\ensuremath\boldsymbol{V}^\top + \ensuremath\boldsymbol{\mu}_x \nonumber \\
&\implies \widehat{\ensuremath\boldsymbol{X}} = \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{V}^\top + \ensuremath\boldsymbol{\mu}_x. \label{equation_reconstruction_dual}
\end{align}
The Eq. (\ref{equation_reconstruction_dual}) can be used for reconstruction of data instead of Eq. (\ref{equation_reconstruction_severalPoint}).
This is reconstruction of training data in dual PCA.

\subsection{Out-of-sample Projection}

Recall Eq. (\ref{equation_outOfSample_projection_PCA}) for projection of an out-of-sample point $\ensuremath\boldsymbol{x}_t$ onto PCA subspace.
According to Eq. (\ref{equation_dual_U}), we have:
\begin{align}
&\ensuremath\boldsymbol{U}^\top \overset{(\ref{equation_dual_U})}{=} \ensuremath\boldsymbol{\Sigma}^{-\top}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \overset{(a)}{=} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \label{equation_U_transpose_dual}\\
&\overset{(\ref{equation_outOfSample_projection_PCA})}{\implies} \widetilde{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{x}}_t, \label{equation_outOfSample_projection_dual}
\end{align}
where $(a)$ is because $\ensuremath\boldsymbol{\Sigma}^{-1}$ is diagonal and thus symmetric.
The Eq. (\ref{equation_outOfSample_projection_dual}) can be used for projecting out-of-sample data point onto PCA subspace instead of Eq. (\ref{equation_outOfSample_projection_PCA}).
This is out-of-sample projection in dual PCA.

Considering all the $n_t$ out-of-sample data points, the projection is:
\begin{align}
\widetilde{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{X}}_t.
\end{align}

\subsection{Out-of-sample Reconstruction}

Recall Eq. (\ref{equation_outOfSample_reconstruct_PCA}) for reconstruction of an out-of-sample point $\ensuremath\boldsymbol{x}_t$.
According to Eqs. (\ref{equation_dual_U}) and (\ref{equation_U_transpose_dual}), we have:
\begin{align}
&\ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top = \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-1} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \nonumber\\
&\overset{(\ref{equation_outOfSample_reconstruct_PCA})}{\implies} \widehat{\ensuremath\boldsymbol{x}}_t = \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{x}}_t + \ensuremath\boldsymbol{\mu}_x. \label{equation_outOfSample_reconstruct_dual}
\end{align}
The Eq. (\ref{equation_outOfSample_reconstruct_dual}) can be used for reconstruction of an out-of-sample data point instead of Eq. (\ref{equation_outOfSample_reconstruct_PCA}). This is out-of-sample reconstruction in dual PCA.

Considering all the $n_t$ out-of-sample data points, the reconstruction is:
\begin{align}
\widehat{\ensuremath\boldsymbol{X}}_t = \breve{\ensuremath\boldsymbol{X}}\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{X}}_t + \ensuremath\boldsymbol{\mu}_x.
\end{align}

\subsection{Why is Dual PCA Useful?}

The dual PCA can be useful for two reasons:
\begin{enumerate}
\item As can be seen in Eqs. (\ref{equation_projected_dual}), (\ref{equation_reconstruction_dual}), (\ref{equation_outOfSample_projection_dual}), and (\ref{equation_outOfSample_reconstruct_dual}), the formulae for dual PCA only includes $\ensuremath\boldsymbol{V}$ and not $\ensuremath\boldsymbol{U}$. The columns of $\ensuremath\boldsymbol{V}$ are the eigenvectors of $\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{X}} \in \mathbb{R}^{n \times n}$ and the columns of $\ensuremath\boldsymbol{U}$ are the eigenvectors of $\breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top \in \mathbb{R}^{d \times d}$. In case the dimensionality of data is much high and greater than the sample size, i.e., $n \ll d$, computation of eigenvectors of $\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{X}}$ is easier and faster than $\breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top$ and also requires less storage. Therefore, dual PCA is more efficient than direct PCA in this case in terms of both speed and storage. Note that the result of PCA and dual PCA are exactly the same.
\item Some inner product forms, such as $\breve{\ensuremath\boldsymbol{X}}^\top \breve{\ensuremath\boldsymbol{x}}_t$, have appeared in the formulae of dual PCA. This provides opportunity for kernelizing the PCA to have kernel PCA using the so-called kernel trick. As will be seen in the next section, we use dual PCA in formulation of kernel PCA.
\end{enumerate}

\section{Kernel Principal Component Analysis}

\begin{figure}[!t]
\centering
\includegraphics[width=3.2in]{./images/nonlinear_manifold}
\caption{(a) A 2D nonlinear manifold where the data exist on in the 3D original space. As the manifold is nonlinear, the geodesic distances of points on the manifold are different from their Euclidean distances. (b) The correct unfolded manifold where the geodesic distances of points on the manifold have been preserved. (c) Applying the linear PCA, which takes Euclidean distances into account, on the nonlinear data where the found subspace has ruined the manifold so the far away red and green points have fallen next to each other. The credit of this example is for Prof. Ali Ghodsi.}
\label{figure_nonlinear_manifold}
\end{figure}

The PCA is a linear method because the projection is linear. In case the data points exist on a non-linear sub-manifold, the linear subspace learning might not be completely effective. For example, see Fig. \ref{figure_nonlinear_manifold}.

In order to handle this problem of PCA, we have two options. We should either change PCA to become a nonlinear method or we can leave the PCA to be linear but change the data hoping to fall on a linear or close to linear manifold. Here, we do the latter so we change the data. We increase the dimensionality of data by mapping the data to feature space with higher dimensionality hoping that in the feature space, it falls on a linear manifold. This is referred to as ``blessing of dimensionality'' in the literature \cite{donoho2000high} which is pursued using kernels \cite{hofmann2008kernel}. This PCA method which uses the kernel of data is named ``kernel PCA'' \cite{scholkopf1997kernel}.

\subsection{Kernels and Hilbert Space}

Suppose that $\ensuremath\boldsymbol{\phi}: \ensuremath\boldsymbol{x} \rightarrow \mathcal{H}$ is a function which maps the data $\ensuremath\boldsymbol{x}$ to Hilbert space (feature space). The $\ensuremath\boldsymbol{\phi}$ is called ``pulling function''. In other words, $\ensuremath\boldsymbol{x} \mapsto \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x})$. Let $t$ denote the dimensionality of the feature space, i.e., $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}) \in \mathbb{R}^t$ while $\ensuremath\boldsymbol{x} \in \mathbb{R}^d$. Note that we usually have $t \gg d$.

If $\mathcal{X}$ denotes the set of points, i.e., $\ensuremath\boldsymbol{x} \in \mathcal{X}$, the kernel of two vectors $\ensuremath\boldsymbol{x}_1$ and $\ensuremath\boldsymbol{x}_2$ is $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ and is defined as \cite{hofmann2008kernel,herbrich2001learning}:
\begin{align}
k(\ensuremath\boldsymbol{x}_1, \ensuremath\boldsymbol{x}_2) := \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_1)^\top \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_2),
\end{align}
which is a measure of ``similarity'' between the two vectors because the inner product captures similarity.

We can compute the kernel of two matrices $\ensuremath\boldsymbol{X}_1 \in \mathbb{R}^{d \times n_1}$ and $\ensuremath\boldsymbol{X}_2 \in \mathbb{R}^{d \times n_2}$ and have a ``kernel matrix'' (also called ``Gram matrix''):
\begin{align}
\mathbb{R}^{n_1 \times n_2} \ni \ensuremath\boldsymbol{K}(\ensuremath\boldsymbol{X}_1, \ensuremath\boldsymbol{X}_2) := \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}_1)^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}_2),
\end{align}
where $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}_1) := [\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_1), \dots, \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_n)] \in \mathbb{R}^{t \times n_1}$ is the matrix of mapped $\ensuremath\boldsymbol{X}_1$ to the feature space. The $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}_2) \in \mathbb{R}^{t \times n_2}$ is defined similarly.
We can compute the kernel matrix of dataset $\ensuremath\boldsymbol{X} \in \mathbb{R}^{d \times n}$ over itself:
\begin{align}\label{equation_kernel_matrix_of_X}
\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{K}_x := \ensuremath\boldsymbol{K}(\ensuremath\boldsymbol{X}, \ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}),
\end{align}
where $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) := [\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_1), \dots, \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_n)] \in \mathbb{R}^{t \times n}$ is the pulled (mapped) data.

Note that in kernel methods, the pulled data $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$ are usually not available and merely the kernel matrix $\ensuremath\boldsymbol{K}(\ensuremath\boldsymbol{X}, \ensuremath\boldsymbol{X})$, which is the inner product of the pulled data with itself, is available.

There exist different types of kernels. Some of the most well-known kernels are:
\begin{align}
&\text{Linear:} ~~ k(\ensuremath\boldsymbol{x}_1, \ensuremath\boldsymbol{x}_2) = \ensuremath\boldsymbol{x}_1^\top \ensuremath\boldsymbol{x}_2 + c_1, \\
&\text{Polynomial:} ~~ k(\ensuremath\boldsymbol{x}_1, \ensuremath\boldsymbol{x}_2) = (c_1\ensuremath\boldsymbol{x}_1^\top \ensuremath\boldsymbol{x}_2 + c_2)^{c_3}, \\
&\text{Gaussian:} ~~ k(\ensuremath\boldsymbol{x}_1, \ensuremath\boldsymbol{x}_2) = \exp\big(\!-\frac{||\ensuremath\boldsymbol{x}_1 - \ensuremath\boldsymbol{x}_2||_2^2}{2\sigma^2}\big), \\
&\text{Sigmoid:} ~~ k(\ensuremath\boldsymbol{x}_1, \ensuremath\boldsymbol{x}_2) = \tanh(c_1\ensuremath\boldsymbol{x}_1^\top\ensuremath\boldsymbol{x}_2 + c_2),
\end{align}
where $c_1$, $c_2$, $c_3$, and $\sigma$ are scalar constants. The Gaussian and Sigmoid kernels are also called Radial Basis Function (RBF) and hyperbolic tangent, respectively. Note that the Gaussian kernel can also be written as $\exp\big(\!-\gamma||\ensuremath\boldsymbol{x}_1 - \ensuremath\boldsymbol{x}_2||_2^2\big)$ where $\gamma > 0$.

It is noteworthy to mention that in the RBF kernel, the dimensionality of the feature space is infinite. The reason lies in the Maclaurin series expansion (Taylor series expansion at zero) of this kernel:
\begin{align*}
\exp(-\gamma r) \approx 1 - \gamma r + \frac{\gamma^2}{2!} r^2 - \frac{\gamma^3}{3!} r^3 + \dots,
\end{align*}
where $r := ||\ensuremath\boldsymbol{x}_1 - \ensuremath\boldsymbol{x}_2||_2^2$, which is infinite dimensional with respect to $r$.

It is also worth mentioning that if we want the pulled data $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$ to be centered, i.e.:
\begin{align}
\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) := \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H},
\end{align}
we should double center the kernel matrix (see Appendix \ref{section_appendix_centering}) because if we use centered pulled data in Eq. (\ref{equation_kernel_matrix_of_X}), we have:
\begin{align*}
&\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \big(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\big)^\top \big(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\big) \\
&\overset{(\ref{equation_centeringMatrix_is_symmetric})}{=} \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H} \overset{(\ref{equation_kernel_matrix_of_X})}{=} \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H},
\end{align*}
which is the double-centered kernel matrix.
Thus:
\begin{align}\label{equation_doubleCentered_kernel}
\breve{\ensuremath\boldsymbol{K}}_x := \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H} = \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}),
\end{align}
where $\breve{\ensuremath\boldsymbol{K}}_x$ denotes the double-centered kernel matrix (see Appendix \ref{section_appendix_centeringKernel}).

\subsection{Projection}

We apply incomplete SVD on the centered pulled (mapped) data $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$ (see Appendix \ref{section_appendix_SVD}):
\begin{align}\label{equation_kernelPCA_SVD_Phi}
\mathbb{R}^{t \times n} \ni \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top,
\end{align}
where $\ensuremath\boldsymbol{U} \in \mathbb{R}^{t \times p}$ and $\ensuremath\boldsymbol{V} \in \mathbb{R}^{n \times p}$ contain the $p$ leading left and right singular vectors of $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$, respectively, where $p$ is the number of ``non-zero'' singular values of $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$ and usually $p \ll t$. Here, the $\ensuremath\boldsymbol{\Sigma} \in \mathbb{R}^{p \times p}$ is a square matrix having the $p$ largest non-zero singular values of $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$.

However, as mentioned before, the pulled data are not necessarily available so Eq. (\ref{equation_kernelPCA_SVD_Phi}) cannot be done.
The kernel, however, is available. Therefore, we apply eigen-decomposition \cite{ghojogh2019eigenvalue} to the double-centered kernel:
\begin{align}\label{equation_kernelPCA_eigen_centered_kernel}
\breve{\ensuremath\boldsymbol{K}}_x \ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Lambda},
\end{align}
where the columns of $\ensuremath\boldsymbol{V}$ and the diagonal of $\ensuremath\boldsymbol{\Lambda}$ are the eigenvectors and eigenvalues of $\breve{\ensuremath\boldsymbol{K}}_x$, respectively.
The columns of $\ensuremath\boldsymbol{V}$ in Eq. (\ref{equation_kernelPCA_SVD_Phi}) are the right singular vectors of $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$ which are equivalent to the eigenvectors of $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \breve{\ensuremath\boldsymbol{K}}_x$,
according to Proposition \ref{proposition_SVD} in Appendix \ref{section_appendix_SVD}.
Also, according to that proposition, the diagonal of $\ensuremath\boldsymbol{\Sigma}$ in Eq. (\ref{equation_kernelPCA_SVD_Phi}) is equivalent to the square root of eigenvalues of $\breve{\ensuremath\boldsymbol{K}}_x$.

Therefore, in practice where the pulling function is not necessarily available, we use Eq. (\ref{equation_kernelPCA_eigen_centered_kernel}) in order to find the $\ensuremath\boldsymbol{V}$ and $\ensuremath\boldsymbol{\Sigma}$ in Eq. (\ref{equation_kernelPCA_SVD_Phi}). The Eq. (\ref{equation_kernelPCA_eigen_centered_kernel}) can be restated as:
\begin{align}\label{equation_kernelPCA_eigen_centered_kernel_2}
\breve{\ensuremath\boldsymbol{K}}_x \ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^2,
\end{align}
to be compatible to Eq. (\ref{equation_kernelPCA_SVD_Phi}).
It is noteworthy that because of using Eq. (\ref{equation_kernelPCA_eigen_centered_kernel_2}) instead of Eq. (\ref{equation_kernelPCA_SVD_Phi}), \textit{the projection directions $\ensuremath\boldsymbol{U}$ are not available in kernel PCA to be observed or plotted.}

Similar to what we did for Eq. (\ref{equation_projected_dual}):
\begin{align}
&\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top \nonumber \\
&\implies \ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \underbrace{\ensuremath\boldsymbol{U}^\top\ensuremath\boldsymbol{U}}_{\ensuremath\boldsymbol{I}}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top = \ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top \nonumber \\
&\therefore ~~~~~~~ \ensuremath\boldsymbol{\Phi}(\widetilde{\ensuremath\boldsymbol{X}}) = \ensuremath\boldsymbol{U}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top, \label{equation_projected_kernel_PCA}
\end{align}
where $\ensuremath\boldsymbol{\Sigma}$ and $\ensuremath\boldsymbol{V}$ are obtained from Eq. (\ref{equation_kernelPCA_eigen_centered_kernel_2}).
The Eq. (\ref{equation_projected_kernel_PCA}) is projection of the training data in kernel PCA.

\subsection{Reconstruction}

Similar to what we did for Eq. (\ref{equation_reconstruction_dual}):
\begin{align}
\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\ensuremath\boldsymbol{V}^\top &\implies \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma}\underbrace{\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{V}}_{\ensuremath\boldsymbol{I}} = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Sigma} \nonumber \\
&\implies \ensuremath\boldsymbol{U} = \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-1}. \label{equation_kernel_U}
\end{align}
Therefore, the reconstruction is:
\begin{alignat}{2}
&\ensuremath\boldsymbol{\Phi}(\widehat{\ensuremath\boldsymbol{X}}) &&= \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Phi}(\widetilde{\ensuremath\boldsymbol{X}}) + \ensuremath\boldsymbol{\mu}_x \overset{(\ref{equation_kernel_U})}{=} \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{\Phi}(\widetilde{\ensuremath\boldsymbol{X}}) + \ensuremath\boldsymbol{\mu}_x \nonumber \\
& &&\overset{(\ref{equation_projected_kernel_PCA})}{=} \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\underbrace{\ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{\Sigma}}_{\ensuremath\boldsymbol{I}}\ensuremath\boldsymbol{V}^\top + \ensuremath\boldsymbol{\mu}_x \nonumber \\
&\implies &&\ensuremath\boldsymbol{\Phi}(\widehat{\ensuremath\boldsymbol{X}}) = \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{V}^\top + \ensuremath\boldsymbol{\mu}_x. \label{equation_reconstruction_kernel_PCA}
\end{alignat}
However, the $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$ is not available necessarily; therefore, we cannot reconstruct the training data in kernel PCA.

\subsection{Out-of-sample Projection}

Similar to what we did for Eq. (\ref{equation_outOfSample_projection_dual}):
\begin{align}
&\ensuremath\boldsymbol{U}^\top \overset{(\ref{equation_kernel_U})}{=} \ensuremath\boldsymbol{\Sigma}^{-\top}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \overset{(a)}{=} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \nonumber \\
&\implies \ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{x}}_t) = \ensuremath\boldsymbol{U}^\top \breve{\ensuremath\boldsymbol{\phi}}(\ensuremath\boldsymbol{x}_t) = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \breve{\ensuremath\boldsymbol{\phi}}(\ensuremath\boldsymbol{x}_t), \nonumber \\
&\overset{(\ref{equation_appendix_centered_kernelVector_outOfSample})}{\implies} \ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{x}}_t) = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{k}}_t, \label{equation_outOfSample_projection_kernel_PCA}
\end{align}
where $(a)$ is because $\ensuremath\boldsymbol{\Sigma}^{-1}$ is diagonal and thus symmetric and the $\breve{\ensuremath\boldsymbol{k}}_t \in \mathbb{R}^n$ is calculated by Eq. (\ref{equation_appendix_doubleCentered_outOfSample_kernel_oneSample}) in Appendix \ref{section_appendix_centeringKernel}.

The Eq. (\ref{equation_outOfSample_projection_kernel_PCA}) is the projection of out-of-sample data in kernel PCA.

Considering all the $n_t$ out-of-sample data points, $\ensuremath\boldsymbol{X}_t$, the projection is:
\begin{align}
\ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{X}}_t) = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{K}}_t,
\end{align}
where $\breve{\ensuremath\boldsymbol{K}}_t$ is calculated by Eq. (\ref{equation_appendix_doubleCentered_outOfSample_kernel}).

\subsection{Out-of-sample Reconstruction}

Similar to what we did for Eq. (\ref{equation_outOfSample_reconstruct_dual}):
\begin{align}
&\implies \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \overset{(\ref{equation_kernel_U})}{=} \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-1} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \nonumber\\
&\implies \ensuremath\boldsymbol{\phi}(\widehat{\ensuremath\boldsymbol{x}}_t) = \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})^\top \breve{\ensuremath\boldsymbol{\phi}}(\ensuremath\boldsymbol{x}_t) + \ensuremath\boldsymbol{\mu}_x \nonumber \\
&\overset{(\ref{equation_appendix_centered_kernelVector_outOfSample})}{\implies} \ensuremath\boldsymbol{\phi}(\widehat{\ensuremath\boldsymbol{x}}_t) = \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top \breve{\ensuremath\boldsymbol{k}}_t + \ensuremath\boldsymbol{\mu}_x, \label{equation_outOfSample_reconstruct_kernel_PCA}
\end{align}
where the $\breve{\ensuremath\boldsymbol{k}}_t \in \mathbb{R}^n$ is calculated by Eq. (\ref{equation_appendix_doubleCentered_outOfSample_kernel_oneSample}) in Appendix \ref{section_appendix_centeringKernel}.

Considering all the $n_t$ out-of-sample data points, $\ensuremath\boldsymbol{X}_t$, the reconstruction is:
\begin{align}
\ensuremath\boldsymbol{\phi}(\widehat{\ensuremath\boldsymbol{X}}_t) = \breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{V}\ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top \breve{\ensuremath\boldsymbol{K}}_t + \ensuremath\boldsymbol{\mu}_x,
\end{align}
where $\breve{\ensuremath\boldsymbol{K}}_t$ is calculated by Eq. (\ref{equation_appendix_doubleCentered_outOfSample_kernel}).

In Eq. (\ref{equation_outOfSample_reconstruct_kernel_PCA}), the $\breve{\ensuremath\boldsymbol{\Phi}}(\ensuremath\boldsymbol{X})$, appeared at the left of expression, is not available necessarily; therefore, we cannot reconstruct an out-of-sample point in kernel PCA.
According to Eqs. (\ref{equation_reconstruction_kernel_PCA}) and (\ref{equation_outOfSample_reconstruct_kernel_PCA}), we conclude that kernel PCA is not able to reconstruct any data, whether training or out-of-sample.

\subsection{Why is Kernel PCA Useful?}

Finally, it is noteworthy that as the choice of the best kernel might be hard, the kernel PCA is not ``always'' effective in practice \cite{ghodsi2006dimensionality}.
However, it provides us some useful theoretical insights for explaining the PCA, Multi-Dimensional Scaling (MDS) \cite{cox2008multidimensional}, Isomap \cite{tenenbaum2000global}, Locally Linear Embedding (LLE) \cite{roweis2000nonlinear}, and Laplacian Eigenmap (LE) \cite{belkin2003laplacian} as special cases of kernel PCA with their own kernels (see \cite{ham2004kernel} and chapter 2 in \cite{strange2014open}).

\section{Supervised Principal Component Analysis Using Scoring}

The older version of SPCA used scoring \cite{bair2006prediction}.
In this version of SPCA, PCA is not a special case of SPCA. The version of SPCA, which will be introduced in the next section, is more solid in terms of theory where PCA is a special case of SPCA.

In SPCA using scoring, we compute the similarity of every feature of data with the class labels and then sort the features and remove the features having the least similarity with the labels. The larger the similarity of a feature with the labels, the better that feature is for discrimination in the embedded subspace.

Consider the training dataset $\mathbb{R}^{d \times n} \ni \ensuremath\boldsymbol{X} = [\ensuremath\boldsymbol{x}_1, \dots, \ensuremath\boldsymbol{x}_n] = [\ensuremath\boldsymbol{x}^1, \dots, \ensuremath\boldsymbol{x}^d]^\top$ where $\ensuremath\boldsymbol{x}_i \in \mathbb{R}^d$ and $\ensuremath\boldsymbol{x}^j \in \mathbb{R}^n$ are the $i$-th data point and the $j$-th feature, respectively.
This type of SPCA is only for classification task so we can consider the dimensionality of the labels to be one, $\ell=1$. Thus, we have $\ensuremath\boldsymbol{Y} \in \mathbb{R}^{1 \times n}$. We define $\mathbb{R}^n \ni \ensuremath\boldsymbol{y} := \ensuremath\boldsymbol{Y}^\top$.

The score of the $j$-th feature, $\ensuremath\boldsymbol{x}^j$, is:
\begin{align}
\mathbb{R} \ni s_j := \frac{(\ensuremath\boldsymbol{x}^j)^\top \ensuremath\boldsymbol{y}}{||(\ensuremath\boldsymbol{x}^j)^\top \ensuremath\boldsymbol{x}^j||_2} = \frac{(\ensuremath\boldsymbol{x}^j)^\top \ensuremath\boldsymbol{y}}{\sqrt{(\ensuremath\boldsymbol{x}^j)^\top \ensuremath\boldsymbol{x}^j}},
\end{align}
After computing the scores of all the features, we sort the features from largest to smallest score. Let $\ensuremath\boldsymbol{X}' \in \mathbb{R}^{d \times n}$ denote the training dataset whose features are sorted.
We take the $q \leq d$ features with largest scores and remove the other features. Let:
\begin{align}
\mathbb{R}^{q \times n} \ni \ensuremath\boldsymbol{X}'' := \ensuremath\boldsymbol{X}'(1:q, :),
\end{align}
be the training dataset with $q$ best features.

Then, we apply PCA on the $\ensuremath\boldsymbol{X}'' \in \mathbb{R}^{q \times n}$ rather than $\ensuremath\boldsymbol{X} \in \mathbb{R}^{d \times n}$. Applying PCA and kernel PCA on $\ensuremath\boldsymbol{X}''$ results in SPCA and kernel PCA, respectively.
This type of SPCA was mostly used and popular in bioinformatics for genome data analysis \cite{ma2011principal}.

\section{Supervised Principal Component Analysis Using HSIC}

\subsection{Hilbert-Schmidt Independence Criterion}

Suppose we want to measure the dependence of two random variables. Measuring the correlation between them is easier because correlation is just ``linear'' dependence.

According to \cite{hein2004kernels}, two random variables are independent if and only if any bounded continuous functions of them are uncorrelated. Therefore, if we map the two random variables $\ensuremath\boldsymbol{x}$ and $\ensuremath\boldsymbol{y}$ to two different (``separable'') Reproducing Kernel Hilbert Spaces (RKHSs) and have $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x})$ and $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{y})$, we can measure the correlation of $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x})$ and $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{y})$ in Hilbert space to have an estimation of dependence of $\ensuremath\boldsymbol{x}$ and $\ensuremath\boldsymbol{y}$ in the original space.

The correlation of $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x})$ and $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{y})$ can be computed by the Hilbert-Schmidt norm of the cross-covariance of them \cite{gretton2005measuring}. Note that the squared Hilbert-Schmidt norm of a matrix $\ensuremath\boldsymbol{A}$ is \cite{bell2016trace}:
\begin{align*}
||\ensuremath\boldsymbol{A}||_{HS}^2 := \textbf{tr}(\ensuremath\boldsymbol{A}^\top \ensuremath\boldsymbol{A}),
\end{align*}
and the cross-covariance matrix of two vectors $\ensuremath\boldsymbol{x}$ and $\ensuremath\boldsymbol{y}$ is \cite{gubner2006probability,gretton2005measuring}:
\begin{align*}
\mathbb{C}\text{ov}(\ensuremath\boldsymbol{x}, \ensuremath\boldsymbol{y}) := \mathbb{E}\Big[&\big(\ensuremath\boldsymbol{x} - \mathbb{E}(\ensuremath\boldsymbol{x})\big) \big(\ensuremath\boldsymbol{y} - \mathbb{E}(\ensuremath\boldsymbol{y})\big) \Big].
\end{align*}
Using the explained intuition, an empirical estimation of the Hilbert-Schmidt Independence Criterion (HSIC) is introduced \cite{gretton2005measuring}:
\begin{align}\label{equation_HSIC}
\text{HSIC} := \frac{1}{(n-1)^2}\, \textbf{tr}(\ddot{\ensuremath\boldsymbol{K}}_x\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}),
\end{align}
where $\ddot{\ensuremath\boldsymbol{K}}_x$ and $\ensuremath\boldsymbol{K}_y$ are the kernels over $\ensuremath\boldsymbol{x}$ and $\ensuremath\boldsymbol{y}$, respectively. In other words, $\ddot{\ensuremath\boldsymbol{K}}_x = \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x})^\top \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x})$ and $\ensuremath\boldsymbol{K}_y = \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{y})^\top \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{y})$.
We are using $\ddot{\ensuremath\boldsymbol{K}}_x$ rather than $\ensuremath\boldsymbol{K}_x$ because $\ensuremath\boldsymbol{K}_x$ is going to be used in kernel SPCA in the next sections.
The term $1/(n-1)^2$ is used for normalization.
The $\ensuremath\boldsymbol{H}$ is the centering matrix (see Appendix \ref{section_appendix_centering}):
\begin{align}
\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{H} = \ensuremath\boldsymbol{I} - (1/n) \ensuremath\boldsymbol{1}\ensuremath\boldsymbol{1}^\top.
\end{align}
The $\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}$ double centers the $\ensuremath\boldsymbol{K}_y$ in HSIC.

The HSIC (Eq. (\ref{equation_HSIC})) measures the dependence of two random variable vectors $\ensuremath\boldsymbol{x}$ and $\ensuremath\boldsymbol{y}$. Note that $\text{HSIC}=0$ and $\text{HSIC}>0$ mean that $\ensuremath\boldsymbol{x}$ and $\ensuremath\boldsymbol{y}$ are independent and dependent, respectively. The greater the HSIC, the greater dependence they have.

\subsection{Supervised PCA}

Supervised PCA (SPCA) \cite{barshan2011supervised} uses the HSIC. We have the data $\ensuremath\boldsymbol{X} = [\ensuremath\boldsymbol{x}_1, \dots, \ensuremath\boldsymbol{x}_n] \in \mathbb{R}^{d \times n}$ and the labels $\ensuremath\boldsymbol{Y} = [\ensuremath\boldsymbol{y}_1, \dots, \ensuremath\boldsymbol{y}_n] \in \mathbb{R}^{\ell \times n}$, where $\ell$ is the dimensionality of the labels and we usually have $\ell=1$. However, in case the labels are encoded (e.g., one-hot-encoded) or SPCA is used for regression (e.g., see \cite{ghojogh2019instance}), we have $\ell > 1$.

SPCA tries to maximize the dependence of the projected data points $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}$ and the labels $\ensuremath\boldsymbol{Y}$. It uses a linear kernel for the projected data points:
\begin{align}\label{equation_SPCA_kernel_of_projection}
\ddot{\ensuremath\boldsymbol{K}}_x = (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X})^\top (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X},
\end{align}
and an arbitrary kernel $\ensuremath\boldsymbol{K}_y$ over $\ensuremath\boldsymbol{Y}$.
For classification task, one of the best choices for the $\ensuremath\boldsymbol{K}_y$ is delta kernel \cite{barshan2011supervised} where the $(i,j)$-th element of kernel is:
\begin{align}
\ensuremath\boldsymbol{K}_y = \delta_{\ensuremath\boldsymbol{y}_i, \ensuremath\boldsymbol{y}_j} :=
\left\{
\begin{array}{ll}
  1 & \text{if } \ensuremath\boldsymbol{y}_i = \ensuremath\boldsymbol{y}_j,\\
  0 & \text{if } \ensuremath\boldsymbol{y}_i \neq \ensuremath\boldsymbol{y}_j,
\end{array}
\right.
\end{align}
where $\delta_{\ensuremath\boldsymbol{y}_i, \ensuremath\boldsymbol{y}_j}$ is the Kronecker delta which is one if the $\ensuremath\boldsymbol{x}_i$ and $\ensuremath\boldsymbol{x}_j$ belong to the same class.

Another good choice for kernel in classification task in SPCA is an arbitrary kernel (e.g., linear kernel $\ensuremath\boldsymbol{K}_y = \ensuremath\boldsymbol{Y}^\top \ensuremath\boldsymbol{Y}$) over $\ensuremath\boldsymbol{Y}$ where the columns of $\ensuremath\boldsymbol{Y}$ are one-hot encoded. This is a good choice because the distances of classes will be equal; otherwise, some classes will fall closer than the others for no reason and fairness between classes goes away.

The SPCA can also be used for regression (e.g., see \cite{ghojogh2019instance}) and that is one of the advantages of SPCA. In that case, a good choice for $\ensuremath\boldsymbol{K}_y$ is an arbitrary kernel (e.g., linear kernel $\ensuremath\boldsymbol{K}_y = \ensuremath\boldsymbol{Y}^\top \ensuremath\boldsymbol{Y}$) over $\ensuremath\boldsymbol{Y}$ where the columns of the $\ensuremath\boldsymbol{Y}$, i.e., labels, are the observations in regression. Here, the distances of observations have meaning and should not be manipulated.

The HSIC in SPCA case becomes:
\begin{align}\label{equation_HSIC_SPCA}
\text{HSIC} = \frac{1}{(n-1)^2}\, \textbf{tr}(\ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}).
\end{align}
where $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$ is the unknown projection matrix for projection onto the SPCA subspace and should be found. The desired dimensionality of the subspace is $p$ and usually $p \ll d$.

We should maximize the HSIC in order to maxzimize the dependence of $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}$ and $\ensuremath\boldsymbol{Y}$. Hence:
\begin{equation}
\begin{aligned}
& \underset{\ensuremath\boldsymbol{U}}{\text{maximize}}
& & \textbf{tr}(\ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}), \\
& \text{subject to}
& & \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I},
\end{aligned}
\end{equation}
where the constraint ensures that the $\ensuremath\boldsymbol{U}$ is an orthogonal matrix, i.e., the SPCA directions are orthonormal.

Using Lagrangian \cite{boyd2004convex}, we have:
\begin{align*}
\mathcal{L} &= \textbf{tr}(\ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}) - \textbf{tr}\big(\ensuremath\boldsymbol{\Lambda}^\top (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} - \ensuremath\boldsymbol{I})\big) \\
&\overset{(a)}{=} \textbf{tr}(\ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top) - \textbf{tr}\big(\ensuremath\boldsymbol{\Lambda}^\top (\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} - \ensuremath\boldsymbol{I})\big),
\end{align*}
where $(a)$ is because of the cyclic property of trace and $\ensuremath\boldsymbol{\Lambda} \in \mathbb{R}^{p \times p}$ is a diagonal matrix $\textbf{diag}([\lambda_1, \dots, \lambda_p]^\top)$ including the Lagrange multipliers.
Setting the derivative of Lagrangian to zero gives:
\begin{align}
& \mathbb{R}^{d \times p} \ni \frac{\partial \mathcal{L}}{\partial \ensuremath\boldsymbol{U}} = 2 \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{U} - 2\ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda} \overset{\text{set}}{=} 0 \nonumber \\
& \implies \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Lambda}, \label{eqaution_SPCA_eigendecomposition}
\end{align}
which is the eigen-decomposition of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top$ where the columns of $\ensuremath\boldsymbol{U}$ and the diagonal of $\ensuremath\boldsymbol{\Lambda}$ are the eigenvectors and eigenvalues of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top$, respectively \cite{ghojogh2019eigenvalue}. The eigenvectors and eigenvalues are sorted from the leading (largest eigenvalue) to the trailing (smallest eigenvalue) because we are maximizing in the optimization problem.
As a conclusion, if projecting onto the SPCA subspace or $\textbf{span}\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$, the SPCA directions $\{\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p\}$ are the sorted eigenvectors of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top$.
In other words, the columns of the projection matrix $\ensuremath\boldsymbol{U}$ in SPCA are the $p$ leading eigenvectors of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top$.

Similar to what we had in PCA, the projection, projection of out-of-sample, reconstruction, and reconstruction of out-of-sample in SPCA are:
\begin{align}
&\widetilde{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}, \label{equation_SPCA_projection} \\
&\widetilde{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x}_t, \label{equation_SPCA_projection_outOfSample} \\
&\widehat{\ensuremath\boldsymbol{X}} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X} = \ensuremath\boldsymbol{U} \widetilde{\ensuremath\boldsymbol{X}}, \label{equation_SPCA_reconstruction} \\
&\widehat{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{x}_t = \ensuremath\boldsymbol{U} \widetilde{\ensuremath\boldsymbol{x}}_t, \label{equation_SPCA_reconstruction_outOfSample}
\end{align}
respectively.
In SPCA, there is no need to center the data as the centering is already handled by $\ensuremath\boldsymbol{H}$ in HSIC. This gets more clear in the following section where we see that PCA is a spacial case of SPCA.
Note that in the equations of SPCA, although not necessary, we can center the data and in that case, the mean of embedding in the subspace will be zero.

Considering all the $n_t$ out-of-sample data points, the projection and reconstruction are:
\begin{align}
&\widetilde{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}_t, \\
&\widehat{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X}_t = \ensuremath\boldsymbol{U} \widetilde{\ensuremath\boldsymbol{X}}_t,
\end{align}
respectively.

\subsection{PCA is a spacial case of SPCA!}

Not considering the similarities of the labels means that we do not care about the class labels so we are unsupervised.
if we do not consider the similarities of labels, the kernel over the labels becomes the identity matrix, $\ensuremath\boldsymbol{K}_y = \ensuremath\boldsymbol{I}$.
According to Eq. (\ref{eqaution_SPCA_eigendecomposition}), SPCA is the eigen-decomposition of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top$. In this case, this matrix becomes:
\begin{align*}
\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top &= \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{I}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top \overset{(\ref{equation_centeringMatrix_is_symmetric})}{=} \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{I}\ensuremath\boldsymbol{H}^\top\ensuremath\boldsymbol{X}^\top \\
&= \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{H}^\top\ensuremath\boldsymbol{X}^\top = (\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H})(\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H})^\top \\
&\overset{(\ref{equation_centered_training_data})}{=} \breve{\ensuremath\boldsymbol{X}} \breve{\ensuremath\boldsymbol{X}}^\top \overset{(\ref{equation_covariance_matrix})}{=} \ensuremath\boldsymbol{S},
\end{align*}
which is the covariance matrix whose eigenvectors are the PCA directions.
Thus, if we do not consider the similarities of labels, i.e., we are unsupervised, SPCA reduces to PCA as expected.

\subsection{Dual Supervised PCA}

The SPCA can be formulated in dual form \cite{barshan2011supervised}.
We saw that in SPCA, the columns of $\ensuremath\boldsymbol{U}$ are the eigenvectors of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top$. We apply SVD on $\ensuremath\boldsymbol{K}_y$ (see Appendix \ref{section_appendix_SVD}):
\begin{align*}
\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{K}_y = \ensuremath\boldsymbol{Q} \ensuremath\boldsymbol{\Omega} \ensuremath\boldsymbol{Q}^\top,
\end{align*}
where $\ensuremath\boldsymbol{Q} \in \mathbb{R}^{n \times n}$ includes left or right singular vectors and $\ensuremath\boldsymbol{\Omega} \in \mathbb{R}^{n \times n}$ contains the singular values of $\ensuremath\boldsymbol{K}_y$. Note that the left and right singular vectors are equal because $\ensuremath\boldsymbol{K}_y$ is symmetric and thus $\ensuremath\boldsymbol{K}_y \ensuremath\boldsymbol{K}_y^\top$ and $\ensuremath\boldsymbol{K}_y^\top \ensuremath\boldsymbol{K}_y$ are equal. As $\ensuremath\boldsymbol{\Omega}$ is a diagonal matrix with non-negative entries, we can decompose it to $\ensuremath\boldsymbol{\Omega} = \ensuremath\boldsymbol{\Omega}^{1/2} \ensuremath\boldsymbol{\Omega}^{1/2} = \ensuremath\boldsymbol{\Omega}^{1/2} (\ensuremath\boldsymbol{\Omega}^{1/2})^\top$ where the diagonal entries of $\ensuremath\boldsymbol{\Omega}^{1/2} \in \mathbb{R}^{n \times n}$ are square root of diagonal entries of $\ensuremath\boldsymbol{\Omega}$. Therefore, we can decompose $\ensuremath\boldsymbol{K}_y$ into:
\begin{align}
\ensuremath\boldsymbol{K}_y &= \ensuremath\boldsymbol{Q} \ensuremath\boldsymbol{\Omega}^{1/2} (\ensuremath\boldsymbol{\Omega}^{1/2})^\top \ensuremath\boldsymbol{Q}^\top \nonumber \\
&= (\ensuremath\boldsymbol{Q} \ensuremath\boldsymbol{\Omega}^{1/2}) (\ensuremath\boldsymbol{Q} \ensuremath\boldsymbol{\Omega}^{1/2})^\top = \ensuremath\boldsymbol{\Delta} \ensuremath\boldsymbol{\Delta}^\top, \label{equation_SPCA_decompose_kernelOfLabels}
\end{align}
where:
\begin{align}
\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{\Delta} := \ensuremath\boldsymbol{Q} \ensuremath\boldsymbol{\Omega}^{1/2}.
\end{align}
Therefore, we have:
\begin{align*}
\therefore ~~~~ \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top &\overset{(\ref{equation_SPCA_decompose_kernelOfLabels})}{=} \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta} \ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top \\
&\overset{(\ref{equation_centeringMatrix_is_symmetric})}{=} (\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}) (\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta})^\top = \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{\Psi}^\top,
\end{align*}
where:
\begin{align}\label{equation_Psi_dual_SPCA}
\mathbb{R}^{d \times n} \ni \ensuremath\boldsymbol{\Psi} := \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}.
\end{align}

We apply incomplete SVD on $\ensuremath\boldsymbol{\Psi}$ (see Appendix \ref{section_appendix_SVD}):
\begin{align}\label{equation_Psi_SVD_dual_SPCA}
\mathbb{R}^{d \times n} \ni \ensuremath\boldsymbol{\Psi} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \ensuremath\boldsymbol{V}^\top,
\end{align}
where $\ensuremath\boldsymbol{U} \in \mathbb{R}^{d \times p}$ and $\ensuremath\boldsymbol{V} \in \mathbb{R}^{d \times p}$ include the $p$ leading left or right singular vectors of $\ensuremath\boldsymbol{\Psi}$, respectively, and $\ensuremath\boldsymbol{\Sigma} \in \mathbb{R}^{p \times p}$ contains the $p$ largest singular values of $\ensuremath\boldsymbol{\Psi}$.

We can compute $\ensuremath\boldsymbol{U}$ as:
\begin{align}
\ensuremath\boldsymbol{\Psi} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \ensuremath\boldsymbol{V}^\top &\implies \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \underbrace{\ensuremath\boldsymbol{V}^\top \ensuremath\boldsymbol{V}}_{\ensuremath\boldsymbol{I}} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \nonumber\\
&\implies \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-1} \label{equation_U_dual_SPCA}
\end{align}

The projection of data $\ensuremath\boldsymbol{X}$ in dual SPCA is:
\begin{align}
\widetilde{\ensuremath\boldsymbol{X}} &\overset{(\ref{equation_SPCA_projection})}{=} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X} \overset{(\ref{equation_U_dual_SPCA})}{=} (\ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-1})^\top \ensuremath\boldsymbol{X} = \ensuremath\boldsymbol{\Sigma}^{-\top}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Psi}^\top\ensuremath\boldsymbol{X} \nonumber \\
& \overset{(\ref{equation_Psi_dual_SPCA})}{=} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{X}. \label{equation_projected_dual_SPCA}
\end{align}
Note that $\ensuremath\boldsymbol{\Sigma}$ and $\ensuremath\boldsymbol{H}$ are symmetric.

Similarly, out-of-sample projection in dual SPCA is:
\begin{align}\label{equation_outOfSample_projected_dual_SPCA}
\widetilde{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{x}_t.
\end{align}

Considering all the $n_t$ out-of-sample data points, the projection is:
\begin{align}
\widetilde{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{X}_t.
\end{align}

Reconstruction of $\ensuremath\boldsymbol{X}$ after projection onto the SPCA subspace is:
\begin{align}
\widehat{\ensuremath\boldsymbol{X}} &\overset{(\ref{equation_SPCA_reconstruction})}{=} \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{X} = \ensuremath\boldsymbol{U}\widetilde{\ensuremath\boldsymbol{X}} \nonumber \\
&\overset{(a)}{=} \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{X} \nonumber \\
&= \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{X} \nonumber \\
&\overset{(\ref{equation_Psi_dual_SPCA})}{=} \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{X} \label{equation_reconstruction_dual_SPCA}
\end{align}
where $(a)$ is because of Eqs. (\ref{equation_U_dual_SPCA}) and (\ref{equation_projected_dual_SPCA}).

Similarly, reconstruction of an out-of-sample data point in dual SPCA is:
\begin{align}\label{equation_outOfSample_reconstruction_dual_SPCA}
\widehat{\ensuremath\boldsymbol{x}}_t = \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{x}_t.
\end{align}

Considering all the $n_t$ out-of-sample data points, the reconstruction is:
\begin{align}
\widehat{\ensuremath\boldsymbol{X}}_t = \ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top\ensuremath\boldsymbol{X}_t.
\end{align}

Note that dual PCA was important especially because it provided opportunity to kernelize the PCA. However, as it is explained in next section, kernel SPCA can be obtained directly from SPCA. Therefore, dual SPCA might not be very important for the sake of kernel SPCA.

The dual SPCA has another benefit similar to what we had for dual PCA. In Eqs. (\ref{equation_projected_dual_SPCA}), (\ref{equation_outOfSample_projected_dual_SPCA}), (\ref{equation_reconstruction_dual_SPCA}), and (\ref{equation_outOfSample_reconstruction_dual_SPCA}), $\ensuremath\boldsymbol{U}$ is not used but $\ensuremath\boldsymbol{V}$ exists. In Eq. (\ref{equation_Psi_SVD_dual_SPCA}), the columns of $\ensuremath\boldsymbol{V}$ are the eigenvectors of $\ensuremath\boldsymbol{\Psi}^\top \ensuremath\boldsymbol{\Psi} \in \mathbb{R}^{n \times n}$, according to Proposition \ref{proposition_SVD} in appendix \ref{section_appendix_SVD}. On the other hand, in direct SPCA, we have eigen-decomposition of $\ensuremath\boldsymbol{X}\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{X}^\top \in \mathbb{R}^{d \times d}$ in Eq. (\ref{eqaution_SPCA_eigendecomposition}) which is then used in Eqs. (\ref{equation_SPCA_projection}), (\ref{equation_SPCA_projection_outOfSample}), (\ref{equation_SPCA_reconstruction}), and (\ref{equation_SPCA_reconstruction_outOfSample}).
In case we have huge dimensionality, $d \gg n$, decomposition of and $n \times n$ matrix is faster and needs less storage so dual SPCA will be more efficient.

\subsection{Kernel Supervised PCA}

The SPCA can be kernelized by two approaches, using either direct SPCA or dual SPCA \cite{barshan2011supervised}.

\subsubsection{Kernel SPCA Using Direct SPCA}

According to the representation theory \cite{alperin1993local}, any solution (direction) $\ensuremath\boldsymbol{u} \in \mathcal{H}$ must lie in the span of ``all'' the training vectors mapped to $\mathcal{H}$, i.e., $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) = [\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_1), \dots, \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_n)] \in \mathbb{R}^{t\times n}$ (usually $t \gg d$). Note that $\mathcal{H}$ denotes the Hilbert space (feature space). Therefore, we can state that:
\begin{align*}
\ensuremath\boldsymbol{u} = \sum_{i=1}^n \theta_i\, \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_i) = \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\, \ensuremath\boldsymbol{\theta},
\end{align*}
where $\ensuremath\boldsymbol{\theta} \in \mathbb{R}^n$ is the unknown vector of coefficients, and $\ensuremath\boldsymbol{u} \in \mathbb{R}^t$ is the kernel SPCA direction in Hilbert space here.
The directions can be put together in $\mathbb{R}^{t \times p} \ni \ensuremath\boldsymbol{U} := [\ensuremath\boldsymbol{u}_1, \dots, \ensuremath\boldsymbol{u}_p]$:
\begin{align}\label{equation_U_kernel_SPCA}
\ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\, \ensuremath\boldsymbol{\Theta},
\end{align}
where $\ensuremath\boldsymbol{\Theta} := [\ensuremath\boldsymbol{\theta}_1, \dots, \ensuremath\boldsymbol{\theta}_p] \in \mathbb{R}^{n \times p}$.

The Eq. (\ref{equation_HSIC_SPCA}) in the feature space becomes:
\begin{align*}
\text{HSIC} = \frac{1}{(n-1)^2}\, \textbf{tr}(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}).
\end{align*}
The $\textbf{tr}(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H})$ can be simplified as:
\begin{align}
&\textbf{tr}(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}) \nonumber \\
&= \textbf{tr}(\ensuremath\boldsymbol{U} \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top) \nonumber \\
&= \textbf{tr}(\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top\ensuremath\boldsymbol{U}) \label{equation_traceInHSIC_kernel_SPCA}
\end{align}
Plugging Eq. (\ref{equation_U_kernel_SPCA}) in Eq. (\ref{equation_traceInHSIC_kernel_SPCA}) gives us:
\begin{align}
&\textbf{tr}\big(\ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\, \ensuremath\boldsymbol{\Theta}\big) \nonumber \\
&= \textbf{tr}(\ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta}),
\end{align}
where:
\begin{align}\label{equation_kernel_SPCA_kernel_X}
\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{K}_x := \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}).
\end{align}
Note that the Eqs. (\ref{equation_kernel_SPCA_kernel_X}) and (\ref{equation_SPCA_kernel_of_projection}) are different and should not be confused.

Moreover, for the constraint of orthogonality of projection matrix, i.e., $\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{I}$, in features space becomes:
\begin{align}
\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{U} &= \big(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\, \ensuremath\boldsymbol{\Theta}\big)^\top \big(\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\, \ensuremath\boldsymbol{\Theta}\big) \nonumber \\
&= \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) \ensuremath\boldsymbol{\Theta} = \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x\, \ensuremath\boldsymbol{\Theta}
\end{align}
Therefore, the optimization problem is:
\begin{equation}
\begin{aligned}
& \underset{\ensuremath\boldsymbol{\Theta}}{\text{maximize}}
& & \textbf{tr}(\ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta}), \\
& \text{subject to}
& & \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x\, \ensuremath\boldsymbol{\Theta} = \ensuremath\boldsymbol{I},
\end{aligned}
\end{equation}
where the objective variable is the unknown $\ensuremath\boldsymbol{\Theta}$.

Using Lagrange multiplier \cite{boyd2004convex}, we have:
\begin{align*}
&\mathcal{L} = \\
&\textbf{tr}(\ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta}) - \textbf{tr}\big(\ensuremath\boldsymbol{\Lambda}^\top (\ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x\, \ensuremath\boldsymbol{\Theta} - \ensuremath\boldsymbol{I})\big) \\
&= \textbf{tr}(\ensuremath\boldsymbol{\Theta} \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x) - \textbf{tr}\big(\ensuremath\boldsymbol{\Lambda}^\top (\ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x\, \ensuremath\boldsymbol{\Theta} - \ensuremath\boldsymbol{I})\big),
\end{align*}
where $\ensuremath\boldsymbol{\Lambda} \in \mathbb{R}^{p \times p}$ is a diagonal matrix $\textbf{diag}([\lambda_1, \dots, \lambda_p]^\top)$.
\begin{align}
& \mathbb{R}^{n \times p} \ni \frac{\partial \mathcal{L}}{\partial \ensuremath\boldsymbol{\Theta}} = 2 \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta} - 2\ensuremath\boldsymbol{K}_x\ensuremath\boldsymbol{\Theta} \ensuremath\boldsymbol{\Lambda} \overset{\text{set}}{=} 0 \nonumber \\
& \implies \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta} = \ensuremath\boldsymbol{K}_x\ensuremath\boldsymbol{\Theta} \ensuremath\boldsymbol{\Lambda}, \label{equation_kernel_SPCA_generalized_eigen_problem}
\end{align}
which is the generalized eigenvalue problem $(\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x, \ensuremath\boldsymbol{K}_x)$ \cite{ghojogh2019eigenvalue}. The $\ensuremath\boldsymbol{\Theta}$ and $\ensuremath\boldsymbol{\Lambda}$, which are the eigenvector and eigenvalue matrices, respectively, can be calculated according to \cite{ghojogh2019eigenvalue}.

Note that in practice, we can naively solve Eq. (\ref{equation_kernel_SPCA_generalized_eigen_problem}) by left multiplying $\ensuremath\boldsymbol{K}_x^{-1}$ (hoping that it is positive definite and thus not singular):
\begin{align}
&\underbrace{\ensuremath\boldsymbol{K}_x^{-1}\ensuremath\boldsymbol{K}_x}_{\ensuremath\boldsymbol{I}} \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta} = \ensuremath\boldsymbol{\Theta} \ensuremath\boldsymbol{\Lambda} \nonumber \\
&\implies \ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{\Theta} = \ensuremath\boldsymbol{\Theta} \ensuremath\boldsymbol{\Lambda},
\end{align}
which is the eigenvalue problem \cite{ghojogh2019eigenvalue} for $\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_y\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x$, where columns of $\ensuremath\boldsymbol{\Theta}$ are the eigenvectors of it and $\ensuremath\boldsymbol{\Lambda}$ includes its eigenvalues on its diagonal.

If we take the $p$ leading eigenvectors to have $\ensuremath\boldsymbol{\Theta} \in \mathbb{R}^{n \times p}$, the projection of $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) \in \mathbb{R}^{t \times n}$ is:
\begin{align}
\mathbb{R}^{p \times n} \ni \ensuremath\boldsymbol{\Phi}(\widetilde{\ensuremath\boldsymbol{X}}) &= \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) \nonumber \\
&\overset{(\ref{equation_U_kernel_SPCA})}{=} \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_x,
\end{align}
where $\mathbb{R}^{n \times n} \ni \ensuremath\boldsymbol{K}_x := \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$.
Similarly, the projection of out-of-sample data point $\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_t) \in \mathbb{R}^t$ is:
\begin{align}
\mathbb{R}^{p} \ni \ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{x}}_t) &= \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_t) \nonumber \\
&\overset{(\ref{equation_U_kernel_SPCA})}{=} \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_t) = \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{k}_t,
\end{align}
where $\ensuremath\boldsymbol{k}_t$ is Eq. (\ref{equation_appendix_kernelVector_outOfSample}).

Considering all the $n_t$ out-of-sample data points, $\ensuremath\boldsymbol{X}_t$, the projection is:
\begin{align}
\mathbb{R}^{p \times n_t} \ni \ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{X}}_t) = \ensuremath\boldsymbol{\Theta}^\top \ensuremath\boldsymbol{K}_t,
\end{align}
where $\ensuremath\boldsymbol{K}_t$ is Eq. (\ref{equation_appendix_kernelMatrix_outOfSample}).

As we will show in the following section, in kernel SPCA, as in kernel PCA, we cannot reconstruct data, whether training or out-of-sample.

\subsubsection{Kernel SPCA Using Dual SPCA}

The Eq. (\ref{equation_Psi_dual_SPCA}) in $t$-dimensional feature space becomes:
\begin{align}\label{equation_Psi_kernel_SPCA_using_dual}
\mathbb{R}^{t \times n} \ni \ensuremath\boldsymbol{\Psi} = \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta},
\end{align}
where $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) = [\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_1), \dots, \ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_n)] \in \mathbb{R}^{t \times n}$.

Applying SVD (see Appendix \ref{section_appendix_SVD}) on $\ensuremath\boldsymbol{\Psi}$ of Eq. (\ref{equation_Psi_kernel_SPCA_using_dual}) is similar to the form of Eq. (\ref{equation_Psi_SVD_dual_SPCA}).
Having the same discussion which we had for Eqs. (\ref{equation_kernelPCA_SVD_Phi}) and (\ref{equation_kernelPCA_eigen_centered_kernel_2}), we do not necessarily have $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$ in Eq. (\ref{equation_Psi_kernel_SPCA_using_dual}) so we can obtain $\ensuremath\boldsymbol{V}$ and $\ensuremath\boldsymbol{\Sigma}$ as:
\begin{align}\label{equation_kernel_SPCA_eigen_Delta_centered_kernel}
\big(\ensuremath\boldsymbol{\Delta}^\top \breve{\ensuremath\boldsymbol{K}}_x \ensuremath\boldsymbol{\Delta}\big) \ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^2,
\end{align}
where $\breve{\ensuremath\boldsymbol{K}}_x := \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{\Delta}$ and the columns of $\ensuremath\boldsymbol{V}$ are the eigenvectors of (see Proposition \ref{proposition_SVD} in Appendix \ref{section_appendix_SVD}):
\begin{align*}
\ensuremath\boldsymbol{\Psi}^\top \ensuremath\boldsymbol{\Psi} &\overset{(a)}{=} \ensuremath\boldsymbol{\Delta}^\top \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{\Delta} \overset{(\ref{equation_kernel_SPCA_kernel_X})}{=} \ensuremath\boldsymbol{\Delta}^\top \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{K}_x \ensuremath\boldsymbol{H} \ensuremath\boldsymbol{\Delta} \\
&= \ensuremath\boldsymbol{\Delta}^\top \breve{\ensuremath\boldsymbol{K}}_x \ensuremath\boldsymbol{\Delta},
\end{align*}
where $(a)$ is because of Eqs. (\ref{equation_Psi_kernel_SPCA_using_dual}) and (\ref{equation_centeringMatrix_is_symmetric}).

It is noteworthy that because of using Eq. (\ref{equation_kernel_SPCA_eigen_Delta_centered_kernel}) instead of Eq. (\ref{equation_Psi_kernel_SPCA_using_dual}), \textit{the projection directions $\ensuremath\boldsymbol{U}$ are not available in kernel SPCA to be observed or plotted.}

Similar to equations (\ref{equation_Psi_SVD_dual_SPCA}) and (\ref{equation_U_dual_SPCA}), we have:
\begin{align}
\ensuremath\boldsymbol{\Psi} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \ensuremath\boldsymbol{V}^\top &\implies \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \underbrace{\ensuremath\boldsymbol{V}^\top \ensuremath\boldsymbol{V}}_{\ensuremath\boldsymbol{I}} = \ensuremath\boldsymbol{U} \ensuremath\boldsymbol{\Sigma} \nonumber\\
&\implies \ensuremath\boldsymbol{U} = \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-1}, \label{equation_U_kernel_SPCA_using_dual}
\end{align}
where $\ensuremath\boldsymbol{V}$ and $\ensuremath\boldsymbol{\Sigma}$ are obtained from Eq. (\ref{equation_kernel_SPCA_eigen_Delta_centered_kernel}).

The projection of data $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$ is:
\begin{align}
\ensuremath\boldsymbol{\Phi}(\widetilde{\ensuremath\boldsymbol{X}}) &= \ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) = (\ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-1})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) \nonumber \\
&= \ensuremath\boldsymbol{\Sigma}^{-\top}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Psi}^\top\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) \nonumber \nonumber \\
& \overset{(\ref{equation_Psi_kernel_SPCA_using_dual})}{=} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}), \nonumber \\
& \overset{(\ref{equation_kernel_SPCA_kernel_X})}{=} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x.
\label{equation_projected_kernel_SPCA_using_dual}
\end{align}
Note that $\ensuremath\boldsymbol{\Sigma}$ and $\ensuremath\boldsymbol{H}$ are symmetric.

Similarly, out-of-sample projection in kernel SPCA is:
\begin{align}
\ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{x}}_t) &= \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_t) \nonumber \\
&= \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\,\ensuremath\boldsymbol{k}_t,
\end{align}
where $\ensuremath\boldsymbol{k}_t$ is Eq. (\ref{equation_appendix_kernelVector_outOfSample}).

Considering all the $n_t$ out-of-sample data points, $\ensuremath\boldsymbol{X}_t$, the projection is:
\begin{align}
\ensuremath\boldsymbol{\phi}(\widetilde{\ensuremath\boldsymbol{X}}_t)
= \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\,\ensuremath\boldsymbol{K}_t.
\end{align}
where $\ensuremath\boldsymbol{K}_t$ is Eq. (\ref{equation_appendix_kernelMatrix_outOfSample}).

Reconstruction of $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$ after projection onto the SPCA subspace is:
\begin{align}
\ensuremath\boldsymbol{\Phi}(\widehat{\ensuremath\boldsymbol{X}}) &= \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{U}^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{U}\ensuremath\boldsymbol{\Phi}(\widetilde{\ensuremath\boldsymbol{X}}) \nonumber \\
& \overset{(a)}{=} \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{\Sigma}^{-1}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \nonumber \\
&= \ensuremath\boldsymbol{\Psi}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \nonumber \\
&\overset{(\ref{equation_Psi_kernel_SPCA_using_dual})}{=} \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{K}_x \label{equation_outOfSample_projected_kernel_SPCA_using_dual}
\end{align}
where $(a)$ is because of Eqs. (\ref{equation_U_kernel_SPCA_using_dual}) and (\ref{equation_projected_kernel_SPCA_using_dual}).

Similarly, reconstruction of an out-of-sample data point in dual SPCA is:
\begin{align}
\widehat{\ensuremath\boldsymbol{x}}_t &= \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top\ensuremath\boldsymbol{\phi}(\ensuremath\boldsymbol{x}_t) \nonumber\\
&= \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})\ensuremath\boldsymbol{H}\ensuremath\boldsymbol{\Delta}\ensuremath\boldsymbol{V} \ensuremath\boldsymbol{\Sigma}^{-2}\ensuremath\boldsymbol{V}^\top\ensuremath\boldsymbol{\Delta}^\top\ensuremath\boldsymbol{H}\,\ensuremath\boldsymbol{k}_t, \label{equation_outOfSample_reconstruction_kernel_SPCA_using_dual}
\end{align}
where $\ensuremath\boldsymbol{k}_t$ is Eq. (\ref{equation_appendix_kernelVector_outOfSample}).

However, in Eqs. (\ref{equation_outOfSample_projected_kernel_SPCA_using_dual}) and (\ref{equation_outOfSample_reconstruction_kernel_SPCA_using_dual}), we do not necessarily have $\ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})$; therefore, in kernel SPCA, as in kernel PCA, we cannot reconstruct data, whether training or out-of-sample.

\begin{figure*}[!t]
\centering
\includegraphics[width=6.5in]{./images/project_train_Frey}
\caption{Projection of the Frey dataset onto the subspaces of (a) the PCA, (b) the dual SPCA, (c) the kernel PCA (linear kernel), (d) the kernel PCA (RBF kernel), and (e) the kernel PCA (cosine kernel).}
\label{figure_projection_train_Frey}
\end{figure*}

\section{Simulations}\label{section_simulations}

\subsection{Experiments on PCA and Kernel PCA}

For the experiments on PCA and kernel PCA, we used the Frey face dataset. This dataset includes $1965$ images of one subject with different poses and expressions.
The dataset, except for reconstruction experiments, was standardized so that its mean and variance became zero and one, respectively.
First, we used the whole dataset for training the PCA, dual PCA, and kernel PCA.
Figure \ref{figure_projection_train_Frey} shows the projection of the images onto the two leading dimensions of the PCA, dual PCA, and kernel PCA subspaces.
As can be seen in this figure, the result of PCA and dual PCA are exactly the same but in this dataset, dual PCA is much faster.

The linear kernel is $\ensuremath\boldsymbol{K} = \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X})^\top \ensuremath\boldsymbol{\Phi}(\ensuremath\boldsymbol{X}) = \ensuremath\boldsymbol{X}^\top \ensuremath\boldsymbol{X}$.
According to Eqs. (\ref{equation_SVD_dual}) and (\ref{equation_kernelPCA_SVD_Phi}), dual PCA and kernel PCA with linear kernel are equivalent. Figure \ref{figure_projection_train_Frey} verifies it.
The projections of kernel PCA with RBF and cosine kernels are also shown in this figure.

We then shuffled the Frey dataset and randomly split the data to train/out-of-sample sets with $80\%/20\%$ portions. Figure \ref{figure_projection_test_Frey} shows the projection both training and out-of-sample images onto the PCA, dual PCA, and kernel PCA subspaces. The used kernels were linear, RBF, and cosine.
As can be seen, the out-of-sample data, although were not seen in the training phase, are projected very well. The model, somewhat, has extrapolated the projections.

The top ten PCA directions for the PCA trained on the entire Frey dataset are shown in Fig. \ref{figure_projection_directions}.
As can be seen, the projection directions of a facial dataset are some facial features which are like ghost faces. That is why the facial projection directions are also referred to as ``ghost faces''.
The ghost faces in PCA is also referred to as ``eigenfaces'' \cite{turk1991eigenfaces,turk1991face}.
In this figure, the projection directions have captured different facial features which discriminate the data with respect to the maximum variance. The captured features are eyes, nose, cheeks, chin, lips, and eyebrows, which are the most important facial features.
This figure does not include projection directions of kernel PCA because in kernel PCA, the projection directions are not available as mentioned before.
Note that the face recognition using kernel PCA is referred to as ``kernel eigenfaces'' \cite{yang2000face}.

The reconstruction of training and test images after projection onto PCA are shown in Fig. \ref{figure_reconstruction_images}. We have reconstructed using all and also two top projection directions. As expected, the reconstructions using all projection directions are good enough.

The recognition of faces using PCA and kernel PCA are called eigenfaces \cite{turk1991eigenfaces,turk1991face} and kernel eigenfaces \cite{yang2000face} because PCA uses eigenvalue decomposition \cite{ghojogh2019eigenvalue} of the covariance matrix.

\begin{figure*}[!t]
\centering
\includegraphics[width=6.5in]{./images/project_test_Frey}
\caption{Projection of the training and out-of-sample sets of Frey dataset onto the subspaces of (a) the PCA, (b) the dual SPCA, (c) the kernel PCA (linear kernel), (d) the kernel PCA (RBF kernel), and (e) the kernel PCA (cosine kernel). The images with red frame are the out-of-sample images.}
\label{figure_projection_test_Frey}
\end{figure*}

\begin{figure*}[!t]
\centering
\includegraphics[width=6in]{./images/directions}
\caption{The ghost faces: the leading eigenvectors of PCA and SPCA for Frey, AT\&T, and AT\&T glasses datasets.}
\label{figure_projection_directions}
\end{figure*}

\begin{figure*}[!t]
\centering
\includegraphics[width=3.7in]{./images/reconstruction}
\caption{The reconstructed faces using all and two of the leading eigenvectors of PCA and SPCA for Frey and AT\&T datasets.}
\label{figure_reconstruction_images}
\end{figure*}

\subsection{Experiments on SPCA and Kernel SPCA}

For the experiments of SPCA and kernel SPCA, we used the AT\&T face dataset which includes $400$ images, $40$ subjects, and $10$ images per subject. The images of every person have different poses and expressions.
For better visualization of separation of classes in the projection subspace, we only used the images of the first three subjects.
The dataset, except for reconstruction experiments, was standardized so that its mean and variance became zero and one, respectively.

First, we used the entire $30$ images for training and the projection of images are shown in Fig. \ref{figure_projection_train_ATT} for SPCA, dual SPCA, direct kernel SPCA, and kernel SPCA using dual SPCA. As can be seen, the result of direct kernel SPCA and kernel SPCA using dual SPCA are very similar which is also claimed in \cite{barshan2011supervised} (although for some datasets they differ as we will see in Fig. \ref{figure_projection_test_ATT_glasses} for example). In comparisons, note that rotation, flipping, and scale do not matter in subspace and manifold learning because these impact all the distances of instances similarly.

We then took the first six images of each of the three subjects as training images and the rest as test images. The projection of training and out-of-sample images onto SPCA, dual SPCA, direct kernel SPCA, and kernel SPCA using dual SPCA are shown in Fig. \ref{figure_projection_test_ATT}.
This figure shows that projection of out-of-sample images have been properly carried on in SPCA and its variants.

The top projection directions of SPCA, dual SPCA, direct kernel SPCA, and kernel SPCA using dual SPCA trained by the entire $30$ images are shown in Fig. \ref{figure_projection_directions}. Moreover, the projection directions of PCA are also shown for this dataset. We can refer to the ghost faces (facial projection directions) of SPCA as the ``supervised eigenfaces''. The face recognition using kernel SPCA can also be referred to as ``kernel supervised eigenfaces''.
The Fig. \ref{figure_projection_directions} does not include projection directions of kernel SPCA because in kernel SPCA, the projection directions are not available as mentioned before.
Comparison of PCA and SPCA directions show that both PCA and SPCA are capturing eye glasses as important discriminators. However, some Haar wavelet like features \cite{stankovic2003haar} are captured as the projection directions in SPCA. We know that Haar wavelets are important in face recognition and detection such as Viola-Jones face detector \cite{wang2014analysis} which uses Haar wavelets.

The reconstruction of training and test images after projection onto SPCA are shown in Fig. \ref{figure_reconstruction_images}. We have reconstructed using all and also two top projection directions. As expected, the reconstructions using all projection directions are good enough.

We call the recognition of faces using SPCA and kernel SPCA as supervised eigenfaces and kernel supervised eigenfaces.

\begin{figure*}[!t]
\centering
\includegraphics[width=6.5in]{./images/project_train_ATT_normalized}
\caption{Projection of the AT\&T dataset onto the subspaces of (a) the SPCA, (b) the dual SPCA, (c) the direct kernel SPCA (linear kernel), (d) the direct kernel SPCA (RBF kernel), (e) the direct kernel SPCA (cosine kernel) subspaces, (f) the kernel SPCA using dual (linear kernel), (g) the kernel SPCA using dual (RBF kernel), and (h) the kernel SPCA using direct (cosine kernel).}
\label{figure_projection_train_ATT}
\end{figure*}

\begin{figure*}[!t]
\centering
\includegraphics[width=6.5in]{./images/project_test_ATT_normalized}
\caption{Projection of the training and out-of-sample sets of AT\&T dataset onto the subspaces of (a) the SPCA, (b) the dual SPCA, (c) the direct kernel SPCA (linear kernel), (d) the direct kernel SPCA (RBF kernel), (e) the direct kernel SPCA (cosine kernel) subspaces, (f) the kernel SPCA using dual (linear kernel), (g) the kernel SPCA using dual (RBF kernel), and (h) the kernel SPCA using direct (cosine kernel). The images with red frame are the out-of-sample images.}
\label{figure_projection_test_ATT}
\end{figure*}

\subsection{Comparison of PCA and SPCA}

For comparison of PCA and SPCA, we used $90$ images of AT\&T dataset without eye glasses and $90$ with eye glasses. We used all the $180$ images for training.
The dataset was standardized so that its mean and variance became zero and one, respectively.
Figure \ref{figure_projection_test_ATT_glasses} shows the projection of data onto PCA, SPCA, direct kernel SPCA, and kernel SPCA using dual SPCA. As can be seen, the PCA has not separated the two classes of with and without eye glasses. However, the two classes have been properly separated in SPCA and kernel SPCA because they make use of the class labels.

The projection directions of PCA and SPCA for this experiment are shown in Fig. \ref{figure_projection_directions}. Both PCA and SPCA have captured eyes as discriminators; however, SPCA has also focused on the frame of eye glasses because of the usage of class labels. That is while PCA has also captured other distracting facial features such as forehead, cheeks, hair, mustache, etc, because it is not aware that the two classes are different in terms of glasses and sees the whole dataset as a whole.

\begin{figure*}[!t]
\centering
\includegraphics[width=6.5in]{./images/project_test_ATT_glasses_normalized}
\caption{Projection of the of AT\&T glasses dataset onto the subspaces of (a) the PCA, (b) the SPCA, (c) the direct kernel SPCA (linear kernel), (d) the direct kernel SPCA (RBF kernel), (e) the direct kernel SPCA (cosine kernel) subspaces, (f) the kernel SPCA using dual (linear kernel), (g) the kernel SPCA using dual (RBF kernel), and (h) the kernel SPCA using direct (cosine kernel). The images with red frame are the images with glasses and the other images do not have glasses.}
\label{figure_projection_test_ATT_glasses}
\end{figure*}

\section{Conclusion and Future Work}\label{section_conclusions}

In this paper, the PCA and SPCA were introduced in details of theory. Moreover, kernel PCA and kernel SPCA were covered. The illustrations and experiments on Frey and AT\&T face datasets were also provided in order to analyze the explained methods in practice.

The calculation of $\ensuremath\boldsymbol{K}_y \in \mathbb{R}^{n \times n}$ in SPCA might be challenging for big data in terms of speed and storage. The Supervised Random Projection (SRP) \cite{karimi2018srp,karimi2018exploring} addresses this problem by approximating the kernel matrix $\ensuremath\boldsymbol{K}_y$ using Random Fourier Features (RFF)
\cite{rahimi2008random}. As a future work, we will write a tutorial on SRP.

Moreover, the sparsity is very effective because of the \textit{``bet on sparsity''} principal: ``Use a procedure that does well in sparse problems, since no procedure does well in dense problems \cite{friedman2001elements,tibshirani2015statistical}.''
Another reason for the effectiveness of the sparsity is Occam's razor \cite{domingos1999role} stating that ``simpler solutions are more likely to be correct than complex ones'' or ``simplicity is a goal in itself''.
Therefore, the sparse methods such as sparse PCA \cite{zou2006sparse,shen2008sparse}, sparse kernel PCA \cite{tipping2001sparse}, and Sparse Supervised Principal Component Analysis (SSPCA) \cite{sharifzadeh2017sparse} have been proposed. We will defer these methods to future tutorials.

\section*{Acknowledgment}
The authors hugely thank Prof. Ali Ghodsi (see his great online courses \cite{web_data_visualization,web_classification}), Prof. Mu Zhu, Prof. Wayne Oldford, Prof. Hoda Mohammadzade, and other professors whose courses have partly covered the materials mentioned in this tutorial paper.

Machine Learning (stat.ML); Machine Learning (cs.LG)