As a running example in this paper, consider images of hand-written digits from the MNIST[1]} dataset. For example, using PCA and a user-defined estimate of the number of latent variable \(k = 50,100,200,400, 784\)  we reduce the number of features of digit 0 and reconstruct digit 0 using the formula \(\hat{X} = U \hat{\Sigma } \hat{V}^T\)  for
\(\hat{V}\in \mathbb {R}^{n \times k}\) , to see the effect of various levels of dimension reduction. Visually, even when we reduce the number of dimension of digit 0 from 784 to 50, the reconstructed digit 0 is visually (figure REF ) similar to the original digit 0. Note, already a quite important issue has arisen. How is one to chose \(k\) ? To assist the latent dimension estimation process a scree plot (figure REF ) of the PCA digit 0, which plots the sizes of the singular values \({\sigma _{i}}\) , can be used. Moreover, one may expect that reconstruction error will remain low if \(k\)  is greater than the intrinsic dimensionality of the linear PCA embedding. On the contrary, if \(k\)  goes below the intrinsic dimensionality, the dimensionality reduction may cause a sudden increase in reconstruction error. This is illustrated in digit 0 using a reconstruction error plot(figure REF ) as \(k\)  decreases.
<FIGURE><FIGURE><FIGURE>