<jupyter_start><jupyter_text>Chapter 8 - Dimensionality Reduction  Explained Variance Ratio / Proportion of Variance ExplainedAfter a projection is complete, we ask how much of the information in a given dataset is lost by projection the observations to the first few principal components? In other words, how much of the variance in the data is not contained in the first few principal components? Generally, we want to find the proportion of variance explained (PVE) by each principal component. The total variance in a dataset is defined as:$$\sum_{j=1}^p \text{Var}(X_j) = \sum_{j=1}^p\frac 1n \sum_{i=1}^n x^2_{ij}$$and the variance explained by the $m$th principal component is:$$\frac 1n \sum_{i=1}^n z^2_{im} = \frac 1n \sum_{i=1}^n\begin{pmatrix}\sum_{j=1}^p \phi_{jm}x_{ij}\end{pmatrix}^2$$and hence the PVE of the $m$th principal component is:$$\frac{\sum_{i=1}^n\begin{pmatrix}\sum_{j=1}^p \phi_{jm}x_{ij}\end{pmatrix}^2}{\sum_{j=1}^p \sum_{i=1}^n x^2_{ij}}$$<jupyter_code>import pandas as pd
import numpy as np

from sklearn.model_selection import KFold
from sklearn.decomposition import PCA, IncrementalPCA

import matplotlib.pyplot as plt

def load(fname):
    import pickle
    mnist = None
    try:
        with open(fname, 'rb') as f:
            mnist = pickle.load(f)
            return mnist
    except FileNotFoundError:
        from sklearn.datasets import fetch_openml
        mnist = fetch_openml('mnist_784', version=1, cache=True)
        with open(fname, 'wb') as f:
            mnist = pickle.dump(mnist, f)
        return mnist
# Ingest
mnsit = load('mnist.data.pkl')
mnsit_X, mnsit_y = mnsit['data'], mnsit['target']

kf = KFold(n_splits=10, shuffle=True, random_state=0)
kf.get_n_splits()
t1 = []
for train_index, test_index in kf.split(mnsit_X, mnsit_y):
    t1 = test_index
    break
X = mnsit_X[t1]
y_test = mnsit_y[t1]
# Train the PCA model
pca = PCA(n_components=25**2)
pca.fit(X)<jupyter_output><empty_output><jupyter_text>The PVE of each principal component is a positive quantity. In order to compute the cumulative PVE of the first $M$ principal components, we simply sum the PVE of the expression above.The PVE of each principal component as well as the cumulative PVE can be shown in a scree plot.<jupyter_code># Obtain the proportion of variance explained
var_df = pd.DataFrame({'evr' : pca.explained_variance_ratio_})
var_df['cum_vr'] = var_df['evr'].cumsum()
var_df['PC'] = var_df.index+1
# For testing
# print(var_df[:5])
# Scree plot & Cumulative Variance explained plot 
fig= plt.figure(figsize=(18,5))
ax1, ax2 = fig.add_subplot(121),fig.add_subplot(122)
var_df.plot(kind='line', x='PC', y='evr', ax=ax1)
var_df.plot(kind='line', x='PC', y='cum_vr', ax=ax2)
ax1.set_ylim(0,0.12)
ax1.set_yticks([i/100 for i in range(0,12)])
ax2.set_ylim(0,1.2)
ax2.set_yticks([i/10 for i in range(0,12)])
ax1.grid()
ax2.grid()
plt.show()
# Choosing the right number of dimensions using 95% explanation,
num_features = np.argmax(var_df['cum_vr']>0.95)+1
print(num_features)
# Train with this selection
pca2 = PCA(n_components=num_features)
pca2.fit(X)
# Fit using 95% explained variance directly using parameters
pca3 = PCA(n_components=0.95)
pca3.fit(X)
# For testing
# Validate that the n_components for both pca2 and pca3 are the same
# print(pca2.n_components_)
# print(pca3.n_components_)<jupyter_output><empty_output><jupyter_text>Of course, we aim to use the smallest number of principal components to describe the data aptly. However, there is no good solution to this.Generally, we use a scree plot to help us. Specifically, we look for a point where the proportion of variance explained by each subsequent principal component drops off. It is referred to as the elbow in the scree plot. In the above example, the elbow after the second principal component. The third principal component captures 10% of the variance and the fourth principal component explains less that 5% and is essentially worthless.Another thing to consider is to choose the number of dimensions that add up to a sufficiently large portion of the variance e.g. 95%.  Incremental PCAInstead of fitting the whole training set for the SVD algorithm to run, we can split the training set to mini-batches and run incremental PCA (IPCA), one batch at a time.<jupyter_code># Incremental PCA
n_batches = 100
incpca = IncrementalPCA(n_components=10)
for X_batch in np.array_split(X, n_batches):
    incpca.partial_fit(X_batch)
# Transform using the projection learnt after the incremental PCA
X_proj3 = incpca.transform(X)<jupyter_output><empty_output><jupyter_text>Randomised PCAAnother option for PCA is randomised PCA. This is a stochastic algorithm that quickly finds an approximation of the first $m$ principal components. It is faster when the number of principal components $m$ is much smaller than the number of observations $n$.<jupyter_code># Randomised PCA
rnd_pca = PCA(n_components=10, svd_solver='randomized')
# Transformation after using the projection learnt
X_proj4 = rnd_pca.fit_transform(X)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09896866322887743, 'fraction_numerical': 0.020825069416898056, 'mean_word_length': 5.172582619339045, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '358503', 'license': 'MIT', 'n_tokens_mistral': 1741, 'n_tokens_neox': 1638, 'n_words': 577, 'path': 'chap08/08-textbook-dimensionality-reduction-02.ipynb', 'repo_name': 'bryanblackbee/topic__hands-on-machine-learning', 'size': '5042'}