<jupyter_start><jupyter_text>Intro:Many ML datasets being very large cause extremely slow training coz of curse of dimensionality Here Dimensionality reduction comes to the rescue  Methods: PCA, Kernel PCA and LLE  PCA in sklearn Uses SVD Trying it on mnist<jupyter_code>import tensorflow as tf
import numpy as np
import sklearn
mnist = tf.keras.datasets.mnist.load_data()
(X_train, y_train), (X_test, y_test) = mnist
X_train[0].shape
from sklearn.decomposition import PCA
pca = PCA(n_components = 28)
X2D = pca.fit_transform(X_train[0])
X2D.shape
import matplotlib.pyplot as plt
plt.imshow(X_train[0])
plt.imshow(X2D)
[float(i) for i in list(pca.explained_variance_ratio_)]<jupyter_output><empty_output><jupyter_text>Variance is something that holds the information containing in data, more the variance in PCA, more information is preserved.  Choosing right number of dimensions in PCA Components that add up to make 95% of the variance are considered good<jupyter_code>pca = PCA()
pca.fit(X_train[0])<jupyter_output><empty_output><jupyter_text>calculate cumulative sum<jupyter_code>cumsum = np.cumsum(pca.explained_variance_ratio_)
d = np.argmax(cumsum >= 0.95)
d<jupyter_output><empty_output><jupyter_text>Select 5 principal components<jupyter_code>pca = PCA(n_components=5)
X5D = pca.fit_transform(X_train[0])
plt.imshow(X5D)<jupyter_output><empty_output><jupyter_text>Better option:<jupyter_code>pca = PCA(n_components=0.95)
X95var = pca.fit_transform(X_train[0])<jupyter_output><empty_output><jupyter_text>Another method: elbow method Steps to find n_components:plot Dimensions vs Variance using for loop where dimensions are in range(0, no. of feature columns)find "elbow point" at which variance starts to flat out sharplytake projection of elbow point on Dimension axis and find ideal dimensions  PCA as compressor/decompressor lets try<jupyter_code>pca = PCA(n_components=5)
X_95var = pca.fit_transform(X_train[0])
plt.imshow(X_train[0])
plt.imshow(pca.inverse_transform(X_95var))
plt.imshow(pca.inverse_transform(X_95var))<jupyter_output><empty_output><jupyter_text>Okay, not original Oh, something called reconstruction error caused that thing  Randomized PCA set svd_solver = 'randomized' and it quickly finds approximation of first d principal component's projection It is slightly faster than finding projections using full svd approach only if input data is larger than 500 * 500<jupyter_code>randomized_pca = PCA(n_components=5, svd_solver='randomized')
X_reduced = randomized_pca.fit_transform(X_train[0])<jupyter_output><empty_output><jupyter_text>Incremental PCA If the dataset is too large to load in memory, we can use incremental PCAIt will partially fit the training set in small batches<jupyter_code>from sklearn.decomposition import IncrementalPCA
ipca = IncrementalPCA(n_components=2)
n_batches = 2
for X_batch in np.array_split(X_train[0], n_batches):
    ipca.partial_fit(X_batch)
plt.imshow(ipca.inverse_transform(ipca.transform(X_train[1])))<jupyter_output><empty_output><jupyter_text>Kernel PCA Kernel PCA uses Kernel Trick Kernel Trick: It is a mathematical technique that implicitly maps instances into a very higher dimensional space and thus enables non linear classification with the help of hyperplane Kernel PCA is only useful for X with max. 2 Dimensions(2 columns only) Example:<jupyter_code>from sklearn.decomposition import KernelPCA
rbfpca = KernelPCA(n_components=2,kernel='rbf', gamma=0.05)
rbfpca.fit_transform(X)<jupyter_output><empty_output><jupyter_text>The ideal kernel and its gamma can be found by gridsearching kernel names, gamma values and fitting to a logistic regression classifier All instances should be fitted to the logreg i.e. without any train_test_spilt<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
kpca = KernelPCA(n_components=2)
logreg = sklearn.linear_model.LogisticRegression(n_jobs=-1)
clf = Pipeline([('kpca',kpca),('logreg',logreg)])
param_grid = [{
    "kpca__gamma":np.linspace(0.3,0.5,10),
    "kpca__kernel":["rbf","sigmoid"]
}]
grid_search = GridSearchCV(clf,param_grid,cv=3).fit(X,y)
grid_search.best_params_<jupyter_output><empty_output><jupyter_text>Another method to find best hyperparameters It is fully unsupervised Steps:select manual grid of hyperparameters, kernel and gammastart manual grid searchtake kpcatake inverse kpcafind reconstruction_errorselect best params on finding minimum reconstruction error  LLE: Locally Linear Embedding Very powerful non-linear dimensionality reduction(NLDR) technique It is manifold learning technique that does not rely on projections like pca It works by measuring how each training instance linearly relates to its closest neighbours and then looking for a low-dimensional representation where these local relationships are best preserved It will only work with less noise<jupyter_code>from sklearn.manifold import LocallyLinearEmbedding
lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
X_reduced = lle.fit_transform(X_train[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07559865426479319, 'fraction_numerical': 0.013061547595487829, 'mean_word_length': 7.624573378839591, 'pattern_counts': {'":': 2, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '483559', 'license': 'MIT', 'n_tokens_mistral': 1683, 'n_tokens_neox': 1586, 'n_words': 480, 'path': 'Hands on Machine Learning/Hands on ML ch8 Dimensionality Reduction.ipynb', 'repo_name': 'sonwanesuresh95/Books-to-notebooks', 'size': '5053'}