<jupyter_start><jupyter_text>Exploring 4 dimension reduction techniques on 2 sklearn toy datasets.![dim_red_img](https://i.pinimg.com/originals/77/fa/32/77fa324afc57531dd8898a3ab7e83b87.jpg)In this notebook we explore dimension reduction, specifically 2 linear techniques: Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), as well as 2 manifold learning techniques: t-distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP). The wine and digits datasets from `sklearn.datasets` are used for 2D and 3D projection respectively.  Why reduce?A large dataset with many features may contain less useful/unimportant features, this inherently increases the risk of overfitting.High dimensional data also requires more storage space and is naturally difficult for humans to visualize - a projection into 2 or 3D space is what we can interpret easily.Fewer features reduces the overall computation required to train the model.Finally, less misleading data means better models in the real world.  2D projection of wine datasetThe wine dataset contains 178 samples belonging to 3 classes (labelled 0, 1 and 2) with 13 features. Load the data into a pandas Dataframe.<jupyter_code>import time
import pandas as pd 
import numpy as np
import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import axes3d, Axes3D
from matplotlib.colors import to_hex

from sklearn.datasets import load_digits, load_wine
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler

import umap
wine = load_wine()
x_train = wine.data
y_train = wine.target
x_train = StandardScaler().fit_transform(x_train) # Normalize
features = ["Alcohol","Malic acid", "Ash", "Alcalinity of Ash", "Magnesium",
                        "Total phenols", "Flavanoids", "Nonflavanoid phenols",
                        "Proanthocyanins", "Color intensity", "Hue",
                        "OD280/OD315 of diluted wines", "Proline"]
df = pd.DataFrame(x_train, columns = features)
df['Target'] = y_train
df.head()<jupyter_output><empty_output><jupyter_text>Since we are going to reduce the dimensions of the data, we define `build_reduced_df` to accept the reduced components and rebuild our dataframe with them.<jupyter_code>def build_reduced_df(data, columns, targets):
  
  # data = components from reduction technique
  # columns = name of columns in new dataframe
  # targets = original targets

  red_df = pd.DataFrame(data = data, columns = columns)
  red_df = pd.concat([red_df, targets], axis = 1)

  return red_df<jupyter_output><empty_output><jupyter_text>`plot_projection` accepts a dataframe and plots the components in either 2D or 3D depending on the number of components passed.<jupyter_code>def plot_projection (dataframe, components, targets, title):

  # dataframe = dataframe containing reduced components
  # components = list of column names passed to dataframe
  # targets = list of target classes 
  # title = title of plot

  fig = plt.figure(figsize = (8,8))

  # Set colors to tabular10, converts this to a list of corresponding hex values
  cmap = plt.get_cmap('tab10')
  colors = [to_hex(cmap(i)) for i in np.linspace(0, 1, len(targets))]

  if len(components) == 2:  

    # 2D plot

    ax = fig.add_subplot(1,1,1) 
    ax.set_xlabel(components[0])
    ax.set_ylabel(components[1])
    ax.set_title(title)

    targets = targets

    for target, color in zip(targets,colors):
        indicesToKeep = dataframe['Target'] == target
        ax.scatter(dataframe.loc[indicesToKeep, components[0]],
                   dataframe.loc[indicesToKeep, components[1]],
                   c = color, s = 50)

  else:   

    # 3D plot

    ax = Axes3D(fig)
    ax.set_xlabel(components[0])
    ax.set_ylabel(components[1])
    ax.set_zlabel(components[2])
    ax.set_title(title)

    targets = targets

    for target, color in zip(targets,colors):
        indicesToKeep = dataframe['Target'] == target
        ax.scatter(dataframe.loc[indicesToKeep, components[0]],
                    dataframe.loc[indicesToKeep, components[1]],
                    dataframe.loc[indicesToKeep, components[2]],
                    c = color, s = 50)
        
  ax.legend(targets)
  ax.grid()

  plt.show()<jupyter_output><empty_output><jupyter_text>Reduction via PCAIn essence, PCA captures the points of the data with the highest variance and, using this, rotates and projects the data in the direction of increasing variance. This results in highly correlated features being merged together such that the new components are representative of the overall dataset. PCA is a unsupervised algorithm, hence only `x_train` is passed to the `fit_transform`. PCA has been found to [outperform](https://ieeexplore.ieee.org/document/908974) LDA in classification tasks where the number of samples for a class are relatively small.<jupyter_code>N_COMPONENTS = 2 # Number of components

pca = PCA(n_components = N_COMPONENTS)
p_components = pca.fit_transform(x_train) # fit_transform since we normalized data
pca_df = build_reduced_df(p_components, ['PC 1', 'PC 2'], df['Target'])
plot_projection(pca_df, ["PC 1", "PC 2"], [0, 1, 2], "2D PCA Projection")<jupyter_output><empty_output><jupyter_text>Reduction via LDALDA aims to maximize the class seperability in the embedded space. This is achieved by maximizing the between-class variance whilst minimizing the within-class variance.Unlike PCA, LDA is a supervised algorithm, that is, it takes into account the class labels.<jupyter_code>N_COMPONENTS = 2

lda = LinearDiscriminantAnalysis(n_components = N_COMPONENTS) 
lda_components = lda.fit_transform(x_train, y_train)
lda_df = build_reduced_df(lda_components, ['LDA 1', 'LDA 2'], df['Target'])
plot_projection(lda_df, ["LDA 1", "LDA 2"], [0, 1, 2], "2D LDA Projection")<jupyter_output><empty_output><jupyter_text>Manifold learningWhen our data does not lie in a linear subspace, manifold learning methods (such as t-SNE and UMAP) are better suited. These techniques are based on the [Manifold Hypothesis](https://arxiv.org/abs/1310.0425) which states that the most relevant information in high dimensional data is often concentrated within a small number of lower dimensional manifolds.Both algorithims below necessarily warp the high-dimensional shape when projecting into a lower dimension. This results in cluster sizes and distances between clusters in the embedding to be meaningless.  Reduction via t-SNEIn essence, t-SNE computes the probability that pairs of datapoints within the high dimensional space are related and aims to produce a low dimensional embedding which produces a similar distribution. The use of stochastic neighbors allows t-SNE to consider both the local and global structure.Note: t-SNE is non-convex, that is, we may get stuck in some local minima. In an attempt to avoid this 'early compression' (simply adding an L2 penalty to the cost function during early stages of optimization) is used.Note: t-SNE is non-deterministic, sequential reruns on the same data may not produce the same outputs.<jupyter_code># PERPLEXITY = number of nearest neighbors for each point
# Higher value => Inc. global structure preservation
#              => Inc. training time
PERPLEXITY = 30
N_COMPONENTS = 2

tsne = TSNE(n_components=N_COMPONENTS, verbose=0, perplexity=PERPLEXITY, 
            n_iter=300)
tsne_components = tsne.fit_transform(x_train)
tsne_df = build_reduced_df(tsne_components, ['tsne 1', 'tsne 2'], df['Target'])
plot_projection(tsne_df, ["tsne 1", "tsne 2"], 
                [0,1,2], "t-SNE 2D Projection")<jupyter_output><empty_output><jupyter_text>Reduction via UMAPUMAP is similar to t-SNE, however it is faster, scales better and strikes a better balance between the preservation of local vs global structure: UMAP is better at preserving the global structure in the final projection. This implies that the inter-cluster relations are *potentially* more meaningful than in t-SNE.Note: UMAP is non-deterministic as well.For a great visualization of the trade off between local and global structure preservation between t-SNE and UMAP, have a look at this [article](https://pair-code.github.io/understanding-umap/).<jupyter_code># N_NEIGHBORS = number of initial neighbors for each point
# Higher value => Inc. global structure preservation


# MIN_DST = minimum distance between points in projection
# Higher value => Inc. global structure preservation
#              => 'Looser' clusters


N_NEIGHBORS = 15
MIN_DIST = 0.1
N_COMPONENTS = 2

umap_reducer = umap.UMAP(n_neighbors = N_NEIGHBORS, min_dist = MIN_DIST,
                 n_components = N_COMPONENTS)
umap_reducer = umap_reducer.fit_transform(x_train)
umap_df = build_reduced_df(umap_reducer, ['umap 1', 'umap 2'], df["Target"])
plot_projection(umap_df, ["umap 1", "umap 2"], 
                [0,1,2], "umap 2D Projection")<jupyter_output><empty_output><jupyter_text>3D projection of digits datasetThe digits dataset contains 1797 samples belonging to 10 classes. Each digit is an 8x8 image, that is, there are 64 dimensions.<jupyter_code>mnist = load_digits()
x_train_mnist = mnist.data / 255.0
y_train_mnist = mnist.target
# Just for naming columns
feat_cols = [ 'pixel'+str(i) for i in range(x_train_mnist.shape[1])]
mnist_df = pd.DataFrame(x_train_mnist, columns = feat_cols)
mnist_df['Target'] = y_train_mnist<jupyter_output><empty_output><jupyter_text>PCA<jupyter_code>N_COMPONENTS = 3

time_start = time.time()

pca_mnist = PCA(n_components = N_COMPONENTS)
p_mnist_components = pca_mnist.fit_transform(x_train_mnist)

print('PCA reduction complete! Time elapsed: {} seconds'.format(time.time()-time_start))
pca_mnist_df = build_reduced_df(p_mnist_components, 
                                ['PC 1', 'PC 2', 'PC 3'], 
                                mnist_df['Target'])
plot_projection(pca_mnist_df, 
                ['PC 1', 'PC 2', 'PC 3'], 
                [i for i in range(10)],   # classes  are digits 0-9
                "3D MNIST PCA Projection")<jupyter_output><empty_output><jupyter_text>LDA<jupyter_code>N_COMPONENTS = 3

time_start = time.time()

lda = LinearDiscriminantAnalysis(n_components = N_COMPONENTS) 
lda_mnist_components = lda.fit_transform(x_train_mnist, y_train_mnist)

print('LDA reduction complete! Time elapsed: {} seconds'.format(time.time()-time_start))
lda_mnist_df = build_reduced_df(lda_mnist_components, 
                                ['LDA 1', 'LDA 2', 'LDA 3'], 
                                mnist_df['Target'])
plot_projection(lda_mnist_df, 
                ['LDA 1', 'LDA 2', 'LDA 3'], 
                [i for i in range(10)],
                "3D MNIST LDA Projection")<jupyter_output><empty_output><jupyter_text>t-SNE<jupyter_code>PERPLEXITY = 30
N_COMPONENTS = 3

time_start = time.time()

tsne_mnist = TSNE(n_components=N_COMPONENTS, verbose=0, 
                  perplexity=PERPLEXITY, n_iter=300)

tsne_mnist_components = tsne_mnist.fit_transform(x_train_mnist)

print('t-SNE reduction complete! Time elapsed: {} seconds'.format(time.time()-time_start))
tsne_mnist_df = build_reduced_df(tsne_mnist_components, 
                                 ['tsne 1', 'tsne 2', 'tsne 3'], 
                                 mnist_df['Target'])
plot_projection(tsne_mnist_df, 
                ['tsne 1', 'tsne 2', 'tsne 3'], 
                [i for i in range(10)], 
                "3D MNIST t-SNE Projection")<jupyter_output><empty_output><jupyter_text>UMAP<jupyter_code>N_NEIGHBORS = 15
MIN_DIST = 0.2
N_COMPONENTS = 3

time_start = time.time()

umap_mnist_reducer = umap.UMAP(n_neighbors = N_NEIGHBORS, min_dist = MIN_DIST,
                 n_components = N_COMPONENTS)

umap_mnist_components = umap_mnist_reducer.fit_transform(x_train_mnist)

print('UMAP reduction complete! Time elapsed: {} seconds'.format(time.time()-time_start))
umap_mnist_df = build_reduced_df(umap_mnist_components, 
                                 ['umap 1', 'umap 2', 'umap 3'], 
                                 mnist_df['Target'])
plot_projection(umap_mnist_df, 
                ['umap 1', 'umap 2', 'umap 3'], 
                [i for i in range(10)], 
                "3D MNIST UMAP Projection")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08840273259596616, 'fraction_numerical': 0.017241379310344827, 'mean_word_length': 4.02328431372549, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 54, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '39105', 'license': 'MIT', 'n_tokens_mistral': 4087, 'n_tokens_neox': 3722, 'n_words': 1187, 'path': 'ML Concepts/DimReduction&Proj.ipynb', 'repo_name': 'Dev-data-fun/TensorFlow-notebooks', 'size': '12296'}