<jupyter_start><jupyter_text>Week 3 - Classical ML Models - Part II Principal Component AnalysisPrincipal components analysis (or PCA) is a technique used for dimensionality reduction enabling to identify signifficant correlations in the dataset. Consequently, it allows to reduce the number of dimensions within the dataset without lossing important information. Why do we need PCA?As we have seen so far, ML models has a tendency to work better with larger datasets: the good amount of data allows training of more accurate model. On the other hand, as we increase the dataset dimensions, we observe a few negative effects:- In large dimensional datasets, there are a lot of inconsistencies reducing model's accuracy- The usage of redundant features increases the computational time.This is where the dimensional reduction comes in - it helps to extract only the most signifficant correlations and reduce the number of dimensions. PCA computation steps 1. StandardizationIn short, the data standardization involves taking values and scalling them into a similar range. It ensures less biased model training process as larger values no longer shift whole model.It is carried out by subtracting each value by the mean and dividing by deviation.![standardization](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2019/08/Standardization-Principal-Component-Analysis-Edureka-300x77.png) 2. Computing covariance matrixAs it has been mentioned earlier, PCA helps to find the correlations within the dataset. These correlations between different dataset variables can be expressed in a covariance matrix.Mathematically speaking, covariance matrix can be imagined as a  p x p matrix, where p represent the dimensions.For example, let's say we have a 2-dimensional dataset consisting of variables a and b. In such case, the covariance matrix can be expressed as:![covariance matrix](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2019/08/Covariance-Matrix-Principal-Component-Analysis-Edureka-150x61.png)- $Cov(a, a)$ shows the covariance between the variable and itself- $Cov(a, b)$ shows the covariance between two variables 3. Eigenvalues and eigenvectorsLet's say we have a 3-dimensional matrix **A** containing dataset variables:![matrix](https://miro.medium.com/max/139/1*OBDgTXEUlUt3wfKblp47BQ.png)According to theory, if we multiply the matrix by a vector and the resulting vector differs from the original vector by a scalar value, such vector is called **eigenvector**. The scalar value in such expression becomes **eigenvalue**. In mathematical terms - $A.x = lambda.x$ or  $A.x - lambda.x = 0$.In such condition, the determinant of the characteristic function has to be equal to 0 or in other words:$det|A - lambda.I|$, where $I$ is identity function.To better understand this, let's analyze an example with 2-dimensions:![matrix](https://miro.medium.com/max/130/1*4htJZnnvPc6CLad3IqPLbw.png)After multilplying $\lambda$ by identity matrix and subtracting from the 2-dimensional matrix, we get:![matrix](https://miro.medium.com/max/144/1*eZSqgvsRvB8-sahbKqseGQ.png)After determining the determinant and solving it, we get the following eigenvalues: $\frac{5}{2} + i\frac{\sqrt{15}}{2}$ and $\frac{5}{2} - i\frac{\sqrt{15}}{2}$.To get the eigenvectors, we have to simply substitute the eigenvalues into the equation: $det|A - \lambda.I|$ 4. Computing principal componentsIn short, principal components are new set of variables obtained from the initial dataset that are signifficant an independent of each other. Once we computed the eigenvectors and eigenvalues, we have to order them in the descending order (first component is formed from eigenvector with the heighest eigenvalue and so on). 5. Reducing the dimensions of the datasetThe last step is to re-arrange the original data accoring to variables' signifficance. In order to do so, we simply multiply the transpose of the original data by the transpose of the feature vector.  Python implementation Such pipeline would take the following code form:<jupyter_code>import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

def pca(X):
    #Scaling values
    X = StandardScaler().fit_transform(X)
    
    #Computing covariance matrix
    mean = np.mean(X, axis = 0)
    cov_mat = (X - mean).T.dot((X - mean)) / (X.shape[0]-1)
    
    #Calculating eigenvectors and eigenvalues
    cov_mat = np.cov(X.T)
    eig_vals, eig_vecs = np.linalg.eig(cov_mat)
    
    #Computing feature vectors
    eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
    
    return eig_pairs<jupyter_output><empty_output><jupyter_text>However, as in the previous examples, we Scikit-learn library provides functions for performing PCA computations which reduces the coding time.<jupyter_code>#PCA using scikit-learn
def pca(X):
    
    pca = PCA(n_components=2)
    pca.fit_transform(X)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07226013649136893, 'fraction_numerical': 0.015455640305098354, 'mean_word_length': 4.9605263157894735, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '452349', 'license': 'MIT', 'n_tokens_mistral': 1433, 'n_tokens_neox': 1309, 'n_words': 596, 'path': 'Week-03/2_Principal Component Analysis.ipynb', 'repo_name': 'Davidyz/ML-Tutorials-Season-2', 'size': '4982'}