<jupyter_start><jupyter_text>Principal Component AnalysisThe principal application of PCA is **dimension reduction**. If we have high dimensional data, PCA allows us to reduce the dimensionality of the data so the bulk of the variation that exists in your data across many high dimensions is captured in fewer dimensions. The key idea for PCA is to help reducing dimension of the dataset to preserve the most information (variation) of the dataset. Curse of Dimensionality**Sparseness in N-Dimensional Space:** Curse of dimensionality refers to a set of problems that arise when working with high-dimensional data. The dimension of a dataset corresponds to the number of attributes/features that exist in a dataset. The popular aspects of the curse of dimensionality includes "data sparsity" and "distance concentration".**Data Sparsity**:When the training samples do not capture all combinations of the attributes, is referred to as "data sparsity" or simply "sparsity" in high dimensional data. Training a model with sparse data could lead to high-variance or overfitting condition because the model has learned from the frequently occurring combinations of the attributes and can predict the outcome accurately, but failed to predict less frequently occurring combinations in the reality (unseen data).**Distance Concentration**:Distance concentration refers to the problem of all the pairwise distances between different samples/points in the space converging to the same value as the dimensionality of the data increases. Several machine learning models such as clustering or nearest neighbours’ methods use distance-based metrics to identify similar or proximity of the samples. Due to distance concentration, the concept of proximity or similarity of the samples may not be qualitatively relevant in higher dimensions. Steps to Conducting PCA Mathematical Approach:1. Center each feature by subtracting the feature mean2. Calculate the covariance matrix for your normalized dataset (Eigen Decomposition)3. Calculate the eigenvectors/eigenvalues for the covariance matrix4. Take the dot product of the transpose of the eigenvectors with the transpose of the normalized data Graphical Approach:Assume two predictors / features case: $X_1$ and $X_2$1. Plot the two predictor on the graph with scatter plot![PCA0](img/pca0.png)![PCA1](img/pca1.png)2. Standard scale both $X_1$ and $X_2$ and fit the best PC line to the data![PCA2](img/pca2.png)3. Maximize the Sum of Squared Distance (SSD) for PC1![PCA3](img/pca3.png)Note: SSD for PC1 is the Eigenvalue for PC14. Calculate the Eigenvector for PC1![PCA4](img/pca4.png)Note: Eigenvector for PC1 is the called "singular vector", which consists of $\frac{\Delta X_1}{m}$ and $\frac{\Delta X_2}{m}$, where $m$ is the slope of PC1.5. PC2 is just a perpendicular line of PC1 in this case![PCA5](img/pca5.png)Note: the Eigenvalue and Eigenvector can be calculated using the same logic to the PC2 line.6. Rotate the plot to make PC1 the horizontal axis and PC2 the vertical axis![PCA6](img/pca6.png)At this stage, we can calculate the variation for PC1 and PC2:$Variation_{PC1} = \frac{SSD_{PC1}}{n-1}$$Variation_{PC2} = \frac{SSD_{PC2}}{n-1}$Total Variation = $Variation_{PC1} + Variation_{PC2}$Therefore, the total variation of the data being explained by PC1 and PC 2:$$Explained_{PC1} = \frac{Variation_{PC1}}{TotalVariation}$$$$Explained_{PC2} = \frac{Variation_{PC2}}{TotalVariation}$$For higher dimensional data, the Eigenvector for each principal component consists the total number of elements that matches with the total numbers of the predictors / features in the dataset.e.g. Principal Component Functions$$PC_1 = ev_{11}(x_1) + ev_{12}(x_2) + ... + ev_{1p}(x_p)$$$$PC_2 = ev_{21}(x_1) + ev_{22}(x_2) + ... + ev_{2p}(x_p)$$$$PC_3 = ev_{31}(x_1) + ev_{32}(x_2) + ... + ev_{3p}(x_p)$$...where - $ev_{11}, ev_{12}, ... ev_{1p}$ are the values in the eigenvector (which is calculated by $\frac{\Delta X_1}{m}, \frac{\Delta X_2}{m}$, ... in our example)   - $x_1, x_2, ... x_p$ are the predictors columns  - $p$ refers to the number of predictors in the dataset  Let's check out this example to see how the eigenvalues and eigenvectors are interpreted in the Principal Component Analysis.<jupyter_code># Import dependencies
import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
# Importing cars dataset
cars = pd.read_csv('data/cars.csv')

# Data Cleaning
cars[' cubicinches'].replace(' ', np.nan, inplace=True)
cars[' cubicinches'] = cars[' cubicinches'].astype(float)
cars[' weightlbs'].replace(' ', np.nan, inplace=True)
cars[' weightlbs'] = cars[' weightlbs'].astype(float)

# Train/Test Split
X_train, X_test, y_train, y_test = train_test_split(cars.drop('mpg', axis=1),
                                                    cars['mpg'],
                                                   random_state=20)

# Data Preparation
ct1 = ColumnTransformer(transformers=[
    ('imputer', SimpleImputer(), [1, 3])],
    remainder='passthrough')

ct2 = ColumnTransformer(transformers=[
    ('scaler', StandardScaler(), [0, 1, 2, 3, 4, 5]),
    ('ohe', OneHotEncoder(), [6])],
    remainder='passthrough')

# Pipeline for transformations
pipe = Pipeline(steps=[
    ('ct1', ct1),
    ('ct2', ct2)
])

# Transform the data
X_tr_pp = pipe.transform(X_train)
X_te_pp = pipe.transform(X_test)

# Fit the training data with LinearRegression()
lr = LinearRegression()
lr.fit(X_tr_pp, y_train)

# Training R^2
lr.score(X_tr_pp, y_train)

# Testing R^2
lr.score(X_te_pp, y_test)
# Create the PCA object for 3 PCs
cars_pca = PCA(n_components=3) 

# Fit and transform the data into PCs
X_train_new = cars_pca.fit_transform(X_tr_pp)
X_test_new = cars_pca.transform(X_te_pp)
# Calculate the values in each eigenvector 
cars_pca.components_
# Calculate the explained variance ratio for each PC
cars_pca.explained_variance_ratio_<jupyter_output><empty_output><jupyter_text>The results of our PCA are as follows:**PC1** = 0.465 * cubicinches_sd + 0.435 * weightlbs_sd + 0.449 * cylinders_sd + 0.454 * hp_sd - 0.349 * time-to-60_sd - 0.187 * year_sd - 0.068 * Europe - 0.073 * Japan + 0.140 * US**PC2** = -0.099 * cubicinches_sd - 0.196 * weightlbs_sd - 0.131 * cylinders_sd + 0.006 * hp_sd - 0.125 * time-to-60_sd - 0.937 * year_sd + 0.129 * Europe + 0.022 * Japan - 0.152 * US**PC3** = 0.141 * cubicinches_sd + 0.342 * weightlbs_sd + 0.187 * cylinders_sd - 0.144 * hp_sd + 0.851 * time-to-60_sd - 0.239 * year_sd + 0.043 * Europe - 0.132 * Japan + 0.089 * USNow that we have optimized our features, we can build a new model with them!<jupyter_code># Fit the new PC features with LinearRegression()
lr_pca = LinearRegression()
lr_pca.fit(X_train_new, y_train)

# Training R^2
lr_pca.score(X_train_new, y_train)

# Testing R^2
lr_pca.score(X_test_new, y_test)<jupyter_output><empty_output><jupyter_text>In this exercise, we should realize that1. PCA reduces the dimensions (number of features) in the modeling process and preserve majority of the information from the original predictors / features.2. PCA creates new dimensions (new features) that has zero correlation, which avoid multi-collinearity issue.3. PCA reduces the overfitting issue of the model.  Python CodeSKLearn Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html Calculate the Explained Variance by the PCA``` Python Import dependencifrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCA Standard Scale the predictorsX = StandardScaler().fit_transform(X) Create a PCA object that reduce dimensions into 2 principal componentspca = PCA(n_components = 2) Fit the predictors to the PCA objectprincipalComponents = pca.fit_transform(X) Putting the components into a dataframeprincipal_df = pd.DataFrame(data = principalComponents,                            columns = ['principal component 1', 'principal component 2']) Explained variance by the PCApca.explained_variance_ratio_``` Diagnostic of the Principal ComponentsWe usually use the **scree plot** to diagnose the principal components to determine the number of components to include in the model.``` Python Import dependenciesimport numpy as npimport matplotlib.pyplot as plt Create the PC values for the horizontal axisPCs = ['PC1', 'PC2', 'PC3', ....] Create the explained variance values for each componentexp_var = pca.explained_variance_ratio_.tolist() Create bar plot with the explained variance ratioplt.bar(PCs, exp_var, color='blue')plt.title('Scree Plot')plt.xlabel('Principal Components')plt.ylabel('Proportion of Variance Explained')plt.show()``` Dimension Reduction by Minimum Explained Variance Threshold``` Python Import dependenciesfrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAfrom sklearn.linear_model import LogisticRegression Standard Scale the predictorsX = StandardScaler().fit_transform(X) Create a PCA object that choose the minimum number of principal component such that 90% of the variance is retainedpca_90 = PCA(n_components = 0.90) Fit PCA on training setpca_90.fit(X_train) Transform both training set and test set with the pca_90 objectpca_X_train = pca.transform(X_train)pca_X_Test = pca.transform(X_test) Apply transformed training set for ML model, like logistic regressionlr = LogisticRegression(solver='lbfgs')lr.fit(pca_X_train, y_train) Make prediction based on the testing setlr.predict(pca_X_test) Calculate the evaluation metric (consider using other measure based on situation)lr.score(pca_X_test, y_test)```You can try setting different ```n_components``` values, such as 0.95, 0.9, 0.85, 0.8, 0.75, 0.7 ... to compare the number of components (features or predictors) applied to the model, time for training, and testing accuracy rate to see if it makes a significant improvement on training time without too much of accuracy rate.Now, lets play around with this image dataset from SKLearn and see if you can reduce the number of features for accuracy prediction.<jupyter_code># Import dependencies
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.datasets import fetch_openml
# Import the image dataset from SKLearn
mnist = fetch_openml('mnist_784')
# Test Train Split
train_img, test_img, train_lbl, test_lbl = train_test_split(
    mnist.data, mnist.target, test_size=0.15, random_state=999)
# Create the Scaling object
scaler = StandardScaler()

# Fit on training set only.
scaler.fit(train_img)

# Apply transform to both the training set and the test set.
train_img = scaler.transform(train_img)
test_img = scaler.transform(test_img)
# Create a PCA object with 500 principle components


# Fit the training data to the PCA


# Create the PC values for the horizontal axis


# Create the explained variance for each component


# Create bar plot with the explained variance ratio


# Create the PCA object to preserve minimum 90% of variance 


# Fit on training set only


# Transform the training set and testing set with the PCA object



# Build your own model for this classification problem


# Validate the prediction with CV or testing set


# Try to create a for loop to run the process with different n_component values and track the accuracy and run time<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08361204013377926, 'fraction_numerical': 0.024097418746248177, 'mean_word_length': 5.0268733850129195, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '335111', 'license': 'MIT', 'n_tokens_mistral': 3595, 'n_tokens_neox': 3268, 'n_words': 1339, 'path': 'Principal Component Analysis_Lecture.ipynb', 'repo_name': 'NormanLo4319/Principal-Component-Analysis', 'size': '11661'}