<jupyter_start><jupyter_text>Principal Component AnalysisExistence of large number of features or dimension makes analysis computationally intensive and hard to performance machine learning tasks for pattern identification. PCA is the most popular unsupervised linear transformation technique for dimensionality reduction. PCA finds the directions of maximum variance in high-dimensional data such that most of the information is retained and project it onto a smaller dimensional subspace.<jupyter_code>from IPython.display import Image
Image(filename='../Chapter 3 Figures/PCA.png', width=300)
import numpy as np
from sklearn.datasets import load_iris
from matplotlib import pyplot as plt
%matplotlib inline

from sklearn.decomposition import PCA

iris = load_iris()
X = iris.data<jupyter_output><empty_output><jupyter_text>Standardize data<jupyter_code>from sklearn.preprocessing import StandardScaler

X_std = StandardScaler().fit_transform(X)<jupyter_output><empty_output><jupyter_text>Eigens Decomposition<jupyter_code># eigendecomposition on covariance matrix

# create covariance matrix
cov_mat = np.cov(X_std.T)

print('Covariance matrix \n%s' %cov_mat)

eig_vals, eig_vecs = np.linalg.eig(cov_mat)
print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)
# sort eigenvalues in decreasing order
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]

print('Eigenvalues in descending order:')
for i in eig_pairs:
    print(i[0])<jupyter_output>Eigenvalues in descending order:
2.93035377559
0.927403621517
0.148342226482
0.0207460139956<jupyter_text>Plot principal components vs explained variance<jupyter_code>tot = sum(eig_vals)
var_exp = [(i / tot)*100 for i in sorted(eig_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print "Cummulative Variance Explained", cum_var_exp

plt.figure(figsize=(6, 4))

plt.bar(range(4), var_exp, alpha=0.5, align='center',
        label='Individual explained variance')
plt.step(range(4), cum_var_exp, where='mid',
         label='Cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc='best')
plt.tight_layout()
plt.show()<jupyter_output>Cummulative Variance Explained [  72.77045209   95.80097536   99.48480732  100.        ]<jupyter_text>You perform the PCA using sklearn library and plot eigenvaluesIn the above plot we can see that first 3 principal components are explaining 99% of the variance.Let's perform PCA using scikit-learn and plot the 1st 3 eigenvector<jupyter_code># source: http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html#
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn import datasets
from sklearn.decomposition import PCA

# import some data to play with
iris = datasets.load_iris()
Y = iris.target

# To getter a better understanding of interaction of the dimensions
# plot the first three PCA dimensions
fig = plt.figure(1, figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)
X_reduced = PCA(n_components=3).fit_transform(iris.data)
ax.scatter(X_reduced[:, 0], X_reduced[:, 1], X_reduced[:, 2], c=Y, cmap=plt.cm.Paired)
ax.set_title("First three PCA directions")
ax.set_xlabel("1st eigenvector")
ax.w_xaxis.set_ticklabels([])
ax.set_ylabel("2nd eigenvector")
ax.w_yaxis.set_ticklabels([])
ax.set_zlabel("3rd eigenvector")
ax.w_zaxis.set_ticklabels([])

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10014513788098693, 'fraction_numerical': 0.035703918722786644, 'mean_word_length': 6.540481400437637, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '612546', 'license': 'MIT', 'n_tokens_mistral': 1263, 'n_tokens_neox': 1145, 'n_words': 313, 'path': 'jupyter_notebooks/machine_learning/ebook_mastering_ml_in_6_steps/Chapter_3_Code/Code/PCA.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '3445'}