<jupyter_start><jupyter_text>Almost all the variation in the data is from left to right.In this case, we can take the 2-D data and project it on 1-D graph without any significant loss of data.Both graph would signify that the variation in data is from left to right.This can be proved from the covariance matrix below<jupyter_code>cov_mat = np.cov([x2,y2])
cov_mat<jupyter_output><empty_output><jupyter_text>This shows X varies in X by 9.1667 units and other units there is no variation at all.<jupyter_code>x3 = np.array([2]*10)
y3 = np.arange(0,10)
plt.plot(x3,y3)
np.cov([x3,y3])<jupyter_output><empty_output><jupyter_text>Similarly, This shows y varies in y by 9.1667 units and other units there is no variation at all.  Considering the IRIS DatasetAbout IrisFor the following tutorial, we will be working with the famous “Iris” dataset that has been deposited on the UCI machine learning repository (https://archive.ics.uci.edu/ml/datasets/Iris).The iris dataset contains measurements for 150 iris flowers from three different species.The three classes in the Iris dataset are:Iris-setosa (n=50)Iris-versicolor (n=50)Iris-virginica (n=50)And the four features of in Iris dataset are:sepal length in cmsepal width in cmpetal length in cmpetal width in cm<jupyter_code>iris = load_iris()
iris_df = pd.DataFrame(iris.data,columns=[iris.feature_names])
iris_df.head()
X = iris.data
X.shape<jupyter_output><empty_output><jupyter_text>StandardizingWhether to standardize the data prior to a PCA on the covariance matrix depends on the measurement scales of the original features. Since PCA yields a feature subspace that maximizes the variance along the axes, it makes sense to standardize the data, especially, if it was measured on different scales. Although, all features in the Iris dataset were measured in centimeters, let us continue with the transformation of the data onto unit scale (mean=0 and variance=1), which is a requirement for the optimal performance of many machine learning algorithms.<jupyter_code>from sklearn.preprocessing import StandardScaler
X_std = StandardScaler().fit_transform(X)
print X_std[0:5]
print "The shape of Feature Matrix is -",X_std.shape
X_covariance_matrix = np.cov(X_std.T)
X_covariance_matrix<jupyter_output><empty_output><jupyter_text>![alt text](eigen.png "Eigen Values") Eigenvectors and eigenvalues.Example of a a 2×2 covariance matrix that is given by$$\begin{array}{rrr}1.07 & 0.63 \\0.63 & 0.64 \end{array}$$What this means is that- The variance of the x variable is 1.07- The variance of the y variable is 0.64- The covariance between them is 0.63.> As it is a square symmetric matrix, it can be diagonalized by choosing a new orthogonal coordinate system, given by its eigenvectors (incidentally, this is called spectral theorem); corresponding eigenvalues will then be located on the diagonal. In this new coordinate system, the covariance matrix is diagonal and looks like that:$$\begin{array}{rrr}1.52 & 0 \\0 & 0.19 \end{array}$$- This means that the correlation between points is now zero. It becomes clear that the variance of any projection will be given by a weighted average of the eigenvalues.- Consequently, the maximum possible variance (1.52) will be achieved if we simply take the projection on the first coordinate axis. It follows that the direction of the first principal component is given by the first eigenvector of the covariance matrix.<jupyter_code>eig_vals, eig_vecs = np.linalg.eig(X_covariance_matrix)

print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)
# Make a list of (eigenvalue, eigenvector) tuples
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]

# Sort the (eigenvalue, eigenvector) tuples from high to low
eig_pairs.sort(key=lambda x: x[0], reverse=True)

# Visually confirm that the list is correctly sorted by decreasing eigenvalues
print('Eigenvalues in descending order:')
for i in eig_pairs:
    print(i[0])
tot = sum(eig_vals)
var_exp = [(i / tot)*100 for i in sorted(eig_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print "Variance captured by each component is \n",var_exp
print(40 * '-')
print "Cumulative variance captured as we travel each component \n",cum_var_exp<jupyter_output>Variance captured by each component is 
[72.770452093801353, 23.030523267680632, 3.6838319576273828, 0.51519268089063464]
----------------------------------------
Cumulative variance captured as we travel each component 
[  72.77    95.801   99.485  100.   ]<jupyter_text>The array above clearly shows that most of the variance (72.77% of the variance to be precise) can be explained by the first principal component alone. The second principal component still bears some information (23.03%) while the third and fourth principal components can safely be dropped without losing to much information. Together, the first two principal components contain 95.8% of the information.<jupyter_code>print "All Eigen Values along with Eigen Vectors"
pprint.pprint(eig_pairs)
print(40 * '-')
matrix_w = np.hstack((eig_pairs[0][1].reshape(4,1),
                      eig_pairs[1][1].reshape(4,1)))

print 'Matrix W:\n', matrix_w
Y = X_std.dot(matrix_w)
print Y[0:5]
pl.figure()
target_names = iris.target_names
y = iris.target
for c, i, target_name in zip("rgb", [0, 1, 2], target_names):
    pl.scatter(Y[y==i,0], Y[y==i,1], c=c, label=target_name)
pl.xlabel('Principal Component 1')
pl.ylabel('Principal Component 2')
pl.legend()
pl.title('PCA of IRIS dataset')
pl.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09373306844861838, 'fraction_numerical': 0.035398230088495575, 'mean_word_length': 5.153333333333333, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '67403', 'license': 'MIT', 'n_tokens_mistral': 1842, 'n_tokens_neox': 1678, 'n_words': 668, 'path': 'Principal_Component_Analysis_Iris_DataSet.ipynb', 'repo_name': 'amcodec1/pca_from_scratch_iris_dataset', 'size': '5537'}