<jupyter_start><jupyter_text>Unsupervised Learning**Goal** : - To discover interesting things about themeasurements:     - Is there an informative way to visualize thedata?     - Can we discover subgroups among the variables oramong the observations?- We discuss two methods:    1. **Principal Components Analysis**, a tool used for datavisualization or data pre-processing. It is also a linear feature extraction technique    2. **Clustering**, a broad class of methods for discoveringunknown subgroups in data.<jupyter_code>norm_data = data.values
norm_data = StandardScaler().fit_transform(norm_data) # normalizing the features
np.mean(norm_data), np.std(norm_data)
# convert the normalized features into a tabular format
feat_cols = ['feature'+str(i) for i in range(norm_data.shape[1])]
normalised_data = pd.DataFrame(norm_data,columns=feat_cols)
normalised_data.tail()
pca_NMR = PCA(n_components=2)
principalComponents_NMR = pca_NMR.fit_transform(normalised_data)
pca_NMR.explained_variance_
principal_NMR = pd.DataFrame(data = principalComponents_NMR
             , columns = ['PC1', 'PC2'])
print('Explained variation per principal component: {}'.format(pca_NMR.explained_variance_ratio_))
npoints = 10000
idx = range(0,npoints,10)

plt.figure()
plt.figure(figsize=(9,7))
plt.xticks(fontsize=12)
plt.yticks(fontsize=14)
plt.xlabel('Principal Component-1',fontsize=20)
plt.ylabel('Principal Component-2',fontsize=20)
plt.title("PCA of Magnetization curves",fontsize=20)


plt.scatter(principal_NMR.iloc[idx,0],principal_NMR.iloc[idx,1])
plt.show()
# Choosing the number of components
fig,axes = plt.subplots(nrows=2,ncols=2,figsize=(10,8))
pca = PCA().fit(normalised_data)

axes[0,0].plot(np.cumsum(pca.explained_variance_ratio_))
axes[0,1].plot(np.cumsum(pca.explained_variance_ratio_)[:200])
axes[1,0].plot(np.cumsum(pca.explained_variance_ratio_)[:50])
axes[1,1].plot(np.cumsum(pca.explained_variance_ratio_)[:10])

for _,ax in np.ndenumerate(axes):
    ax.set_xlabel('Number of Principal components')
plt.suptitle('Cumulative Variance Expalined by PCA')
plt.show()<jupyter_output><empty_output><jupyter_text>Some conclusions- This curve quantifies how much of the total, 942-dimensional variance is contained within the first N components. For example, we see that within first 10 components contain approximately 85% of the variance, while you need around 50 components to describe close to 100% of the variance.- Here we see that our two-dimensional projection loses a lot of information (as measured by the explained variance) and that we'd need about 20 components to retain 90% of the variance. Looking at this plot for a high-dimensional dataset can help you understand the level of redundancy present in multiple observations.  Future things to do1. Dimensionality Reduction    - Randomised PCA    - t-SNE2. Clustering [(Scikit-API)](https://scikit-learn.org/stable/modules/clustering.html)    - Affinity Propagation    - Agglomerative Clustering    - BIRCH    - DBSCAN    - K-Means    - Mini-Batch K-Means    - Mean Shift    - OPTICS    - Spectral Clustering    - Mixture of Gaussians<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.101716465352829, 'fraction_numerical': 0.02161474888747616, 'mean_word_length': 5.71002132196162, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '110002', 'license': 'MIT', 'n_tokens_mistral': 1057, 'n_tokens_neox': 969, 'n_words': 275, 'path': 'Week1_2/Jun14_pca.ipynb', 'repo_name': 'Anantha-Rao12/NMR-quantumstates-GSOC21', 'size': '3146'}