<jupyter_start><jupyter_text>**Unsupervised Learning Assignment**In this Jupyter Notebook, you can find the code that I've used to create the dendrograms and to calculate the optimal number of clusters. The notebook is divided in 2 parts: the first one where I load and process all the data and the second part where I calculate all the things that you required us. First of all, let's import all the functions that we need to work with:<jupyter_code>import pandas as pd # To work with dataframes
import numpy as np # To work with arrays
import seaborn as sns # To make scatterplot
from scipy.cluster import hierarchy # To make hierarchical clustering
from scipy.cluster.hierarchy import dendrogram # To make the dendograms
from sklearn.cluster import KMeans # KMeans function
from sklearn.metrics import silhouette_score # To choose the best parameter K
import matplotlib.pyplot as plt # To plot dendograms and elbow plots
from yellowbrick.cluster import SilhouetteVisualizer # To plot Silouhette score<jupyter_output><empty_output><jupyter_text>**Data management**To do our job, first we need to load our data and prepare it.<jupyter_code>df = pd.read_csv('drive/MyDrive/Colab Notebooks/nci_var_filtered.csv', header = 'infer', sep = '\t')
df.head(5)<jupyter_output><empty_output><jupyter_text>Set the gene name as the index of our dataframe.<jupyter_code>df = df.set_index('gene')
df.head(5)<jupyter_output><empty_output><jupyter_text>And then, we can retrieve all the cell lines and store them into a list.<jupyter_code>cell_lines = list(df.columns.values)
cell_lines[1:8]<jupyter_output><empty_output><jupyter_text>To work with cluster models we need to transpose our data, because it uses the rows.<jupyter_code>df_t= df.transpose()
df_t.head(5)<jupyter_output><empty_output><jupyter_text>**Hierarchical Clustering**It is a method that classifies the classes into different hierarchical clusters. In our case of study, it will group genes in different groups based on their similarity.[*In all of the hierarchical studies I set *optimal_ordering = True* *because it will take the minimal distance between successive leaves and the classification will be more accurate*]  Single method with different metricsThe single method is the most famous and oldest of all, also known as the nearest neighbour method, where the distance between two groups is the distance defined between the two nearest members of these groups.<jupyter_code>cluster = hierarchy.linkage(df_t, method='single', metric='euclidean', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Single method, euclidean distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')
cluster = hierarchy.linkage(df_t, method='single', metric='cityblock', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Single method, manhattan distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')
cluster = hierarchy.linkage(df_t, method='single', metric='chebyshev', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Single method, chebyshev distance")
plt.ylabel("Distance")
plt.xlabel('Cell Lines')
cluster = hierarchy.linkage(df_t, method='single', metric='minkowski', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Single method, minkowski distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')<jupyter_output><empty_output><jupyter_text>Most of the genes are classify as the same group (blue color), while the other ones are highly distributed in different groups (green, orange, purple...) that are smaller ones (most of them are just compound by 2 members).Because of we are taking the minimal distance between two points of the two groups, it makes sense that most of the genes are grouped in the same group.  Complete method with different metricsThe complete method sorts in the opposite way to the previous method, choosing those members of two groups that are farthest apart. Because of this, it makes the clusters formed compact and well separated from each other.<jupyter_code>cluster = hierarchy.linkage(df_t, method='complete', metric='euclidean', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Complete method, euclidean distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')
cluster = hierarchy.linkage(df_t, method='complete', metric='cityblock', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Complete method, manhattan distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')
cluster = hierarchy.linkage(df_t, method='complete', metric='chebyshev', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Complete method, chebyshev distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')
cluster = hierarchy.linkage(df_t, method='complete', metric='minkowski', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Complete method, minkowski distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')<jupyter_output><empty_output><jupyter_text>In contrast with the single method, the complete method set more groups between the different genes, being this classification more heterogeneus.This makes sense because when you are grouping by farthest distance, the number of groups are higher because there are less similarity.  Average method with different metricsThe average method classifies clusters according to the distance between the centroids of each group (i.e. the most representative point of each group). This ranking method can introduce classifications that are difficult to interpret due to the fact that the merging of new clusters occurs at different levels.<jupyter_code>cluster = hierarchy.linkage(df_t, method='average', metric='euclidean', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Average method, euclidean distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')
cluster = hierarchy.linkage(df_t, method='average', metric='cityblock', optimal_ordering=True)
# Dendrogram plotting using linkage matrix
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Average method, manhattan distance")
plt.ylabel("Distance")
plt.xlabel('Cell lines')<jupyter_output><empty_output><jupyter_text>In this case, we see a distribution similar to the single method, where most of the genes are classified in the same group while the other ones are classified into smaller ones.As I commented in the PDF, this method creates clusters at different levels making it difficult to understand the real classification of the data.  Ward methodThe last method seen in class is Ward's method where he classifies the groups in such a way that the sum of squares within the group is minimised, i.e. groups that result in a smaller increase in the sum of squares are merged.<jupyter_code># Ward
cluster = hierarchy.linkage(df_t, method='ward', metric='euclidean', optimal_ordering=True)
plt.figure(figsize=(15,8))
dendrogram = hierarchy.dendrogram(cluster, labels = cell_lines)
plt.title("Ward method")
plt.ylabel("Distance")
plt.xlabel('Cell lines')<jupyter_output><empty_output><jupyter_text>This method is the most used (and best) to quantitative variables (the same that we are using). It classifies in a better way the data because it makes 2 large groups and then it creates subgroups between them. We see a better distribution of the clusters than with the other methods.  **Partitional Clustering**In this part I will analyze the optimal number of clusters to classify the different cell lines and try to understand the connexion between them.  Performing some K-means clustering by varying the number of clusters To start, I will create a loop to store the SSW and score values for each k and print the results. With this data, later, I will be able to do an elbow plot and select the best number of cluster by choosing the one with the lowest difference in SSW value between clusters and the one with the highest Silouhette score.<jupyter_code>K = range(2, 10, 1) # Set a range between 2 clusters to 9 
SSW = []
scores = []

for k in K:
  model = KMeans(n_clusters= k, random_state = 42) # Setting number of clusters
  model.fit(df_t)
  label_k = model.predict(df_t)
  score_k = silhouette_score(df_t, label_k)
  SSW.append(model.inertia_)
  scores.append(score_k)
  print('For', k, 'clusters, the SSW is', model.inertia_,
        '\nAnd the Silhouete score is', score_k, '\n')<jupyter_output>For 2 clusters, the SSW is 350044.4926600001 
And the Silhouete score is 0.10592698479474807 

For 3 clusters, the SSW is 319569.4965012501 
And the Silhouete score is 0.10545716423873563 

For 4 clusters, the SSW is 294544.4497448833 
And the Silhouete score is 0.12194642870622228 

For 5 clusters, the SSW is 277026.14662874327 
And the Silhouete score is 0.10873902837778296 

For 6 clusters, the SSW is 267113.58551294886 
And the Silhouete score is 0.0851575909045804 

For 7 clusters, the SSW is 255234.48240821442 
And the Silhouete score is 0.07985842512770369 

For 8 clusters, the SSW is 245006.97674878815 
And the Silhouete score is 0.08223364725460675 

For 9 clusters, the SSW is 236420.17700833335 
And the Silhouete score is 0.08345853295935533<jupyter_text>Let's build an elbow plot for SSW and scores to decide:<jupyter_code>plt.plot(K, SSW, 'bo-')
plt.xlabel('Number of clusters')
plt.xlim([2, 9])
plt.ylabel('SSW')
plt.title('The Elbow Method showing the optimal k')
plt.show()
plt.plot(K, scores, 'ro-')
plt.xlim([2, 9])
plt.xlabel('Number of clusters')
plt.ylabel('scores')
plt.title('The Elbow Method showing the optimal k')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's visualize the average Silouhette Score and the clusters:<jupyter_code>model = KMeans(4, random_state=42)
visualizer = SilhouetteVisualizer(model, colors='yellowbrick')
visualizer.fit(df_t)
visualizer.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08450442066077245, 'fraction_numerical': 0.0316426244765007, 'mean_word_length': 6.295315682281059, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '315651', 'license': 'MIT', 'n_tokens_mistral': 3492, 'n_tokens_neox': 3108, 'n_words': 1190, 'path': 'Machine_Learning/Unsupervised_Learning_FinalWork.ipynb', 'repo_name': 'biocompt/My_Work_In_The_Master', 'size': '10745'}