<jupyter_start><jupyter_text>IntroductionIn this notebook we will use a clustering algorithm to analyze our data (i.e. YouTube comments of a single video).This will help us extract topics of discussion.We use the embeddings generated in Assignment 4 as input. (This notebook will not run without first running the assignment 4 Notebook, as it relies on the data in the folder 'output/')Each of our comments has been assigned a vector that encodes information about its meaning.The closer two vectors are, the more similar the meaning.Each vector is of 512 Dimensions.Before we can cluster our data we need to reduce the embeddings' dimensionality to overcome the curse of dimensionality.We use the UMAP ALgorithm for this.After that we use the KMedoids Algorithm to partition the embedding space and generate our clusters this way.We need to define the number of clusters we want to have. To find the optimal number of clusters, we use a simple optimization scheme.Once the clusters are created, we visualize them.To do this we reduce the dimensionality of the embeddings again to two dimensions.Then we render a scatterplot of our data.Furthermore we want to analyze and interpret our clusters.To do this, we:- print some statistics about each of the clusters- print cluster's medoid (the central sample)- print the cluster(s) we want to analyze further Check to see if jupyter lab uses the correct python interpreter with '!which python'.It should be something like '/opt/anaconda3/envs/[environment name]/bin/python' (on Mac).If not, try this: https://github.com/jupyter/notebook/issues/3146issuecomment-352718675<jupyter_code>!which python<jupyter_output>/opt/anaconda3/envs/csma3/bin/python<jupyter_text>Install dependencies:<jupyter_code>install_packages = False
if install_packages:    
    !conda install -c conda-forge umap-learn -y
    !conda install -c conda-forge scikit-learn-extra -y<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code>#imports

import pandas as pd
import numpy as np
import os
import time
import matplotlib.pyplot as plt
import umap
from sklearn_extra.cluster import KMedoids
import seaborn as sns
#from sklearn.cluster import AgglomerativeClustering, DBSCAN, KMeans, OPTICS
from sklearn.metrics import silhouette_samples, silhouette_score, pairwise_distances<jupyter_output><empty_output><jupyter_text>Functions to Save and load manually<jupyter_code># Save and load your data after clustering

def save_results():
    data.to_pickle(output_path+'data_clustered'+'.pkl')
    
def load_results():
    data = pd.read_pickle(output_path+'data_clustered'+'.pkl')<jupyter_output><empty_output><jupyter_text>Set pandas print options This will improve readability of printed pandas dataframe.<jupyter_code>pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', None)<jupyter_output><empty_output><jupyter_text>Set global ParametersSet your parameters here:output_path: Files generated in this notebook will be saved here.model_type: Define which model was used to produce the embeddings. (Check the name of the .npy-file containing the embeddings)<jupyter_code>output_path = "./output/"
model_type = 'Transformer' #@param ['DAN','Transformer','Transformer_Multilingual']<jupyter_output><empty_output><jupyter_text>Load DataLoad the preprocessed data as a pandas dataframe.And load the embeddings as a numpy ndarray (a matrix in our case).<jupyter_code>data = pd.read_pickle(output_path+'data_preprocessed'+'.pkl')
labels_default = np.zeros(len(data.index))-1
data['label_manual'] = labels_default
embeddings = np.load(output_path+'/embeddings'+model_type+'.npy', mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII')<jupyter_output><empty_output><jupyter_text>Dimensionality reduction with UMAPWe reduce the number of dimensions of our embeddings to make possibly present clusters more pronounced. The number of dimensions (num_dimensions) depends on the number of samples<jupyter_code># Set the number of dimensions to reduce to
num_dimensions =100

reducer_clustering = umap.UMAP(n_neighbors=50, 
                     n_components=num_dimensions, 
                     metric='cosine', 
                     #n_epochs=200, 
                     learning_rate=.5,
                     init='spectral', 
                     min_dist=0,
                     #spread=5.0, 
                     #set_op_mix_ratio=1.0, 
                     #local_connectivity=1.0, 
                     #negative_sample_rate=5, 
                     #transform_queue_size=4.0, 
                     force_approximation_algorithm=True, 
                     unique=True)
embeddings_umap = reducer_clustering.fit_transform(embeddings)<jupyter_output><empty_output><jupyter_text>Optimize the Number of Clusters<jupyter_code>#optimize number of clusters
optimize_number_of_clusters = True#@param {type:'boolean'}

min_clusters= 5 
max_clusters=55
step=5

if optimize_number_of_clusters:
    rows_list = []
    inertias = []
    n_clusters = []
    silouette_scores = []
    init_param = 'k-medoids++' #@param ['random', 'heuristic', 'k-medoids++']
    random_state_param=134 #@param {type:'number'}
    for i in range(min_clusters,max_clusters, step):
        temp_clustering = KMedoids(n_clusters=i, metric='euclidean', init=init_param, max_iter=300, random_state=random_state_param).fit(embeddings_umap)
        silhouette_avg = silhouette_score(embeddings_umap, temp_clustering.labels_)
        print("n_clusters:",i, "silhouette_avg:",silhouette_avg)
        silhouette_dict = {'number of clusters': i, 'silhouette average': silhouette_avg}
        rows_list.append(silhouette_dict)
    results = pd.DataFrame(rows_list)
    sns.lineplot(x = 'number of clusters', y = 'silhouette average',data = results)<jupyter_output>n_clusters: 5 silhouette_avg: 0.282379
n_clusters: 10 silhouette_avg: 0.3443897
n_clusters: 15 silhouette_avg: 0.38070863
n_clusters: 20 silhouette_avg: 0.36556098
n_clusters: 25 silhouette_avg: 0.3519204
n_clusters: 30 silhouette_avg: 0.3452696
n_clusters: 35 silhouette_avg: 0.34582657
n_clusters: 40 silhouette_avg: 0.34683618
n_clusters: 45 silhouette_avg: 0.34131685
n_clusters: 50 silhouette_avg: 0.34406087<jupyter_text>Clustering with KMedoids<jupyter_code>number_of_clusters =  20
init_param = 'k-medoids++' #@param ['random', 'heuristic', 'k-medoids++']
clustering_model = KMedoids(n_clusters=number_of_clusters,
                            metric='cosine', 
                            init=init_param, 
                            max_iter=150, 
                            random_state=None).fit(embeddings_umap)
clustering_model
labels = clustering_model.labels_
data["label_kmedoids"] = labels
print("cluster","members", data["label_kmedoids"].value_counts().sort_values())

clustering_model.inertia_

medoids_indices = clustering_model.medoid_indices_
#calculate distances
distances = np.diag(pairwise_distances(X = clustering_model.cluster_centers_[labels], Y = embeddings_umap[:], metric='cosine'))
data["distance_kmedoids"] = distances<jupyter_output><empty_output><jupyter_text>Dimensionality Reduction for Visualization<jupyter_code>num_dimensions =2

reducer_visualization = umap.UMAP(n_neighbors=50, 
                                  n_components=num_dimensions, 
                                  metric='cosine', 
                                  output_metric='euclidean', 
                                  #n_epochs=200, 
                                  learning_rate=.5,
                                  init='spectral', 
                                  min_dist=.1,
                                  spread=5.0, 
                                  set_op_mix_ratio=1.0, 
                                  local_connectivity=1.0, 
                                  negative_sample_rate=5, 
                                  transform_queue_size=4.0, 
                                  force_approximation_algorithm=True, 
                                  unique=True)
embeddings_umap_2d = reducer_visualization.fit_transform(embeddings)<jupyter_output>/usr/local/lib/python3.8/site-packages/umap/umap_.py:1158: RuntimeWarning: divide by zero encountered in power
  return 1.0 / (1.0 + a * x ** (2 * b))<jupyter_text>Visualize clustering results<jupyter_code>#@markdown Set the color palette used for visualizing different clusters
palette_param = "Spectral" #@param ['Accent','cubehelix', "tab10", 'Paired', "Spectral"]
#@markdown Set opacity of data points (1 = opaque, 0 = invisible)
alpha_param = 1 #@param {type:"slider", min:0, max:1, step:0.01}


sns.relplot(x = embeddings_umap_2d[:, 0], y = embeddings_umap_2d[:, 1],  hue = data['label_kmedoids'], palette = palette_param,alpha = alpha_param,height = 10)<jupyter_output><empty_output><jupyter_text>Highlight one cluster<jupyter_code>## Choose a cluster to higlight:
cluster_num = 15
cluster_num_2 = 14

data['highlight'] = np.zeros(len(data.index))
data.loc[data['label_kmedoids'] == cluster_num, 'highlight'] = 1
data.loc[data['label_kmedoids'] == cluster_num_2, 'highlight'] = 2

sns.relplot(x = embeddings_umap_2d[:, 0], y = embeddings_umap_2d[:, 1],  hue = data['highlight'], palette = "Accent",alpha = 0.8,height = 10)<jupyter_output><empty_output><jupyter_text>Print Medoids and cluster statistics<jupyter_code># print the medoids 
data.iloc[medoids_indices]      
# print statistics for each cluster
data['label_kmedoids'].value_counts().sort_values()
for k,g in data.groupby(by = 'label_kmedoids'):
      print(g.iloc[0]['label_kmedoids'],"number of samples: ",len(g.index),"mean distance from center: ", 100*np.mean(g['distance_kmedoids']), "Proportion of replies:", 100*np.sum(g['isReply'])/len(g.index))<jupyter_output>0 number of samples:  256 mean distance from center:  0.005472637712955475 Proportion of replies: 27.734375
1 number of samples:  149 mean distance from center:  0.004751170126837678 Proportion of replies: 81.20805369127517
2 number of samples:  135 mean distance from center:  0.005244281419436447 Proportion of replies: 35.55555555555556
3 number of samples:  164 mean distance from center:  0.005976565080345608 Proportion of replies: 50.609756097560975
4 number of samples:  163 mean distance from center:  0.0031442363251699135 Proportion of replies: 78.52760736196319
5 number of samples:  87 mean distance from center:  0.005120450077811256 Proportion of replies: 83.9080459770115
6 number of samples:  102 mean distance from center:  0.0019379691366339102 Proportion of replies: 63.72549019607843
7 number of samples:  161 mean distance from center:  0.005908301318413578 Proportion of replies: 31.055900621118013
8 number of samples:  153 mean distance from center:  0.00507103068230208 Prop[...]<jupyter_text>Print ClusterPrint the comments within a cluster. Comments are sorted by their distance from the cluster medoid<jupyter_code># Choose a cluster to print
cluster_number =   14
# Choose the number of samples to print
number_of_samples_to_print = 1000

data['label_kmedoids'] = data['label_kmedoids'].astype('category')
cluster = data[data['label_kmedoids']==cluster_number]
if cluster["text"].count()<=number_of_samples_to_print:
  number_of_samples_to_print = cluster["text"].count()

cluster = cluster.sort_values(by='distance_kmedoids')

print("Number of samples in the cluster:", cluster["text"].count())
print("Average Distance from cluster center:", np.mean(cluster['distance_kmedoids']))
cluster['text']<jupyter_output>Number of samples in the cluster: 148
Average Distance from cluster center: 7.213894e-05<jupyter_text>Assign Cluster labels manuallycluster_number: which cluster would you like to assign labels to?min_distance: the minimum distance from the cluster medoid be for a data point to still get the specified labelmax_distance: the maximum distance from the cluster medoid be for a data point to still get the specified labellabel_manual: your label<jupyter_code>#which cluster would you like to assign labels to?
cluster_number = 15
#your label
label_manual = 'rationality'
#the minimum distance from the cluster medoid be for a data point to still get the specified label
min_distance = 0
#the maximum distance from the cluster medoid be for a data point to still get the specified label
max_distance = 1000

# 2. Filter data by cluster label and specified label to filtered data
data.loc[(data['label_kmedoids']==cluster_number) & (data['distance_kmedoids'] <= max_distance) & (data['distance_kmedoids'] >= min_distance),  'label_manual'] = label_manual
data[data['label_kmedoids']==cluster_number].sort_values(by='distance_kmedoids')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09007802033257152, 'fraction_numerical': 0.04586649854204429, 'mean_word_length': 4.336417157275021, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '38806', 'license': 'CC0-1.0', 'n_tokens_mistral': 4320, 'n_tokens_neox': 3797, 'n_words': 1053, 'path': 'Pipeline/Assignment_5/isabel_schmuck_assignment_5/Assignment-5_Clustering.ipynb', 'repo_name': 'FUB-HCC/criticalsocialmedia', 'size': '12689'}