<jupyter_start><jupyter_text>IntroductionIn this notebook we will use a clustering algorithm to analyze our data (i.e. YouTube comments of a single video). This will help us extract topics of discussion. We use the embeddings generated in Assignment 4 as input. (This notebook will not run without first running the assignment 4 Notebook, as it relies on the data in the folder 'output/') Each of our comments has been assigned a vector that encodes information about its meaning. The closer two vectors are, the more similar the meaning.Each vector is of 512 Dimensions.Before we can cluster our data we need to reduce the embeddings' dimensionality to overcome the curse of dimensionality. We use the UMAP ALgorithm for this.After that we use the KMedoids Algorithm to partition the embedding space and generate our clusters this way.We need to define the number of clusters we want to have. To find the optimal number of clusters, we use a simple optimization scheme.Once the clusters are created, we visualize them. To do this we reduce the dimensionality of the embeddings again to two dimensions. Then we render a scatterplot of our data.Furthermore we want to analyze and interpret our clusters. To do this, we:print some statistics about each of the clustersprint cluster's medoid (the central sample)print the cluster(s) we want to analyze furtherCheck to see if jupyter lab uses the correct python interpreter with '!which python'. It should be something like '/opt/anaconda3/envs/[environment name]/bin/python' (on Mac). If not, try this: https://github.com/jupyter/notebook/issues/3146issuecomment-352718675<jupyter_code>!python --version<jupyter_output>Python 3.8.5<jupyter_text>Install dependencies:<jupyter_code>install_packages = True
if install_packages:    
    !conda install -c conda-forge umap-learn -y
    !conda install -c conda-forge scikit-learn-extra -y<jupyter_output>Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\Zaher11\anaconda3

  added / updated specs:
    - umap-learn


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.9.2                |   py38haa244fe_0         3.1 MB  conda-forge
    python_abi-3.8             |           1_cp38           4 KB  conda-forge
    umap-learn-0.4.6           |   py38h32f6830_0         111 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         3.2 MB

The following NEW packages will be INSTALLED:

  python_abi         conda-forge/win-64::python_abi-3.8-1_cp38
  tbb                conda-forge/win-64::tbb-2020.1-he980bc4_0
  umap-learn         conda-forge/win-64::umap-learn-0.4.6-py38h32f6830_0

The following p[...]<jupyter_text>Imports<jupyter_code>#imports

import pandas as pd
import numpy as np
import os
import time
import matplotlib.pyplot as plt
import umap
from sklearn_extra.cluster import KMedoids
import seaborn as sns
#from sklearn.cluster import AgglomerativeClustering, DBSCAN, KMeans, OPTICS
from sklearn.metrics import silhouette_samples, silhouette_score, pairwise_distances<jupyter_output><empty_output><jupyter_text>Functions to Save and load manually<jupyter_code># Save and load your data after clustering

def save_results():
    data.to_pickle(output_path+'data_clustered'+'.pkl')<jupyter_output><empty_output><jupyter_text>Set pandas print optionsThis will improve readability of printed pandas dataframe.<jupyter_code>pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', None)<jupyter_output><empty_output><jupyter_text>Set global ParametersSet your parameters here:output_path: Files generated in this notebook will be saved here.model_type: Define which model was used to produce the embeddings. (Check the name of the .npy-file containing the embeddings)<jupyter_code>output_path = "./output/"
model_type = 'Transformer' #@param ['DAN','Transformer','Transformer_Multilingual']<jupyter_output><empty_output><jupyter_text>Load DataLoad the preprocessed data as a pandas dataframe. And load the embeddings as a numpy ndarray (a matrix in our case).<jupyter_code>data = pd.read_pickle(output_path+'data_preprocessed'+'.pkl')
labels_default = np.zeros(len(data.index))-1
data['label_manual'] = labels_default
embeddings = np.load(output_path+'/embeddings'+model_type+'.npy', mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII')<jupyter_output><empty_output><jupyter_text>Dimensionality reduction with UMAPWe reduce the number of dimensions of our embeddings to make possibly present clusters more pronounced. The number of dimensions (num_dimensions) depends on the number of samples<jupyter_code># Set the number of dimensions to reduce to
num_dimensions =100

reducer_clustering = umap.UMAP(n_neighbors=50, 
                     n_components=num_dimensions, 
                     metric='cosine', 
                     #n_epochs=200, 
                     learning_rate=.5,
                     init='spectral', 
                     min_dist=0,
                     #spread=5.0, 
                     #set_op_mix_ratio=1.0, 
                     #local_connectivity=1.0, 
                     #negative_sample_rate=5, 
                     #transform_queue_size=4.0, 
                     force_approximation_algorithm=True, 
                     unique=True)
embeddings_umap = reducer_clustering.fit_transform(embeddings)<jupyter_output><empty_output><jupyter_text>Optimize the Number of Clusters<jupyter_code>#optimize number of clusters
optimize_number_of_clusters = True#@param {type:'boolean'}

min_clusters=2 
max_clusters=1000 
step=100 

if optimize_number_of_clusters:
    rows_list = []
    inertias = []
    n_clusters = []
    silouette_scores = []
    init_param = 'k-medoids++' #@param ['random', 'heuristic', 'k-medoids++']
    random_state_param=1234 #@param {type:'number'}
    for i in range(min_clusters,max_clusters, step):
        temp_clustering = KMedoids(n_clusters=i, metric='euclidean', init=init_param, max_iter=200, random_state=random_state_param).fit(embeddings_umap)
        silhouette_avg = silhouette_score(embeddings_umap, temp_clustering.labels_)
        print("n_clusters:",i, "silhouette_avg:",silhouette_avg)
        silhouette_dict = {'number of clusters': i, 'silhouette average': silhouette_avg}
        rows_list.append(silhouette_dict)
    results = pd.DataFrame(rows_list)
    sns.lineplot(x = 'number of clusters', y = 'silhouette average',data = results)<jupyter_output>n_clusters: 2 silhouette_avg: 0.47220892
n_clusters: 102 silhouette_avg: 0.34595862
n_clusters: 202 silhouette_avg: 0.3305257
n_clusters: 302 silhouette_avg: 0.31658703
n_clusters: 402 silhouette_avg: 0.29387975
n_clusters: 502 silhouette_avg: 0.26846132
n_clusters: 602 silhouette_avg: 0.2424068
n_clusters: 702 silhouette_avg: 0.19251211
n_clusters: 802 silhouette_avg: 0.14201015
n_clusters: 902 silhouette_avg: 0.09095344<jupyter_text>Clustering with KMedoids<jupyter_code>number_of_clusters =  100
init_param = 'k-medoids++' #@param ['random', 'heuristic', 'k-medoids++']
clustering_model = KMedoids(n_clusters=number_of_clusters,
                            metric='cosine', 
                            init=init_param, 
                            max_iter=150, 
                            random_state=None).fit(embeddings_umap)
clustering_model
labels = clustering_model.labels_
data["label_kmedoids"] = labels
print("cluster","members", data["label_kmedoids"].value_counts().sort_values())

clustering_model.inertia_

medoids_indices = clustering_model.medoid_indices_
#calculate distances
distances = np.diag(pairwise_distances(X = clustering_model.cluster_centers_[labels], Y = embeddings_umap[:], metric='cosine'))
data["distance_kmedoids"] = distances<jupyter_output><empty_output><jupyter_text>Dimensionality Reduction for Visualization<jupyter_code>num_dimensions =2

reducer_visualization = umap.UMAP(n_neighbors=50, 
                                  n_components=num_dimensions, 
                                  metric='cosine', 
                                  output_metric='euclidean', 
                                  #n_epochs=200, 
                                  learning_rate=.5,
                                  init='spectral', 
                                  min_dist=.1,
                                  spread=5.0, 
                                  set_op_mix_ratio=1.0, 
                                  local_connectivity=1.0, 
                                  negative_sample_rate=5, 
                                  transform_queue_size=4.0, 
                                  force_approximation_algorithm=True, 
                                  unique=True)
embeddings_umap_2d = reducer_clustering.fit_transform(embeddings)<jupyter_output><empty_output><jupyter_text>Visualize clustering results<jupyter_code>#@markdown Set the color palette used for visualizing different clusters
palette_param = "Paired" #@param ['Accent','cubehelix', "tab10", 'Paired', "Spectral"]
#@markdown Set opacity of data points (1 = opaque, 0 = invisible)
alpha_param = 0.16 #@param {type:"slider", min:0, max:1, step:0.01}


sns.relplot(x = embeddings_umap_2d[:, 0], y = embeddings_umap_2d[:, 1],  hue = data['label_kmedoids'], palette = palette_param,alpha = alpha_param,height = 10)<jupyter_output><empty_output><jupyter_text>Highlight one cluster<jupyter_code>## Choose a cluster to higlight:
cluster_num = 60

data['highlight'] = np.zeros(len(data.index))
data.loc[data['label_kmedoids'] == cluster_num, 'highlight'] = 1

sns.relplot(x = embeddings_umap_2d[:, 0], y = embeddings_umap_2d[:, 1],  hue = data['highlight'], palette = "Accent",alpha = 0.8,height = 10)<jupyter_output><empty_output><jupyter_text>Print Medoids and cluster statistics<jupyter_code># print the medoids 
data.iloc[medoids_indices]
# print statistics for each cluster
data['label_kmedoids'].value_counts().sort_values()
for k,g in data.groupby(by = 'label_kmedoids'):
      print(g.iloc[0]['label_kmedoids'],"number of samples: ",len(g.index),"mean distance from center: ", 100*np.mean(g['distance_kmedoids']), "Proportion of replies:", 100*np.sum(g['isReply'])/len(g.index))<jupyter_output>0 number of samples:  7 mean distance from center:  0.0015624931620550342 Proportion of replies: 57.142857142857146
1 number of samples:  8 mean distance from center:  0.00025406479835510254 Proportion of replies: 0.0
2 number of samples:  15 mean distance from center:  0.0015823046851437539 Proportion of replies: 26.666666666666668
3 number of samples:  6 mean distance from center:  0.0006198883056640625 Proportion of replies: 0.0
4 number of samples:  10 mean distance from center:  0.0029534101486206055 Proportion of replies: 10.0
5 number of samples:  25 mean distance from center:  0.0007543563697254285 Proportion of replies: 96.0
6 number of samples:  31 mean distance from center:  0.00029110140076227253 Proportion of replies: 83.87096774193549
7 number of samples:  12 mean distance from center:  0.0008543332114641089 Proportion of replies: 8.333333333333334
8 number of samples:  29 mean distance from center:  0.0004213431566313375 Proportion of replies: 96.55172413793103
9 number [...]<jupyter_text>Print ClusterPrint the comments within a cluster. Comments are sorted by their distance from the cluster medoid<jupyter_code># Choose a cluster to print
cluster_number =   60
# Choose the number of samples to print
number_of_samples_to_print = 10000

data['label_kmedoids'] = data['label_kmedoids'].astype('category')
cluster = data[data['label_kmedoids']==cluster_number]
if cluster["text"].count()<=number_of_samples_to_print:
    number_of_samples_to_print = cluster["text"].count()

cluster = cluster.sort_values(by='distance_kmedoids')

print("Number of samples in the cluster:", cluster["text"].count())
print("Average Distance from cluster center:", np.mean(cluster['distance_kmedoids']))
cluster<jupyter_output>Number of samples in the cluster: 11
Average Distance from cluster center: 7.125464435375761e-06<jupyter_text>Assign Cluster labels manuallycluster_number: which cluster would you like to assign labels to? min_distance: the minimum distance from the cluster medoid be for a data point to still get the specified label max_distance: the maximum distance from the cluster medoid be for a data point to still get the specified label label_manual: your label<jupyter_code>#which cluster would you like to assign labels to?
cluster_number = 60
#your label
label_manual = ' climate_change_prevention'
#the minimum distance from the cluster medoid be for a data point to still get the specified label
min_distance = 0
#the maximum distance from the cluster medoid be for a data point to still get the specified label
max_distance = 1000

# 2. Filter data by cluster label and specified label to filtered data
data.loc[(data['label_kmedoids']==cluster_number) & (data['distance_kmedoids'] <= max_distance) & (data['distance_kmedoids'] >= min_distance),  'label_manual'] = label_manual
data[data['label_kmedoids']==cluster_number].sort_values(by='distance_kmedoids')<jupyter_output><empty_output><jupyter_text>Print Cluster 2<jupyter_code># Choose a cluster to print
cluster_number =  64
# Choose the number of samples to print
number_of_samples_to_print = 10000

data['label_kmedoids'] = data['label_kmedoids'].astype('category')
cluster = data[data['label_kmedoids']==cluster_number]
if cluster["text"].count()<=number_of_samples_to_print:
    number_of_samples_to_print = cluster["text"].count()

cluster = cluster.sort_values(by='distance_kmedoids')

print("Number of samples in the cluster:", cluster["text"].count())
print("Average Distance from cluster center:", np.mean(cluster['distance_kmedoids']))
cluster
#which cluster would you like to assign labels to?
cluster_number = 64
#your label
label_manual = 'global_warming'
#the minimum distance from the cluster medoid be for a data point to still get the specified label
min_distance = 0
#the maximum distance from the cluster medoid be for a data point to still get the specified label
max_distance = 1000

# 2. Filter data by cluster label and specified label to filtered data
data.loc[(data['label_kmedoids']==cluster_number) & (data['distance_kmedoids'] <= max_distance) & (data['distance_kmedoids'] >= min_distance),  'label_manual'] = label_manual
data[data['label_kmedoids']==cluster_number].sort_values(by='distance_kmedoids')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09638390429581294, 'fraction_numerical': 0.04438553561718325, 'mean_word_length': 3.984078590785908, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 66, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '474647', 'license': 'CC0-1.0', 'n_tokens_mistral': 4937, 'n_tokens_neox': 4350, 'n_words': 1221, 'path': 'Pipeline/Assignment_5/rahaf_assignment_5/rahaf_gharz_assignment_5.ipynb', 'repo_name': 'FUB-HCC/criticalsocialmedia', 'size': '14712'}