<jupyter_start><jupyter_text>Finding K in K-MeansSo when applying K-Means Clustring, one comes at a point where he/she has to decide how many cluster they want, now one can not go and tell 2,3, or 4 clusters there should be a some evidence that clustering the data into *'k'* clusters will yeild good results. So I met this problem, after searching for what can be done about this problem I stumble upon a something called **'Silhouette Analysis'** Silhouette AnalysisSilhouette Analysis takes into considration of how well a particular data point lies within a given cluster. So lets take an example to understand how this works, Say we are using K-Means Clustring for *'k'* number of clusters. Now for a data point *'i'*, we define average distance from all points in the same cluster as *avg(i) = a(i)*, so we can interpret *a(i)* as measure of how well that point belongs in the cluster or to put it simply how well that point belongs in that cluster, so smaller the value, better the assignment of that data point to that cluster. Similarly, lets say *b(i)* is the average dissimlarity, that means *b(i)* the lowest average distance of data point *i*  to all points in any other cluster, of which *i* is not a member of. The cluster with this lowest average dissimilarity is said to be the "neighbouring cluster" of *i* because it is the next best fit cluster for point *i*<jupyter_code>from IPython.display import Math
Math(r's(i) = \frac{b(i)-a(i)}{max(b(i),a(i))}')
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import numpy as np
from sklearn.metrics import silhouette_score, adjusted_rand_score
import pickle 

with open('/home/jay/class-level-docs.pkl', 'rb') as docs:
    class_docs =pickle.load(docs)
    
with open('/home/jay/class-level-label.pkl', 'rb') as labs:
    class_label = pickle.load(labs)
class_label
# running the k-means for 32 clusters
# creating tf-idf matrix

tfidf_matrix_constructor = TfidfVectorizer(max_df=0.8, max_features=200000, min_df=0.2, ngram_range=(1,3))

tfidf_matrix = tfidf_matrix_constructor.fit_transform(class_docs)

# printing the shape of the resultant matrix to ensure I am right
print(tfidf_matrix.shape)
with open('/home/jay/k-find/pickles/tf-idf-class-level.pkl', 'wb') as tf:
    pickle.dump(tfidf_matrix, tf)
# now applying k-means for range 1-32 and calculating silhouette_score for each cluster

for k in range(2 ,33):
    
    # define k-means constructor
    kmeans = KMeans(n_clusters=k,random_state=10)
    
    cluster_labels = kmeans.fit_predict(tfidf_matrix.toarray())
    
    # Calculating silhouette_score for k
    
    score = silhouette_score(tfidf_matrix.toarray(), cluster_labels, random_state=10)
    print("The silhouette score for {} clusters is {}".format(k,score))
# The elbow analysis 

# define list to store cluster errors

cluster_errors = []

for k in range(1 ,33):
    
    # define k-means constructor
    kmeans = KMeans(n_clusters=k,random_state=10,n_jobs=-1)
    
    cluster_ = kmeans.fit(tfidf_matrix.toarray())
    
    cluster_errors.append(cluster_.inertia_)
import matplotlib.pyplot as plt
%matplotlib inline
len(cluster_errors)
plt.figure(figsize=(12,6))
plt.plot(range(1,33), cluster_errors, marker = "o" )
a=plt.xticks(np.arange(33))
plt.grid()<jupyter_output><empty_output><jupyter_text>ClusteringThe first one is based on the silhouette score that gave the number of clusters to be 10, but the clustering is not as good as expected. The distribution of classes doesnt make a lot of sense to to me.<jupyter_code>kmeas = KMeans(n_clusters=10, random_state=10,n_jobs=-1)

cluster_label =kmeas.fit(tfidf_matrix.toarray())
for i, j in zip(class_label, cluster_label.labels_):
    print(i,j)
Counter(cluster_label.labels_)<jupyter_output><empty_output><jupyter_text>Clustering Based on Elbow AnalysisThe elbow analysis suggested that number of clusters should be between 3 and 5, however it was not clear that what it should be. So performed on 3,4,5. The results are down below. I think 3 cluster for class level clustering makes more sense as there is some sense to it. The cluster of classes itself makes more sense than in 4 and 5<jupyter_code># using elbow score

elbow = KMeans(n_clusters=3, random_state=10,n_jobs=-1)
elbow_model = elbow.fit(tfidf_matrix.toarray())
for i, j in zip(class_label, elbow_model.labels_):
    print(i,j)
elbow_4 = KMeans(n_clusters=4, random_state=10,n_jobs=-1)
elbow_model_4 = elbow_4.fit(tfidf_matrix.toarray())
for i, j in zip(class_label, elbow_model_4.labels_):
    print(i,j)
elbow_5 = KMeans(n_clusters=5, random_state=10,n_jobs=-1)
elbow_model_5 = elbow_5.fit(tfidf_matrix.toarray())
for i, j in zip(class_label, elbow_model_5.labels_):
    print(i,j)<jupyter_output>employment_and_social_policy 0
audiovisual_and_media 0
institutional_affairs 0
enlargement 0
customs 0
enterprise 0
internal_market 0
humanitarian_aid 4
justice_freedom_security 2
energy 2
consumers 2
maritime_affairs_and_fisheries 2
foreign_and_security_policy 2
information_society 2
education_training_youth 2
human_rights 2
external_relations 1
development 1
agriculture 1
food_safety 1
fight_against_fraud 1
economic_and_monetary_affairs 1
competition 1
transport 3
culture 3
environment 3
public_health 3
taxation 3
regional_policy 3
external_trade 3
research_innovation 3
budget 3
{'fraction_non_alphanumeric': 0.07981481481481481, 'fraction_numerical': 0.02, 'mean_word_length': 5.316959064327485, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '360020', 'license': 'MIT', 'n_tokens_mistral': 1868, 'n_tokens_neox': 1743, 'n_words': 612, 'path': 'notebooks/Finding K in  K-Means-Clusturing-Class-Level.ipynb', 'repo_name': 'jdvala/website', 'size': '5400'}