<jupyter_start><jupyter_text>run K-means based on silhoeutte score and inertiafirst check hopkins statisticthe null hypothesis (no meaningfull cluster) happens when the hopkins test is around 0.5 the hopkins test tends to 0 when meaningful cluster exists in the space existence of clusters when the hopkins score is bellow 0.25. the hopkins test is highly influenced by outliers, use with normalised data. https://www.kaggle.com/lachhebo/hopkins-test<jupyter_code>from pyclustertend import hopkins 
from sklearn.preprocessing import scale

hopkins(scale(df), df.shape[0])

### Data conversion

categorical to continuous
scaling with zscore

https://datascience.stackexchange.com/questions/16700/confused-about-how-to-apply-kmeans-on-my-a-dataset-with-features-extracted
    
    
https://datascience.stackexchange.com/questions/22/k-means-clustering-for-mixed-numeric-and-categorical-data

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

sil = []
for k in range(2,10):
    kmeans = KMeans(n_clusters  = k).fit(df)
    sil.append([k,silhouette_score(df, kmeans.labels_)])
    
plt.plot(pd.DataFrame(sil)[0], pd.DataFrame(sil)[1])
plt.vlines(x=3, ymax=0.35, ymin=0.25, colors="r", linestyles="--")
plt.hlines(y=0.30, xmax=10, xmin=0, colors="g", linestyles="--")
plt.show()

ssd = []
for k in range(2,10):
    k_mean = KMeans(n_clusters = k).fit(df_countries_pca)
    ssd.append([k, k_mean.inertia_])
    
plt.plot(pd.DataFrame(ssd)[0], pd.DataFrame(ssd)[1])

kmean5 = KMeans(n_clusters = 3, random_state=42)
kmean5.fit(df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12626262626262627, 'fraction_numerical': 0.026515151515151516, 'mean_word_length': 5.167315175097277, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '302536', 'license': 'Apache-2.0', 'n_tokens_mistral': 615, 'n_tokens_neox': 574, 'n_words': 127, 'path': 'kmeans.ipynb', 'repo_name': 'sanjosh/machine_learning', 'size': '1584'}