<jupyter_start><jupyter_text>Praca domowa 6<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import preprocessing
import seaborn as sns
from sklearn.cluster import KMeans, DBSCAN
from sklearn.metrics import silhouette_samples, silhouette_score, davies_bouldin_score
from sklearn.mixture import GaussianMixture
from mpl_toolkits.mplot3d import Axes3D<jupyter_output><empty_output><jupyter_text>Przygotowanie danych<jupyter_code>data = pd.read_csv("..\\..\\clustering_R3.csv")
data.describe()<jupyter_output><empty_output><jupyter_text>Jak widać w podsumowaniu danych, kolumny X1, X2 i X3 różnią się rzędem wielkości, więc standaryzacja danych powinna znacznie ułatwić rozróżnianie klastrów.<jupyter_code>data = pd.DataFrame(preprocessing.scale(data))
data.columns = ['X1', 'X2', 'X3']
data.describe()<jupyter_output><empty_output><jupyter_text>Wizualizacja danych<jupyter_code>fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(data.X1, data.X2, data.X3, s=50)

ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('X3')

plt.tight_layout()
plt.show()
data.plot()
sns.pairplot(data, height=3)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Po przedstawieniu danych na wykresach widzimy, że dane ze zbioru clustering_R3 układają się w pewne naturalne, wyraźnie odzielone klastry. Do wykrycia tych klastrów posłużę się 3 algorytmami: KMeans, DBSCAN i GaussianMixture  KMeans<jupyter_code>labels_kmeans = []
sil_scores_kmeans = []
dbi_scores_kmeans = []
for k in range(2, 21):
    model_kmeans = KMeans(n_clusters = k)
    prediction_kmeans = model_kmeans.fit_predict(data)
    labels_kmeans.append(prediction_kmeans)
    
    sil_scores_kmeans.append(silhouette_score(data, prediction_kmeans))
    dbi_scores_kmeans.append(davies_bouldin_score(data, prediction_kmeans))
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.plot(range(2, 21), sil_scores_kmeans, '-o')
ax1.set_xlabel("Number of clusters")
ax1.set_ylabel("Silhouette score")

ax2.plot(range(2, 21), dbi_scores_kmeans, '-o')
ax2.set_xlabel("Number of clusters")
ax2.set_ylabel("DBI score")
ax2.yaxis.set_label_position("right")

plt.show()
[sil_scores_kmeans[0], dbi_scores_kmeans[0]]<jupyter_output><empty_output><jupyter_text>Z wykresów 2 miar wynika, że kmeans naturalnie rozdziela zbiór na 2 klastry. Spróbujmy je zwizualizować<jupyter_code>model_kmeans = KMeans(n_clusters = 2)
labels_kmeans = model_kmeans.fit_predict(data)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(data.X1, data.X2, data.X3, s=50, c = labels_kmeans)

ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('X3')

plt.tight_layout()
plt.show()
data['k'] = labels_kmeans
sns.pairplot(data, height=3, hue = 'k')
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>DBSCAN<jupyter_code>labels_dbscan = []
sil_scores_dbscan = []
dbi_scores_dbscan = []
for eps in np.arange(0.05, 1.05, 0.05):
    dbs = DBSCAN(eps)
    prediction_dbscan = dbs.fit_predict(data)
    labels_dbscan.append(prediction_dbscan)
    
    sil_scores_dbscan.append(silhouette_score(data, prediction_dbscan))
    dbi_scores_dbscan.append(davies_bouldin_score(data, prediction_dbscan))
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.plot(np.arange(0.05, 1.05, 0.05), sil_scores_dbscan, '-o')
ax1.set_xlabel("Epsilon")
ax1.set_ylabel("Silhouette score")

ax2.plot(np.arange(0.05, 1.05, 0.05), dbi_scores_dbscan, '-o')
ax2.set_xlabel("Epsilon")
ax2.set_ylabel("DBI score")
ax2.yaxis.set_label_position("right")

plt.show()
[sil_scores_dbscan[10], dbi_scores_dbscan[10]]<jupyter_output><empty_output><jupyter_text>Ze wstępnej analizy parametru epsilon w algorytmie DBSCAN domyślamy się, że najlepszym parametrem będzie eps = 0.55. Ponieważ parametr epsilon jest bardzo czuły i znaczący dla wyniku algorytmu, sprawdzę jak zachowuje się w okolicach wyniku 0.5.<jupyter_code>labels_dbscan_extra = []
sil_scores_dbscan_extra = []
dbi_scores_dbscan_extra = []
for eps in np.arange(0.45, 0.55, 0.01):
    dbs_extra = DBSCAN(eps)
    prediction_dbscan_extra = dbs_extra.fit_predict(data)
    labels_dbscan_extra.append(prediction_dbscan_extra)
    
    sil_scores_dbscan_extra.append(silhouette_score(data, prediction_dbscan_extra))
    dbi_scores_dbscan_extra.append(davies_bouldin_score(data, prediction_dbscan_extra))
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.plot(np.arange(0.45, 0.55, 0.01), sil_scores_dbscan_extra, '-o')
ax1.set_xlabel("Epsilon")
ax1.set_ylabel("Silhouette score")

ax2.plot(np.arange(0.45, 0.55, 0.01), dbi_scores_dbscan_extra, '-o')
ax2.set_xlabel("Epsilon")
ax2.set_ylabel("DBI score")
ax2.yaxis.set_label_position("right")

plt.show()<jupyter_output><empty_output><jupyter_text>Moje przypuszczenia się potwierdziły. Epsilon równy 0.55 daje najlepsze wyniki miar Silhouette i DBI.<jupyter_code>dbs = DBSCAN(eps=0.55)
labels_dbscan = dbs.fit_predict(data)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(data.X1, data.X2, data.X3, s=50, c = labels_dbscan)

ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('X3')

plt.tight_layout()
plt.show()
data['k'] = labels_dbscan
sns.pairplot(data, height=3, hue = 'k')
plt.tight_layout()<jupyter_output>C:\Users\jwisn\anaconda3\lib\site-packages\seaborn\distributions.py:288: UserWarning: Data must have variance to compute a kernel density estimate.
  warnings.warn(msg, UserWarning)
C:\Users\jwisn\anaconda3\lib\site-packages\seaborn\distributions.py:288: UserWarning: Data must have variance to compute a kernel density estimate.
  warnings.warn(msg, UserWarning)
C:\Users\jwisn\anaconda3\lib\site-packages\seaborn\distributions.py:288: UserWarning: Data must have variance to compute a kernel density estimate.
  warnings.warn(msg, UserWarning)<jupyter_text>Tak prezentują się klastry utworzone przy pomocy tego algorytmu. W przeciwieństwie do algorytmu KMeans, DBSCAN dzieli zbiór na 3 klastry.  GaussianMixture<jupyter_code>labels_gmm = []
sil_scores_gmm = []
dbi_scores_gmm = []
for k in range(2, 21):
    gmm = GaussianMixture(n_components=k, covariance_type='full')
    prediction_gmm = gmm.fit_predict(data)
    labels_gmm.append(prediction_gmm)
    
    sil_scores_gmm.append(silhouette_score(data, prediction_gmm))
    dbi_scores_gmm.append(davies_bouldin_score(data, prediction_gmm))
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1.plot(range(2, 21), sil_scores_gmm, '-o')
ax1.set_xlabel("Number of clusters")
ax1.set_ylabel("Silhouette score")

ax2.plot(range(2, 21), dbi_scores_gmm, '-o')
ax2.set_xlabel("Number of clusters")
ax2.set_ylabel("DBI score")
ax2.yaxis.set_label_position("right")

plt.show()
[sil_scores_gmm[0], dbi_scores_gmm[0]]<jupyter_output><empty_output><jupyter_text>Wyniki tego algorytmu, podobnie jak w przypadku KMeans, proponują podział zbioru na 2 klastry.<jupyter_code>gmm = GaussianMixture(n_components=2, covariance_type='full')
labels_gmm = gmm.fit_predict(data)
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(data.X1, data.X2, data.X3, s=50, c = labels_gmm)

ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('X3')

plt.tight_layout()
plt.show()
data['k'] = labels_gmm
sns.pairplot(data, height=3, hue = 'k')
plt.tight_layout()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1277685492801772, 'fraction_numerical': 0.030038759689922482, 'mean_word_length': 6.904814004376368, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '441497', 'license': 'Apache-2.0', 'n_tokens_mistral': 3158, 'n_tokens_neox': 3022, 'n_words': 506, 'path': 'Prace_domowe/Praca_domowa6/Grupa3/WisniewskiJacek/pd6.ipynb', 'repo_name': 'welkierk/2020L-WUM', 'size': '7224'}