<jupyter_start><jupyter_text>Import preprocessed data<jupyter_code>df = pd.read_csv(join('..', 'data', 'tugas_preprocessed.csv'))
df.head()
df.columns
# Splitting feature names into groups
non_metric_features = df.columns[df.columns.str.startswith('x')]
pc_features = df.columns[df.columns.str.startswith('PC')]
metric_features = df.columns[~df.columns.str.startswith('x') & ~df.columns.str.startswith('PC')]<jupyter_output><empty_output><jupyter_text>K-Means ClusteringWhat is K-Means clustering? How does it work? How is it computed?![](../figures/kmeans.png)  Characteristics:- *Number of clusters* need to be set apriori- One of the *fastest* clustering algorithms- The results *depend on the initialization* (stochastic)- Prone to *local optima*- Favors *convex* (round shape) and *isotropic* (same shape) clusters  How to apply K-Means clustering?<jupyter_code>kmclust = KMeans(init='random')
# the fit method
kmclust.fit(df[metric_features])
# the predict method
kmclust.predict(df[metric_features])
#shape
kmclust.predict(df[metric_features]).shape
# the transform method
pd.DataFrame(kmclust.transform(df[metric_features]))<jupyter_output><empty_output><jupyter_text>How can we improve the initialization step?<jupyter_code># Better initialization method and provide more n_init
kmclust = KMeans(n_clusters=8, init='k-means++', n_init=15, random_state=1)
kmclust.fit(df[metric_features])
kmclust.predict(df[metric_features])<jupyter_output><empty_output><jupyter_text>*init='k-means++'* initializes the centroids to be (generally) distant from each other, leading to probably better results than random initialization. *n_init=K* allows to initialize KMeans K times and pick the best clustering in terms of Inertia. This can been shown in the link below.**Empirical evaluation of the impact of k-means initialization:**https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_stability_low_dim_dense.htmlsphx-glr-auto-examples-cluster-plot-kmeans-stability-low-dim-dense-py  Defining the number of clusters:<jupyter_code>"""we can use hierarquical cluster and defininf the numbers of cluster from the dendogram"""

range_clusters = range(1,15)
inertia = []
for n_clus in range_clusters:  # iterate over desired ncluster range
    kmclust = KMeans(n_clusters=n_clus, init='k-means++', n_init=15, random_state=1)
    # fit the clusterer
    kmclust.fit(df[metric_features])
    inertia.append(kmclust.inertia_)  # save the inertia of the given cluster solution
len(inertia)<jupyter_output><empty_output><jupyter_text>**Inertia (within-cluster sum-of-squares distance) Formula:**$$\sum_{j=0}^{C}\sum_{i=0}^{n_j}(||x_i - \mu_j||^2)$$, where:$C$: Set of identified clusters.$n_j$: Set of observations belonging to cluster $j$.$x_i$: Observation $i$.$\mu_j$: Centroid of cluster $j$.<jupyter_code># The inertia plot
plt.figure(figsize=(9,5))
# plt.plot(inertia)
x = range(1,15)
xi = list(range(len(x)))
y = inertia
# plot the index for the x-values
plt.plot(xi, y, marker='o', linestyle='--', color='r', label='Square')

plt.xlabel('N_clusters')
plt.ylabel('Inertia: SSw') 
plt.xticks(xi, x)
plt.title("Inertia plot over clusters", size=15)
plt.show()<jupyter_output><empty_output><jupyter_text>**Silhouette Coefficient formula for a single sample:**$$s = \frac{b - a}{max(a, b)}$$, where:- $a$: The mean distance between a sample and all other points in the same cluster.- $b$: The mean distance between a sample and all other points in the next nearest cluster<jupyter_code># Adapted from:
# https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html#sphx-glr-auto-examples-cluster-plot-kmeans-silhouette-analysis-py

# Storing average silhouette metric
avg_silhouette = []
for nclus in range_clusters:
    # Skip nclus == 1
    if nclus == 1:
        continue
    
    # Create a figure
    fig = plt.figure(figsize=(13, 7))

    # Initialize the KMeans object with n_clusters value and a random generator
    # seed of 10 for reproducibility.
    kmclust = KMeans(n_clusters=nclus, init='k-means++', n_init=15, random_state=1)
    cluster_labels = kmclust.fit_predict(df[metric_features])

    # The silhouette_score gives the average value for all the samples.
    # This gives a perspective into the density and separation of the formed clusters
    silhouette_avg =  silhouette_score(df[metric_features], cluster_labels)
    avg_silhouette.append(silhouette_avg)
    print(f"For n_clusters = {nclus}, the average silhouette_score is : {silhouette_avg}")

    # Compute the silhouette scores for each sample
    sample_silhouette_values =  silhouette_samples(df[metric_features], cluster_labels)

    y_lower = 10
    for i in range(nclus):
        # Aggregate the silhouette scores for samples belonging to cluster i, and sort them
        ith_cluster_silhouette_values = sample_silhouette_values[cluster_labels == i]
        ith_cluster_silhouette_values.sort()
        
        # Get y_upper to demarcate silhouette y range size
        size_cluster_i = ith_cluster_silhouette_values.shape[0]
        y_upper = y_lower + size_cluster_i
        
        # Filling the silhouette
        color = cm.nipy_spectral(float(i) / nclus)
        plt.fill_betweenx(np.arange(y_lower, y_upper),
                          0, ith_cluster_silhouette_values,
                          facecolor=color, edgecolor=color, alpha=0.7)

        # Label the silhouette plots with their cluster numbers at the middle
        plt.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))

        # Compute the new y_lower for next plot
        y_lower = y_upper + 10  # 10 for the 0 samples

    plt.title("The silhouette plot for the various clusters.")
    plt.xlabel("The silhouette coefficient values")
    plt.ylabel("Cluster label")

    # The vertical line for average silhouette score of all the values
    plt.axvline(x=silhouette_avg, color="red", linestyle="--")
    
    # The silhouette coefficient can range from -1, 1
    xmin, xmax = np.round(sample_silhouette_values.min() -0.1, 2), np.round(sample_silhouette_values.max() + 0.1, 2)
    plt.xlim([xmin, xmax])
    
    # The (nclus+1)*10 is for inserting blank space between silhouette
    # plots of individual clusters, to demarcate them clearly.
    plt.ylim([0, len(df[metric_features]) + (nclus + 1) * 10])

    plt.yticks([])  # Clear the yaxis labels / ticks
    plt.xticks(np.arange(xmin, xmax, 0.1))
# The average silhouette plot
# The inertia plot
plt.figure(figsize=(9,5))
plt.plot(avg_silhouette)
plt.ylabel("Average silhouette")
plt.xlabel("Number of clusters")
plt.title("Average silhouette plot over clusters", size=15)
plt.show()<jupyter_output><empty_output><jupyter_text>Final KMeans clustering solution<jupyter_code># final cluster solution
number_clusters =  3
kmclust = KMeans(n_clusters=number_clusters, init='k-means++', n_init=15, random_state=1)
km_labels = kmclust.fit_predict(df[metric_features])
km_labels
# Characterizing the final clusters
df_concat = pd.concat((df, pd.Series(km_labels, name='labels')), axis=1)
df_concat.groupby('labels').mean()<jupyter_output><empty_output><jupyter_text>How can we combine the 2 algorithms?  Exercise:**Apply Hierarchical Clustering and K-means on the Principal Components.**Choose the appropriate parameters and number of clusters for each algorithm and interpret each cluster based on the Principal Components interpretation:<jupyter_code>from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram<jupyter_output><empty_output><jupyter_text>defining n_clusters for HC<jupyter_code># setting distance_threshold=0 and n_clusters=None ensures we compute the full tree
linkage = 'ward'
distance = 'euclidean'
hclust = AgglomerativeClustering(linkage=linkage, affinity=distance, distance_threshold=0, n_clusters=None)
hclust.fit_predict(df[pc_features])
# Adapted from:
# https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html#sphx-glr-auto-examples-cluster-plot-agglomerative-dendrogram-py

# create the counts of samples under each node (number of points being merged)
counts = np.zeros(hclust.children_.shape[0])
n_samples = len(hclust.labels_)

# hclust.children_ contains the observation ids that are being merged together
# At the i-th iteration, children[i][0] and children[i][1] are merged to form node n_samples + i
for i, merge in enumerate(hclust.children_):
    # track the number of observations in the current cluster being formed
    current_count = 0
    for child_idx in merge:
        if child_idx < n_samples:
            # If this is True, then we are merging an observation
            current_count += 1  # leaf node
        else:
            # Otherwise, we are merging a previously formed cluster
            current_count += counts[child_idx - n_samples]
    counts[i] = current_count

# the hclust.children_ is used to indicate the two points/clusters being merged (dendrogram's u-joins)
# the hclust.distances_ indicates the distance between the two points/clusters (height of the u-joins)
# the counts indicate the number of points being merged (dendrogram's x-axis)
linkage_matrix = np.column_stack(
    [hclust.children_, hclust.distances_, counts]
).astype(float)

# Plot the corresponding dendrogram
sns.set()
fig = plt.figure(figsize=(11,5))
# The Dendrogram parameters need to be tuned
y_threshold = 100
dendrogram(linkage_matrix, truncate_mode='level', p=5, color_threshold=y_threshold, above_threshold_color='k')
plt.hlines(y_threshold, 0, 1000, colors="r", linestyles="dashed")
plt.title(f'Hierarchical Clustering - {linkage.title()}\'s Dendrogram', fontsize=21)
plt.xlabel('Number of points in node (or index of point if no parenthesis)')
plt.ylabel(f'{distance.title()} Distance', fontsize=13)
plt.show()
# Performing Hierarquical Clustering
hclust = AgglomerativeClustering(linkage='ward', affinity='euclidean', n_clusters=4)
hc_labels = hclust.fit_predict(df[pc_features])
hc_labels
# Characterizing the clusters obtained by HC
df_concat_HC = pd.concat((df, pd.Series(hc_labels, name='labels_HC')), axis=1)
df_concat_HC.groupby('labels_HC').mean()
"""não usar PCA e metric features porque eles tem alta colinearidade entre elas, 
já que PCA são baseadas na variância das metric features.
O ideal é testar a clusterização separadamente em cada set de features e comparar os resultados.
Não é muito comum utilizar as variáveis categóricas em Clustering K-means ou Hierarquical já que os algoritmos
são baseados em cálculo de distâncias (Euclidean) e isso se aplica às variáveis contínuas.
K-Modes pode ser aplicado para considerar as variáveis categóricas
"""<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1044383972975509, 'fraction_numerical': 0.009195833724312659, 'mean_word_length': 5.090285714285714, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '486058', 'license': 'MIT', 'n_tokens_mistral': 3633, 'n_tokens_neox': 3423, 'n_words': 953, 'path': 'notebooks/lab10_kmeans_clustering.ipynb', 'repo_name': 'gabriel-martins-souza/Data-Mining-21-22', 'size': '10657'}