<jupyter_start><jupyter_text>Chat Intents Clustering without dimensionality reduction **Summary**This notebook briefly explores a few methods for representing documents with different embedding methods and clustering the embeddings with different algorithsm. In contrast to notebooks 04 and 05 in this repo, here no dimensionality reduction is used. As shown below, the biggest challene here remains selecting the right hyperparameters when the number of clusters isn't already known.<jupyter_code>import random

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import silhouette_score
from sklearn.metrics.cluster import normalized_mutual_info_score
from sklearn.metrics.cluster import adjusted_rand_score
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from sklearn.cluster import AgglomerativeClustering
import hdbscan
import tensorflow as tf
import tensorflow_hub as hub
from sentence_transformers import SentenceTransformer

pd.set_option("display.max_rows", 500)
pd.set_option("display.max_columns", 500)
pd.set_option("max_colwidth", 400)
data_sample = pd.read_csv('../data/processed/data_sample.csv')
data_sample.head()<jupyter_output><empty_output><jupyter_text>Helper functions<jupyter_code>def plot_kmeans(embeddings, k_range):
    '''
    Plot SSE and silhouette score for kmeans clustering for a range of k values
    Arguments:
        embeddings: array, sentence embeddings
        k_range: range, values of k to evaluate for kmeans clustering  
    '''
    sse = []
    silhouette_avg_n_clusters = []

    for k in k_range:
        kmeans = KMeans(n_clusters=k, init='k-means++', max_iter=100, n_init=1)
        kmeans.fit(embeddings)
        sse.append(kmeans.inertia_)

        silhouette_avg = silhouette_score(embeddings, kmeans.predict(embeddings))
        silhouette_avg_n_clusters.append(silhouette_avg)


    # plot sse
    fig, axes = plt.subplots(1, 2, figsize=(14, 6))
    axes[0].plot(k_range, sse)
    axes[0].set(xlabel = 'k clusters', ylabel = 'SSE', title = 'Elbow plot')
    axes[0].grid()
    
    # plot avg silhouette score
    axes[1].plot(k_range, silhouette_avg_n_clusters)
    axes[1].set(xlabel = 'k clusters', ylabel = 'Silhouette score', title = 'Silhouette score')
    axes[1].grid()

    plt.show()
def search_cluster_size(embeddings, size_range):
    '''
    Scan HDBSCAN min_cluster_size values and return results
    Arguments:
        embeddings: embeddings to use
        size_range: range of min_cluster_size hyperparameter values to scan
    Returns:
        result_df: dataframe of min_cluster_size, total number of clusters,
                   and percent of data labeled as noise
    '''
    
    results = []
    
    for i in size_range:
        min_cluster_size = i
        
        clusters_hdbscan = (hdbscan.HDBSCAN(min_cluster_size=min_cluster_size,
                                           metric='euclidean', 
                                           cluster_selection_method='eom') 
                                           .fit(embeddings))
    
        labels = clusters_hdbscan.labels_
        label_count = len(np.unique(labels))
        
        total_num = len(clusters_hdbscan.labels_)
        cost = (np.count_nonzero(clusters_hdbscan.probabilities_ < 0.05)/total_num) 
                
        results.append([min_cluster_size, label_count, cost])
    
    result_df = pd.DataFrame(results, columns=['min_cluster_size', 
                                               'label_count', 'noise'])
    
    return result_df
def random_search(embeddings, space, num_evals):
    '''
    Random search of HDBSCAN hyperparameter spcae
    Arguments:
        embeddings: embeddings to use
        space: dict, hyperparameter space to search with keys of
               'min_cluster_size' and 'min_samples' and values as ranges
        num_evals: int, number of trials to run
    Returns:
        result_df: dataframe of run_id, min_cluster_size, min_samples, 
                   total number of clusters, and percent of data labeled as noise
    
    '''
    
    results = []
    
    for i in range(num_evals):
        min_cluster_size = random.choice(space['min_cluster_size'])
        min_samples = random.choice(space['min_samples'])
        
        clusters_hdbscan = (hdbscan.HDBSCAN(min_cluster_size=min_cluster_size,
                                           min_samples = min_samples,
                                           metric='euclidean', 
                                           cluster_selection_method='eom') 
                                           .fit(embeddings))
    
        labels = clusters_hdbscan.labels_
        label_count = len(np.unique(labels))
        
        total_num = len(clusters_hdbscan.labels_)
        cost = (np.count_nonzero(clusters_hdbscan.probabilities_ < 0.05)/total_num) 
                
        results.append([i, min_cluster_size, min_samples, label_count, cost])
    
    result_df = pd.DataFrame(results, columns=['run_id', 'min_cluster_size', 'min_samples', 
                                               'label_count', 'cost'])
    
    return result_df.sort_values(by='cost')<jupyter_output><empty_output><jupyter_text>TF-IDF embeddings with K-means<jupyter_code>cleaned_text = data_sample['cleaned_text']
cleaned_text.head()
tfidf_vectorizer = TfidfVectorizer()
tfidf_transformed_vector = tfidf_vectorizer.fit_transform(cleaned_text)
tfidf_transformed_vector.shape
k = 10
kmeans = KMeans(n_clusters=k, init='k-means++', max_iter=100, n_init=1)
kmeans.fit(tfidf_transformed_vector)
len(kmeans.labels_)
%%time

k_range = range(5, 141)
plot_kmeans(tfidf_transformed_vector, k_range)<jupyter_output><empty_output><jupyter_text>There's no clear elbow in the scree plot of the sum of squared errors as a function of the number of clusters (k). Additionally, the silhouette score appears to continue increasing over the range considered. Thus, neither approach makes it clear what value to select for k.  Sentence Embeddings The sentence embedding models typically expect raw sentences (without preprocessing), so we'll use those here.<jupyter_code>full_text = data_sample['text']<jupyter_output><empty_output><jupyter_text>USE Sentence Embeddings + Kmeans Start with Universal Sentence Encoder (USE) as an example sentence embedding model.<jupyter_code>module_url = "https://tfhub.dev/google/universal-sentence-encoder/4"
model_use = hub.load(module_url)

print(f"module {module_url} loaded")
use_embeddings = model_use(full_text)
use_embeddings.shape
%%time

k_range = range(5, 151)
plot_kmeans(use_embeddings, k_range)<jupyter_output><empty_output><jupyter_text>As with the TF-IDF embeddings above, it's not obvious what number of clusters to select from these plots. The silhouette score continues to slowly increase as the number of clusters increases. Even at 150 clusters though, the silhouette score has only reached 0.08, which is quite low.  Sentence Transformer Embeddings + Kmeans Use a sentence-transformer pretrained model that was made in a way to preserve the usefulness of Euclidean distances (https://www.sbert.net/docs/pretrained_models.html).<jupyter_code>model_st = SentenceTransformer('all-mpnet-base-v2')
st_embeddings = model_st.encode(full_text)
st_embeddings.shape
%%time

k_range = range(5, 151)
plot_kmeans(st_embeddings, k_range)<jupyter_output><empty_output><jupyter_text>As with the USE model, it still isn't obvious what value of k to select. From the silhouette score plot we see that the values seem to somewhat level off starting at around 100 clusters, but it also continues to slowly rise from there.  Hierarchical clustering<jupyter_code># Normalize the embeddings to unit length
st_embeddings_norm = st_embeddings /  np.linalg.norm(st_embeddings, axis=1, keepdims=True)
# Perform clustering
agglom_model = AgglomerativeClustering(n_clusters=None, distance_threshold=1.5)
agglom_model.fit(st_embeddings_norm)
cluster_assignment = agglom_model.labels_

len(np.unique(cluster_assignment))<jupyter_output><empty_output><jupyter_text>While we can obvoiusly create clusters using hierarchical clustering, we still have no way to intuitively select the necessary hyperparameters (e.g. distance_threshold in this case).  Sentence Embeddings + HDBSCAN<jupyter_code>clusters_hdbscan = hdbscan.HDBSCAN(min_cluster_size=2, 
                                   metric='euclidean', 
                                   cluster_selection_method='eom').fit(st_embeddings)

labels = clusters_hdbscan.labels_
len(np.unique(labels))
pd.Series(labels).value_counts().head(10)
size_range = range(2,11)
search_cluster_size(st_embeddings, size_range=size_range)<jupyter_output><empty_output><jupyter_text>Scanning through different min_cluster_size values for HDBSCAN reveals some of the challenges of using the raw sentence embeddings. Either all the datapoints are lumped into the same, small number of clusters (e.g. 3 total), or a large percentage of datapoints are not clustered (e.g. 38% of the data is left out as noise with a min_cluster_size of 3).<jupyter_code>%%time

space = {
    "min_cluster_size": range(2, 30),
    "min_samples": range(2, 30)
}

random_search(st_embeddings, space, num_evals=100).head(20)<jupyter_output>CPU times: user 2min 7s, sys: 816 ms, total: 2min 8s
Wall time: 2min 17s
{'fraction_non_alphanumeric': 0.07898102749946706, 'fraction_numerical': 0.01033894691963334, 'mean_word_length': 3.631293188548865, 'pattern_counts': {'":': 2, '<': 38, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '52694', 'license': 'MIT', 'n_tokens_mistral': 2937, 'n_tokens_neox': 2748, 'n_words': 759, 'path': 'notebooks/03-clustering_without_dim_reduction.ipynb', 'repo_name': 'dborrelli/chat-intents', 'size': '9382'}