<jupyter_start><jupyter_text>SummaryStraightforward notebook. Apply HDBSCAN to 100,000 row subset and save the model. I've provided the data w/ vectorized sample on kaggle.Note that algorithmic clusters do not necessarily match up perfectly with human intuitions of 'clusters'. A more precise way of thinking about purpose of this notebook and next couple is that we are trying to separate comments similar enough to be from the same campaign, and comments that are not. Thus a campaign may be broken up into several smaller clusters. Additional Notes* I did not re-run this on the entire 100,000 datapoints as fit_predict() takes a few hours to finish running.* Like most clustering algorithms, HDBSCAN does not scale well with dimensionality or sample size - I tried PCA on the doc vecs but it gave me very muddy results* My intuition is that for this step if one had train the word vectors on the corpus using word2vec or use another encoding method with fewer dimensions you might be able to scale up HDBSCAN better. The issue with other encoding methods, however, is that many require taking the entire corpus into memory, which is not feasible on my desktop<jupyter_code>import pandas as pd
import numpy as np
from sklearn.preprocessing import normalize
import hdbscan
%%time
## import a sample of the comments we didn't cull out from the original, vectorized
X = pd.read_csv('proc_17_108_copy_uniques_level0_unclustered_sample_vectorized.csv', index_col=0)
X_sample = X.sample(20000) #try a small subset to see it working
# X_sample = X
### hdbscan doesn't support cosine distance so let's use angular distance instead (euclidean of normalized vectors)
data = X_sample.iloc[:,-300:]
norm_data = normalize(data, norm='l2')
clusterer = hdbscan.HDBSCAN(min_cluster_size=10, prediction_data=True) # would want to scale min_cluster_size with your sample size
%%time
# takes a few hours for all 100,000 data points
dbscan = clusterer.fit_predict(norm_data)
dbscan.max() #make sure it's clustering -- can do some more checking by glomming back onto the dataframe
joblib.dump(clusterer, 'level1-clusterer.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05236091631603553, 'fraction_numerical': 0.017765310892940627, 'mean_word_length': 5.407185628742515, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '576438', 'license': 'MIT', 'n_tokens_mistral': 607, 'n_tokens_neox': 571, 'n_words': 295, 'path': 'proc_17_108_analysis_02_HDBScan_Clustering.ipynb', 'repo_name': 'j2kao/fcc_nn_research', 'size': '2139'}