<jupyter_start><jupyter_text>An attempt at clusteringSo, if you read the unique_words notebook, it's clear thatclustering this in a traditional way is hopeless.The algorithm that seemed to work somewhat was hdbscan which let me have some very small clusters (by setting the method=leaf). This at least clustered a bunch of cut-n-pasted sections.  But even so, mostof the corpus was unclassifiable.<jupyter_code>import pandas as pd
import numpy as np 

from gensim import corpora
from gensim.models import TfidfModel, LsiModel<jupyter_output><empty_output><jupyter_text>Read the data<jupyter_code>from budget_corpus import read_documents, read_raw_corpus
corpus = read_documents()
raw_corpus = read_raw_corpus() # so I can see the original doc for debugging
dictionary = corpora.Dictionary(tokens for tokens in corpus)
tokened_corpus = [dictionary.doc2bow(tokens) for tokens in corpus ]<jupyter_output><empty_output><jupyter_text>TFIDF and LSITFIDF compensates for variation in document length (at least if this were a normal corpus)LSI (called LSA in sklearn) does dimensionality reduction. Went from 4000vocabulary words down to 100 dimensions so that thing would cluster for HDBSCAN.<jupyter_code># first convert words to tfidf values (this helps compensate for the
# uneven document sizes)

tfidf = TfidfModel(dictionary=dictionary)
vectored_corpus = [ tfidf[doc] for doc in tokened_corpus]

# next do dimensionalty reduction - 
lsi = LsiModel(corpus=vectored_corpus, num_topics=100, id2word=dictionary, onepass=False, power_iters=3)
lsi.print_topics()
lsi_corpus = [ lsi[doc] for doc in vectored_corpus ]

# each element of lsi corpus is a topic number and a percentage,
# to turn this into an array of vectors for clustering, need to
# drop the topic numbers
lsi_corpus[0][ :10]
lsi_array = np.array(lsi_corpus)
lsi_array.shape
lsi_array = lsi_array[:,:,1]
lsi_array.shape
# and you can see we have the same numbers as the sample above,
# just in a different shape
lsi_array[0, :10]
# cluster with hdbscan, telling it to go with fine grained clusters
# because the broad ones created with other models make even less sense...

import hdbscan
hdb = hdbscan.HDBSCAN(cluster_selection_method='leaf')
hdb.fit(lsi_array)
hdb.labels_
labels = pd.Series(hdb.labels_)
labels.value_counts()
# The clustering is good at finding repeated texts at least
for idx, lab in enumerate(hdb.labels_):
    if lab == 8:
        print(" " )
        print(raw_corpus[idx])
# Let's take a peek at the bigger category : You can see
# that it's not particularly obvious why they're combined
import random
for idx, lab in enumerate(hdb.labels_):
    if lab == 12 and random.random() > .8:
        print(" " )
        print(raw_corpus[idx][:500])
# Finally, a peek at what the clusterer could not cluster at all
for idx, lab in enumerate(hdb.labels_):
    if lab == -1 and random.random() > .99:
        print(" " )
        print(raw_corpus[idx][:500])
# and where's my butterfly? See above...
raw_corpus[234]
hdb.labels_[234]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0791295746785361, 'fraction_numerical': 0.012858555885262116, 'mean_word_length': 5.007920792079208, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '310205', 'license': 'MIT', 'n_tokens_mistral': 1010, 'n_tokens_neox': 950, 'n_words': 357, 'path': 'src/clustering.ipynb', 'repo_name': 'j-carson/nlp-budget', 'size': '3033'}