<jupyter_start><jupyter_text>CLUSTERING  HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) : is a clustering algorithm  Here, there is no assumptions except for some noise. HDBSCAN uses a density-based approach and it's a non-parametric method that looks for a cluster hierarchy shaped.  Rather than looking for clusters with a particular shape, it looks for regions of the data that are denser than the surrounding space. The underlying distribution is almost always unknown for real-world data estimate the PDF.2 ways : - Counting the number of neighbors of a particular point within its 𝜀-radius (hypersphere around the point and count the number of points within it)- Finding the distance to the K-th nearest neighbor. We determine the number of neighbors we want and find the smallest value of 𝜀 that would contain these K neighbors. Which is what HDBSCAN uses. This distances are called 'core' distances in HDBSCAN. Points with smaller core distances are in denser regions and would have a high estimate for the PDF. Points with larger core distances are in sparser regions because we have to travel larger distances to include enough neighbors.  The hyperparameter K is referred to as min_samples in the HDBSCAN. If K is too low then we have lots of noise. If K is too high it smoothen the PDF too much. The larger the value of min_samples you provide, the more conservative the clustering – more points will be declared as noise.  Important to use min_cluster_size. Distribution graphs may show peaks and bumps. min_cluster_size dictates the maximum size of a “bump” before it is considered a peak. An issue can appear if little bumps are interpreted as mini-clusters. Without removing the bumps, the main clusters may not be seen. The solution is to flatten these small bumps. Main clusters will no longer be distracted by the small bumps First step is to vectorize words so that they can be represented in a graph  First do Word2Vec<jupyter_code>model = word2vec.Word2Vec(processed_description, size=300, window=20, min_count=2, workers=1, iter=100)
# 10 first vectors coordinates associated to the word 'banana'
model.wv['banana'][:10]
w2v_vectors = model.wv.vectors
w2v_indices = {word: model.wv.vocab[word].index for word in model.wv.vocab} # here you load indices - with whom you can find an index of the particular word in your model<jupyter_output><empty_output><jupyter_text>Use the hierarchical density-based clustering method<jupyter_code>clusterer = hdbscan.HDBSCAN(min_cluster_size = 5, min_samples = 10, gen_min_span_tree=True, cluster_selection_method = 'leaf')
clusterer.fit(w2v_vectors)
pal = sns.color_palette('deep', 8)
colors = [sns.desaturate(pal[col], sat) for col, sat in zip(clusterer.labels_,
                                                           clusterer.probabilities_)]
plt.scatter(w2v_vectors.T[0], w2v_vectors.T[1], c=colors, s=1.8)
clusterer.condensed_tree_.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05450874831763122, 'fraction_numerical': 0.010430686406460296, 'mean_word_length': 4.673664122137405, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '73607', 'license': 'MIT', 'n_tokens_mistral': 829, 'n_tokens_neox': 801, 'n_words': 392, 'path': '3_hdbscan_and_conclusion.ipynb', 'repo_name': 'MauPfau/wine-classification', 'size': '2972'}