<jupyter_start><jupyter_text>We visualize the true labels using t-SNE<jupyter_code>projection = TSNE(random_state=2).fit_transform(data)

color_palette = sns.color_palette("Paired", 10)
cluster_colors = [color_palette[x] for x in datasets.load_digits().target]
plt.figure(figsize=(5,5))
plt.scatter(projection.T[0], projection.T[1], s=10, linewidth=0, c=cluster_colors, alpha=0.8)
plt.show()<jupyter_output><empty_output><jupyter_text>Now we cluster the data with gamma-linkage.In order to choose the parameters, we start by looking at a vineyard.<jupyter_code>x = MPSpace(data)
x.fit()

x0 = x.connection_radius()
initial_k = 0.02
final_k = 0.15
N_PARAMETERS = 50

gammas = [ Gamma_curve.linear_interpolator_alpha_s_indexed(k, x0) for k in np.linspace(initial_k, final_k, num=N_PARAMETERS) ]
PDs = [ x.gamma_linkage(curve).PD()[0] for curve in gammas ]
kappa = 0.05
tau = 0.7

plotProminenceVineyard(PDs, N_PARAMETERS, initial_k, final_k, kappa = kappa, tau = tau, s = 10, y_lim = [0.1,10])<jupyter_output><empty_output><jupyter_text>Choose k in the 12 cluster gap.We choose a small k in this gap, because we'll get a larger percentage of points clustered.<jupyter_code>x = MPSpace(data)
x.fit()

gamma = Gamma_curve.linear_interpolator_alpha_s_indexed(kappa, x0)
hc = x.gamma_linkage(gamma)
gamma_labels = hc.persistence_based_flattening(threshold=tau)<jupyter_output><empty_output><jupyter_text>Coloring the points by the gamma-linkage labels<jupyter_code>color_palette = sns.color_palette("Paired", 12)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.85, 0.85, 0.85)
                  for x in gamma_labels[1]]
plt.figure(figsize=(5,5))
plt.scatter(projection.T[0], projection.T[1], s=10, linewidth=0, c=cluster_colors, alpha=0.8)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12673707615341856, 'fraction_numerical': 0.032795997776542525, 'mean_word_length': 6.0588235294117645, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '521115', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 739, 'n_tokens_neox': 692, 'n_words': 144, 'path': 'example-digits.ipynb', 'repo_name': 'LuisScoccola/gamma-linkage', 'size': '1799'}