<jupyter_start><jupyter_text>It is not possible to find a fixed kernel bandwidth that allows us tosample along the s-curve.The reliance on the fixed bandwith kernel is thus problematic for stratified samplingon a manifold embedded in a higher dimensional space.<jupyter_code>num_samples = 200

kernel_scale1d = 5.0 * kernel_scale_factor(dimensionality=1,
                                           num_points=s_curve.shape[0],
                                           num_samples=num_samples)
kernel_scale1d
density_estimator = KernelDensityEstimator(s_curve,
                                           GaussianKernel(sigma=kernel_scale1d))
vc = VoidAndCluster(density_estimator, s_curve.shape[0], num_initial_samples=10)

sample_indices = vc.sample(size=num_samples)
curve_samples = s_curve[sample_indices]
curve_color_samples = s_curve_color[sample_indices]
fig = plt.figure(figsize=(12, 8))
ax = fig.add_subplot(projection='3d')
ax.set_title('We fail to sample along the S-curve')
ax.scatter(curve_samples[:, 0], curve_samples[:, 1], curve_samples[:, 2],
           c=curve_color_samples, cmap=plt.cm.Spectral)
ax.view_init(4, -72)<jupyter_output><empty_output><jupyter_text>Estimating density on the manifoldWe now use ``UMAPDensityEstimator`` to perform density estimation not in the 3D euclideanspace, but directly on the embedded manifold (the S-curve).In detail, the density is estimated on the graph used by the UMAP (https://umap-learn.readthedocs.io/) dimensionality reduction technique. This graph describes the topological connectedness of the embedded manifold. Sampling the graph with the void and cluster technique leads to optimally stratified samples on the embedded manifold.<jupyter_code>from vc_sample.density_estimation import UMAPDensityEstimator

ude = UMAPDensityEstimator(s_curve, n_neighbors=10)
rho = ude.estimate()
fig = plt.figure(figsize=(12, 8))
ax = fig.add_subplot(projection='3d')
ax.set_title('Density estimated on the embedded manifold')
v = ax.scatter(s_curve[:, 0], s_curve[:, 1], s_curve[:, 2], c=rho,
               cmap=plt.cm.Spectral)
ax.view_init(4, -72)
fig.colorbar(v, ax=ax, fraction=0.017, pad=0.0);
vc = VoidAndCluster(ude, s_curve.shape[0], num_initial_samples=100)

sample_indices = vc.sample(size=num_samples)
curve_samples = s_curve[sample_indices]
curve_color_samples = s_curve_color[sample_indices]
fig = plt.figure(figsize=(12, 8))
ax = fig.add_subplot(projection='3d')
ax.set_title('We correctly sample the S-curve')
ax.scatter(curve_samples[:, 0], curve_samples[:, 1], curve_samples[:, 2],
           c=curve_color_samples, cmap=plt.cm.Spectral)
ax.view_init(4, -72)<jupyter_output><empty_output><jupyter_text>Large circle containing a small inner circle<jupyter_code>circles, circles_class = make_circles(n_samples=1000)

fig, ax = plt.subplots(1, 1)
ax.scatter(circles[:, 0], circles[:, 1], c=circles_class, cmap='Paired');
from vc_sample.density_estimation import UMAPDensityEstimator

ude = UMAPDensityEstimator(circles, n_neighbors=15)
rho = ude.estimate()
plt.scatter(circles[:, 0], circles[:, 1], c=rho, cmap='viridis');
plt.colorbar();
vc = VoidAndCluster(ude, circles.shape[0], num_initial_samples=50)

sample_indices = vc.sample(size=num_samples)

circles_samples = circles[sample_indices]
circles_class_samples = circles_class[sample_indices]

plt.scatter(circles_samples[:, 0], circles_samples[:, 1], c=circles_class_samples,
            cmap='Paired');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11082175925925926, 'fraction_numerical': 0.021122685185185185, 'mean_word_length': 5.437616387337058, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '455264', 'license': 'MIT', 'n_tokens_mistral': 1204, 'n_tokens_neox': 1124, 'n_words': 261, 'path': 'notebooks/sampling_higherdim.ipynb', 'repo_name': 'TobiasRp/vc_sample', 'size': '3456'}