<jupyter_start><jupyter_text>*Plot utils*<jupyter_code>def _convert_sparse_df_to_grid(df):
    col = df.columns
    plot_df = df.pivot(*col)
    x, y = np.meshgrid(plot_df.index.values, plot_df.columns.values)
    z = plot_df.T.values
    return x, y, z
# For manipulating the plots
%matplotlib tk
%matplotlib inline
fig_size = [12, 9]
plt.rcParams["figure.figsize"] = fig_size<jupyter_output><empty_output><jupyter_text>Initialization<jupyter_code>x_max, ndim = 100., 2
X = np.random.rand(15000, ndim) * x_max
X_t = np.random.rand(15000, ndim) * x_max

bw_list = [20.] * ndim

X[:5]<jupyter_output><empty_output><jupyter_text>Compare resultsIn order to compare the probability provided by sklearn implementation and bt_kdeWe query a sample:<jupyter_code>sk_model = BallTree(X)
sk_model.kernel_density(X_t[:10], bw_list[0])
bt_kde_eval = bt_kde(X, bw_list)
bt_kde_eval(X_t[:10])<jupyter_output><empty_output><jupyter_text>We have an approximation but we have the same results.  Compare perf  Init<jupyter_code>%timeit -n1 -r1 BallTree(X)
%timeit -n1 -r1 bt_kde(X, bw_list)<jupyter_output>1 loop, best of 1: 4.36 ms per loop
1 loop, best of 1: 584 ms per loop<jupyter_text>Not realy conclusive perf wise on the initialization.The real benefit comes in the query.  Query a few points<jupyter_code>%timeit -n1 -r1 sk_model.kernel_density(X_t[:10], bw_list[0])
%timeit -n1 -r1 bt_kde_eval(X_t[:10])<jupyter_output>1 loop, best of 1: 17.1 ms per loop
1 loop, best of 1: 5.48 ms per loop<jupyter_text>We start to see the benefit of it.  Query more points<jupyter_code>%timeit -n1 -r1 sk_model.kernel_density(X_t, bw_list[0])
%timeit -n1 -r1 bt_kde_eval(X_t)<jupyter_output>1 loop, best of 1: 21.9 s per loop
1 loop, best of 1: 752 ms per loop<jupyter_text>Here we go!The implementation which is aggregating the distributions smash the perfs of classical KDE  Accuracy study We need to evaluate the approximation we are doing in order to understand the tradeoff we are dealing with.  Tol parameter impact<jupyter_code>p_x = X_t[:100]

plot_df = pd.DataFrame(index=[x[0] for x in p_x])
plot_df['sk'] = BallTree(X).kernel_density(p_x, bw_list[0])
plot_df['bt_kde_1'] = bt_kde_eval(p_x, pred_tol=2.)
plot_df['bt_kde_2'] = bt_kde_eval(p_x, pred_tol=4.)

plot_df.sort_index(inplace=True)

plot_df.plot()
plt.show()<jupyter_output><empty_output><jupyter_text>Here we can also note the impact of the tolerance parameter.  Plot error<jupyter_code>res = 50

plot_x = np.array([pt for pt in it.product(np.linspace(0., x_max, res), np.linspace(0., x_max, res))])

plot_df = pd.DataFrame(plot_x, columns=['x', 'y'])
plot_df['sk'] = BallTree(X).kernel_density(plot_x, bw_list[0])
plot_df['bt_kde'] = bt_kde_eval(plot_x)
plot_df['err'] = plot_df['bt_kde'] - plot_df['sk']

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.plot_wireframe(*_convert_sparse_df_to_grid(plot_df[['x', 'y', 'sk']]), rstride=1, cstride=1,
                                                                          alpha=0.2, color='g', label='bt')

ax.plot_wireframe(*_convert_sparse_df_to_grid(plot_df[['x', 'y', 'bt_kde']]), rstride=1, cstride=1,
                                                                          alpha=0.2, color='b', label='sk')

ax.plot_wireframe(*_convert_sparse_df_to_grid(plot_df[['x', 'y', 'err']]), rstride=1, cstride=1,
                                                                            alpha=0.2, color='r', label='err')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13139735480161013, 'fraction_numerical': 0.029614721104082805, 'mean_word_length': 4.239457831325301, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '634517', 'license': 'MIT', 'n_tokens_mistral': 1392, 'n_tokens_neox': 1345, 'n_words': 304, 'path': 'demo.ipynb', 'repo_name': 'pelodelfuego/bt-kde', 'size': '3478'}