<jupyter_start><jupyter_text>In the cell below you can specify the parameters for the experiments.- timeout: In seconds. Choose how long a summary is allowed to run before stoping its execution.- dataset: Uncomment one of the three available datasets.- frac: float in [0.0, 0.1]. Percentage of the dataset to be uniformely sampled. The experiments will be run on that sample. The default value is 0.10 a smaller value will make the experiments run faster due to the smaller size. However the results might not be that reliable. Higher values will lead to slower execution times.<jupyter_code>timeout = 120

dataset = 'cars'
#dataset = 'capital'
#dataset = 'hours'
if dataset == 'hours':
    file = '../data/hours_per_week.csv'
elif dataset == 'cars':
    file = '../data/cars_year_produced.csv'
elif dataset == 'capital':
    file = '../data/capital_gain.csv'

frac = 0.1
    
data_name = file.split('/')[2].rstrip('.csv')
print(f'Dataset: {data_name}')<jupyter_output>Dataset: cars_year_produced<jupyter_text>Loading the dataset<jupyter_code>df = pd.read_csv(file, header=None)
df = df.sample(frac = frac, random_state=0)
df = df.loc[:,0].value_counts()
df = df.sort_index()
values = df.index.values
f = df.values<jupyter_output><empty_output><jupyter_text>We are now going to create a histogram on the sample of the dataset that we got with 10 buckets and 2% of the dataset size as deletions, as those are the default parameters used in the paper. Then we are going to visualise the histograms that the different algorithms provide.<jupyter_code>beta = 10
del_percentage = 0.02
k = int(np.sum(f) * del_percentage)


err, elapsed, s = exec_summary('SD CON', values, f, k, beta, timeout=timeout)
sd_con_left, sd_con_right, sd_con_n = s.getBucketRanges()
sd_con_err = err
sd_con_time = elapsed

err, elapsed, s = exec_summary('NO DEL', values, f, k, beta, timeout=timeout)
nd_left, nd_right, nd_n = s.getBucketRanges()
no_del_err = err
no_del_time = elapsed


err, elapsed, s = exec_summary('CON', values, f, k, beta,jit=True ,timeout=timeout)
con_left, con_right, con_n = s.getBucketRanges()
con_err = err
con_time = elapsed<jupyter_output><empty_output><jupyter_text>First we are going to compare a v-optimal histogram with no deletions to the algorithm SD CON that deletes outliers on the buckets created by the no deletion algorithm in order ot minimize variance<jupyter_code>left_edges = [values[nd_left], values[sd_con_left]]
right_edges = [values[nd_right], values[sd_con_right]]
ns = [nd_n, sd_con_n]
color = ['tab:gray', 'tab:orange']
labels = ['NO DEL', 'SD CON']
j = 0
plt.figure(figsize=(6,4))
plt.title('Histogram comparison No Deletions vs SD CON')
for left, right, n in zip(left_edges, right_edges, ns):
    buckets = []
    widths = []
    for i in range(len(left)):
        buckets.append(left[i]+right[i])
        width = right[i]-left[i]
        if width == 0:
            width = 0.1
        widths.append(width)
    plt.bar(np.array(buckets)/2, n, width=widths, color=color[j], alpha=0.7, label=labels[j])
    plt.legend()
    j += 1
plt.savefig(f'../results/vis_nodel_sdcon{data_name}.png')<jupyter_output><empty_output><jupyter_text>Next we are going to compare a v-optimal histogram with no deletions to the algorithm CON that deletes outliers while creating the histogram.<jupyter_code>left_edges = [values[nd_left], values[con_left]]
right_edges = [values[nd_right], values[con_right]]
ns = [nd_n, con_n]
color = ['tab:gray', 'tab:blue']
labels = ['NO DEL', 'CON']
j = 0
plt.figure(figsize=(6,4))
plt.title('Histogram comparison No Deletions vs CON')
for left, right, n in zip(left_edges, right_edges, ns):
    buckets = []
    widths = []
    for i in range(len(left)):
        buckets.append(left[i]+right[i])
        width = right[i]-left[i]
        if width == 0:
            width = 0.1
        widths.append(width)
    plt.bar(np.array(buckets)/2, n, width=widths, color=color[j], alpha=0.5, label=labels[j])
    plt.legend()
    j += 1
plt.savefig(f'../results/vis_nodel_con{data_name}.png')<jupyter_output><empty_output><jupyter_text>Finally lets see a simple execution example<jupyter_code>values = np.array([1, 2, 5, 7, 9 ,10, 25])
f = np.array([1, 5, 3, 1, 10 ,10, 3])
beta = 3
k = 5
err, elapsed, s = exec_summary('CON', values, f, k, beta,jit=True ,timeout=timeout)
con_left, con_right, con_n = s.getBucketRanges()
plot_histogram(values[con_left], values[con_right], con_n)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1287998198603918, 'fraction_numerical': 0.014185994145462734, 'mean_word_length': 4.709511568123394, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '522074', 'license': 'MIT', 'n_tokens_mistral': 1605, 'n_tokens_neox': 1509, 'n_words': 458, 'path': 'experiments/demo.ipynb', 'repo_name': 'AlexandrosKal/opt_hist', 'size': '4441'}