<jupyter_start><jupyter_text>Sequential classification with bandit feedback$85$ multi-class classification from OpenML(https://www.openml.org), see below for the list of darasets Policies:1. __optimal contextual policy__ that assigns each covariate to its correct label;2. __best fixed_arm policy__ that always assigns each covariates to the label with most samples;3. __contrast__ between the above policies.We demonstrate the contrast estimation here. Compared estimation methods:- DM: direct method;- DR: doubly-robust method with weight $h_t=1$;- non-contextual `MinVar` $h_t = 1/E[\sum_w\frac{\pi^2(x,w)}{e_t(x,w)}]$- non-contextual `StableVar` $h_t = 1/\sqrt{E[\sum_w\frac{\pi^2(x,w)}{e_t(x,w)}]}$- contextual `MinVar` $h_t(x) = 1/\sum_w\frac{\pi^2(x,w)}{e_t(x,w)}$- contextual `StableVar` $h_t(x) = 1/\sum_w\frac{\pi^2(x,w)}{e_t(x,w)}$  Read saving results<jupyter_code>def read_data(name):
    results = read_files(f'{name}*')
    print(len(results))
    if os.path.exists(f"{name}_stats.csv"):
        df_stats = pd.read_csv(f"{name}_stats.csv")
    else:
        df_stats, _, _ = generate_data_frames(results)
        df_stats.to_csv(f"{name}_stats.csv")
    return df_stats
df_stats =  read_data("results/classification")
df_lite = merge_dataset_df(df_stats)
df_lite['dgp'].unique()<jupyter_output><empty_output><jupyter_text>Analyze characteristics of datasets: classes, features, observations<jupyter_code>analyze_characteristics(df_lite)<jupyter_output>#Classes: =2: 66; >2,<=8: 16; >8: 4
#Features: <=10: 46; >10, <=50: 36; >50: 4 
#Observations: <=1k: 44; >1k, <=10k: 32; >10k: 10<jupyter_text>Plots  Use DR as a baseline and compare it with other methods; evaluating contrastEvery subplot corresponds to an estimation method, wherein each point denotes a dataset, and its $(x,y)$ coordinates are bias and standard error normalized by RMSE of DR estimate on the same dataset.<jupyter_code>plot_radius(df_lite, 'uniform', 'optimal-best_arm')<jupyter_output><empty_output><jupyter_text>Compare contextual weighting versus non-contextual weighting; evaluating contrastIn the left panel, each point denotes a dataset, and its $(x,y)$ coordinates are bias and standard error of contextual `MinVar` normalized by RMSE of non-contextual `MinVar` on the same dataset. Same visualization is applied to contextual and non-contextual `StableVar`  in the right panel.<jupyter_code>plot_radius_comparison(df_lite, 'optimal-best_arm')<jupyter_output><empty_output><jupyter_text>Compare MinVar versus StableVar; evaluating contrastEach point corresponds to a dataset. The $(x,y)$ coordinates are bias and standard error of `MinVar` normalized by its `StableVar` counterpart in forms of both noncontextual weighting (left panel) and contextual weighting (right panel).<jupyter_code>plot_radius_comparison_2(df_lite, 'optimal-best_arm')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1193717277486911, 'fraction_numerical': 0.017102966841186736, 'mean_word_length': 6.50261780104712, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '42970', 'license': 'MIT', 'n_tokens_mistral': 989, 'n_tokens_neox': 923, 'n_words': 266, 'path': 'experiments/statistics_classification.ipynb', 'repo_name': 'gsbDBI/contextual_bandits_evaluation', 'size': '2865'}