<jupyter_start><jupyter_text>Which test to use? On what data? Use the nonparametric Wilcoxon Signed-Rank Test, it:Tests whether the distributions of two paired samples are equal or not.Assumptions:-Observations in each sample are independent and identically distributed (iid).-Observations in each sample can be ranked.-Observations across each sample are paired.Use on the delta_mean data because it is paired***What can we conclude? 2EXP***-all delta_mean values for know_programming users are greater than absolutely_none in regard to distinct attempts (for correctness, this is true for all but one question)-two p-values << 0.05 indicates that there is a statistically sigificant difference between the medians of the delta_means for both distinct attempts and correctness-"The difference in delta means is not equal to zero"-Look at each question separately (thousands of data points instead of 21), or is this sufficient? Could alternatively look at means instead of delta_means, does it matter?***What can we conclude? 3EXP***-know vs. other : no significant difference in median delta_means for distinct attempts, but there is a significant difference in median delta_means for correctness -other vs. none : significant difference in median delta_means for distinct attempts AND a significant difference in median delta_means for correctness (same for know vs. none)   2EXP<jupyter_code>df_know = pd.read_csv(r'delta_know_dist.csv')
df_none = pd.read_csv(r'delta_none_dist.csv')

data_distinct_know = df_know.delta_mean.tolist()
data_distinct_none = df_none.delta_mean.tolist()

df_know_corr = pd.read_csv(r'delta_know_corr.csv')
df_none_corr = pd.read_csv(r'delta_none_corr.csv')

data_corr_know = df_know_corr.delta_mean.tolist()
data_corr_none = df_none_corr.delta_mean.tolist()

print(len(data_corr_know))
print(len(data_corr_none))
print(len(data_distinct_know))
print(len(data_distinct_none))
wilcoxon_test(data_distinct_know, data_distinct_none, data_corr_know, data_corr_none)
# pyplot.hist(data_distinct_know, bins=20)
# plt.xlim(-5, 2)
# pyplot.show()
# pyplot.hist(data_distinct_none, bins = 20)
# plt.xlim(-5, 2)
# pyplot.show()
# pyplot.hist(data_corr_know, bins=20)
# plt.xlim(-.15, .05)
# pyplot.show()
# pyplot.hist(data_corr_none, bins = 20)
# plt.xlim(-.15, .05)
# pyplot.show()<jupyter_output><empty_output><jupyter_text>3EXP<jupyter_code>df_know = pd.read_csv(r'delta_know_dist_3exp.csv')
df_other = pd.read_csv(r'delta_other_dist_3exp.csv')

data_distinct_know = df_know.delta_mean.tolist()
data_distinct_other = df_other.delta_mean.tolist()

df_know_corr = pd.read_csv(r'delta_know_corr_3exp.csv')
df_other_corr = pd.read_csv(r'delta_other_corr_3exp.csv')

data_corr_know = df_know_corr.delta_mean.tolist()
data_corr_other = df_other_corr.delta_mean.tolist()

print(len(data_distinct_know))
print(len(data_distinct_other))
print(len(data_corr_know))
print(len(data_corr_other))
#compare know and other
#run with Bonferroni correction
wilcoxon_test_bon(data_distinct_know, data_distinct_other, data_corr_know, data_corr_other)
#compare other and none
wilcoxon_test_bon(data_distinct_other, data_distinct_none, data_corr_other, data_corr_none)
#compare know and none
wilcoxon_test_bon(data_distinct_know, data_distinct_none, data_corr_know, data_corr_none)<jupyter_output>Delta Mean Distinct Attempts:
  Statistic=0.000, p=0.00005957
  Samples do not look equal (reject H0)

Delta Mean Correctness:
  Statistic=3.000, p=0.00009221
  Samples do not look equal (reject H0)
{'fraction_non_alphanumeric': 0.08859674192626464, 'fraction_numerical': 0.0174335524435553, 'mean_word_length': 7.083140877598153, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '599812', 'license': 'MIT', 'n_tokens_mistral': 1270, 'n_tokens_neox': 1166, 'n_words': 306, 'path': 'significance_tests.ipynb', 'repo_name': 'jyoonkim/Test', 'size': '3499'}