<jupyter_start><jupyter_text>When Null is True* p-value is uniformly distributed.* p-value distribution is independent of sample size.* t-statistic follows unit normal (for sample size >= 40).<jupyter_code>alpha=0.05
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=10000, mu0=5, sample_size=100)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))
alpha=0.05
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=10000, mu0=5, sample_size=10)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))<jupyter_output>100%|██████████| 10000/10000 [00:02<00:00, 4257.71it/s]<jupyter_text>When Null is False* smaller effect size reduces power* large sample size increases power* large alpha increases power* number of samples does not affect power<jupyter_code>alpha=0.05
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=10000, mu0=5.5, sample_size=100)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))
# smaller effect size decreases power
alpha=0.05
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=10000, mu0=5.1, sample_size=100)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))
# large sample size increases power (reduces SE)
alpha=0.05
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=10000, mu0=5.1, sample_size=1000)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))
# large alpha increases power
alpha = 0.1
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=10000, mu0=5.1, sample_size=1000)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))
# number of samples does not affect power
alpha = 0.1
p_vals, t_vals, se_vals = two_tailed_t_test(n_samples=100000, mu0=5.1, sample_size=1000)
plot_sampling_distribution(p_vals, t_vals, se_vals, alpha=alpha, save=None)
print("reject null for %.2f%% of the trials"%(100 * sum(np.array(p_vals) < alpha) / len(p_vals)))<jupyter_output>100%|██████████| 100000/100000 [00:21<00:00, 4642.66it/s]
{'fraction_non_alphanumeric': 0.14357197844495767, 'fraction_numerical': 0.07043879907621248, 'mean_word_length': 7.383870967741935, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '589417', 'license': 'MIT', 'n_tokens_mistral': 1117, 'n_tokens_neox': 976, 'n_words': 230, 'path': 'statistics_test/t_test_one_sample_simulation.ipynb', 'repo_name': 'shawlu95/Data_Science_Toolbox', 'size': '2598'}