<jupyter_start><jupyter_text>Statistical Power **Type I error**    Rejecting the null when it is actaully true, denoted by $\alpha$. It is essentially a false positive because we  recommend a treatment when it actually does not work.**Type II error**   Do not reject the null when it is actaully false, denoted by $\beta$. It is a false negative because we end up not recomending a treatment that works.**Significance level** A significance level of 0.05 means that there is a 5% chance of a false positive. Choosing level of significance is an arbitrary task, but for many applications, a level of 5% is chosen, for no better reason than that it is conventional**Power**Power of 0.80 means that there is an 80% chance that if there was an effect, we would detect it (or a 20% chance that we'd miss the effect). In other words, power is equivalent to $1 - \beta$. There are no formal standards for power, most researchers assess the power of their tests using 0.80 for adequacy |   Scenario   |          $H_0$ is true              |       $H_0$ is false     ||--------------|:-----------------------------------:|-------------------------:|| Accept $H_0$ |         Correct Decision            | Type 2 Error (1 - power) || Reject $H_0$ | Type 1 Error (significance level)   |      Correct decision    |  IntuitionA good way to get a feel for the underlying mechanics is to plot the probability distribution of $z$ assuming that the null hypothesis is true. Then do the same assuming that the alternative hypothesis is true, and overlay the two plots.Consider the following example:* $H_0$: $p_a = p_b$* $H_1$: $p_a > p_b$$n_1 = 2500, n_2 = 2500, p_1 = 0.08, p_2 = 0.10$<jupyter_code>def plot_power(
    n1: int,
    n2: int,
    p1: float,
    p2: float,
    significance: float = 0.05
)-> plt.Figure:
    counts= np.array([p1*n1, p2*n2])
    nobs = np.array([n1, n2])
    zscore, _ = proportions_ztest(counts, nobs, alternative = 'larger')
    
    #calucalte distributions
    h0_dist = stats.norm(loc = 0, scale = 1)
    h1_dist = stats.norm(loc = zscore, scale = 1)
    
    # calculate threshold and power
    x = np.linspace(-5, 6, num = 100)
    threshold = h0_dist.ppf(1 - significance)
    mask = x > threshold
    power = np.round(1 - h1_dist.cdf(threshold), 2)
    
    #plot figure    
    fig = plt.figure(figsize=(8, 8))
    ax = plt.subplot(1, 1, 1)
    
    sns.lineplot(x=x, y=h1_dist.pdf(x), label="$H_1$ is true", ax=ax, color="navy")
    ax.fill_between(x = x[mask], y1 = 0.0, y2 = h1_dist.pdf(x)[mask],alpha = 0.2, color="navy")
    
    sns.lineplot(x=x, y=h0_dist.pdf(x), label="$H_0$ is true", ax=ax, color="green")
    ax.fill_between(x = x[mask], y1 = 0.0, y2 = h0_dist.pdf(x)[mask],alpha = 0.2, color="green")
    
    ax.fill_between(x = x[mask], y1 = 0.0, y2 = h1_dist.pdf(x)[mask],alpha = 0.2)
    ax.set_title(f"n1: {n1}, n2: {n2}, p1: {p1*100}%, p2: {p2*100}%, power: {power*100}%", fontsize=16)

    plt.show()

plot_power(n1=2500, n2=2500, p1=0.10, p2=0.08)<jupyter_output><empty_output><jupyter_text>The shaded green area denotes the significance region, while the shaded blue area denotes the power (note that it includes the shaded green area). Note that if we pick a smaller N, or a smaller probability difference between the control and experiment group, the power drops (the shaded blue area decreases), meaning that if there’s is in fact a change, there’s lesser percent chance that we’ll detect it.<jupyter_code>plot_power(n1=1250, n2=1250, p1=0.10, p2=0.08)  
plot_power(n1=2500, n2=2500, p1=0.10, p2=0.099)<jupyter_output><empty_output><jupyter_text>Power AnalysisStatistical power is one piece in a puzzle that has four related parts:* **Effect Size:** The quantified magnitude of a result present in the population* **Sample Size:** The number of observations in the sample.* **Significance:** The significance level used in the statistical test, e.g. alpha. Often set to 5% or 0.05.* **Statistical Power:** The probability of accepting the alternative hypothesis if it is true.Say we've followed the rule of thumb and required the significance level to be 5% and the power to be 80%. This means we have now specified the two key components of a power analysis.    A decision rule of when to reject the null hypothesis. We reject the null when the p-value is less than 5%.    Our tolerance for committing type 2 error (1−80%=20%).To actually solve for the equation of finding the suitable sample size, we also need to specify the detectable difference, i.e. the level of impact we want to be able to detect with our test.In order to explain the dynamics behind this, we'll return to the definition of power: the power is the probability of rejecting the null hypothesis when it is false. Hence for us to calculate the power, we need to define what "false" means to us in the context of the study. In other words, how much impact, i.e., difference between test and control, do we need to observe in order to reject the null hypothesis and conclude that the action worked?Let's consider two illustrative examples: if we think that an event rate reduction of, say, $10^{-10}$ is enough to reject the null hypothesis, then we need a very large sample size to get a power of 80%. This is pretty easy to deduce from the charts above: if the difference in event rates between test and control is a small number like $10^{-10}$, the null and alternative probability distributions will be nearly indistinguishable. Hence we will need to increase the sample size in order to move the alternative distribution to the right and gain power. Conversely, if we only require a reduction of 0.02 in order to claim success, we can make do with a much smaller sample size.    The smaller the detectable difference, the larger the required sample size  Student’s t Test Power Analysis<jupyter_code>alpha = 0.05
beta = 0.2
power = 1 - beta
effect = 0.80

# perform power analysis
analysis = TTestIndPower()
result = analysis.solve_power(
    effect, 
    power=power, 
    nobs1=None, 
    ratio=1.0, 
    alpha=alpha
)

print(f"Sample Size: {np.ceil(result)}")<jupyter_output>Sample Size: 26.0<jupyter_text>Power Curves<jupyter_code># parameters for power analysis
effect_sizes = np.array([0.2, 0.5, 0.8])
sample_sizes = np.array(range(5, 100))

analysis = TTestIndPower()
analysis.plot_power(dep_var='nobs', nobs=sample_sizes, effect_size=effect_sizes)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10545171339563864, 'fraction_numerical': 0.035358255451713394, 'mean_word_length': 3.763353115727003, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '511491', 'license': 'MIT', 'n_tokens_mistral': 2059, 'n_tokens_neox': 1905, 'n_words': 855, 'path': 'statistics/power_analysis.ipynb', 'repo_name': 'AndonisPavlidis/portfolio', 'size': '6420'}