<jupyter_start><jupyter_text>Two Samples Tests This notebook will use parametric and non-parametric statistical methods to compare two samples: independent and paired.<jupyter_code>import numpy as np
from scipy import stats as st
from statsmodels.stats import weightstats as stests
import matplotlib.pyplot as plt
import seaborn as sns;
sns.set_style("whitegrid")<jupyter_output><empty_output><jupyter_text>Parametric Tests for Two Independent Samples  Two independent samples t-test This notebook will use parametric and non-parametric statistical methods to compare two samples: independent and paired.The Independent Samples t-Test or 2-sample t-test compares the means of two independent groups to determine whether there is statistical evidence that the associated population means are significantly different. The Independent Samples t-Test is a parametric test. This test is also known as the Independent t-Test. **Example 1**: Two different methods taught two sections of a class in statistics. Students’ scores on a standardized test are shown below. Do the results present evidence of a difference in the effectiveness of the two methods?<jupyter_code># Generating the data
np.random.seed(123)
classA = np.random.normal(86,6,23)
classB = np.random.normal(88,5,25)
# Plotting densities 
ax=sns.kdeplot(x=classA, color='royalblue', shade=True, label='Class A')
ax=sns.kdeplot(x=classB, color='salmon', shade=True, alpha=0.5, label='Class B')
plt.legend();
def two_ind_ttest(sample1, sample2, alpha=0.05, alternative='two-sided'):
    '''
    It compares two independend groups.
    sample1:     sample of values of population 1 (normally distributed)
    sample2:     sample of values of population 2 (normally distributed)
    alpha:       significance level
    alternative: one of the three values: 'two-sided', 'smaller', and 'larger'  
    '''
    sigHa = {'two-sided':'!=', 'smaller':' <', 'larger':' >'}   
    print('--- Two independent sample t-test ---')
    print('    Ho: mean(sample1)  =', 'mean(sample2)')
    print('    Ha: mean(sample1)', sigHa[alternative], 'mean(sample2)')
    print('Sample 1:   n = %i\tMean = %.2f\tStdDev = %.2f' %(len(sample1),np.mean(sample1), np.std(sample1)))
    print('Sample 2:   n = %i\tMean = %.2f,\tStdDev = %.2f' %(len(sample2),np.mean(sample2), np.std(sample2)))
    t_stat, pval, df = stests.ttest_ind(x1=sample1, x2=sample2, usevar='unequal', alternative=alternative)
    print('t-value = %.3f    df = %i    p-value = %.4f' %(t_stat,  df, pval))
    if pval < alpha:    
        print("%.4f <= %.2f We are rejecting null hypothesis." %(pval, alpha))
    else:
        print("%.4f > %.2f We have no reason to reject the null hypothesis." %(pval, alpha))
two_ind_ttest(classA, classB)
two_ind_ttest(classA, classB, alternative='smaller')<jupyter_output>--- Two independent sample t-test ---
    Ho: mean(sample1)  = mean(sample2)
    Ha: mean(sample1)  < mean(sample2)
Sample 1:   n = 23	Mean = 86.93	StdDev = 7.23
Sample 2:   n = 25	Mean = 86.77,	StdDev = 5.42
t-value = 0.085    df = 40    p-value = 0.5336
0.5336 > 0.05 We have no reason to reject the null hypothesis.<jupyter_text>Two independent samples z-test A `z-test` is a statistical test used to determine whether two population means are different when the variances are known and with large sample size. Usually, 30 cases per group are enough.<jupyter_code>from statsmodels.stats import weightstats as stests<jupyter_output><empty_output><jupyter_text>We would use a `z-test` if:- Our sample size is greater than 30. Otherwise, use a t-test.- Data points should be independent of each other- Our data should be normally distributed. - Our data should be randomly selected from a population where each item has an equal chance of being selected.In two samples `z-test,` or `t-test`, we are checking two independent data groups and deciding whether the sample means are similar or not.<jupyter_code>def two_ind_ztest(sample1, sample2, alpha=0.05, alternative='two-sided'):
    '''
    Compare two independend groups
    sample1:     sample of values of population 1 (normally distributed)
    sample2:     sample of values of population 2 (normally distributed)
    alpha:       significance level
    alternative: one of the three values: 'two-sided', 'smaller', and 'larger'      
    '''
    sigHa = {'two-sided':'!=', 'smaller':' <', 'larger':' >'}       
    print('--- Two independent sample z-test ---')
    print('    Ho: mean(sample1)  =', 'mean(sample2)')
    print('    Ha: mean(sample1)', sigHa[alternative], 'mean(sample2)')
    print('Sample 1:   n = %i\tMean = %.2f\tStdDev = %.2f' %(len(sample1),np.mean(sample1), np.std(sample1)))
    print('Sample 2:   n = %i\tMean = %.2f,\tStdDev = %.2f' %(len(sample2),np.mean(sample2), np.std(sample2)))
    z_stat, pval = stests.ztest(sample1, sample2, value=0, alternative=alternative)
    print('z-value = %.3f    p-value = %.4f' %(z_stat, pval))
    if pval < alpha:    
        print("%.4f <= %.2f We are rejecting null hypothesis" %(pval, alpha))
    else:
        print("%.4f > %.2f We have no reason to reject the null hypothesis" %(pval, alpha))<jupyter_output><empty_output><jupyter_text>Let's generate values for classes C and D:<jupyter_code>np.random.seed(123)
classC = np.random.normal(80, 3, 100)
classD = np.random.normal(90, 3, 95)
ax=sns.kdeplot(x=classC, color='limegreen', shade=True, label='Class C')
ax=sns.kdeplot(x=classD, color='orange', shade=True, alpha=0.3, label='Class D')
plt.legend();<jupyter_output><empty_output><jupyter_text>Let's compare class C and D with the z-test:<jupyter_code>two_ind_ztest(classC, classD)
two_ind_ztest(classC, classD, alternative='smaller')
two_ind_ztest(classC, classD, alternative='larger')<jupyter_output>--- Two independent sample z-test ---
    Ho: mean(sample1)  = mean(sample2)
    Ha: mean(sample1)  > mean(sample2)
Sample 1:   n = 100	Mean = 80.08	StdDev = 3.38
Sample 2:   n = 95	Mean = 90.07,	StdDev = 2.81
z-value = -22.243    p-value = 1.0000
1.0000 > 0.05 We have no reason to reject the null hypothesis<jupyter_text>Non-paramnetric Tests for Two Independent Samples  Mann-Whitney U Test It is a non-parametric test.Mann-Whitney tests whether the distributions of two independent samples are equal or not.Assumptions- Observations in each sample are independent and identically distributed (iid).- Observations in each sample can be ranked.Interpretation- Ho: the distributions of both samples are equal.- Ha: the distributions of both samples are not equal.<jupyter_code>def m_w(sample1, sample2, alpha=0.05, alternative='two-sided'):
    '''
    The Mann-Whitney U test is a non-parametric two independent groups.
    sample1: sample of values of population 1 
    sample2: sample of values of population 2 
    alpha: significance level
    alternative: one of the three values: 'two-sided', 'smaller', and 'larger'
    '''
    alt={'two-sided':'two-sided','smaller':'less','larger':'greater'}
    sigHa = {'two-sided':'!=', 'smaller':' <', 'larger':' >'}
    print('--- Non-parametric Mann-Whitney U Test ---')
    print('    Ho: mean_rank(sample1)  =', 'mean_rank(sample2)')
    print('    Ha: mean_rank(sample1)', sigHa[alternative], 'mean_rank(sample2)')
    print('Sample 1:  n = %i\tMean = %.2f\tStdDev = %.2f' %(len(sample1),np.mean(sample1), np.std(sample1)))
    print('Sample 2:  n = %i\tMean = %.2f\tStdDev = %.2f' %(len(sample2),np.mean(sample2), np.std(sample2)))
    stat, pval = st.mannwhitneyu(sample1, sample2, alternative=alt[alternative])
    print('stat = %.2f    p-value = %.4f' %(stat, pval))
    if pval < alpha:    
        print("%.4f <= %.2f We are rejecting null hypothesis" %(pval, alpha))
    else:
        print("%.4f > %.2f We have no reason to reject the null hypothesis" %(pval, alpha))<jupyter_output><empty_output><jupyter_text>Remember Mann-Whitney U is a nonparametric test. That means the test does not work with original data. It works with ranks. Non-parametric tests are helpful when the usual analysis of variance assumption of normality is not viable.<jupyter_code>m_w(classA, classB)
m_w(classC, classD)
m_w(classC, classD, alternative='smaller')<jupyter_output>--- Non-parametric Mann-Whitney U Test ---
    Ho: mean_rank(sample1)  = mean_rank(sample2)
    Ha: mean_rank(sample1)  < mean_rank(sample2)
Sample 1:  n = 100	Mean = 80.08	StdDev = 3.38
Sample 2:  n = 95	Mean = 90.07	StdDev = 2.81
stat = 106.00    p-value = 0.0000
0.0000 <= 0.05 We are rejecting null hypothesis<jupyter_text>Kolmogorov Smirnov Test The Kolmogorov-Smirnov test (KS-test) determines if two datasets differ significantly without assuming anything about the data distribution. - Ho: the distributions of both samples are equal.- Ha: the distributions of both samples are not equal.<jupyter_code>def k_s(sample1, sample2, alpha=0.05, alternative='two-sided'):
    '''
    The Kolmogorov Smirnov test is a non-parametric two independent groups.
    sample1:     sample of values of population 1 
    sample2:     sample of values of population 2 
    alpha:       significance level
    alternative: one of the three values: 'two-sided', 'smaller', and 'larger'
    '''
    alt={'two-sided':'two-sided','smaller':'less','larger':'greater'}
    sigHa = {'two-sided':'!=', 'smaller':' <', 'larger':' >'}
    print('--- Non-parametric Kolmogorov Smirnov Test ---')
    print('    Ho: mean_rank(sample1)  =', 'mean_rank(sample2)')
    print('    Ha: mean_rank(sample1)', sigHa[alternative], 'mean_rank(sample2)')
    print('Sample 1:  n = %i\tMean = %.2f\tStdDev = %.2f' %(len(sample1),np.mean(sample1), np.std(sample1)))
    print('Sample 2:  n = %i\tMean = %.2f\tStdDev = %.2f' %(len(sample2),np.mean(sample2), np.std(sample2)))
    stat, pval = st.ks_2samp(sample1, sample2, alternative=alt[alternative])
    print('stat = %.2f    p-value = %.4f' %(stat, pval))
    if pval < alpha:    
        print("%.4f <= %.2f We are rejecting null hypothesis" %(pval, alpha))
    else:
        print("%.4f > %.2f We have no reason to reject the null hypothesis" %(pval, alpha))
k_s(classA, classB)
k_s(classC, classD)
k_s(classC, classD, alternative='smaller')<jupyter_output>--- Non-parametric Kolmogorov Smirnov Test ---
    Ho: mean_rank(sample1)  = mean_rank(sample2)
    Ha: mean_rank(sample1)  < mean_rank(sample2)
Sample 1:  n = 100	Mean = 80.08	StdDev = 3.38
Sample 2:  n = 95	Mean = 90.07	StdDev = 2.81
stat = 0.00    p-value = 1.0000
1.0000 > 0.05 We have no reason to reject the null hypothesis<jupyter_text>Parametric Tests for Two Paired Samples  Paired sample t-test The paired sample t-test is also called dependent sample t-test. It is a univariate test that tests for a significant difference between two related variables. An example of this is if we were to compare students' grades before and after some training or courses.- Ho: the difference in the means between two samples is 0- Ha: the difference in the means between two samples is different from 0 Generating fictitious data.<jupyter_code>np.random.seed(123)
grade_before = np.random.normal(60,7,20)
grade_after  = np.random.normal(85,5,20)<jupyter_output><empty_output><jupyter_text>Presenting some visualizations<jupyter_code>sns.jointplot(x=grade_before, y=grade_after);<jupyter_output><empty_output><jupyter_text>Let's create a function to use a t-test for our paired data.<jupyter_code>def two_rel_ttest(sample1, sample2, alpha=0.05, alternative='two-sided'):
    '''
    Compare paired groups
    sample1:     sample of values at time 1 (normally distributed)
    sample2:     sample of values at time 2 (normally distributed)
    alpha:       significance level
    alternative: one of the three values: 'two-sided', 'smaller', and 'larger'
    '''
    alt={'two-sided':'two-sided','smaller':'less','larger':'greater'}
    sigH1 = {'two-sided':'!=', 'smaller':' <', 'larger':' >'}
    print('--- Two paired sample t-test ---')
    print('    Ho: mean(sample1)  = mean(sample2)')
    print('    Ha: mean(sample1)', sigH1[alternative], 'mean(sample2)')
    print('Sample 1:  n =%i\tMean = %.2f\tStdDev = %.2f' %(len(sample1),np.mean(sample1), np.std(sample1)))
    print('Sample 2:  n =%i\tMean = %.2f,\tStdDev = %.2f' %(len(sample2),np.mean(sample2), np.std(sample2)))
    t_stat, pval = st.ttest_rel(sample1, sample2)
    print('t-value = %.2f    p-value = %.4f' %(t_stat, pval))
    if pval < alpha:    
        print("%.4f <= %.2f We are rejecting null hypothesis" %(pval, alpha))
    else:
        print("%.4f > %.2f We have no reason to reject the null hypothesis" %(pval, alpha))
two_rel_ttest(grade_before, grade_after)
two_rel_ttest(grade_before, grade_after, alternative='smaller')<jupyter_output>--- Two paired sample t-test ---
    Ho: mean(sample1)  = mean(sample2)
    Ha: mean(sample1)  < mean(sample2)
Sample 1:  n =20	Mean = 60.80	StdDev = 8.58
Sample 2:  n =20	Mean = 83.59,	StdDev = 5.34
t-value = -9.30    p-value = 0.0000
0.0000 <= 0.05 We are rejecting null hypothesis<jupyter_text>As we can see, we are rejecting the null hypothesis. There are differences between the grades before and after.Let us generate grade_after2, with values similar to grade_before.<jupyter_code>grade_after2 = np.random.normal(64,5,20)
two_rel_ttest(grade_before, grade_after2)<jupyter_output>--- Two paired sample t-test ---
    Ho: mean(sample1)  = mean(sample2)
    Ha: mean(sample1) != mean(sample2)
Sample 1:  n =20	Mean = 60.80	StdDev = 8.58
Sample 2:  n =20	Mean = 66.23,	StdDev = 5.80
t-value = -2.35    p-value = 0.0300
0.0300 <= 0.05 We are rejecting null hypothesis<jupyter_text>p-value > alpha, therefore there are not significant differences.  Non-parametric Test for Two Paired Data  Wilcoxon Signed-Rank Test It is a non-parametric test.It tests whether the distributions of two paired samples are equal or not.Assumptions- Observations in each sample are independent and identically distributed (iid).- Observations in each sample can be ranked.- Observations across each sample are paired.Interpretation- Ho: the distributions of both samples are equal.- Ha: the distributions of both samples are not equal.<jupyter_code>def wilcoxon(sample1, sample2, alpha=0.05, alternative='two-sided'):
    '''
    The Wilcoxon test is a nonparametric statistical test that compares two paired groups.
    sample1: sample of values at time 1 
    sample2: sample of values at time 2 
    alpha:   significance level
    '''
    alt={'two-sided':'two-sided','smaller':'less','larger':'greater'}
    sigH1 = {'two-sided':'!=', 'smaller':' <', 'larger':' >'}
    print('--- Non-parametric Wilcoxon Signed-Rank Test ---')
    print('    Ho: mean_rank(sample1)  = mean_rank(sample2)')
    print('    Ha: mean_rank(sample1)', sigH1[alternative], 'mean_rank(sample2)')
    print('Sample 1:\tn =%i\tMean = %.2f\tStdDev = %.2f' %(len(sample1),np.mean(sample1), np.std(sample1)))
    print('Sample 2:\tn =%i\tMean = %.2f,\tStdDev = %.2f' %(len(sample2),np.mean(sample2), np.std(sample2)))
    stat, pval = st.wilcoxon(sample1, sample2)
    print('stat = %.3f    p-value = %.4f' %(stat, pval))
    if pval < alpha:    
        print("%.4f <= %.2f We are rejecting null hypothesis" %(pval, alpha))
    else:
        print("%.4f > %.2f We have no reason to reject the null hypothesis" %(pval, alpha))
wilcoxon(grade_before, grade_after)
wilcoxon(grade_before, grade_after, alternative='smaller')
wilcoxon(grade_before, grade_after2)<jupyter_output>--- Non-parametric Wilcoxon Signed-Rank Test ---
    Ho: mean_rank(sample1)  = mean_rank(sample2)
    Ha: mean_rank(sample1) != mean_rank(sample2)
Sample 1:	n =20	Mean = 60.80	StdDev = 8.58
Sample 2:	n =20	Mean = 66.23,	StdDev = 5.80
stat = 48.000    p-value = 0.0328
0.0328 <= 0.05 We are rejecting null hypothesis
{'fraction_non_alphanumeric': 0.13437400446001913, 'fraction_numerical': 0.03561643835616438, 'mean_word_length': 4.07140549273021, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '602503', 'license': 'MIT', 'n_tokens_mistral': 5602, 'n_tokens_neox': 5160, 'n_words': 1624, 'path': '04-Inferential Statistics/06_Two_Samp_Test.ipynb', 'repo_name': 'Goliath-Research/Computational-Statistics', 'size': '15695'}