<jupyter_start><jupyter_text>Hypothesis Testing1. **t-test independent:** parametric version of 2 groups paired data2. **ANOVA:** parametric version of more than 2 groups and unpaired data - To find which group or groups cause the difference, we need to perform a *posthoc test/pairwise comparison* e.g.`Bonferroni adjustment`3. **<NAME>:** nonparametric version of 2 group comparison for unpaired data4. **Kruskal-Wallis:** the nonparametric version of ANOVA for unpaired data (the data is collected from different sources)5. **t-test dependent:** The data is paired since data is collected from the same individuals and assumptions are satisfied6. **Wilcoxon signed-rank test:** The normality assumption is not satisfied; therefore, we need to use the nonparametric version for 2 groups of paired data7. **<NAME>:** nonparametric version of ANOVA for more than 2 groups paired data  Q1. t-test independent**Background Story:** A university professor gave online lectures instead of face-to-face classes due to Covid-19. Later, he uploaded recorded lectures to the cloud for students who followed the course asynchronously (those who did not attend the lesson but later watched the records). However, he believes that the students who attend class at the class time and participate in the process are more successful. Therefore, he recorded the average grades of the students at the end of the semester. The data is below.- synchronous = [94. , 84.9, 82.6, 69.5, 80.1, 79.6, 81.4, 77.8, 81.7, 78.8, 73.2, 87.9, 87.9, 93.5, 82.3, 79.3, 78.3, 71.6, 88.6, 74.6, 74.1, 80.6]- asynchronous = [77.1, 71.7, 91. , 72.2, 74.8, 85.1, 67.6, 69.9, 75.3, 71.7, 65.7, 72.6, 71.5, 78.2]**Goal:** Conduct the hypothesis testing to check whether the professor's belief is statistically significant by using 0.05 significance level to evaluate the null and alternative hypotheses. Before doing hypothesis testing, check the related assumptions.  1. Defining HypothesisSince the grades are obtained from the different individuals, the data is unpaired.- H₀: μₛ≤μₐ- H₁: μₛ>μₐ  2. Assumption Check- Observations in each sample are independent and identically distributed (IID).- Observations in each sample are normally distributed.- Observations in each sample have the same variance. Independence check:- H₀: The data is normally distributed.- H₁: The data is not normally distributed. Normality check:- Shapiro-Wilk’s W test which is generally preferred for **smaller samples** however there are other options like Kolmogorov-Smirnov and D’Agostino and Pearson’s test.- **Shapiro-Wilk Test:** A statistical test used to check if a continuous variable follows a normal distribution. The null hypothesis states that the variable normally distributed, and the alternative hypothesis states that the variable is NOT normally distributed.<jupyter_code>def check_normality(data):
    test_stat_normality, p_value_normality = stats.shapiro(data)
    print("p value:%.4f" % p_value_normality)
    if p_value_normality < 0.05:
        print("Reject null hypothesis >> The data is not normally distributed")
    else:
        print('Fail to reject null hypothesis >> The data is normally distributed')
import numpy as np
from scipy import stats

sync = np.array([94. , 84.9, 82.6, 69.5, 80.1, 79.6, 81.4, 77.8, 81.7, 78.8, 73.2,
       87.9, 87.9, 93.5, 82.3, 79.3, 78.3, 71.6, 88.6, 74.6, 74.1, 80.6])
asyncr = np.array([77.1, 71.7, 91. , 72.2, 74.8, 85.1, 67.6, 69.9, 75.3, 71.7, 65.7, 72.6, 71.5, 78.2])
check_normality(sync)
check_normality(asyncr)<jupyter_output>p value:0.6556
Fail to reject null hypothesis >> The data is normally distributed
p value:0.0803
Fail to reject null hypothesis >> The data is normally distributed<jupyter_text>Variance check:- H₀: The variances of the samples are the same.- H₁: The variances of the samples are different.<jupyter_code># Levene's test 
def check_variance_homogeneity(group1, group2):
    test_stat_var, p_value_var = stats.levene(group1, group2)
    print("p value:%.4f" % p_value_var)
    if p_value_var < 0.05:
        print('Reject null hypothesis >> The variance of the samples are different.')
    else:
        print('Fail to reject null hypothesis >> The variance of the samples are same.')<jupyter_output><empty_output><jupyter_text>It tests the null hypothesis that the the population variance are equal (called homogeneity of variance or homoscedasticity). Suppose the resulting p-value of Levene's test is less than the siginificant level (typically 0.05). In that case, the obtained differences in sample variances are unlikely to occurred based on random sampling from a population with equal variances. **Note:** If we want to compare 2(+) groups on a quantitative variable, we usually want to know if they have equal mean scores. For this we often use:- an independent samples t-test for 2 groups- a one-way ANOVA for 3(+) groupsBoth tests requre the homogeneity assumption: the population variances of the dependent variable must equal for all groups. You can ignore this assumption if you have roughly equal sample sizes for each group. However, if you have sharply different sample sizes, then you do need to make sure that homogeneity of variances is met by your data. **That's why we need to check variance homogeneity here.**<jupyter_code>check_variance_homogeneity(sync, asyncr)<jupyter_output>p value:0.8149
Fail to reject null hypothesis >> The variance of the samples are same.<jupyter_text>Selecting the Proper TestSince assumptions are satisfied, we can perform the parametric version of the test for 2 groups and unpaired data. ![1_LYkUpSAgFPuxlLFjGAUMbA.png](attachment:1_LYkUpSAgFPuxlLFjGAUMbA.png)<jupyter_code>ttest, p_value = stats.ttest_ind(sync, asyncr)
print("p value:%.8f" % p_value)
print("since the hypothesis is one sided >> use p_value/2 >> p_value_one_sided:%.4f" % (p_value/2))
if p_value/2 < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')<jupyter_output>p value:0.00753598
since the hypothesis is one sided >> use p_value/2 >> p_value_one_sided:0.0038
Reject null hypothesis<jupyter_text>4. Decision and Conclusion At this significance level, there is enough evidence to conclude that the average grade of students who follow the course synchronously is higher than the students who follow the course asynchronously.  Q2. ANOVA**Background Story:** A pediatrician wants to see the effect of formula consumption on the average monthly weight gain (in gr) of babies. For this reason, she collected data from three different groups. The first group is exclusively breastfed children (receives only breast milk), the second group is children who are fed with only formula and the last group is both formula and breastfed children. These data are as below.- only_breast = [794.1, 716.9, 993. , 724.7, 760.9, 908.2, 659.3 , 690.8, 768.7, 717.3 , 630.7, 729.5, 714.1, 810.3, 583.5, 679.9, 865.1]- only_formula = [ 898.8, 881.2, 940.2, 966.2, 957.5, 1061.7, 1046.2, 980.4, 895.6, 919.7, 1074.1, 952.5, 796.3, 859.6, 871.1 , 1047.5, 919.1 , 1160.5, 996.9]- both = [976.4, 656.4, 861.2, 706.8, 718.5, 717.1, 759.8, 894.6, 867.6, 805.6, 765.4, 800.3, 789.9, 875.3, 740. , 799.4, 790.3, 795.2 , 823.6, 818.7, 926.8, 791.7, 948.3]**Goal:** According to this information, conduct the hypothesis testing to check whether there is a difference between the average monthly gain of these three groups by using a 0.05 significance level. If there is a significant difference, perform further analysis to find what caused the difference. Before doing hypothesis testing, check the related assumptions.<jupyter_code>only_breast = np.array([794.1, 716.9, 993. , 724.7, 760.9, 908.2, 659.3 , 690.8, 768.7, 717.3 , 630.7, 729.5, 714.1, 810.3, 583.5, 679.9, 865.1])

only_formula = np.array([ 898.8, 881.2, 940.2, 966.2, 957.5, 1061.7, 1046.2, 980.4, 895.6, 919.7, 1074.1, 952.5, 796.3, 859.6, 871.1 , 1047.5, 919.1 , 1160.5, 996.9])

both = np.array([976.4, 656.4, 861.2, 706.8, 718.5, 717.1, 759.8, 894.6, 867.6, 805.6, 765.4, 800.3, 789.9, 875.3, 740. , 799.4, 790.3, 795.2 , 823.6, 818.7, 926.8, 791.7, 948.3])<jupyter_output><empty_output><jupyter_text>1. Defining Hypothesis- H₀: μ₁=μ₂=μ₃ or The mean of the samples is the same.- H₁: At least one of them is different.  2. Assumption Check - H₀: The data is normally distributed.- H₁: The data is not normally distributed.- H₀: The variances of the samples are the same.- H₁: The variances of the samples are different.<jupyter_code>check_normality(only_breast)
check_normality(only_formula)
check_normality(both)

stat, pvalue_levene = stats.levene(only_breast, only_formula, both)
print(" p value:%.4f" % pvalue_levene)
if pvalue_levene < 0.05:
    print('Reject null hypothesis >> The variance of the samples are different.')
else:
    print('Fail to reject null hypothesis >> The variance of the samples are same.')<jupyter_output>p value:0.4694
Fail to reject null hypothesis >> The data is normally distributed
p value:0.8879
Fail to reject null hypothesis >> The data is normally distributed
p value:0.7973
Fail to reject null hypothesis >> The data is normally distributed
 p value:0.7673
Fail to reject null hypothesis >> The variance of the samples are same.<jupyter_text>3. Selecting the Proper TestSince assumptions are satisfied, we can perform the parametric version of the test for more than 2 groups and unpaired data.<jupyter_code>F, p_value = stats.f_oneway(only_breast, only_formula, both)
print("p value:%.6f" % p_value)
if p_value < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')<jupyter_output>p value:0.000000
Reject null hypothesis<jupyter_text>4. Decision and ConclusionAt this significance level, it can be concluded that at least one of the groups has a different average monthly weight gain. To find which group or groups cause the difference, we need to perform a **posthoc test/pairwise comparison** as below.**Note:** - To avoid family-wise p-value inflation, here uses *Bonferroni adjustment*.- *Pairwise Comparison*: the process of comparing candidates in pairs to judge which of each candidate is preferred overall.<jupyter_code>#pip install scikit-posthocs
# Pairwise T test for multiple comparisons of independent groups. 
# It may be used after a parametric ANOVA to do pairwise comparisons.
import scikit_posthocs as sp
posthoc_df = sp.posthoc_ttest([only_breast, only_formula, both], equal_var = True, p_adjust = 'bonferroni')

group_names = ['only breast', 'only formula','both']
posthoc_df.columns = group_names
posthoc_df.index = group_names
posthoc_df.style.applymap(lambda x: 'background-color:violet' if x<0.05 else 'background-color:white')<jupyter_output><empty_output><jupyter_text>At this significant level, it can be concluded that:- 'only breast' is different than 'only formula'- 'only formula' is different than both 'only breast' and 'both'- 'both' is different than 'only formula'  Q3 <NAME> **Background Story:** A human resource specialist working in a technology company is interested in the overwork time of different teams. To investigate whether there is a difference between overtime of the software development team and the test team, she selected 17 employees randomly in each of the two teams and recorded their weekly average overwork time in terms of an hour. The data is below.- test_team=[6.2, 7.1, 1.5, 2,3 , 2, 1.5, 6.1, 2.4, 2.3, 12.4, 1.8, 5.3, 3.1, 9.4, 2.3, 4.1]- developer_team=[2.3, 2.1, 1.4, 2.0, 8.7, 2.2, 3.1, 4.2, 3.6, 2.5, 3.1, 6.2, 12.1, 3.9, 2.2, 1.2 ,3.4]**Goal:** According to this information, conduct the hypothesis testing to check whether there is a difference between the overwork time of two teams by using a 0.05 significance level. Before doing hypothesis testing, check the related assumptions.<jupyter_code>test_team = np.array([6.2, 7.1, 1.5, 2,3 , 2, 1.5, 6.1, 2.4, 2.3, 12.4, 1.8, 5.3, 3.1, 9.4, 2.3, 4.1])
developer_team = ([2.3, 2.1, 1.4, 2.0, 8.7, 2.2, 3.1, 4.2, 3.6, 2.5, 3.1, 6.2, 12.1, 3.9, 2.2, 1.2 ,3.4])<jupyter_output><empty_output><jupyter_text>1. Defining Hypothesis- H₀: μ₁≤μ₂- H₁: μ₁>μ₂  2. Assumption Check- H₀: The data is normally distributed.- H₁: The data is not normally distributed.- H₀: The variances of the samples are the same.- H₁: The variances of the samples are different.<jupyter_code>check_normality(test_team)
check_normality(developer_team)
check_variance_homogeneity(test_team, developer_team)<jupyter_output>p value:0.0046
Reject null hypothesis >> The data is not normally distributed
p value:0.0005
Reject null hypothesis >> The data is not normally distributed
p value:0.5410
Fail to reject null hypothesis >> The variance of the samples are same.<jupyter_text>3. Selecting the Proper TetsThere are two groups, and data is collected from different individuals, so it is not paired. However, the normality assumption is not satisfied; therefore, we need to use the **nonparametric** version of 2 group comparison for unpaired data: the Mann-Whitney U Test.<jupyter_code>ttest, pvalue = stats.mannwhitneyu(test_team, developer_team, alternative = 'two-sided')
print('p-value:%.4f' % pvalue)
if pvalue < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')<jupyter_output>p-value:0.8226
Fail to reject null hypothesis<jupyter_text>At this significant level, it can be said that there is no statistically significant difference between the average overwork time of the two teams.  Q4. Kruskal-Wallis **Background Story:** An e-commerce company regularly advertises on YouTube, Instagram, and Facebook for its campaigns. However, the new manager was curious about if there was any difference between the number of customers attracted by these platforms. Therefore, she started to use Adjust, an application that allows you to find out where your users come from. The daily numbers reported from Adjust for each platform are as below.- Youtube=[1913, 1879, 1939, 2146, 2040, 2127, 2122, 2156, 2036, 1974, 1956, 2146, 2151, 1943, 2125]- Instagram = [2305., 2355., 2203., 2231., 2185., 2420., 2386., 2410., 2340., 2349., 2241., 2396., 2244., 2267., 2281.]- Facebook = [2133., 2522., 2124., 2551., 2293., 2367., 2460., 2311., 2178., 2113., 2048., 2443., 2265., 2095., 2528.]**Goal:** According to this information, conduct the hypothesis testing to check whether there is a difference between the average customer acquisition of these three platforms using a 0.05 significance level. If there is a significant difference, perform further analysis to find that caused the difference. Before doing hypothesis testing, check the related assumptions.<jupyter_code>youtube = np.array([1913, 1879, 1939, 2146, 2040, 2127, 2122, 2156, 2036, 1974, 1956, 2146, 2151, 1943, 2125])
instagram = np.array([2305., 2355., 2203., 2231., 2185., 2420., 2386., 2410., 2340., 2349., 2241., 2396., 2244., 2267., 2281.])
facebook = np.array([2133., 2522., 2124., 2551., 2293., 2367., 2460., 2311., 2178., 2113., 2048., 2443., 2265., 2095., 2528.])<jupyter_output><empty_output><jupyter_text>1. Defining Hypothesis - H₀: μ₁=μ₂=μ₃ or The mean of the samples is the same.- H₁: At least one of them is different.  2. Assumption Check- H₀: The data is normally distributed.- H₁: The data is not normally distributed.- H₀: The variances of the samples are the same.- H₁: The variances of the samples are different.<jupyter_code>check_normality(youtube)
check_normality(instagram)
check_normality(facebook)

stat, pvalue_levene = stats.levene(youtube, instagram, facebook)

print("p value:%.4f" % pvalue_levene)
if pvalue_levene < 0.05:
    print('Reject null hypothesis >> The variance of the samples are different.')
else:
    print('Fail to reject null hypothesis >> The variance of the samples are same.')<jupyter_output>p value:0.0285
Reject null hypothesis >> The data is not normally distributed
p value:0.4156
Fail to reject null hypothesis >> The data is normally distributed
p value:0.1716
Fail to reject null hypothesis >> The data is normally distributed
p value:0.0012
Reject null hypothesis >> The variance of the samples are different.<jupyter_text>3. Selecting the Proper TestThe normality and variance homogeneity assumptions are not satisfied, therefore we need to use the nonparametric version of ANOVA for unpaired data (the data is collected from different sources).<jupyter_code>F, p_value = stats.kruskal(youtube, instagram, facebook)
print("p value:%.6f" % p_value)
if p_value < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')<jupyter_output>p value:0.000015
Reject null hypothesis<jupyter_text>At this significance level, at least one of the average customer acquisition number is different.**Note:** Since the data is not normal, the nonparametric version of posthoc test is used.<jupyter_code>posthoc_df = sp.posthoc_mannwhitney([youtube,instagram, facebook], p_adjust = 'bonferroni')
group_names= ["youtube", "instagram","facebook"]
posthoc_df.columns= group_names
posthoc_df.index= group_names
posthoc_df.style.applymap(lambda x: "background-color:violet" if x<0.05 else "background-color: white")<jupyter_output><empty_output><jupyter_text>The average number of customers coming from YouTube is different than the other (actually smaller than the others).  5. t-test dependent**Background Story:** The University Health Center diagnosed eighteen students with high cholesterol in the previous semester. Healthcare personnel told these patients about the dangers of high cholesterol and prescribed a diet program. One month later, the patients came for control, and their cholesterol level was reexamined. Test whether there is a difference in the cholesterol levels of the patients.- test_results_before_diet=[224, 235, 223, 253, 253, 224, 244, 225, 259, 220, 242, 240, 239, 229, 276, 254, 237, 227]- test_results_after_diet=[198, 195, 213, 190, 246, 206, 225, 199, 214, 210, 188, 205, 200, 220, 190, 199, 191, 218]**Goal:** According to this information, conduct the hypothesis testing to check whether there is a decrease in the cholesterol levels of the patients after the diet by using a 0.05 significance level. Before doing hypothesis testing, check the related assumptions. Comment on the results.<jupyter_code>test_results_before_diet=np.array([224, 235, 223, 253, 253, 224, 244, 225, 259, 220, 242, 240, 239, 229, 276, 254, 237, 227])
test_results_after_diet = np.array([198, 195, 213, 190, 246, 206, 225, 199, 214, 210, 188, 205, 200, 220, 190, 199, 191, 218])<jupyter_output><empty_output><jupyter_text>1. Defining Hypothesis- H₀: μd>=0 or The true mean difference is equal to or bigger than zero.- H₁: μd<0 or The true mean difference is smaller than zero.  2. Assumption Check- The dependent variable must be continuous (interval/ratio)- The observations are independent of one another.- The dependent variable should be approximately normally distributed.- H₀: The data is normally distributed.- H₁: The data is not normally distributed.<jupyter_code>check_normality(test_results_before_diet)
check_normality(test_results_after_diet)<jupyter_output>p value:0.1635
Fail to reject null hypothesis >> The data is normally distributed
p value:0.1003
Fail to reject null hypothesis >> The data is normally distributed<jupyter_text>3. Selecting the Proper TestThe data is paired since data is collected from the same individuals and assumptions are satisfied, then we can use the dependent t-test.<jupyter_code>test_stat, p_value_paired = stats.ttest_rel(test_results_before_diet, test_results_after_diet)
print("p value:%.6f" % p_value_paired, "one tailed p value:%.6f" %(p_value_paired/2))

if p_value_paired < 0.05:
    print('Reject null hypothesis')
else:
    print('Fall to reject null hypothesis')<jupyter_output>p value:0.000008 one tailed p value:0.000004
Reject null hypothesis<jupyter_text>4. Decision and ConclusionAt this siginificant level, there is enough evidence to conclude mean cholesterol level of patients has decreased after the diet.  6. Wilcoxon signed-rank test**Background Story:** A venture capitalist wanted to invest in a startup that provides data compression without any loss in quality, but there are two competitors: PiedPiper and EndFrame. Initially, she believed the performance of the EndFrame could be better but still wanted to test it before the investment. Then, she gave the same files to each company to compress and recorded their performance scores. The data is below.- piedpiper=[4.57, 4.55, 5.47, 4.67, 5.41, 5.55, 5.53, 5.63, 3.86, 3.97, 5.44, 3.93, 5.31, 5.17, 4.39, 4.28, 5.25]- endframe = [4.27, 3.93, 4.01, 4.07, 3.87, 4. , 4. , 3.72, 4.16, 4.1 , 3.9 , 3.97, 4.08, 3.96, 3.96, 3.77, 4.09]**Goal:** According to this information, conduct the related hypothesis testing by using a 0.05 significance level. Before doing hypothesis testing, check the related assumptions. Comment on the results.<jupyter_code>piedpiper=np.array([4.57, 4.55, 5.47, 4.67, 5.41, 5.55, 5.53, 5.63, 3.86, 3.97, 5.44, 3.93, 5.31, 5.17, 4.39, 4.28, 5.25])
endframe = np.array([4.27, 3.93, 4.01, 4.07, 3.87, 4. , 4. , 3.72, 4.16, 4.1 , 3.9 , 3.97, 4.08, 3.96, 3.96, 3.77, 4.09])<jupyter_output><empty_output><jupyter_text>1. Defining HypothesisSince the performance scores are obtained from the same files, the data is paired.- H₀: μd>=0 or The true mean difference is equal to or bigger than zero.- H₁: μd<0 or The true mean difference is smaller than zero.  2. Assumption Check- The dependent variable must be continuous (interval/ratio)- The observations are independent of one another.- The dependent variable should be approximately normally distributed.- H₀: The data is normally distributed.- H₁: The data is not normally distributed.<jupyter_code>check_normality(piedpiper)
check_normality(endframe)<jupyter_output>p value:0.0304
Reject null hypothesis >> The data is not normally distributed
p value:0.9587
Fail to reject null hypothesis >> The data is normally distributed<jupyter_text>3. Selecting the Proper TestThe normality assumption is not satisfied; therefore, we need to use the nonparametric version of the paired test, namely the Wilcoxon Signed Rank test.<jupyter_code>test, pvalue = stats.wilcoxon(endframe, piedpiper) # alternative defalt two sided
print('p-value:%.6f' % pvalue,'>> one_tailed_pval:%.6f'%(pvalue/2))

test,one_sided_pvalue = stats.wilcoxon(endframe, piedpiper, alternative = 'less')
print('one sided pvalue:%.6f' % (one_sided_pvalue))

if pvalue < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')<jupyter_output>p-value:0.000214 >> one_tailed_pval:0.000107
one sided pvalue:0.000107
Reject null hypothesis<jupyter_text>4. Decision and ConclusionAt this significance level, there is enough evidence to conclude that the performance of the PiedPaper is better than the EndFrame.  7. <NAME>**Background Story:** A researcher was curious about whether there is a difference between the methodology she developed, C, and baseline methods A and B in terms of performance. Therefore, she decided to design different experiments and recorded the achieved accuracy by each method. The below table shows the achieved accuracy on test sets by each method. Please note that the same train and test sets were used for each method.**Goal:** According to this information, conduct the hypothesis testing to check whether there is a difference between the performance of the methods by using a 0.05 significance level. If there is a significant difference, perform further analysis to find which one caused the difference. Before doing hypothesis testing, check the related assumptions. Comment on the results.<jupyter_code># Import pandas library
import pandas as pd

# Initialize list of lists
data = [['E1',89.8,90.0,91.5],['E2',89.9,90.1,90.7],['E3',88.6,88.8,90.3],
        ['E4',88.7,88.9,90.4],['E5',89.6,89.9,90.2],['E6',89.7,90.0,90.3],
        ['E7',89.2,89.0,90.2],['E8',89.3,89.2,90.3]]

# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['Experiment','A','B','C'])
df
# Convert the columns from df to np array
method_A = df[['A']].to_numpy()
method_B = df[['B']].to_numpy()
method_C = df[['C']].to_numpy()<jupyter_output><empty_output><jupyter_text>1. Defining Hypothesis- H₀: μ₁=μ₂=μ₃ or The mean of the samples is the same.- H₁: At least one of them is different. 2. Assumption Check- H₀: The data is normally distributed.- H₁: The data is not normally distributed.- H₀: The variances of the samples are the same.- H₁: The variances of the samples are different.<jupyter_code>check_normality(method_A)
check_normality(method_B)
check_normality(method_C)

print("p value:%.4f" % pvalue_levene)
if pvalue_levene < 0.05:
    print('Reject null hypothesis >> The variance of the samples are different.')
else:
    print('Fail to reject null hypothesis >> The variance of the samples are same.')<jupyter_output>p value:0.3076
Fail to reject null hypothesis >> The data is normally distributed
p value:0.0515
Fail to reject null hypothesis >> The data is normally distributed
p value:0.0016
Reject null hypothesis >> The data is not normally distributed
p value:0.0012
Reject null hypothesis >> The variance of the samples are different.<jupyter_text>3. Selecting the Proper TestThere are three groups, but the normality assumption is violated. So, we need to use the nonparametric version of ANOVA for paired data since the accuracy scores are obtained from the same test sets.<jupyter_code>test_stat, p_value = stats.friedmanchisquare(method_A, method_B, method_C)
print('p value: %.4f' % p_value)
if p_value < 0.05:
    print('Reject null hypothesis')
else:
    print('Fail to reject null hypothesis')

print(np.round(np.mean(method_A),2), np.round(np.mean(method_B),2), np.round(np.mean(method_C),2))<jupyter_output>p value: 0.0000
Reject null hypothesis
89.35 89.49 90.49<jupyter_text>At this significance level, at least one of the methds has a different performance.**Note:** Since the data is not normal, the nonparametric version of the posthoc test is used.<jupyter_code>data = np.array([method_A, method_B, method_C]) 
posthoc_df=sp.posthoc_wilcoxon(data, p_adjust="holm")
# posthoc_df = sp.posthoc_nemenyi_friedman(data.T) ## another option for the posthoc test

group_names= ["Method A", "Method B","Method C"]
posthoc_df.columns= group_names
posthoc_df.index= group_names
posthoc_df.style.applymap(lambda x: "background-color:violet" if x<0.05 else "background-color: white")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10030257186081694, 'fraction_numerical': 0.07655068078668684, 'mean_word_length': 4.8433149171270715, 'pattern_counts': {'":': 0, '<': 107, '<?xml version=': 0, '>': 170, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '355014', 'license': 'MIT', 'n_tokens_mistral': 9675, 'n_tokens_neox': 8130, 'n_words': 3344, 'path': 'hypotest/Hypothesis Testing.ipynb', 'repo_name': 'les1smore/Stat-Notes', 'size': '26482'}