<jupyter_start><jupyter_text>Variants of Hypothesis testing<jupyter_code>from IPython.display import Image
Image(filename='C:\\Users\\User\\Desktop\\Types.png')<jupyter_output><empty_output><jupyter_text>Process of Hypothesis testing<jupyter_code>Image(filename='C:\\Users\\User\\Desktop\\flow.png')<jupyter_output><empty_output><jupyter_text>lets do some practical implemetation:  Normality test using Shapiro-Wilk Test : tests If data is normally distributed Assumption : Observations are identically distributed<jupyter_code>#Data Import
import pandas as pd
WineData = pd.read_csv("C:\\Users\\User\\Downloads\\winequality-red.csv")
WineData.head()
#Cheking Histogram
import matplotlib
from matplotlib import pyplot 
%matplotlib inline
pyplot.figure(figsize=(14,6))
pyplot.hist(WineData['volatile acidity'])
pyplot.show()
#Help from Python
from scipy.stats import shapiro

DataToTest = WineData['volatile acidity']

stat, p = shapiro(DataToTest)

print('stat=%.2f, p=%.30f' % (stat, p))

if p > 0.05:
    print('Normal distribution')
else:
    print('Not a normal distribution')
#Lets genrate normally distributed data from Python
from numpy.random import randn
DataToTest = randn(100)
DataToTest
stat, p = shapiro(DataToTest)

print('stat=%.2f, p=%.30f' % (stat, p))

if p > 0.05:
    print('Normal distribution')
else:
    print('Not a normal distribution')<jupyter_output>stat=0.99, p=0.640997469425201416015625000000
Normal distribution<jupyter_text>Normality test using K^2 Normality Test Test : tests If data is normally distributed Assumption : Observations are identically distributed<jupyter_code># Example of the D'Agostino's K^2 Normality Test
from scipy.stats import normaltest
DataToTest = WineData['volatile acidity']

stat, p = normaltest(DataToTest)

print('stat=%.10f, p=%.10f' % (stat, p))

if p > 0.05:
    print('Normal')

else:
    print('Not Normllay distributed')<jupyter_output>stat=143.4193435598, p=0.0000000000
Not Normllay distributed<jupyter_text>Correlation Test - Pearson and Spearman’s Rank CorrelationAsumption - Identical and Normal Distribution<jupyter_code>FirstSample = WineData[1:30]['total sulfur dioxide']
SecondSample = WineData[1:30]['free sulfur dioxide']

pyplot.plot(FirstSample,SecondSample)
pyplot.show()
#Spearman Rank Correlation
from scipy.stats import spearmanr
stat, p = spearmanr(FirstSample, SecondSample)

print('stat=%.3f, p=%5f' % (stat, p))
if p > 0.05:
    print('independent samples')
else:
    print('dependent samples')
#pearson correlation
from scipy.stats import pearsonr
stat, p = pearsonr(FirstSample, SecondSample)

print('stat=%.3f, p=%5f' % (stat, p))
if p > 0.05:
    print('independent samples')
else:
    print('dependent samples')
WineData[1:30].corr(method="pearson")<jupyter_output><empty_output><jupyter_text>Correlation of categorical variable - Chi square test<jupyter_code>#Tests whether two categorical variables are related or independent.
#Assumptions - independent observation, size in each box of contingency table > 25
# Example of the Chi-Squared Test

LoanData = pd.read_csv("C:\\Users\\User\\Downloads\\loan_status.csv")
LoanData.head(10)
contingency_data = pd.crosstab(LoanData['purpose'], LoanData['loan_status'],margins = False)
contingency_data
from scipy.stats import chi2_contingency
stat, p, dof, expected = chi2_contingency(contingency_data)
print('stat=%.3f, p=%.3f' % (stat, p))
if p > 0.05:
    print('independent categories')
else:
    print('dependent categories')
contingency_data = [[25,125],[1200,240]] #Observe the numbers carefully
stat, p, dof, expected = chi2_contingency(contingency_data)
print('stat=%.3f, p=%.3f' % (stat, p))
if p > 0.05:
    print('independent categories')
else:
    print('dependent categories')
##To be Continued with many more examples<jupyter_output><empty_output><jupyter_text>Parametic test 1- T-test  Definiton of Parametric test - Main assumption - data is normally distributed<jupyter_code>Image(filename='C:\\Users\\User\\Desktop\\T-test.png') 
#Scores of me and Virat
my_score = [23,21,31,20,19,35,26,22,21,19]
virat_score = [46,42,62,40,38,70,52,44,42,38]
#Lets check mean of our scores
import numpy as np
print('Aman mean score:', np.mean(my_score))
print('Virat mean score:', np.mean(virat_score))
#One Sample T-test
import scipy
scipy.stats.ttest_1samp(my_score,20)
#Independent Sample T-test
scipy.stats.ttest_ind(my_score,virat_score)
my_score_second_Tour = [46,42,62,40,38,70,52,44,42,38]
#Apired sample t-test
scipy.stats.ttest_rel(my_score,my_score_second_Tour)<jupyter_output><empty_output><jupyter_text>Parametic test 2 - Anova - Tests whether the means of two or more independent samples are significantly different.<jupyter_code># Assumption -  Normal distributon, same variance, identical distribution
average_score = [40,44,60,50,48,68,55,46,44,54]
my_score
average_score
virat_score
tstat, p = scipy.stats.f_oneway(my_score, average_score, virat_score)
print('stat=%.3f, p=%.3f' % (stat, p))
if p > 0.05:
    print('Same distribution of scores')
else:
    print('Different distributions of scores')<jupyter_output>stat=-0.949, p=0.000
Different distributions of scores<jupyter_text>Non Parametric test 1 - Mann-Whitney U Test-Tests whether the distributions of two or more independent samples are equal or not.<jupyter_code>#Assumptions - Idential distribution, observations can be ranked
class_1_score = [91,90,81,80,76]
class_2_score = [88,86,85,84,83]
tstat, p = scipy.stats.mannwhitneyu(class_1_score, class_2_score)
print('stat=%.3f, p=%.3f' % (stat, p))
if p > 0.05:
    print('Same distribution')
else:
    print('Different distributions')
# Similarly check for Wilcoxon Signed-Rank Test/Kruskal-Wallis H Test<jupyter_output><empty_output><jupyter_text>Test of Stationarity - very Important for time series analysis<jupyter_code>#Augmented Dickey-Fuller Test -  null hypothesis - Series is non stationary
#Definition of stationary time series - constant mean and variance
from statsmodels.tsa.stattools import adfuller
stock_price_data = [121,131,142,121,131,142,121,131,142]
stat, p, lags, obs, crit, t = adfuller(stock_price_data)

print('stat=%.3f, p=%.3f' % (stat, p))
if p > 0.05:
    print('Series is not Stationary')
else:
    print('Series is stationary')
#Also check for Kwiatkowski-Phillips-Schmidt-Shin<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12333544705136334, 'fraction_numerical': 0.04644895370957514, 'mean_word_length': 6.3703271028037385, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '591627', 'license': 'MIT', 'n_tokens_mistral': 2394, 'n_tokens_neox': 2158, 'n_words': 538, 'path': 'Examples/Hypothesis testing/Hypothesis Test Practical Implementation .ipynb', 'repo_name': 'bdelaiglesia/DataMiningExamples', 'size': '6308'}