<jupyter_start><jupyter_text>Hypothesis Testing**Outline*** Statistical and Hypothesis Testing   - What is a Hypothesis?   - What is a Hypothesis Statement?   - Null vs Alternative Hypothesis      * What is a typical examples of Ho and Ha?   - Type I and Type II Errors      * Significance Level      * P-value   - Types of Hypothesis Tests   - Is your data normal?      * Graphical Checks      * Statistical Testing   - Hypothesis Testing Process* Demo: Normality Test of a Dataset   - Graphical Normality Checks      * Histogram Plot      * Normal Probability Plot   - Statistical Normality Tests      * Shapiro-Wilk Test      * D’Agostino’s K^2 Test* Demo: Hypothesis Testing   - T-test      * One sample T-test      * Paired Sampled T-test   - Z-test      * One Sample Z-test      * Two Sample Z-test        Statistical and Hypothesis TestingA statistical hypothesis test is a method of statistical inference. Commonly, two statistical data sets are compared, or a data set obtained by sampling is compared against a synthetic data set from an idealized model. An **alternative hypothesis** is proposed for the statistical-relationship between the two data-sets, and is compared to an idealized **null hypothesis** that proposes no relationship between these two data-sets. This comparison is deemed **statistically significant** if the relationship between the data-sets would be an unlikely realization of the null hypothesis according to a threshold probability—the significance level. Hypothesis tests are used when determining what outcomes of a study would lead to a rejection of the null hypothesis for a pre-specified level of significance.The main purpose of statistics is to test a hypothesis. For example, you might run an experiment and find that a certain drug is effective at treating headaches. But if you can’t repeat that experiment, no one will take your results seriously.   What is a Hypothesis?A hypothesis is an educated guess about something in the world around you. It should be testable, either by experiment or observation. For example:* A new medicine you think might work.* A way of teaching you think might be better.* A possible location of new species.* A fairer way to administer standardized tests.It can really be anything at all as long as you can put it to the test. What is a Hypothesis Statement?If you are going to propose a hypothesis, it’s customary to write a statement. Your statement will look like this:“If I…(do this to an independent variable)….then (this will happen to the dependent variable).”For example:* If I (decrease the amount of water given to herbs) then (the herbs will increase in size).* If I (give patients counseling in addition to medication) then (their overall depression scale will decrease).* If I (give exams at noon instead of 7) then (student test scores will improve).* If I (look in this certain location) then (I am more likely to find new species).A good hypothesis statement should:* Include an “if” and “then” statement (according to the University of California).* Include both the independent and dependent variables.* Be testable by experiment, survey or other scientifically sound technique.* Be based on information in prior research (either yours or someone else’s).* Have design criteria (for engineering or programming projects).  Null vs Alternative HypothesisThe **null hypothesis** reflects that there will be **no observed effect** in our experiment. In a mathematical formulation of the null hypothesis, there will typically be an equal sign.The **alternative hypothesis** reflects that there **will be an observed effect** for our experiment. In a mathematical formulation of the alternative hypothesis, there will typically be an inequality, or not equal to symbol. " What is a typical examples of Ho and Ha?Ho: Often we are measuring the means of something, and the null hypothesis is that two means are the same.Ha: When measuring the means of something (average height of American vs Mexican corn, for example), the alternative hypothesis is that the means are not the same.  Type I and Type II ErrorsThe process of distinguishing between the **null hypothesis** and the **alternative hypothesis** is aided by considering two conceptual types of errors. The first type of error occurs when the null hypothesis is wrongly rejected. The second type of error occurs when the null hypothesis is wrongly not rejected. (The two types are known as **Type I and Type II Errors**.) Significance LevelThe significance level, also denoted as **alpha(α)**, is the probability of rejecting the null hypothesis when it is true. In other words, the level of significance is the probability of a Type I Error occurring.  P-valueThe P-value, or calculated probability, is the probability of finding the observed, or more extreme, results when the null hypothesis (Ho) of a study question is true — the definition of ‘extreme’ depends on how the hypothesis is being tested  Types of Hypothesis TestsThere are a number of different types of hypothesis tests, useful for different hypothesis scenarios and data samples. Below is a list of some of the more commonly used hypothesis tests.* **T-test**: Used for comparing means (or another summary statistic) of 1 group to a number, or between two groups* **Z-test**: Used for same thing as a t-test, but typically for large samples and known population variance* **F-test**: Used for testing if different levels of a factor have a significant effect on a summary statistic* **ANOVA**: Tests for the difference between group means after any other variance in the outcome variable is accounted for (e.g., controlling for sex, income, or age)* **Chi Square**: Tests for the strength of the association between two categorical variables.  Is your data normal?Most hypothsesis tests make an assumption about the distribution of the underlying data is normally distributed,and we’re often told that we must carefully check the assumptions!There are a number of techniques that you can use to check if your data sample is normally or sufficiently normal to use standard techniques (ie: hyypothesis testing).* **Graphical Methods**: These are methods for plotting the data and qualitatively evaluating whether the data looks "normally distributed". * **Statistical Tests**: These are methods that calculate statistics on the data and quantify how likely it is that the data was drawn from a normal distribution. Methods of this type are often called normality tests.  Hypothesis Testing Process* Collect your data* Define your null and alternative hypothesis* Determine your alpha* Check the distribution of your data* Calculate some test statistic and compare to alpha* Draw a conclusion about Ho (reject or accept) References: * https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/hypothesis-testing/* https://en.wikipedia.org/wiki/Statistical_hypothesis_testing* https://www.thoughtco.com/null-hypothesis-vs-alternative-hypothesis-3126413  Demo: Normality Test of a Dataset As stated above, there are a number of techniques that we can use to check to see whether our dataset is normally distributed or not.  For this demo, we will be following along with [A Gentle Introduction to Normality Tests in Python](https://machinelearningmastery.com/a-gentle-introduction-to-normality-tests-in-python/) tutorial. Generate a DatasetBefore we start looking at normality tests, let’s first develop a test dataset that we can use throughout this tutorial.<jupyter_code># some of the common imports that we will need
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import pandas as pd

%matplotlib inline
sns.set()  
# generate normally distributed data
from numpy.random import seed
from numpy.random import randn
from numpy import mean
from numpy import std

# seed the random number generator
seed(1)

# generate univariate observations
data = 5 * randn(100) + 50

# summarize
print('mean=%.3f stdv=%.3f' % (mean(data), std(data)))<jupyter_output>mean=50.303 stdv=4.426<jupyter_text>Graphical Normality ChecksWe can create plots of the data to check whether it is Gaussian (aka normally distributed)These checks are qualitative, so less accurate than the statistical methods we will calculate in the next section. Nevertheless, they are fast and like the statistical tests, must still be interpreted before you can make a call about your data sample. Histogram PlotA simple and commonly used plot to quickly check the distribution of a sample of data is the histogram.<jupyter_code># histogram plot
plt.hist(data)<jupyter_output><empty_output><jupyter_text>We can see a Gaussian-like shape to the data, that although is not strongly the familiar bell-shape, is a rough approximation.Helpful Hint:: I prefer seaborn for displaying graphival information. Let's see the difference between matplotlib and seaborn.<jupyter_code># histogram plot - the kde argument adds the line to show the general shape of the distribution
sns.displot(data,kde=True)<jupyter_output><empty_output><jupyter_text>I've found that the seaborn plots are just nicer overall.  Normal Probability PlotThe normal probability plot is a graphical technique for assessing whether or not a data set is approximately normally distributed.The data are plotted against a theoretical normal distribution in such a way that the points should form an approximate straight line. Departures from this straight line indicate departures from normality.The normal probability plot is a special case of the probability plot. We cover the normal probability plot separately due to its importance in many applications.<jupyter_code>import scipy.stats as stats
res = stats.probplot(data, plot=plt)<jupyter_output><empty_output><jupyter_text>There are a few small deviations, especially at the bottom of the plot, which is to be expected given the small data sample. Statistical Normality TestsThere are many statistical tests that we can use to quantify whether a sample of data looks as though it was drawn from a normal distribution.Each test makes different assumptions and considers different aspects of the data. However, each of these tests returns at least two things:* **Statistic**: A quantity calculated by the test that can be interpreted in the context of the test via comparing it to critical values from the distribution of the test statistic.* **p-value**: Used to interpret the test, in this case whether the sample was drawn from a normal distribution.Each test calculates a test-specific statistic. This statistic can aid in the interpretation of the result, although it may require a deeper proficiency with statistics and a deeper knowledge of the specific statistical test. Instead, the p-value can be used to quickly and accurately interpret the statistic in practical applications.The tests assume that that the sample was drawn from a normal distribution. Technically this is called the null hypothesis, or Ho. A threshold level is chosen called alpha, typically 5% (or 0.05), that is used to interpret the p-value.In the SciPy implementation of these tests, you can interpret the p value as follows.* **p <= alpha**: reject Ho, not normal.* **p > alpha**: accept Ho, normal.This means that, in general, we are seeking results with a larger p-value to confirm that our sample was likely drawn from a normal distribution.  Shapiro-Wilk TestThe Shapiro-Wilk test evaluates a data sample and quantifies how likely it is that the data was drawn from a Gaussian distribution, named for <NAME> and <NAME>.In practice, the Shapiro-Wilk test is believed to be a reliable test of normality, although there is some suggestion that the test may be suitable for smaller samples of data, e.g. thousands of observations or fewer.<jupyter_code>from scipy.stats import shapiro

# normality test
stat, p = shapiro(data)
print('Statistics=%.3f, p=%.3f' % (stat, p))

# interpret
alpha = 0.05
if p > alpha:
    print('Sample looks normally distributed (fail to reject H0)')
else:
    print('Sample does not look normally distributed (reject H0)')<jupyter_output>Statistics=0.992, p=0.822
Sample looks normally distributed (fail to reject H0)<jupyter_text>D’Agostino’s K^2 TestThe D’Agostino’s K^2 test calculates summary statistics from the data, namely kurtosis and skewness, to determine if the data distribution departs from the normal distribution, named for <NAME>.* Skewness is a quantification of how much a distribution is pushed left or right, a measure of asymmetry in the distribution.* Kurtosis quantifies how much of the distribution is in the tail. It is a simple and commonly used statistical test for normality.The D’Agostino’s K^2 test is available via the normaltest() SciPy function and returns the test statistic and the p-value. The test statistic is calculated as s^2 + k^2, where s is the z-score returned by skewtest and k is the z-score returned by kurtosis test.<jupyter_code>from scipy.stats import normaltest

# normality test
stat, p = normaltest(data)
print('Statistics=%.3f, p=%.3f' % (stat, p))

# interpret
alpha = 0.05
if p > alpha:
    print('Sample looks normally distributed (fail to reject H0)')
else:
    print('Sample does not look normally distributed (reject H0)')<jupyter_output>Statistics=0.102, p=0.950
Sample looks normally distributed (fail to reject H0)<jupyter_text>Helpful Hint:: If your data is stored in a pandas dataframe skewness and kurtosis are methods of the class Dataframe. Here's an example: &emsp;print(f'Skewness: {train["SalePrice"].skew()}') &emsp;print(f'Kurtosis: {train["SalePrice"].kurt()}')          Demo: Hypothesis TestingAs stated above, hypothesis testing is a statistical method that is used in making statistical decisions using experimental data. In this demo we will demostrate several different types of hypothesis testing and will be following along with [Hypothesis testing in Machine learning using Python](https://towardsdatascience.com/hypothesis-testing-in-machine-learning-using-python-a0dc89e169ce). T-testA t-test is a type of inferential statistic which is used to determine if there is a significant difference between the means of two groups which may be related in certain features. It is mostly used when the data sets, like the set of data recorded as outcome from flipping a coin a 100 times, would follow a normal distribution and may have unknown variances. T test is used as a hypothesis testing tool, which allows testing of an assumption applicable to a population. T-test has 2 types 1. one sampled t-test2. two-sampled t-test. One Sample T-testThe One Sample T-test determines whether the sample mean is statistically different from a known or hypothesised population mean.Example: You have 14 ages and you are checking whether average age is 30 or not.<jupyter_code>from scipy.stats import ttest_1samp
import numpy as np

ages = [32, 34, 29, 29, 22, 39, 38, 37, 38, 36, 30, 26, 22, 22]
print(f'Ages: {ages}')
ages_mean = np.mean(ages)
print(f'Mean: {ages_mean}')

#t-test, where we are defining the Ho = 30
tset, pval = ttest_1samp(ages, 30)
print(f'p-value: {pval}')

# eval p-value
if pval < 0.05:    # alpha value is 0.05 or 5%
   print('Reject null hypothesis')
else:
  print('Accept null hypothesis')<jupyter_output>Ages: [32, 34, 29, 29, 22, 39, 38, 37, 38, 36, 30, 26, 22, 22]
Mean: 31.0
p-value: 0.5605155888171379
Accept null hypothesis<jupyter_text>So, what does all of that mean. The results show that the p-value (.5605) is greater than our defined alpha(α) of .05. This suggests that the null hypothesis(Ho) cannot not be rejected, and the average age of our dataset **is not significantly different** than 30. Even though the calculated mean is 31.  Paired Sampled T-testThe paired sample T-test is also called dependent sample t-test. It’s an uni variate test that tests for a significant difference between 2 related variables. * H0: mean difference between two sample is 0* H1: mean difference between two sample is not 0For this exercise, we have 10 athletes that were testing before and after the same coach used a new training method.   Our first test will be to compare the effectiveness of the new training method. For this comparison, Ho = there was no improvement from the new training method.<jupyter_code>import pandas as pd
from scipy import stats

df = pd.read_csv('data_wk3/coaching.csv')
df
#paired t-test
ttest,pval = stats.ttest_rel(df['before'], df['after'])
print(f'p-value: {pval}')

# eval p-value
if pval < 0.05:    # alpha value is 0.05 or 5%
   print('Reject null hypothesis')
else:
  print('Accept null hypothesis')<jupyter_output>p-value: 0.8358400057139277
Accept null hypothesis<jupyter_text>Since the p-value > 0.05, we do not reject Ho.  This means that the new training method **did not significantly** impact the athlete’s performance, positively or negatively.  There was no overall change.Let's take a look at another example. This time we will look at the blood pressure of individuals before and after some sort of a treatment.<jupyter_code>df2 = pd.read_csv('data_wk3/blood_pressure.csv')
df2
#paired t-test
ttest,pval = stats.ttest_rel(df2['bp_before'], df2['bp_after'])
print(f'p-value: {pval}')

# eval p-value
if pval < 0.05:    # alpha value is 0.05 or 5%
   print('Reject null hypothesis')
else:
  print('Accept null hypothesis')<jupyter_output>p-value: 0.0011297914644840823
Reject null hypothesis<jupyter_text>We got a different result this time and Ho is rejected. This means that treatment **did significantly affect** the blood pressure of this group of individuals.  Z-testA Z-test is very similar to a T-test, but is generally used when we have a larger sample size.You would use a Z test if:* Your sample size is greater than 30. Otherwise, use a t test.* Data points should be independent from each other. In other words, one data point isn’t related or doesn’t affect another data point.* Your data should be normally distributed. However, for large sample sizes (over 30) this doesn’t always matter.* Your data should be randomly selected from a population, where each item has an equal chance of being selected.* Sample sizes should be equal if at all possible  One Sample Z-testWe will reuse our blood pressure dataset from above and we are going to define our Ho: average starting blood pressure of 156 for our group of individuals.<jupyter_code># verify that we still have our data
df2['bp_before']
from statsmodels.stats import weightstats as stests

# z-test
ztest ,pval = stests.ztest(df2['bp_before'], x2=None, value=156)
print(f'p-vale: {pval}')

# eval p-value
if pval < 0.05:    # alpha value is 0.05 or 5%
   print('Reject null hypothesis')
else:
  print('Accept null hypothesis')<jupyter_output>p-vale: 0.6651614730255063
Accept null hypothesis<jupyter_text>To recap, this means that our null hypothesis (Ho) cannot not be rejected, and the average blood pressure of our dataset is not significantly different than 156. Two Sample Z-testIn two sample z-test , similar to t-test here we are checking two independent data groups and deciding whether sample mean of two group is equal or not.* H0 : mean of two group is 0* H1 : mean of two group is not 0Let's redo the blood pressure paired t-test from above. Our dataset does have more than 30 individuals in it.<jupyter_code># verify we still have our dataset
df2
# ztest
ztest ,pval = stests.ztest(df2['bp_before'], x2=df2['bp_after'],value=0,alternative='two-sided')
print(f'p-value: {pval}')

# eval p-value
if pval < 0.05:    # alpha value is 0.05 or 5%
   print('Reject null hypothesis')
else:
  print('Accept null hypothesis')<jupyter_output>p-value: 0.002162306611369422
Reject null hypothesis
{'fraction_non_alphanumeric': 0.06166683626742648, 'fraction_numerical': 0.016688714765442147, 'mean_word_length': 4.813368825791186, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 44, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '166302', 'license': 'MIT', 'n_tokens_mistral': 5295, 'n_tokens_neox': 4895, 'n_words': 2779, 'path': 'week4/4_Hypothesis_Testing.ipynb', 'repo_name': 'kscully-dotcom/msds650', 'size': '19677'}