<jupyter_start><jupyter_text>d-sandbox   -sandbox Hypothesis Testing (25 mins)**Objective**: *Perform a two-sample, two-sided t-test to compare sample means.*In this lab, you will complete a series of exercises to perform a t-test.We want to determine whether the population mean of daily steps taken for the athlete health tracker users is equal to the population mean of daily steps taken for the cardio enthusiast health tracker users.The null and alternative hypotheses for this test are:* H0 = the mean of daily steps taken for athlete users is equal to the mean of daily steps taken for cardio enthusiast users.* H1 = the mean of daily steps taken for athlete users is not equal to the mean of daily steps taken for cardio enthusiast users.<jupyter_code>%run "../../Includes/Classroom-Setup"<jupyter_output><empty_output><jupyter_text>Exercise 1Compute the sample mean of daily steps taken for sedentary users and cardio enthusiast users.<jupyter_code>%sql
-- ANSWER
SELECT lifestyle, avg(steps) AS mean
FROM dsfda.ht_daily_metrics
WHERE lifestyle = "Athlete" OR lifestyle = "Cardio Enthusiast"
GROUP BY lifestyle<jupyter_output><empty_output><jupyter_text>Exercise 2Compute the sample variance of daily steps taken for sedentary users and cardio enthusiast users.<jupyter_code>%sql
-- ANSWER
SELECT lifestyle, var_samp(steps) AS variance
FROM dsfda.ht_daily_metrics
WHERE lifestyle = "Athlete" OR lifestyle = "Cardio Enthusiast"
GROUP BY lifestyle<jupyter_output><empty_output><jupyter_text>Exercise 3Compute the sample size for sedentary users and cardio enthusiast users.<jupyter_code>%sql
-- ANSWER
SELECT lifestyle, count(*) AS sample_size
FROM dsfda.ht_daily_metrics
WHERE lifestyle = "Athlete" OR lifestyle = "Cardio Enthusiast"
GROUP BY lifestyle<jupyter_output><empty_output><jupyter_text>Exercise 4Compute the T-statistic using the sample statistics.<jupyter_code># ANSWER
from math import sqrt

athlete_mean = 11001.597413366928
athlete_variance = 9623550.2344004
athlete_size = 313535

cardio_mean = 13235.376990421259
cardio_variance = 13171492.338618632
cardio_size = 388360

test_statistic = (athlete_mean - cardio_mean) / sqrt((athlete_variance / athlete_size) + (cardio_variance / cardio_size))
print(f"T-statistic = {test_statistic}")<jupyter_output><empty_output><jupyter_text>Exercise 5Compute the degrees of freedom using the sample statistics.<jupyter_code># ANSWER
df_numerator = ((athlete_variance / athlete_size) + (cardio_variance / cardio_size))**2
df_denominator = (athlete_variance / athlete_size)**2 / (athlete_size - 1) + ((cardio_variance / cardio_size)**2 / (cardio_size - 1))
df = df_numerator / df_denominator
print(f"Degrees-of-freedom = {df}")<jupyter_output><empty_output><jupyter_text>Exercise 6Compute the p-value for this T-test by passing in the `test_statistic` and the `df` to `t.cdf()`.<jupyter_code># ANSWER
from scipy.stats import t
p_value = t.cdf(test_statistic, df)
print(f"p-value = {p_value}")<jupyter_output><empty_output><jupyter_text>-sandbox Exercise 7Determine whether we should reject the null hypothesis.  Use a significance level of 0.05.<jupyter_code># ANSWER
print(f"The p-value {p_value} is less than 0.05. Thus, we reject the null hypothesis.")<jupyter_output><empty_output><jupyter_text>Exercise 8Phew! That was a lot of work to answer a simple question.Luckily, Python's `scipy` module makes this process a bit easier than it already has.Check out the demonstration below showing how to perform this same test in a single step only using Python.<jupyter_code>from scipy.stats import ttest_ind

athlete_daily_steps = spark.sql("SELECT steps FROM dsfda.ht_daily_metrics WHERE lifestyle = 'Athlete'").toPandas()["steps"]
cardio_daily_steps = spark.sql("SELECT steps FROM dsfda.ht_daily_metrics WHERE lifestyle = 'Cardio Enthusiast'").toPandas()["steps"]

ttest_ind(athlete_daily_steps, cardio_daily_steps, equal_var = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0836094825388733, 'fraction_numerical': 0.02549069589599796, 'mean_word_length': 6.40377358490566, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '469114', 'license': 'MIT', 'n_tokens_mistral': 1354, 'n_tokens_neox': 1239, 'n_words': 397, 'path': 'Data Science Fundamentals for Data Analysts/ipynb/3.3.1 Lab - Hypothesis Testing.ipynb', 'repo_name': 'bsimmooes/Data-Science-with-Databricks-for-Data-Analysts-Coursera', 'size': '3923'}