<jupyter_start><jupyter_text>Topics in Statistics Sampling Distribution of the meanThe mean of the sampling (say, sampling 10 from the population) will not equal to the population mean.Repeat this process (take 10 samples and compute the mean), and construct a **frequency distribution** of those means.Given a population with a mean of $\mu$ and a standard deviation of $\sigma$, the sampling distribution of the mean has:- $\mu_M = \mu$- $\sigma_M = \frac{\sigma}{\sqrt{n}}$where $n$ is the sample size. $\sigma_M$ is called the *standard error of the mean*.<jupyter_code>import numpy as np
import math
import matplotlib.pyplot as plt

# Generate population
population_size = 1000
population = np.random.randint(100, size=population_size)

# Sampling size
n = 8

means = []
for N in range(1000):
  # Sampling
  samples = np.random.choice(population, size=n, replace=False)
  means.append(np.mean(samples))
  
plt.subplot(1,3,1)
plt.hist(means[:10], bins=10)
plt.ylabel('Frequency')
plt.title('N=10')

plt.subplot(1,3,2)
plt.hist(means[:100], bins=10)
plt.ylabel('Frequency')
plt.title('N=100')

plt.subplot(1,3,3)
plt.hist(means[:1000], bins=10)
plt.ylabel('Frequency')
plt.title('N=1000')


plt.show()

from scipy.stats import norm

sizes = (2, 4, 8, 16, 100, 500)
xmin, xmax = (1, 100)
x = np.linspace(xmin, xmax, 100)
  

for i, n in enumerate(sizes):
  # Sampling
  means = []
  for N in range(1000):
    samples = np.random.choice(population, size=n, replace=False)
    means.append(np.mean(samples))

  mu, std = norm.fit(means)

  plt.subplot(2, math.ceil(len(sizes) / 2), i+1)
  
  plt.ylim(0, 500)
  plt.xlim(1, 100)
  plt.hist(means, bins=10)
  p = norm.pdf(x, mu, std) * 1000      # TODO: how to plot on graph? * 1000?

  plt.plot(x, p, 'k', linewidth=1)
  plt.ylabel('Frequency')
  plt.title('n=%s' % (n))

plt.show()<jupyter_output><empty_output><jupyter_text>Student's t-Test Try to use [Mall Customers](https://www.kaggle.com/vjchoudhary7/customer-segmentation-tutorial-in-python) from Kaggle to show the t-Test.<jupyter_code>from scipy import stats
import matplotlib.pyplot as plt
import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/charlee/practicalML/master/datasets/Mall_Customers.csv')
df.columns = ['id', 'gender', 'age', 'income', 'score']
df = df.dropna()
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Now we compare the spending score of two groups: "Male" and "Female".First we determine a null hypothesis and an alternative hypothesis:- $H_0$: spending score of men and women are the same- $H_a$: spending score of men and women are differentNext determine a confidence interval $\alpha = 0.05$.<jupyter_code># Collect data
male = df[df.gender=='Male'].score
male.describe()
female = df[df.gender=='Female'].score
female.describe()
plt.boxplot([male, female], labels=['Male', 'Female'])
plt.title('Spending score')<jupyter_output>/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:52: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
  return getattr(obj, method)(*args, **kwds)<jupyter_text>We can see that sample sizes of male and female are different. Use `scipy.stats.ttest_ind(equal_var=False)` to perform a two-sided t-test.<jupyter_code>t, p = stats.ttest_ind(male.score, female.score, equal_var=False)
print(t, p)<jupyter_output>-0.8048781809684525 0.42199772447783646<jupyter_text>$p = 0.42 > \alpha = 0.05$, we failed to reject $H_0$, thus there is no evidence that the spending score of men and women are different.  UNUSED<jupyter_code>from scipy import stats
import pandas as pd

df = pd.read_csv('https://github.com/charlee/practicalML/raw/master/datasets/suicide-rates-overview-1985-to-2016.csv')
df = df.dropna()

# Only use US data for this test
df = df[df.country == 'United States'].reset_index().drop(['index', 'country', 'country-year', 'generation'], axis=1)
df.head()
# Group data by sex<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1437875751503006, 'fraction_numerical': 0.03832665330661322, 'mean_word_length': 5.124233128834356, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '596143', 'license': 'MIT', 'n_tokens_mistral': 1562, 'n_tokens_neox': 1386, 'n_words': 388, 'path': '03_Statistics.ipynb', 'repo_name': 'charlee/tftest', 'size': '3992'}