<jupyter_start><jupyter_text>What is statistics? Probability theory computes probabilities of complex events given the underlying base probabilities. Statistics takes us in the opposite direction. We are given **data** that was generated by a **Stochastic process** We **infer** properties of the underlying base probabilities.  Example:  deciding whether a coin is biased. In a previous video we discussed the distribution of the number of heads when flipping a fair coin many times. Let's turn the question around: we flip a coin 1000 times and get 570 heads.  Can we conclude that the coin is biased (not fair) ? What can we conclude if we got 507 heads?  The Logic of Statistical inferenceThe answer uses the following logic. * Suppose that the coin is fair.  * Use **probability theory** to compute the probability of getting at least 570 (or 507) heads. * If this probability is very small, then we can **reject** with confidence the hypothesis that the coin is fair.  Calculating the answerRecall the simulations we did in the video "What is probability". We used $x_i=-1$ for tails and $x_i=+1$ for heads. We looked at the sum $S_k=\sum_{i=1}^k x_i$, here $k=1000$. If number of heads is $570$ then $S_{1000} = 570-430 = 140$   It is very unlikely that $|S_{1000}| > 4\sqrt{k} \approx 126.5$<jupyter_code>from math import sqrt
4*sqrt(1000)<jupyter_output><empty_output><jupyter_text>It is very unlikely that the coin is unbiased.  What about 507 heads? 507 heads = 493 tails $ \Rightarrow S_n = 14$,       $\;\;\;14 \ll 126.5$ We cannot conclude that coin is biased.  ConclusionThe probability that an unbiased coin would generate a sequence with 570 or more heads is extremely small. From which we can conclude, with high confidence, that the coin **is** biased. On the other hand, $\big| S_{1000} \big | \geq 507$ is quite likely. So getting 507 heads does not provide evidence that the coin is biased.  Real-World examplesYou might ask "why should I care whether a coin is biased?" * This is a valid critique. * We will give a few real-world cases in which we want to know whether a "coin" is biased or not.  Case I: Polls* Suppose elections will take place in a few days and we want to know how people plan to vote.* Suppose there are just two parties: **D** and **R**. * We could try and ask **all** potential voters. * That would be very expensive. * Instead, we can use a poll: call up a small randomly selected set of people. * Call $n$ people at random and count the number of **D** votes. * Can you say with confidence that there are more **D** votes, or more **R** votes? * Mathematically equivalent to flipping a biased coin and  * asking whether you can say with confidence that it is biased towards "Heads" or towards "Tails"  Case 2: A/B testingA common practice when optimizing a web page is to perform A/B tests. * A/B refer to two alternative designs for the page.![AB](images/AB.png) * To see which design users prefer we randomly present design A or design B. * We measure how long the user stayed on a page, or whether the user clicked on an advertisement. * We want to decide, with confidence, which of the two designs is better. * Again: similar to making a decision with confidence on whether "Heads" is more probably than "Tails" or vice versa.<jupyter_code>import pandas as pd
# Import data from GitHub (or from your local computer)
df = pd.read_csv("https://raw.githubusercontent.com/kirenz/datasets/master/wage.csv")
df.head()
df.columns
df.shape
# mode of age
df['age'].mode()
# mode of wage
df['wage'].mode()
# calculation of the mean (e.g. for age)
df["age"].mean()
# calculation of the mean (e.g. for wage)
df["wage"].mean()
# calculation of the mean (e.g. for age) and round the result
round(df["age"].mean(), 0)
# calculation of the mean (e.g. for wage) and round the result
round(df["wage"].mean(), 3)
# calculation of the median (e.g. for age)
df["age"].median()
# calculation of the median (e.g. for wage)
df["wage"].median()<jupyter_output><empty_output><jupyter_text>Measures of dispersion<jupyter_code># quantiles
df['age'].quantile([.25, .5, .75])
# Range
df['age'].max() - df['age'].min()
# standard deviation
round(df['age'].std(),2)<jupyter_output><empty_output><jupyter_text>Summary statistics<jupyter_code># summary statistics for all numerical columns
round(df.describe(),2)<jupyter_output><empty_output><jupyter_text>Compare summary statistics for specific groups in the data<jupyter_code># summary statistics by groups
df['age'].groupby(df['education']).describe()
# summary statistics by groups
df['wage'].groupby(df['race']).describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10010810810810811, 'fraction_numerical': 0.02010810810810811, 'mean_word_length': 4.711111111111111, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '309879', 'license': 'Unlicense', 'n_tokens_mistral': 1405, 'n_tokens_neox': 1277, 'n_words': 650, 'path': '2.What-is-Statistics_.ipynb', 'repo_name': 'vvchauit/Data-Science', 'size': '4625'}