<jupyter_start><jupyter_text>The $t$-distribution and confidence intervals: an election example Suppose you want to know the proportion of an electorate that will vote for a certain candidate. (For this example, we will assume there are only two candidates.)  The electorate is too large to ask everybody (this is a poll, not the actual election), so we take samples instead.  Suppose we take $N=10$  samples and the proportion of votes for our candidate (after removing blank/invalid responses) are as follows:<jupyter_code>import numpy
x = numpy.array([0.53,0.56,0.47,0.50,0.59,0.42,0.65,0.62,0.53,0.56])<jupyter_output><empty_output><jupyter_text>The sample mean $\bar{x}$ of the above samples is$$\bar{x}=\frac{\sum_{i=1}^N x_i}{N}$$<jupyter_code>m = numpy.mean(x)
m<jupyter_output><empty_output><jupyter_text>This is larger than 50 percent. But how sure are we that our candidate will actually win the election? Assume that our samples come from a normal distribution with unknown mean and variance.  The sample mean is $\bar{x}$ which we computed above.  The **sample standard deviation** is$$\sigma_x=\sqrt{\frac{\sum_{i=1}^N \left(x_1 - \bar{x}\right)^2}{N-1}}$$<jupyter_code>sd = numpy.std(x,ddof=1)
sd<jupyter_output><empty_output><jupyter_text>Note that the "ddof" parameter above stands for "delta degrees of freedom", which corresponds to the "- 1" term in the above formula. The **standard error of the mean** is$$s_\bar{x}=\frac{\sigma_x}{\sqrt{N}}$$<jupyter_code>import scipy.stats
sem = scipy.stats.sem(x) # note this defaults to using ddof = 1 when computing
                         # the sample standard deviation
sem<jupyter_output><empty_output><jupyter_text>The construction of a confidence interval is as follows:1.   Construct a **Student's $t$-distribution** with $N-1$ degrees of freedom.2.   Shift the distribution so that its new mean is $\bar{x}$.3.   Scale the distribution about its mean $\bar{x}$ by a factor of$s_\bar{x}$. A value of $x$ in the new distribution now corresponds to a value of $(x-\bar{x})/s_\bar{x}$ in the original distribution from Step 1.4.   What interval has an area of 95% (or some other confidence level) of the entire $t$-distribution from Step 3?All of this can be done in a single line of Python:<jupyter_code>scipy.stats.t.interval(0.95,len(x)-1,loc=m,scale=sem)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11602209944751381, 'fraction_numerical': 0.02337441563960901, 'mean_word_length': 5.035897435897436, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '38471', 'license': 'MIT', 'n_tokens_mistral': 765, 'n_tokens_neox': 724, 'n_words': 276, 'path': 'tutorial_01_confidence_intervals.ipynb', 'repo_name': 'yinchi/simpy-examples', 'size': '2353'}