<jupyter_start><jupyter_text>**Homework 10: Central Limit Theorem**<jupyter_code>#Don't change this cell; just run it. 

import numpy as np
from datascience import *

# These lines do some fancy plotting magic.
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
import warnings
warnings.simplefilter('ignore', FutureWarning)

from client.api.notebook import Notebook
ok = Notebook('hw10.ok')
import pandas as pd
#df = pd.read_csv (r'/content/restaurant_inspection_scores.txt')
df = pd.read_csv (r'/content/restaurant_inspection_scores.csv')
df
#df_united = pd.read_csv (r'/content/united_summer2015.txt')
df_united = pd.read_csv (r'/content/united_summer2015.csv')
df_united
#df.to_csv (r'/content/restaurant_inspection_scores.csv', index=None)
#df_united.to_csv (r'/content/united_summer2015.csv', index=None)<jupyter_output><empty_output><jupyter_text>**1. The Bootstrap and The Normal Curve** In this exercise, we will explore a dataset that includes the safety inspection scores for restaurants in the city of Austin, Texas. We will be interested in determining the average restaurant score for the city from a random sample of the scores; the average restaurant score is out of 100. We'll compare two methods for computing a confidence interval for that quantity: the bootstrap resampling method, and an approximation based on the Central Limit Theorem.<jupyter_code># Just run this cell.
pop_restaurants = Table.read_table('/content/restaurant_inspection_scores.csv').drop('Facility ID','Process Description')
pop_restaurants

#plot a histogram of the scores from pop_restaurants
pop_restaurants.hist('Score')
#population mean
pop_mean = np.mean(pop_restaurants.column('Score'))
pop_mean
#random sample of 100 restaurant inspections, called restaurant_sample
#using this sample to estimate the population mean
restaurant_sample = pop_restaurants.sample(100, with_replacement=False)
restaurant_sample
#plot a histogram of the sample scores from restaurant_sample
restaurant_sample.hist('Score')
#sample mean
sample_mean = np.mean(restaurant_sample.column('Score'))
sample_mean<jupyter_output><empty_output><jupyter_text>**1.1 Question 1**Complete the function one_resampled_mean below. It should take in an original table data, with a column Score, and return the mean score of one resampling from data.Remember to call your function and check the output before moving on to autograder tests.<jupyter_code>def one_resampled_mean(data):
    resampled_data=data.select('Score').sample().column(0)
    return np.mean(resampled_data)
this_mean=one_resampled_mean(restaurant_sample)
this_mean
import matplotlib.pyplot as plots
restaurant_sample.select('Score',).hist(bins=10)
observed_statistic = 91.26 
plots.scatter(observed_statistic, 0, color='red', s=200)<jupyter_output><empty_output><jupyter_text>**1.2 Question 2**Complete the function bootstrap_scores below. It should take no arguments. It should simulate drawing 5000 resamples from restaurant_sample and compute the mean restaurant score in each resample. It should return an array of those 5000 resample means.<jupyter_code>def bootstrap_scores():
    #main_sample=restaurant_sample.select('Score')
    resampled_means = make_array()
    for i in range(5000):
        resampled_mean =np.mean(restaurant_sample.select('Score').sample().column(0))
        resampled_means =np.append(resampled_means,resampled_mean)
    return resampled_means

resampled_means = bootstrap_scores()
resampled_means<jupyter_output><empty_output><jupyter_text>**1.3 Question 3**Compute a 95 percent confidence interval for the average restaurant score using the array resampled_means.<jupyter_code>lower_bound = percentile(2.5, resampled_means)
upper_bound = percentile(97.5, resampled_means)
print("95% confidence interval for the average restaurant score, computed by bootstrapping:\n(",lower_bound, ",", upper_bound, ")")
#histogram of the resampled means
Table().with_column('Resampled Means', resampled_means).hist()
#plots.scatter(lower_bound, 0, color='red', s=100)
#plots.scatter(upper_bound, 0, color='red', s=100)
plots.plot(make_array(lower_bound, upper_bound), make_array(0, 0), color='yellow', lw=8)<jupyter_output><empty_output><jupyter_text>**1.4 Question 4**What distribution is the histogram between question 2 and 3 displaying (that is, what data are plotted), and why does it have that shape? The data that are plotted are the mean of all resampling that are the outcome of the bootstrapping the sample and then based on the 95% confidence interval the yellow line indicate the range between the lower bond and upper bond of the confidence interval.The histogram is very close to the normal distribution shape which means almost all the sample can represent the main dataset. **1.5 Question 5**Does the distribution of the sampled scores look normally distributed? State "yes" or "no" and describe in one sentence why you should expect this result.Hint: Remember that we are no longer talking about the resampled means!For the last question, you'll need to recall two facts.If a group of numbers has a normal distribution, around 95% of them lie within 2 standard deviations of their mean.The Central Limit Theorem tells us the quantitative relationship between the following:the standard deviation of an array of numbers.the standard deviation of an array of means of samples taken from those numbers.Also recall the standard deviation of sample means:$\texttt{sd of sample means from many samples from a distribution}  =   $$\frac{\texttt{sd of the original distribution}}{\sqrt{\texttt{sample size}}}$$ $ I think 'yes', but it is not the completely symmetric.It is approximately follows the normal distribution, because the shape of the histogram is very close to the normal distribution and it shows that in the range of the 95% confidence interval with consideration of the 2 standard diviation the distribution is almost symmetric. Also, when we consider the sd of the sample mean and the confidence interval around that they prove the claim of having almost normal distribution. **1.6 Question 6**Without referencing the array resampled_means or performing any new simulations, calculate an interval around the sample_mean that covers approximately 95% of the numbers in the resampled_means array. You may use the following values to compute your result, but you should not perform additional resampling - think about how you can use the CLT to accomplish this.<jupyter_code>np.std(pop_restaurants.column('Score'))
import math
sample_mean = np.mean(restaurant_sample.column('Score'))
sample_sd = np.std(restaurant_sample.column('Score'))
sample_size = restaurant_sample.num_rows
pop_sd=np.std(pop_restaurants.column('Score'))

sd_of_means = pop_sd/math.sqrt(sample_size)
sd_of_means
lower_bound_normal =np.mean(restaurant_sample.column('Score')) - (2*sd_of_means) 
upper_bound_normal = np.mean(restaurant_sample.column('Score')) + (2*sd_of_means)
print("95% confidence interval for the average restaurant score, computed by a normal approximation:\n(",lower_bound_normal, ",", upper_bound_normal, ")")<jupyter_output>95% confidence interval for the average restaurant score, computed by a normal approximation:
( 90.44618327076104 , 93.51381672923897 )<jupyter_text>This confidence interval should look very similar to the one you computed in Question 3. **2. Testing the Central Limit Theorem** To recap the properties we just saw: The Central Limit Theorem tells us that the probability distribution of the sum or average of a large random sample drawn with replacement will be roughly normal, regardless of the distribution of the population from which the sample is drawn.That's a pretty big claim, but the theorem doesn't stop there. It further states that the standard deviation of this normal distribution is given by$$\frac{\texttt{sd of the original distribution}}{\sqrt{\texttt{sample size}}}$$In other words, suppose we start with any distribution that has standard deviation $x$, take a sample of size $n$ (where $n$ is a large number) from that distribution with replacement, and compute the mean of that sample. If we repeat this procedure many times, then those sample means will have a normal distribution with standard deviation $\frac{x}{\sqrt{n}}$.That's an even bigger claim than the first one! The proof of the theorem is beyond the scope of this class, but in this exercise, we will be exploring some data to see the CLT in action. **2.1 Question 1**. Define the function one_statistic_prop_heads which should return exactly one simulated statistic of the proportion of heads from $n$ coin flips.<jupyter_code>coin_proportions = make_array('Heads', 'Tails') # our coin is fair

def one_statistic_prop_heads(n):
    simulated_proportions = np.random.choice(coin_proportions, 100)
    prop_heads = np.count_nonzero(simulated_proportions == 'Heads')
    return prop_heads
one_statistic_prop_heads(100)<jupyter_output><empty_output><jupyter_text>**2.2 Question 2.**The CLT only applies when sample sizes are "sufficiently large." This isn't a very precise statement. Is 10 large? How about 50? The truth is that it depends both on the original population distribution and just how "normal" you want the result to look. Let's use a simulation to get a feel for how the distribution of the sample mean changes as sample size goes up.Consider a coin flip. If we say Heads is $1$ and Tails is $0$, then there's a 50% chance of getting a 1 and a 50% chance of getting a 0, which definitely doesn't match our definition of a normal distribution. The average of several coin tosses, where Heads is 1 and Tails is 0, is equal to the proportion of heads in those coin tosses (which is equivalent to the mean value of the coin tosses), so the CLT should hold true if we compute the sample proportion of heads many times.Write a function called sample_size_n that takes in a sample size $n$. It should return an array that contains 5000 sample proportions of heads, each from $n$ coin flips.<jupyter_code>def sample_size_n(n):
    coin_proportions = make_array('Heads', 'Tails') # our coin is fair
    heads_proportions = make_array()
    for i in np.arange(5000):
        prop_heads = np.count_nonzero(np.random.choice(coin_proportions, 100) == 'Heads')
        heads_proportions =np.append(heads_proportions,prop_heads )
    return heads_proportions
#heads_proportions

sample_size_n(100)<jupyter_output><empty_output><jupyter_text>Now we will test the second claim of the CLT: That the SD of the sample mean is the SD of the original distribution, divided by the square root of the sample size.We have imported the flight delay data and computed its standard deviation for you.<jupyter_code>united = Table.read_table('/content/united_summer2015.csv')
united_std = np.std(united.column('Delay'))
united_std<jupyter_output><empty_output><jupyter_text>**2.3 Question 3** Write a function called empirical_sample_mean_sd that takes a sample size n as its argument. The function should simulate 500 samples with replacement of size n from the flight delays dataset, and it should return the standard deviation of the means of those 500 samples.<jupyter_code>def empirical_sample_mean_sd(n):
    sample_means = make_array()
    for i in np.arange(500):
        sample = united.sample(100, with_replacement=False)
        sample_mean = np.mean(sample.select('Delay').sample().column(0))
        sample_means = np.append(sample_means,sample_mean)
    return np.std(sample_means)

empirical_sample_mean_sd(100)<jupyter_output><empty_output><jupyter_text>**2.4	Question 4**Now, write a function called predict sample mean sd to find the predicted value of the standard deviation of means according to the relationship between the standard deviation of the sample mean and sample size that is discussed in the textbook. It takes a sample size n (a number) as its argument. It returns the predicted value of the standard deviation of the mean delay time for samples of size n from the flight delays (represented in the table united).<jupyter_code>def predict_sample_mean_sd(n):   
   sd_of_means = united_std/math.sqrt(n)
   return sd_of_means   
predict_sample_mean_sd(10)
sd_table = Table().with_column('Sample Size', np.arange(1,101))
predicted = sd_table.apply(predict_sample_mean_sd, 'Sample Size')
empirical = sd_table.apply(empirical_sample_mean_sd, 'Sample Size')
sd_table = sd_table.with_columns('Predicted SD', predicted, 'Empirical SD', empirical)
sd_table.scatter('Sample Size')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0745911447945752, 'fraction_numerical': 0.016912644595133627, 'mean_word_length': 5.891698735568994, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '448805', 'license': 'MIT', 'n_tokens_mistral': 3684, 'n_tokens_neox': 3403, 'n_words': 1429, 'path': 'INFO_5502_Assign_6.ipynb', 'repo_name': 'Kate5-7-2021/INFO-5502-SU2021', 'size': '12535'}