<jupyter_start><jupyter_text>Lab 3: Asking a statistical question Problem 1<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import scipy
import random
import math
from scipy import stats
plt.rcParams["figure.figsize"] = (15,10)
d = np.append(stats.norm.rvs(loc = 12., scale = 0.4, size = 100000), [10., 10.3, 2.1, 0., 0., 15.6, 22.3, 12.7])
fig, ax = plt.subplots(1, 1)
ax.hist(d,100, density=True)
plt.tick_params(labelsize = 24)
plt.yscale('log')
#x = np.linspace(4.95,5.05,1000)
#ax.plot(x,stats.norm.pdf(x,loc = 5., scale = 0.01),linewidth = 8,alpha = 0.7)
plt.show()<jupyter_output><empty_output><jupyter_text>**A)** **1)** By look at data, there are appearently exist some anomaly data exist, distributed on the both side of the distribution. **2)** I believe the data that are beyond $+-5\sigma$ away from the mean are bad data. **3)** Since the standard diviation （$\sigma$）is 0.4, and the mean is 12, so $+-5\sigma$ correspond to $12 + 0.4*5 = 14$ and $12 - 0.4*5 = 10$. Therefore, for data that are smaller than 10 and bigger than 14 are bad data. **4)** Count the good and bad data as follow.<jupyter_code>BadT = 0
for x in d:
    if x < 10 or x > 14:
        BadT += 1
TrueBadT = 0
for x in [10., 10.3, 2.1, 0., 0., 15.6, 22.3, 12.7]:
    if x < 10 or x > 14:
        TrueBadT += 1
FalseBadT = BadT - TrueBadT
FalseGoodT = 8 - TrueBadT
TrueGoodT = d.size - BadT - FalseGoodT
print(TrueGoodT)
print(FalseGoodT)
print(FalseBadT)
print(TrueBadT)<jupyter_output>100000
3
0
5<jupyter_text>||True T|Bad T||---|---|---||Test Good|100000|3||Test Bad|0|5|  **B)** If the threshold is lower, that the $\sigma$ value is smaller, the number of omissions will increase. If the $\sigma$ value is bigger, the number of omissions will decrease. **C)** There are some mistakes of commission, and not all of them are avoidable, since if the bad data got lucky and get a similar value with the true distribution, they will be accepted to be the true data. Usually, if we set the statistical threshold to be smaller, then bad data are more likely to be omit.  Problem 2 **1)** What is the distance of the movement in arcsecond that correspond to 5 sigma away from the mean. **2)** For what distance $\sqrt{x^2 +y^2}$ away from the mean includes area of the distribution that the sum (integral) of pdf within the area is correspond to the significance of 5 sigma, which can be calculated form $$5\sigma = \int\int pdf(x,y) dydx$$ **3)** In the following code, I will try to find the 5 sigma percential of the distance from the mean, and find that value of distance correspond to that percentile.<jupyter_code>a = np.vstack((stats.norm.rvs( scale = 1, size = 100000), stats.norm.rvs( scale = 1, size = 100000)))
r = []
for i in range (0, 100000):
    r.append(math.sqrt(a[0,i]**2 + a[1,i]**2))
r.sort()
prob = stats.norm.cdf(5)
n = math.floor(prob*100000)
print(r[n])<jupyter_output>200000
5.159039047446082<jupyter_text>The distance in arcseconds is 5.159  Problem 3 **1)** If there is no moon, than we expect the background to have the pdf() of $60*8*15=7200$ convolved (summed) poisson distribution. The stronger the moon shadow, the less cosmic rays we expect to detect. **So what is the probability of the number of cosmic rays we see is <= 6800?** **2)** Find the pdf() of back ground by convolving (summing) a poisson distribtion with mean 1 with itself for 7200 times. Then integrate $$ \int \limits _{-\infty} ^{6800} pdf(x)dx $$<jupyter_code>d1 = stats.poisson.rvs(mu = 1, size = 10000)
for i in range(1,7200):
    d1 +=stats.poisson.rvs(mu = 1, size = 10000)
fig, ax = plt.subplots(1,2)
fig.set_size_inches(20,7)
ax[0].hist(d1, 100)
ax[0].set_title("Sum poisson with mean 1 7200 times", fontsize=24)
ax[0].set_yscale("log")
ax[0].tick_params(labelsize = 24)
ax[0].set_xlabel("x", fontsize=24)
ax[0].set_ylabel("Count", fontsize=24)

d2 = stats.poisson.rvs(mu = 7200, size = 10000)
ax[1].hist(d2, 100)
ax[1].set_title("Poisson with mean 7200", fontsize=24)
ax[1].set_yscale("log")
ax[1].tick_params(labelsize = 24)
ax[1].set_xlabel("x", fontsize=24)
ax[1].set_ylabel("Count", fontsize=24)<jupyter_output><empty_output><jupyter_text>I calculate and plot the semi-log graph of summing poisson with mean 1 7200 times and the poisson distribution with mean 7200 above. I found that two distribution is acutally same. So, I would simply use a poisson distribution with mean 7200. **3)** Calculation as below, I am trying to find the probability of the distribution having a value smaller than 6800 and convert it to sigma value.<jupyter_code>d2 = stats.poisson.rvs(mu = 7200, size = 10000000)
Count = 0
for x in d1:
    if x <= 6800:
        Count += 1
prob = Count/10000000
print(prob)
print(stats.norm.ppf(prob))<jupyter_output>0.0
-inf<jupyter_text>Probability is too small, that I unable to find the probability through sampling distribution, so I will use cdf below.<jupyter_code>prob = stats.poisson.cdf(6800, mu = 7200)
print(stats.norm.ppf(prob))<jupyter_output>-4.750747965777188
{'fraction_non_alphanumeric': 0.12847498014297062, 'fraction_numerical': 0.07525814138204924, 'mean_word_length': 4.559602649006623, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '368875', 'license': 'MIT', 'n_tokens_mistral': 1998, 'n_tokens_neox': 1726, 'n_words': 657, 'path': 'LAB3/LAB3.ipynb', 'repo_name': 'HaowenGuan/PHYS-434', 'size': '5036'}