<jupyter_start><jupyter_text>Lab 1When reporting probabilities in a scientific context, the standard is to convert this probabilty into a sigma value, which represents the probabilty of the data To convert sigma values back to a probabilty, some convoluted integration is required. Luckily, python has these capabilites built in. The *erfc* function is one of several scipy functions which returns a probabilty given an input sigma value.<jupyter_code>for x in range(1, 6):
    print(sp.special.erfc(x))<jupyter_output>0.15729920705028516
0.004677734981047266
2.2090496998585445e-05
1.541725790028002e-08
1.5374597944280347e-12<jupyter_text>Here we can see that *erfc* values roughly match up with the negative 1, 2, 3, and 4 sigma values on a z table (or the positive values subtracted from 1), giving us the probabilty that our target event happened outside of our measured sigma value. As previously mentioned, the accepted standard is to convert this probabilty into a sigma value, but a probabilty can also be converted back into a sigma, as is shown below.<jupyter_code>sp.stats.norm.ppf(3e-7)<jupyter_output><empty_output><jupyter_text>Here we see the sigma value reported as a negative number. This is probably due to the fact that the built in function uses the left side of the normal distribution to find the associated sigma value as this would come up first when searching from negative to positive.  Rayleigh Distribution A Rayliegh distribution is distinct in that it is not identical on each side of its peak. Applications of Rayleigh distributions are most common in places where long-lived or large events are less common than those of shorter length. Examples are wave height or product lifespan. Let's create a sample set of data with built-in Python functions.<jupyter_code>#def prob_graph(loc, scale, xleft, xright, size)

d = sp.stats.rayleigh.rvs(loc = 2.0, scale = 1, size = 100000)

xleft = 1.95
xright = 7

fig, ax = plt.subplots(1, 1)
ax.hist(d,50, density=True)
plt.tick_params(labelsize = 20)
plt.xlim(xleft, xright)
x = np.linspace(xleft,xright,1000)
ax.plot(x, sp.stats.rayleigh.pdf(x,loc = 2, scale = 1),linewidth = 7,alpha = 0.6)
plt.show()<jupyter_output><empty_output><jupyter_text>Looks pretty good! The 100,000 sample size seems to have created a pretty accurate distrobution. However, towards the top end (~X = 6), we can't really tell what is going on. The height of the distribution is controlled by the 'scale' factor in the pdf, with a higher scale representing a wider and shorter distribution. Plotting our data on a semilog graph reveals some interesting secrets.<jupyter_code>#def prob_graph(loc, scale, xleft, xright, size)

d = sp.stats.rayleigh.rvs(loc = 2.0, scale = 1, size = 100000)

xleft = 1.95
xright = 7

fig, ax = plt.subplots(1, 1)
ax.set_yscale('log')
ax.hist(d,50, density=True)
plt.tick_params(labelsize = 20)
plt.xlim([xleft, xright])
x = np.linspace(xleft,xright,1000)
ax.plot(x,sp.stats.rayleigh.pdf(x,loc = 2, scale = 1),linewidth = 7,alpha = 0.6)
plt.show()<jupyter_output><empty_output><jupyter_text>Even with the large sample size, there is a suprisingly large amount of innacuracy towards the tail of the distribution. The theoretical data above could represent the lifetime of a company's product in years. If a similar, slightly redesigned product has a lifespan of 4.5 years, what is the chance that this is not an improvement over our original product?<jupyter_code>sp.stats.rayleigh.cdf(4.5)
sp.stats.norm.ppf(0.999959934702607)<jupyter_output><empty_output><jupyter_text>This comes out to be a sigma value of 3.94, which while very significant, would not be accepted by the scientific community in a physics-related context.  Binomial Distribution<jupyter_code>fig, ax = plt.subplots(1,1)
n, p = 100, .45

m, v, s, k = sp.stats.binom.stats(n, p, moments="mvsk")

x = np.arange(sp.stats.binom.ppf(0.01, n, p), sp.stats.binom.ppf(0.99, n, p))
plt.xlim(30, 60)

ax.plot(x, sp.stats.binom.pmf(x, n, p), 'o')

ax.vlines(x, 0, sp.stats.binom.pmf(x, n, p))
fig, ax = plt.subplots(1,1)
n, p = 100, .45

m, v, s, k = sp.stats.binom.stats(n, p, moments="mvsk")

x = np.arange(sp.stats.binom.ppf(0.01, n, p), sp.stats.binom.ppf(0.99, n, p))
plt.xlim(30, 60)

ax.plot(x, sp.stats.binom.pmf(x, n, p), 'o')
ax.set_yscale('log')
ax.vlines(x, 0, sp.stats.binom.pmf(x, n, p))<jupyter_output><empty_output><jupyter_text>In a semilog plot, the distribution takes on the shape of a slightly skewed parabola, looking very similar to, but slightly different from a Gaussian curve. Using the distrubition above, let's assume we flip a coin that is slightly biased to one side. We'd assume most outcomes would land around 45 on one side to 55 on the other, which is reflected in the graph. One difference in comparison to our previous question, which dealt with a continuous probabilty, is that our probabilty only takes on integer values, which makes the binomial distribution good for counting events if we know the general probability that it should happen. Unlike individual data points, statistics about the binomial distribution don't have to necessarily be an integer value. If the average family has 1.9 kids, that clearly does not mean that any familiy has that exact value. So what happens if we get 60 heads on a second coin with unknown properties? Could it be the same type of coin?<jupyter_code>sp.stats.binom.cdf(60, n, p)
sp.stats.norm.ppf(0.9990617681011207)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08924317390507605, 'fraction_numerical': 0.04617921935129192, 'mean_word_length': 5.030939226519337, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '283845', 'license': 'MIT', 'n_tokens_mistral': 1948, 'n_tokens_neox': 1715, 'n_words': 712, 'path': 'labs/.ipynb_checkpoints/lab1-checkpoint.ipynb', 'repo_name': 'deanlr/phys434-lab1', 'size': '5457'}