<jupyter_start><jupyter_text>Bias----------------------------------------Bias - HyperStat Onlinehttp://davidmlane.com/hyperstat/A9257.htmlA statistic is biased if, in the long run, it consistently over or underestimates the parameter it is estimating. More technically it is biased if its expected value is not equal to the parameter. A stop watch that is a little bit fast gives biased estimates of elapsed time. Bias in this sense is different from the notion of a biased sample. A statistic is positively biased if it tends to overestimate the parameter; a statistic is negatively biased if it tends to underestimate the parameter. An unbiased statistic is not necessarily an accurate statistic. If a statistic is sometimes much too high and sometimes much too low, it can still be unbiased. It would be very imprecise, however. A slightly biased statistic that systematically results in very small overestimates of a parameter could be quite efficient. -----------------------https://web.microsoftstream.com/video/fbf7508f-99a2-4054-bc54-1cdca68450e8Cognitive biases in data analytics: https://web.microsoftstream.com/video/581a9928-3899-43f4-83ef-137d7904eeaf<jupyter_code>%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as ss
import seaborn as sns

plt.style.use('ggplot') # R statistical style
plt.rcParams['figure.figsize'] = 14, 10<jupyter_output><empty_output><jupyter_text>Location and scale<jupyter_code>x = np.linspace(-10.0, 10.0, 1000)

plt.fill(x, ss.norm.pdf(x, loc= 0.0, scale=1.0), label="$\mu = 0.0, \sigma = 1.0$", c='b', alpha=0.6, lw=3.0)
plt.fill(x, ss.norm.pdf(x, loc= 2.0, scale=1.0), label="$\mu = 2.0, \sigma = 1.0$", c='r', alpha=0.6, lw=3.0)
plt.fill(x, ss.norm.pdf(x, loc= 0.0, scale=2.0), label="$\mu = 0.0, \sigma = 2.0$", c='g', alpha=0.6, lw=3.0)
plt.fill(x, ss.norm.pdf(x, loc=-1.0, scale=0.6), label="$\mu =-1.0, \sigma = 0.6$", c='y', alpha=0.6, lw=3.0)

plt.title("Normal distribution for different $\mu$ and $\sigma$")
plt.legend()<jupyter_output><empty_output><jupyter_text>These curves represent bell shape curves. The blue and red curves go on to infinity on the x axis. All we are really interested in is the M and SD. No matter what the M and SD are, if you go out 1 SD left and right, 68.27% of the distribution scores lie here, for 2 SD its 95.45%, 3 SDs is 99.7%. Probability<jupyter_code>x = np.linspace(-3.0, 3.0, 1000)
y = ss.norm.pdf(x, loc= 0.0, scale=1.0)

xseg = x[np.logical_and(-1.0 < x,  x < 1.4)]
yseg = y[np.logical_and(-1.0 < x,  x < 1.4)]

plt.plot(x, y, color='k', alpha=0.5)
plt.fill_between(xseg, yseg, color='b', alpha=0.5)
plt.axvline(x=-1.0, color='grey', linestyle=':')
plt.axvline(x= 1.4, color='grey', linestyle=':')

plt.text(0.2, 0.15,
         r'$P (a \leq X \leq b) = \int_a^b \frac{1}{\sqrt{2\pi\sigma^2} } e^{ -\frac{(x-\mu)^2}{2\sigma^2} } \, dx$',
         horizontalalignment='center', size=17)

plt.axhline(y= 0.0, color='black', linestyle='-')

plt.title("Probability of $x$ between $a$ and $b$");
x = np.linspace(-5.0, 5.0, 10000)
plt.plot(x, ss.norm.pdf(x, loc=0.0, scale=1.0), 'k-', lw=1.0)

xseg = x[np.logical_and(-5.0 <= x, x <= -3.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='y', alpha=0.5, linewidth=0)
xseg = x[np.logical_and(-3.0 <= x, x <= -2.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='r', alpha=0.5, linewidth=0)
xseg = x[np.logical_and(-2.0 <= x, x <= -1.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='g', alpha=0.5, linewidth=0)
xseg = x[np.logical_and(-1.0 <= x, x <=  1.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='b', alpha=0.5, linewidth=0, label="$1 \sigma = 68.27\%$")
xseg = x[np.logical_and( 1.0 <= x, x <=  2.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='g', alpha=0.5, linewidth=0, label="$2 \sigma = 95.45\%$")
xseg = x[np.logical_and( 2.0 <= x, x <=  3.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='r', alpha=0.5, linewidth=0, label="$3 \sigma = 99.73\%$")
xseg = x[np.logical_and( 3.0 <= x, x <=  5.0)]
plt.fill_between(xseg, ss.norm.pdf(xseg), color='y', alpha=0.5, linewidth=0)

plt.title("Normal Disribution - Probability Distribution Function")
plt.legend();<jupyter_output><empty_output><jupyter_text>Sampling distribution<jupyter_code>np.set_printoptions(formatter={'float': lambda x: "{0:6.3f}".format(x)})

sampsize = 10
nosamps = 1000 # Thus 1000 samples or 10 persons heights

samp = np.random.standard_normal((nosamps, sampsize))

print(samp)
mean = samp.sum(axis=1) / sampsize
print(mean) # prints out all of the different means from all of the different samples

# Doing this over and over will eventually lead us back to exactly ZERO<jupyter_output>[ 0.479 -0.178  0.184 -0.333 -0.348 -0.043  0.163 -0.295  0.205 -0.632
  0.417  0.045 -0.240 -0.081  0.191  0.357 -0.013  0.085  0.044 -0.132
  0.131 -0.104 -0.128 -0.186  0.364 -0.480 -0.080  0.745 -0.404  0.454
  0.199 -0.154  0.073  0.208 -0.473 -0.155 -0.123  0.136 -0.284 -0.153
  0.512  0.019 -0.221  0.120 -0.366  0.186 -0.353 -0.374 -0.112 -0.383
 -0.111  0.557  0.106  0.287 -0.855  0.338 -0.005  0.131 -0.533 -0.226
 -0.419  0.135  0.098 -0.271 -0.525 -0.365  0.191  0.284  0.350  0.288
 -0.455  0.300 -0.553 -0.299  0.706  0.041 -0.194  0.388 -0.363  0.080
  0.011 -0.406 -0.219 -0.048 -0.085  0.093 -0.235  0.359 -0.174 -0.285
  0.562  0.357  0.665  0.023  0.063 -0.298 -0.368 -0.605 -0.109 -0.234
  0.297 -0.069  0.386 -0.031 -0.187  0.060 -0.081 -0.245 -0.388  0.004
 -0.390 -0.222 -0.221 -0.585 -0.058 -0.508 -0.063  0.173  0.203 -0.546
  0.031  0.490 -0.246  0.296  0.069  0.253 -0.122 -0.378  0.018 -0.454
  0.046  0.083 -0.396 -0.077  0.270  0.218  0.084  0.021  0.451 -0.155
  0.41[...]<jupyter_text>Heres where it ties into BIAS:If you work out a population variance.<jupyter_code># Calculate the variance. (Square of the SD)
# Q: How do we get the variance?:
# ANS: Take the mean from each of the values then square all othe values and add them together and then 
# divide by the number of values you started with....
# BUT !! This method is biased because it typically UNDERESTIMATES the real sampling distribution variance
# LOOK:

vari = (samp - mean[:, np.newaxis])**2
vari = vari.sum(axis=1) / (sampsize)

# Without the correction.
mean_vari = vari.sum() / nosamps
print(mean_vari)

# Using Bessel's correction: https://en.wikipedia.org/wiki/Bessel%27s_correction
print(mean_vari * (sampsize / (sampsize - 1.0)))
# Our variance first output
# The variance is supposed to be 1 (second output)
# We divide by -1 to have an unbiased estimate 

sns.distplot(vari)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17657794676806082, 'fraction_numerical': 0.12653992395437264, 'mean_word_length': 3.4016064257028114, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '667102', 'license': 'Apache-2.0', 'n_tokens_mistral': 3104, 'n_tokens_neox': 2638, 'n_words': 753, 'path': 'Statistical-bias.ipynb', 'repo_name': 'Datagatherer2357/Statistical-bias', 'size': '6575'}