<jupyter_start><jupyter_text>Workshop 5: Statistics (Optional)<jupyter_code># standard preamble
import numpy as np
import scipy as sp      
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>2d distributionsYou can create two independent samples of events and plot their distribution as a *scatter* plot:<jupyter_code>x = np.random.standard_normal(size=1000)
y = np.random.standard_normal(size=1000)
plt.scatter(x,y)
plt.xlabel('x')
plt.ylabel('y')<jupyter_output><empty_output><jupyter_text>You can compute the correlation matrix for two variables:<jupyter_code>print (sp.corrcoef(x,y))<jupyter_output><empty_output><jupyter_text>Although more instructive perhaps is to print the full covariance matrix:<jupyter_code>print (sp.cov(x,y))<jupyter_output><empty_output><jupyter_text>Here is a cute example of plotting projection histograms together with the scatter plot:(from http://matplotlib.org/examples/pylab_examples/scatter_hist.html )<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import NullFormatter

# the random data
x = np.random.randn(1000)
y = np.random.randn(1000)

nullfmt = NullFormatter()         # no labels

# definitions for the axes
left, width = 0.1, 0.65
bottom, height = 0.1, 0.65
bottom_h = left_h = left + width + 0.02

rect_scatter = [left, bottom, width, height]
rect_histx = [left, bottom_h, width, 0.2]
rect_histy = [left_h, bottom, 0.2, height]

# start with a rectangular Figure
plt.figure(1, figsize=(8, 8))

axScatter = plt.axes(rect_scatter)
axHistx = plt.axes(rect_histx)
axHisty = plt.axes(rect_histy)

# no labels
axHistx.xaxis.set_major_formatter(nullfmt)
axHisty.yaxis.set_major_formatter(nullfmt)

# the scatter plot:
axScatter.scatter(x, y)

# now determine nice limits by hand:
binwidth = 0.25
xymax = np.max([np.max(np.fabs(x)), np.max(np.fabs(y))])
lim = (int(xymax/binwidth) + 1) * binwidth

axScatter.set_xlim((-lim, lim))
axScatter.set_ylim((-lim, lim))

bins = np.arange(-lim, lim + binwidth, binwidth)
axHistx.hist(x, bins=bins)
axHisty.hist(y, bins=bins, orientation='horizontal')

axHistx.set_xlim(axScatter.get_xlim())
axHisty.set_ylim(axScatter.get_ylim())

axScatter.set_xlabel('x')
axScatter.set_ylabel('y')

plt.show()<jupyter_output><empty_output><jupyter_text>You can also create a correlated sample:<jupyter_code># mean values of two variables
mean = [0, 0] 

#  covariance matrix 
# Note that the covariance matrix must be positive semidefinite (a.k.a. nonnegative-definite). 
# Otherwise, the behavior of this method is undefined and backwards compatibility is not guaranteed.
cov = [[1, 0.8], [0.8, 1]]  

# produce a sample 
x, y = np.random.multivariate_normal(mean, cov, 1000).T   

# plot -- this looks like a streak
plt.scatter(x,y)
plt.xlabel('x')
plt.ylabel('y')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12227687983134224, 'fraction_numerical': 0.018973998594518624, 'mean_word_length': 5.960880195599022, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '180577', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1111, 'n_tokens_neox': 1040, 'n_words': 251, 'path': 'Week06/WS05/Workshop05_optional.ipynb', 'repo_name': 'ds-connectors/Physics-88-Fa21', 'size': '2846'}