<jupyter_start><jupyter_text>Random Variables in Python Import basic libraries needed, namely numpy (array manipulation, etc.), scipy (random numbers, etc.) and matplotlib.pyplot (plotting).<jupyter_code>import numpy as np
import scipy.stats as sp
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Basic example: generate binomial random variable Bin(1000,1/2). Good programming practice is to declare the parameters first so that you can change them if needed in one single place.<jupyter_code>n=1000
p=1/2
sp.binom.rvs(n, p)<jupyter_output><empty_output><jupyter_text>Obviously, the number is random and it may change in the following run. Meaning other instances of the code may get different answer. For reproducibility, one can fix the number generator to give the same (but random) result by specifying a seed (initial setup of the random number generator).<jupyter_code>sp.binom.rvs(n, p, random_state = 12345)<jupyter_output><empty_output><jupyter_text>This gives 529 in all implementations. Next, generate 1,000,000 such random variables. For illustration, we measure the time required to do so.<jupyter_code>simulations = 1000000
import time
start = time.time()
a = sp.binom.rvs(n, p, size = simulations, random_state = 12345)
end = time.time()
print (end - start)
a<jupyter_output><empty_output><jupyter_text>Basic statistics: average, max, min.<jupyter_code>np.mean(a)
np.min(a)
np.max(a)<jupyter_output><empty_output><jupyter_text>Question: What is a probability that you get exactly 500, namely P(X=500)? That is extracted from .pmf (probabilty mass function).<jupyter_code>k=500
sp.binom.pmf(k, n, p)<jupyter_output><empty_output><jupyter_text>Can we plot these probabilities? Yes, but we need to specify the range of values, plus keep in mind that this is an integer variable. While the entire range of values is [0,1000], the values outside some reasonable region, say [450, 550], are remotely unlikely.<jupyter_code>plt.figure(2)
x =  np.arange(450,551) # Python chops off the last element, so 551 is not a part of the array
plt.plot(x,sp.binom.pmf(x, n, p),'o', ms=2.5)
plt.xlabel('Heads')
plt.ylabel('Probability')
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>Histogram of 1,000,000 simulations overlay.<jupyter_code>plt.figure(3)
plt.hist(a,density=True,bins=x)
plt.plot(x,sp.binom.pmf(x, n, p))
plt.xlabel('Heads')
plt.ylabel('Probability')
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>The concept of cumulative distribution function: F(x) = P(X <= x). For Bin(1000,1/2), k=500:<jupyter_code>np.sum(a==500)/simulations
k=500
sp.binom.cdf(k,n,p)<jupyter_output><empty_output><jupyter_text>Make the plot.<jupyter_code>plt.figure(3)
plt.plot(x,sp.binom.cdf(x, n, p),'o', ms=2.5)
plt.xlabel('Heads')
plt.ylabel('Probability')
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>A cut between "likely" and "unlikely" values of the random variable X. This corresponds to finding a value "Q" such that P(X<=Q) is small. Let us try some values between [460, 480]?<jupyter_code>test_vals = np.arange(460,481)
sp.binom.cdf(test_vals, n, p)
k = 469
sp.binom.cdf(k, n, p)
sp.binom.cdf(474, 1000, 0.5)<jupyter_output><empty_output><jupyter_text>Suppose you are given the value of probability alpha such thatF(Q(alpha))=P(X<=Q(alpha)) <= alpha.Then Q(alpha) = F^(-1)(alpha)is called quantile. The quantile is computed using .ppf in scipy.<jupyter_code>alpha = 0.025
sp.binom.ppf(alpha,n,p)
alpha = 0.05
sp.binom.ppf(alpha,n,p)<jupyter_output><empty_output><jupyter_text>Median = 50% Quantile<jupyter_code>alpha = 0.5
sp.binom.ppf(alpha,n,p)<jupyter_output><empty_output><jupyter_text>The unlikely region is also on the right side of the distribution, that corresponds to high alphas.<jupyter_code>alpha = 0.95
sp.binom.ppf(alpha,n,p)
alpha = 0.975
sp.binom.ppf(alpha,n,p)<jupyter_output><empty_output><jupyter_text>95% confidence interval<jupyter_code>alpha = 0.95
print(sp.binom.interval(alpha, n, p))<jupyter_output>(469.0, 531.0)<jupyter_text>Other statistics: mean, variance.<jupyter_code>sp.binom.mean(n,p)
sp.binom.var(n,p)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12882991556091677, 'fraction_numerical': 0.03691194209891435, 'mean_word_length': 6.88212927756654, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '396410', 'license': 'MIT', 'n_tokens_mistral': 1666, 'n_tokens_neox': 1493, 'n_words': 390, 'path': 'lecture_notes/Lecture 2.ipynb', 'repo_name': 'coinflip112/stoch_models_for_finance', 'size': '4145'}