<jupyter_start><jupyter_text>Bayes Rule** Data $x$ : ** * knowns / observed** Parameters $\theta$ : ** * hypotheses / unknowns / estimated** Prior $p(\theta)$ : *** what do we know about $\theta_i$ prior to observing $x$?**Likelihood $\mathcal{L}(x)$ = $p( x | \theta_i )$ : ** * what is the probability of observing $x$ given $\theta_i$ ?* $\mathcal{L}(x)$ = $p( x | \theta_i )$** Posterior $p(\theta | x)$ : *** what is the probability of $\theta_i$, given that we've observed $x$?$$p( \theta_i | x ) = \frac{p(x|\theta_i)p(\theta_i)}{\sum_i^n p(x|\theta_i)}$$  Simple Example:You take 10-question True/False quiz and answer 9 questions correctly. Based on this data, what can we infer about your true rate of answering T/F questions correctly? (Example from Lee & Wagenmakers, Cognitive Bayesian Modeling) Data* number of questions attempted (n)* number of successes (s)* $x = \{s, n\}$ Parameter of interest * Rate of answering a question correct, $\theta$ Prior,   $p(\theta)$ * $\theta \sim \mathcal{U}(0,1)$ Likelihood function: * $\mathcal{L}(x) = p(s|\theta,n) = \binom{n}{s}\theta^s(1-\theta)^{n-s}$   Estimating the Posterior $p(\theta|x)$ Markov Chain Monte-Carlo (MCMC) Sampling* Estimates Posterior probability density for all\* values of $\theta$ \* **all** meaning all values within specified range and precision   Bayesian Regression Example<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import pymc3 as pymc
from pymc3 import Model, Normal, HalfNormal
from scipy import optimize<jupyter_output><empty_output><jupyter_text>Generate Dataset (two predictors)<jupyter_code># "True" parameter values
#    alpha: intercept
#    beta: weights of X1 and X2 in predicting Y
#    sigma: some random noise  
alpha = 1
sigma = 1
beta = [1, 2.5]
# Number of observations (e.g., "subjects")
size = 100
# Predictor variables
X1 = np.random.randn(size)
X2 = np.random.randn(size) * 0.2
# Simulate outcome variable
Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(size)*sigma
plt.scatter(X1, Y, color='Gray')<jupyter_output><empty_output><jupyter_text>Define multiple regression model (two predictors)<jupyter_code>basic_model = Model()
with basic_model:
    # Priors for unknown model parameters
    alpha = Normal('alpha', mu=0, sd=10)
    beta = Normal('beta', mu=0, sd=10, shape=2)
    sigma = HalfNormal('sigma', sd=1)
    # Expected value of outcome
    mu = alpha + beta[0]*X1 + beta[1]*X2
    # Likelihood (sampling distribution) of observations
    Y_obs = Normal('Y_obs', mu=mu, sd=sigma, observed=Y)<jupyter_output><empty_output><jupyter_text>Maximum a-posterior estimation (MAP)<jupyter_code>map_estimate = pymc.find_MAP(model=basic_model)
for k,v in map_estimate.items(): 
    print('{}={}'.format(k,v))

a = float(map_estimate['alpha'])
b1, b2 = map_estimate['beta']
x = np.linspace(-2, 2, 2)
yhat = a + b1*x + b2*x*.2

plt.scatter(X1, Y, color='Gray')
plt.plot(x, yhat, 'k')<jupyter_output>Optimization terminated successfully.
         Current function value: 148.054964
         Iterations: 19
         Function evaluations: 26
         Gradient evaluations: 26
alpha=0.918699201077
beta=[ 1.0133  2.3048]
sigma_log_=-0.0422637602261<jupyter_text>MCMC sampling of $\alpha$, $\beta_1$, $\beta_2$ posteriors<jupyter_code>from pymc3 import NUTS, sample, traceplot
with basic_model:
    # draw 2000 posterior samples
    trace = sample(2000)
traceplot(trace);
# get MCMC sample traces 
aSamp = trace.get_values('alpha')
b1Samp, b2Samp = trace.get_values('beta').T

# calc mean of posterior samples
aMu = aSamp.mean()
b1Mu = b1Samp.mean()
b2Mu = b2Samp.mean()

# MCMC samples of posterior converge to MAP 
print(np.isclose(a, aMu, 1e-3))
print(np.isclose(b1, b1Mu, 1e-3))
print(np.isclose(b2, b2Mu, 1e-3))<jupyter_output>True
True
True
{'fraction_non_alphanumeric': 0.14478918058870327, 'fraction_numerical': 0.03367806947759215, 'mean_word_length': 4.812018489984592, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '607734', 'license': 'MIT', 'n_tokens_mistral': 1419, 'n_tokens_neox': 1311, 'n_words': 378, 'path': 'chapter_5/intro_bayesian_stats.ipynb', 'repo_name': 'pbeukema/ML_CNBC', 'size': '3771'}