<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Practical Assignment 51.1&nbsp;&nbsp;Objective1.2&nbsp;&nbsp;Gaussian Process regression1.3&nbsp;&nbsp;Model 11.4&nbsp;&nbsp;Model 2: Shared Values  Practical Assignment 5Write a probabilistic programme in pymc3 which implements a Gaussian Processes Regress to find functions that model the relationship between the following.x = [0, 0.1, 2, 4] & y = [4, 0.9, 3.6, 5] Objective- Write a probabilistic programme in pymc3 which implements a Gaussian Processes Regress to find functions that model the relationship the given points   Gaussian Process regressionGaussian Process regression is a non-parametric approach to regression or data fitting that assumes that observed data points y are generated by some unknown latent function `f(x)`. The latent function `f(x)` is modeled as being multivariate normally distributed (a Gaussian Process)Denoted as: `f(x)∼GP(m(x;θ),k(x,x′;θ)).``m(x;θ)` is the mean function, and `k(x,x′;θ)` is the covariance function. In many applications, the mean function is set to 0 because the data can still be fit well using just covariances.`θ`  is the set of hyperparameters for either the mean or covariance function. These are the unknown variables. They are usually found by maximizing the marginal likelihood. This approach is much faster computationally than MCMC, but produces a point estimate, `θMAP`.Source: [Pymc3](https://pymc3-testing.readthedocs.io/en/rtd-docs/notebooks/GP-introduction.html)<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import pymc3 as pm
import arviz as az
import theano.tensor as tt
import theano as T

sns.set_style('whitegrid')
# Regression 
x = np.array([0, 0.1, 2, 4])
y = np.array([4, 0.9, 3.6, 5])
plt.plot(x,y, color = 'b')
plt.scatter(x,y, color = 'r')
plt.title('Regression')
plt.xlabel('x')
plt.ylabel('y')
plt.show()<jupyter_output><empty_output><jupyter_text>Model 1 **Gaussian Process** The `Gaussian processes model` is a probabilistic supervised machine learning framework that has been widely used for regression and classification tasks. A Gaussian processes regression (GPR) model can make predictions incorporating prior knowledge (kernels) and provide uncertainty measures over predictions. `Gaussian Processes` are convenient for building Bayesian non-parametric models capability of them being utilized as prior distributions over functions.**Model** The aim is to implement a model according to the specification of the following equations:$p(f(X_*) | X_*,X,y) ~ N (\mu, \sum)$       $\mu = K_*^T K^{-1} y$         where: $K = K(X,X)$ $K_* = K(X_*,X)$ $K_{**} = K(X_*,X_*)$ In building the model, the parameters of interest will be $\epsilon$ and $\ell$. Whereby  $\epsilon$ will represent the variance of the normal likelihood and $\ell$ will be the length-scale parameter of the exponentiated quadratic kernel. The marginal likelihood is used as a likelihood, the marginal likelihood is the integral of the likelihood and prior as follows: $p(y | X, \theta) ~ \[ \int p(y | f, X, \theta) p(f | X, \theta) \,df \]$ where: - $X$ is the independent variable - $y$ is the dependent variable - $f$ is a function**Probabilistic Modeling**- In probabilistic modeling, the parameters are random variables (RVs) thus the probabilistic programs consists of observed and unobserved random variables.- The observed random variables are defined through the likelihood distributions.- The unobserved random variables are defined through the prior distributions<jupyter_code># A one dimensional column vector of inputs.
X = x[:, None]
with pm.Model() as model_reg:
    # hyperprior for lengthscale kernel parameter
    ℓ = pm.Gamma('ℓ', 2, 0.5)
    # instanciate a covariance function
    cov = pm.gp.cov.ExpQuad(1, ls=ℓ)
    # instanciate a GP prior
    gp = pm.gp.Marginal(cov_func=cov)
    # prior
    ϵ = pm.HalfNormal('ϵ', 0.5)
    # likelihood
    y_pred = gp.marginal_likelihood('y_pred', X=X, y=y, noise=ϵ)
    trace_reg = pm.sample(2000)
sns.set_palette('Set1')  
az.plot_trace(trace_reg);
az.summary(trace_reg)<jupyter_output>C:\Users\Student\anaconda3\lib\site-packages\arviz\data\io_pymc3.py:96: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
  warnings.warn(<jupyter_text>The `HDI` indicates which points of a distribution are most credible, and which cover most of the distribution.The plots below provide a 94% HDI for the model parameters that is, $\ell$ and  $\epsilon$ . In terms of interpreting the output below. `X1` can be defined as the left end of the black line and `X2` can be defined as the right end of the black line. The output below, indicates the subjective probability of a value being in between `X1` and `X2` with a 94% credible interval.<jupyter_code>pm.plot_posterior(trace_reg, figsize=(12, 5));
X_new = np.linspace(np.floor(x.min()), np.ceil(x.max()), 100)[:,None]
# Posterior Sampling 
with model:
    gp_samples = pm.gp.sample_gp(trace_reg, y_pred, X_new, samples=100, random_seed=42)
fig, ax = plt.subplots(figsize=(14,5))

[ax.plot(X_new, x, color=cm(0.3), alpha=0.3) for x in gp_samples]
# overlay the observed data
ax.plot(X, y, 'ok', ms=10);
ax.set_xlabel("x");
ax.set_ylabel("f(x)");
ax.set_title("Posterior predictive distribution");
with model_reg:
    f_pred = gp.conditional('f_pred', X_new)
with model_reg:
    pred_samples = pm.sample_posterior_predictive(trace_reg, vars=[f_pred], samples=82)
_, ax = plt.subplots(figsize=(12,5))
ax.plot(X_new, pred_samples['f_pred'].T, 'C1-', alpha=0.3)
ax.plot(X, y, 'ko')
ax.set_xlabel('X')<jupyter_output><empty_output><jupyter_text>Model 2: Shared Values<jupyter_code># A one dimensional column vector of inputs.
X = x[:, None]

x_shared = T.shared(X)
y_shared = T.shared(y)
with pm.Model() as model_reg_2:
    # hyperprior for lengthscale kernel parameter
    ℓ = pm.Gamma('ℓ', 2, 0.5)
    # instanciate a covariance function
    cov = pm.gp.cov.ExpQuad(1, ls=ℓ)
    # instanciate a GP prior
    gp = pm.gp.Marginal(cov_func=cov)
    # prior
    ϵ = pm.HalfNormal('ϵ', 0.5)
    # likelihood
    y_pred = gp.marginal_likelihood('y', X=x_shared, y=y_shared, noise=ϵ)
    trace_reg_2 = pm.sample(2000)
az.plot_trace(trace_reg_2);
az.summary(trace_reg_2)
pm.plot_posterior(trace_reg_2, figsize=(12, 5));
X_new = np.linspace(np.floor(x.min()), np.ceil(x.max()), 100)[:,None]
x_shared.set_value(X_new)
y_shared.set_value(np.zeros(len(X_new)))
with model_reg_2:
    pred_samples_2 = pm.sample_posterior_predictive(trace_reg_2, samples=100)
_, ax = plt.subplots(figsize=(12,5))
ax.plot(X_new, pred_samples_2['y'].T, 'C1-', alpha=0.3)
ax.plot(X, y, 'ko')
ax.set_xlabel('X');
# plot the results
#_, ax = plt.subplots(figsize=(12,5))

# predict
# point = {'ℓ': trace_reg['ℓ'].mean(), 'ϵ': trace_reg['ϵ'].mean()}
# mu, var = gp.predict(X_new, point=point, diag=True)
# sd = var**0.5

# plot mean and 1σ and 2σ intervals
# ax.plot(X_new, mu, 'C1')
# ax.fill_between(X_new.flatten(),
#                 mu - sd, mu + sd,
#                 color="C1",
#                 alpha=0.3)

#ax.fill_between(X_new.flatten(),
#                 mu - 2*sd, mu + 2*sd,
#                 color="C1",
#                 alpha=0.3)

#ax.plot(X, y, 'ko')
#ax.set_xlabel('X');
#pred_samples['y_pred'].mean(axis=0).T<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12331868424557198, 'fraction_numerical': 0.019709681715275002, 'mean_word_length': 4.663650075414782, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '305415', 'license': 'MIT', 'n_tokens_mistral': 2624, 'n_tokens_neox': 2480, 'n_words': 818, 'path': '.ipynb_checkpoints/Practical_Assignment_5-checkpoint.ipynb', 'repo_name': 'Ellie190/Advanced-Machine-Learning', 'size': '7509'}