<jupyter_start><jupyter_text>Lecture 24 - Bayesian Model Selection using Sequential Monte Carlo  Objectives+ Compute the model evidence using [pysmc](https://github.com/PredictiveScienceLab/pysmc).+ Do model selection with [pysmc](https://github.com/PredictiveScienceLab/pysmc).<jupyter_code>import numpy as np
import math
import scipy.stats as st
import scipy
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
mpl.rcParams['figure.dpi'] = 300
import seaborn as sns
sns.set_style('white')
sns.set_context('talk')
#import design
#import orthpol
import warnings
warnings.filterwarnings('ignore')
#from tqdm import tqdm # pip install tqdm (or conda)
import sklearn
import pymc as pm
import pysmc as ps


# We need a generic function that computes the design matrix
def compute_design_matrix(X, phi):
    """
    Arguments:
    
    X   -  The observed inputs (1D array)
    phi -  The basis functions.
    """
    num_observations = X.shape[0]
    num_basis = phi.num_basis
    Phi = np.ndarray((num_observations, num_basis))
    for i in xrange(num_observations):
        Phi[i, :] = phi(X[i, :])
    return Phi

class PolynomialBasis(object):
    """
    A set of linear basis functions.
    
    Arguments:
    degree  -  The degree of the polynomial.
    """
    def __init__(self, degree):
        self.degree = degree
        self.num_basis = degree + 1
    def __call__(self, x):
        return np.array([x[0] ** i for i in range(self.degree + 1)])
    

class FourierBasis(object):
    """
    A set of linear basis functions.
    
    Arguments:
    num_terms  -  The number of Fourier terms.
    L          -  The period of the function.
    """
    def __init__(self, num_terms, L):
        self.num_terms = num_terms
        self.L = L
        self.num_basis = 2 * num_terms
    def __call__(self, x):
        res = np.ndarray((self.num_basis,))
        for i in xrange(num_terms):
            res[2 * i] = np.cos(2 * i * np.pi / self.L * x[0])
            res[2 * i + 1] = np.sin(2 * (i+1) * np.pi / self.L * x[0])
        return res
    

class RadialBasisFunctions(object):
    """
    A set of linear basis functions.
    
    Arguments:
    X   -  The centers of the radial basis functions.
    ell -  The assumed lengthscale.
    """
    def __init__(self, X, ell):
        self.X = X
        self.ell = ell
        self.num_basis = X.shape[0]
    def __call__(self, x):
        return np.exp(-.5 * (x - self.X) ** 2 / self.ell ** 2).flatten()<jupyter_output><empty_output><jupyter_text>Sanity check - Does the calculation of the evidence with PySMC work?Let$$p(\theta) = \mathcal{N}(\theta|0, 1),$$and$$p(y|\theta) = \mathcal{N}(y|\theta,0).$$The posterior of $\theta$ given $y$ is:$$p(\theta|y) = \frac{p(y|\theta)p(\theta)}{Z},$$where$$Z = \int_{-\infty}^{\infty} p(y|\theta)p(\theta)d\theta.$$Let's first calculate $Z$ analytically.<jupyter_code>import sympy
import sympy.stats
sympy.init_printing()
y, t = sympy.symbols('y \\theta')
q = 1. / sympy.sqrt(2. * sympy.pi) * sympy.exp(-0.5 * (y - t) ** 2) * \
    1. / sympy.sqrt(2. * sympy.pi) * sympy.exp(-0.5 * t ** 2)
sympy.simplify(sympy.integrate(q, (t, -sympy.oo, sympy.oo)))<jupyter_output><empty_output><jupyter_text>So, if the observed $y$ was zero, then the Z should be:$$Z = \frac{1}{2\sqrt{\pi}}.$$<jupyter_code>Z = 1 /  2. / np.sqrt(np.pi)
print 'log Z = {0:1.3e}'.format(np.log(Z))<jupyter_output>log Z = -1.266e+00<jupyter_text>All, right. Now, let's program this thing in pysmc and compare the results.We start with the model:<jupyter_code>def make_model(y=0.):
    # The gamma parameter
    gamma = 1.
    
    # The prior over theta
    theta = pm.Normal('$\theta$', mu=0., tau=1., value=0.)
    
    # The likelihood term - which must contain gamma in it
    @pm.stochastic(observed=True)
    def likelihood(value=y, gamma=gamma, theta=theta):
        return gamma * pm.normal_like(value, theta, 1.)
    
    return locals()<jupyter_output><empty_output><jupyter_text>Now we initialize the model and run PySMC.<jupyter_code>model = make_model(y=0.)
mcmc = pm.MCMC(model)
mcmc.use_step_method(ps.RandomWalk, model['theta'])
smc = ps.SMC(mcmc, num_particles=100, num_mcmc=1, verbose=4, ess_reduction=0.9)
smc.initialize(0.)
smc.move_to(1.)<jupyter_output>------------------------
START SMC Initialization
------------------------
- initializing at gamma : 0.0
- initializing by sampling from the prior: SUCCESS
----------------------
END SMC Initialization
----------------------
-----------------
START SMC MOVE TO
-----------------
initial  gamma : 0.0
final gamma : 1.0
ess reduction:  0.9
- tuning the MCMC parameters:
	- tuning step method: <pysmc._step_methods.RandomWalk object at 0x107b1a990>
- finding next gamma.
- moving to gamma : 0.885585220454
- performing 1 MCMC step(s) per particle
- ESS  = 90.00 %
- logZ = -1.089e+00
 [-----------------100%-----------------] 99 of 99 complete in 0.0 sec
- acceptance rate for each step method:
	- <pysmc._step_methods.RandomWalk object at 0x107b1a990> : 0.93
- tuning the MCMC parameters:
	- tuning step method: <pysmc._step_methods.RandomWalk object at 0x107b1a990>

		adaptive_scale_factor: 1.1

		SUCCESS
- finding next gamma.
- 	we can move directly to the target gamma...
- moving to gamma : 1.0
-[...]<jupyter_text>Here are the log Z ratios that we gathered along the way:<jupyter_code>print smc.log_Zs<jupyter_output>[-1.0886775023625104, -0.12993101938408813]<jupyter_text>The evidence is the sum of these:<jupyter_code>print 'Log Z SMC estiamte: {0:1.6e}'.format(np.sum(smc.log_Zs))<jupyter_output>Log Z SMC estiamte: -1.218609e+00<jupyter_text>Which is close to the truth.  Questions+ Repeat the calculations above for a varying number of SMC particles. Start from 10 and go up to 10,000.  Polynomial Regression We are going to implement a standard Bayesian linear regression using pymc and train it with pysmc.We will compute the evidence in order to select the best class of basis functions.The model is as follows. The output $y$ conditioned on the input $x$, the weights of the basis functions $w$ andthe noise variance $\sigma^2$ has likelihood:$$p(y|x,w,\sigma, \mathcal{M}) = \mathcal{N}(y|w^T\phi_{\mathcal{M}}(x), \sigma^2),$$where $\phi_{\mathcal{M},1}(\cdot), \dots, \phi_{\mathcal{M},m_{\mathcal{M}}}(\cdot)$ are the$m_{\mathcal{M}}$ basis functions of the model $\mathcal{M}$.We put a normal prior on the weights:$$p(w|\alpha) = \mathcal{N}(w|0, \alpha I_{m_{\mathcal{M}}}),$$and an inverse Gamma prior for $\sigma$ and $\alpha$:$$p(\sigma^2) = \mbox{IG}(\sigma^2|1, 1),$$and$$p(\alpha) = \mbox{IG}(\alpha|1,1).$$Assume that the data we have observed are:$$x_{1:n} = \{x_1,\dots,x_n\},\;\mbox{and}\;y_{1:n} = \{y_1,\dots,y_n\}.$$Consider the design matrix $\Phi_{\mathcal{M}}\in\mathbb{R}^{n\times m}$:$$\Phi_{\mathcal{M},ij} = \phi_{\mathcal{M},j}(x_i).$$The likelihood of the data is:$$p(y_{1:n} | x_{1:n}, w, \sigma, \mathcal{M}) = \mathcal{N}(y_{1:n}|\Phi_{\mathcal{M}}w, \sigma^2I_n).$$Let's turn this into pymc code.<jupyter_code>def make_model(Phi, y):
    # The parameter gamma
    gamma = 1.
    # The prior for sigma
    sigma2 = pm.InverseGamma('sigma2', .5, 0.5)
    # The prior for alpha
    alpha = pm.InverseGamma('alpha', .5, .5)
    # The prior of the weights
    w = pm.Normal('w', mu=np.zeros(Phi.shape[1]), tau=alpha)
    # The likelihood
    @pm.stochastic(observed=True)
    def likelihood(value=y, Phi=Phi, gamma=gamma, w=w, sigma2=sigma2):
        return gamma * pm.normal_like(value, np.dot(Phi, w), 1. / sigma2)
    return locals()<jupyter_output><empty_output><jupyter_text>Let's generate some fake data to play with:<jupyter_code>np.random.seed(12345)
num_samples = 50
sigma2 = 1e-6
X = 2 * np.random.rand(num_samples) - 1.
y = 0.5 * X  ** 3 - 0.3 * X ** 2 + np.sqrt(sigma2) * np.random.rand(num_samples)
plt.plot(X, y, 'kx', markeredgewidth=2)<jupyter_output><empty_output><jupyter_text>Now, let's create a function that trains the model using pysmc for a polynomial basis with a given order.<jupyter_code>def fit_poly(phi, X, y, num_particles=100, verbose=0):
    Phi = compute_design_matrix(X[:, None], phi)
    model = make_model(Phi, y)
    mcmc = pm.MCMC(model)
    mcmc.use_step_method(ps.RandomWalk, model['w'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['alpha'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['sigma2'])
    smc = ps.SMC(mcmc, num_particles=num_particles, num_mcmc=10, verbose=verbose, ess_reduction=0.95)
    smc.initialize(0.)
    smc.move_to(1.)
    return smc<jupyter_output><empty_output><jupyter_text>Now, let's define a function that plots the posteriors of everything.<jupyter_code>def plot_posteriors(smc):
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    fig, ax = plt.subplots()
    for j in xrange(W.shape[1]):
        ax.hist(W[:, j], normed=True, alpha=0.5, label='$w_{%d}$' % j)
    plt.legend(loc='best')
    fig, ax = plt.subplots()
    alpha = pa.alpha
    ax.hist(alpha, normed=True, label='$\alpha$', alpha=0.5)
    ax.set_xlabel('alpha')
    fig, ax = plt.subplots()
    sigma2 = pa.sigma2
    ax.hist(sigma2, normed=True, label='$\sigma^2$', alpha=0.5)
    ax.set_xlabel('sigma2')<jupyter_output><empty_output><jupyter_text>Let's also have a function that plots the predictive distribution:<jupyter_code>def plot_predictive(phi, smc, num_samples=100):
    Xt = np.linspace(-1, 1, 100)
    Phit = compute_design_matrix(Xt[:, None], phi)
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    sigma2 = pa.sigma2
    # We find the percantiles numerically
    Ys = np.ndarray((num_samples * pa.num_particles, Phit.shape[0]))
    for i in xrange(W.shape[0]):
        w = W[i, :]
        sigma = np.sqrt(sigma2[i])
        Yt = np.dot(Phit, w)
        for j in xrange(num_samples):
            Ys[i * num_samples + j, :] = Yt + sigma * np.random.randn(*Yt.shape)
    Ym = np.percentile(Ys, 50, axis=0)
    Yu = np.percentile(Ys, 97.5, axis=0)
    Yl = np.percentile(Ys, 2.5, axis=0)
    fig, ax = plt.subplots()
    ax.plot(Xt, Ym, color=sns.color_palette()[0])
    ax.fill_between(Xt, Yl, Yu, color=sns.color_palette()[0], alpha=0.25)
    ax.plot(X, y, 'kx', markeredgewidth=2)
phi = PolynomialBasis(3)
smc = fit_poly(phi, X, y, num_particles=100, verbose=1)
plot_posteriors(smc)
plot_predictive(phi, smc)<jupyter_output>------------------------
START SMC Initialization
------------------------
- initializing at gamma : 0.0
- initializing by sampling from the prior: SUCCESS
----------------------
END SMC Initialization
----------------------
-----------------
START SMC MOVE TO
-----------------
initial  gamma : 0.0
final gamma : 1.0
ess reduction:  0.95
- moving to gamma : 0.00363744153598
- performing 10 MCMC step(s) per particle
- ESS  = 95.00 %
- logZ = -4.089e-01

- moving to gamma : 0.00708773715132
- performing 10 MCMC step(s) per particle
- ESS  = 90.25 %
- logZ = -3.241e-01

- moving to gamma : 0.0104634277779
- performing 10 MCMC step(s) per particle
- ESS  = 85.74 %
- logZ = -2.830e-01

- moving to gamma : 0.0138096797298
- performing 10 MCMC step(s) per particle
- ESS  = 81.45 %
- logZ = -2.455e-01

- moving to gamma : 0.0177813988721
- performing 10 MCMC step(s) per particle
- ESS  = 77.38 %
- logZ = -2.946e-01

- moving to gamma : 0.020943445566
- performing 10 MCMC step(s) per particle
- [...]<jupyter_text>Now, we can compute and compare the evidence of all  models:<jupyter_code># Evaluate the evidence for the various degrees
log_Zs = []
for d in [1, 2, 3, 4, 5]:
    phi = phi = PolynomialBasis(d)
    smc = fit_poly(phi, X, y, num_particles=100)
    log_Z = np.sum(smc.log_Zs)
    print 'degree {0:1d} gives {1:1.3e}'.format(d, log_Z)
    log_Zs.append(log_Z)
X.shape
fig, ax = plt.subplots()
ax.bar(np.arange(1, len(log_Zs) + 1), np.exp(log_Zs) / np.sum(np.exp(log_Zs)))<jupyter_output><empty_output><jupyter_text>Questions+ The model with degree 3 polynomials has the gradest evidence. However, the degree 4 and 5 seem also very plaussible. Is this a problem for the theory of Bayesian model selection? What complicates things here, is that model 3 is included in model 4 which is included in model 5. This requires us to design special priors for the models being right. They have to be consistent in some sense. For example, if model 3 is right then model 4 must be right, etc.+ In the space below, we load the motorcycle data. By copy pasting the code above and by making small modifications, evaluate the model evidence for a 1) Polynomial basis; 2) a Fourier basis; and 3) a Radial basis function basis. Feel free to pick the parameters of these basis functions to be the best we found in [Lecture 9](./lec_09.ipynb).  Revisiting Challenger Disaster Problem (Model Selection)<jupyter_code>from IPython.core.pylabtools import figsize
figsize(12.5, 3.5)
np.set_printoptions(precision=3, suppress=True)
challenger_data = np.genfromtxt("challenger_data.csv", skip_header=1,
                                usecols=[1, 2], missing_values="NA",
                                delimiter=",")
# drop the NA values
challenger_data = challenger_data[~np.isnan(challenger_data[:, 1])]

# plot it, as a function of temperature (the first column)
print("Temp (F), O-Ring failure?")
print(challenger_data)

plt.scatter(challenger_data[:, 0], challenger_data[:, 1])
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Fahrenheit)")
plt.title("Defects of the Space Shuttle O-Rings vs temperature");
def make_model(feat, y):
    """
    PyMC model (wrapping all the data and variables into a single function)
    Inputs:
    challenger data (23 x 2): Pandas dataframe ( Outside Temp vs. Damage Incident)
    Outputs:
    alpha, beta : stochastic parameters associated with the given model
    """
    gamma = 1.
    # Load data
    
    feat_scaled = (feat - np.mean(feat))/np.std(feat) # to ensure proper mixing of MCMC chains
    #D = challenger_data[:, 1]  # defect or not?
    
    # Define Prior
    beta = pm.Normal("beta", 0, 0.1, value = 0) # 0 mean and small 0.001 precision - vague prior
    alpha = pm.Normal("alpha", 0, 0.1, value = 0)
    
    @pm.deterministic
    def p(t=feat_scaled, alpha=alpha, beta=beta):
        return 1.0 / (1. + np.exp(beta * t + alpha))
    
    # Define likelihood
    # connect the probabilities in `p` with our observations through a
    # Bernoulli random variable.
    @pm.stochastic(observed = True)
    def likelihood(value = y, p = p, gamma = gamma):
        return gamma*pm.bernoulli_like(value, p)
    return locals()

import mpi4py.MPI as mpi

def fit_smc(feat,y, num_particles=100, verbose=0):
    model = make_model(feat,y)
    mcmc = pm.MCMC(model)
    mcmc.use_step_method(ps.RandomWalk, model['beta'])
    mcmc.use_step_method(ps.RandomWalk, model['alpha'])
    smc = ps.SMC(mcmc, num_particles=num_particles, num_mcmc=20,
                 verbose=verbose,mpi=mpi, ess_reduction=0.95)
    smc.initialize(0.)
    smc.move_to(1.)
    return smc
temp = challenger_data[:,0]
y = challenger_data[:,1]
smc = fit_smc(temp,y, num_particles= 500, verbose=1)
def plot_posterior(smc):
    pa = smc.get_particle_approximation()
    pa.resample()
    fig, ax = plt.subplots()
    alpha = pa.alpha
    ax.hist(alpha, normed=True, label='$\alpha$', alpha=0.5)
    ax.set_xlabel('alpha')
    fig, ax = plt.subplots()
    beta = pa.beta
    ax.hist(beta, normed=True, label='$\beta^2$', alpha=0.5)
    ax.set_xlabel('beta')

def logistic(x, beta, alpha=0):
    return 1.0 / (1.0 + np.exp(np.dot(beta, x) + alpha))

def plot_predictive(challenger_data, smc, num_samples=500):
    pa = smc.get_particle_approximation()
    pa.resample()
    alpha = pa.alpha
    beta = pa.beta
    
    temp = challenger_data[:, 0]
    D = challenger_data[:, 1]
    t = np.linspace(temp.min() - 5, temp.max() + 5, 50)[:, None]
    t_scaled = (t - np.mean(temp))/np.std(temp)
    p_t = logistic(t_scaled.T, beta[:,None], alpha[:,None])

    mean_prob_t = p_t.mean(axis=0)
    # vectorized bottom and top 2.5% quantiles for "confidence interval"
    qs = mquantiles(p_t, [0.025, 0.975], axis=0)
    plt.fill_between(t[:, 0], *qs, alpha=0.7,
                     color="#7A68A6")

    plt.plot(t[:, 0], qs[0], label="95% CI", color="#7A68A6", alpha=0.7)

    plt.plot(t, mean_prob_t, lw=1, ls="--", color="k",
             label="average posterior \nprobability of defect")

    plt.xlim(t.min(), t.max())
    plt.ylim(-0.02, 1.02)
    plt.legend(loc="lower left")
    plt.scatter(temp, D, color="k", s=50, alpha=0.5)
    plt.xlabel("temp, $t$")

    plt.ylabel("probability estimate")
    plt.title("Posterior probability estimates given temp. $t$");
plot_posterior(smc)
from scipy.stats.mstats import mquantiles
plot_predictive(challenger_data, smc, num_samples=500)<jupyter_output><empty_output><jupyter_text>Feature SelectionWe knew that temperature was a relevant feature for predicting the probability of failure of O-rings since there are numerous studies published which illustrates this fact.But when dealing with your own problem, you might not know which feature is relevant/important. For example, we have not considered Leak-check pressure (psi) for predicting the probability of failure (which is available in whole Challenger disaster data UCI data repository (https://archive.ics.uci.edu/ml/datasets/Challenger+USA+Space+Shuttle+O-Ring). Let us see if we can predict the probability of failure using leak check pressure feature.<jupyter_code>c2_data = np.loadtxt('challenger2.txt')
leak_pressure = c2_data[:,3]
smc2 = fit_smc(leak_pressure,y, num_particles= 500, verbose=1)
log_Zs = [np.sum(smc.log_Zs), np.sum(smc2.log_Zs)]
print 'When we use temp as feature, log evidence is :', str(log_Zs[0])
print 'When we use leak pressure as feature, log evidence is :', str(log_Zs[1])
fig, ax = plt.subplots()
ax.bar(np.arange(1, len(log_Zs) + 1), np.exp(log_Zs) / np.sum(np.exp(log_Zs)))<jupyter_output><empty_output><jupyter_text>Motor Cycle Data Visualization<jupyter_code>data = np.loadtxt('motor.dat')
X = data[:, 0][:, None]
Y = data[:, 1]
fig, ax = plt.subplots()
ax.plot(X, Y, 'x', markeredgewidth=2)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
import pymc as pm
def make_model(Phi, y):
    # The parameter gamma
    gamma = 1.
    # The prior for sigma
    sigma1 = pm.Exponential('sigma1', 0.1)
    sigma2 = pm.Exponential('sigma2', 0.1)
    # The prior of the weights
    w = pm.Normal('w', mu=np.zeros(Phi.shape[1]), tau= 1./sigma1**2)
    # The likelihood
    @pm.stochastic(observed=True)
    def likelihood(value=y, Phi=Phi, gamma=gamma, w=w, sigma2=sigma2):
        return gamma * pm.normal_like(value, np.dot(Phi, w), 1. / sigma2**2)
    return locals()

def fit_radial(ell, X, y, num_particles=100, verbose=0):
    phi = RadialBasisFunctions(X, ell)
    Phi = compute_design_matrix(X, phi)
    model = make_model(Phi, y)
    mcmc = pm.MCMC(model)
    mcmc.use_step_method(ps.RandomWalk, model['w'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['sigma1'])
    mcmc.use_step_method(ps.LognormalRandomWalk, model['sigma2'])
    smc = ps.SMC(mcmc, num_particles=num_particles, num_mcmc=20,
                 verbose=verbose,mpi=mpi, ess_reduction=0.95)
    smc.initialize(0.)
    smc.move_to(1.)
    return smc
def plot_posterior(smc):
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    fig, ax = plt.subplots()
    for j in xrange(W.shape[1]):
        ax.hist(W[:, j], normed=True, alpha=0.5, label='$w_{%d}$' % j)
    #plt.legend(loc='best')
    fig, ax = plt.subplots()
    sigma1 = pa.sigma1
    ax.hist(sigma1, normed=True, label='$\alpha$', alpha=0.5)
    ax.set_xlabel('sigma1')
    fig, ax = plt.subplots()
    sigma2 = pa.sigma2
    ax.hist(sigma2, normed=True, label='$\sigma^2$', alpha=0.5)
    ax.set_xlabel('sigma2')
    
def plot_predictive(phi, smc, num_samples=100):
    Xt = np.linspace(0, 60, 100)
    Phit = compute_design_matrix(Xt[:,None], phi)
    pa = smc.get_particle_approximation()
    pa.resample()
    W = pa.w
    
    p_t = np.dot(W,Phit.T)
    mean_prob_t = np.mean(p_t, axis = 0)

    # vectorized bottom and top 2.5% quantiles for "confidence interval"
    qs = mquantiles(p_t, [0.025, 0.975], axis=0)
    plt.fill_between(Xt, *qs, alpha=0.7,
                     color="#7A68A6")

    plt.plot(Xt, qs[0], label="95% CI", color="#7A68A6", alpha=0.7)

    plt.plot(Xt, mean_prob_t, lw=1, ls="--", color="k",
             label="average posterior")

    plt.xlim(Xt.min(), Xt.max())
    plt.legend(loc="best")
    plt.scatter(X, Y, color="k", s=50, alpha=0.5)
ell = 5.
smc = fit_radial(ell, X, Y, num_particles=500, verbose=1)
plot_posterior(smc)
phi = RadialBasisFunctions(X, ell)
plot_predictive(phi, smc, num_samples=500)
# Evaluate the evidence for the various degrees
log_Zs = []
for ell in [2, 5]:
    smc = fit_radial(ell, X, Y, num_particles=200, verbose=1)
    log_Z = np.sum(smc.log_Zs)
    log_Zs.append(log_Z)
ells = [2, 5]
for i in xrange(2):
    print 'For ell = ' + str(ell) + ', evidence is : ' + str(log_Zs[i])
fig, ax = plt.subplots()
ax.bar(np.arange(1, len(log_Zs) + 1), np.exp(log_Zs) / np.sum(np.exp(log_Zs)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15526452649992908, 'fraction_numerical': 0.034230060044442345, 'mean_word_length': 3.4455653635981505, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '436720', 'license': 'MIT', 'n_tokens_mistral': 8326, 'n_tokens_neox': 7748, 'n_words': 1992, 'path': 'handouts/handout_24.ipynb', 'repo_name': 'bdlafleur/uq-course', 'size': '21151'}