<jupyter_start><jupyter_text>Sampler statisticsWhen checking for convergence or when debugging a badly behavingsampler, it is often helpful to take a closer look at what thesampler is doing. For this purpose some samplers exportstatistics for each generated sample.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import seaborn as sb
import pandas as pd
import pymc3 as pm 

%matplotlib inline<jupyter_output><empty_output><jupyter_text>As a minimal example we sample from a standard normal distribution:<jupyter_code>model = pm.Model()
with model:
    mu1 = pm.Normal("mu1", mu=0, sigma=1, shape=10)
with model:
    step = pm.NUTS()
    trace = pm.sample(2000, tune=1000, init=None, step=step, cores=2)<jupyter_output>100%|██████████| 2000/2000 [00:01<00:00, 1361.25it/s]<jupyter_text>NUTS provides the following statistics:<jupyter_code>trace.stat_names<jupyter_output><empty_output><jupyter_text>- `mean_tree_accept`: The mean acceptance probability for the tree that generated this sample. The mean of these values across all samples but the burn-in should be approximately `target_accept` (the default for this is 0.8).- `diverging`: Whether the trajectory for this sample diverged. If there are many diverging samples, this usually indicates that a region of the posterior has high curvature. Reparametrization can often help, but you can also try to increase `target_accept` to something like 0.9 or 0.95.- `energy`: The energy at the point in phase-space where the sample was accepted. This can be used to identify posteriors with problematically long tails. See below for an example.- `energy_error`: The difference in energy between the start and the end of the trajectory. For a perfect integrator this would always be zero.- `max_energy_error`: The maximum difference in energy along the whole trajectory.- `depth`: The depth of the tree that was used to generate this sample- `tree_size`: The number of leafs of the sampling tree, when the sample was accepted. This is usually a bit less than $2 ^ \text{depth}$. If the tree size is large, the sampler is using a lot of leapfrog steps to find the next sample. This can for example happen if there are strong correlations in the posterior, if the posterior has long tails, if there are regions of high curvature ("funnels"), or if the variance estimates in the mass matrix are inaccurate. Reparametrisation of the model or estimating the posterior variances from past samples might help.- `tune`: This is `True`, if step size adaptation was turned on when this sample was generated.- `step_size`: The step size used for this sample.- `step_size_bar`: The current best known step-size. After the tuning samples, the step size is set to this value. This should converge during tuning.- `model_logp`: The model log-likelihood for this sample. If the name of the statistic does not clash with the name of one of the variables, we can use indexing to get the values. The values for the chains will be concatenated.We can see that the step sizes converged after the 1000 tuning samples for both chains to about the same value. The first 2000 values are from chain 1, the second 2000 from chain 2.<jupyter_code>plt.plot(trace['step_size_bar'])<jupyter_output><empty_output><jupyter_text>The `get_sampler_stats` method provides more control over which values should be returned, and it also works if the name of the statistic is the same as the name of one of the variables. We can use the `chains` option, to control values from which chain should be returned, or we can set `combine=False` to get the values for the individual chains:<jupyter_code>sizes1, sizes2 = trace.get_sampler_stats('depth', combine=False)
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, sharey=True)
ax1.plot(sizes1)
ax2.plot(sizes2)
accept = trace.get_sampler_stats('mean_tree_accept', burn=1000)
sb.distplot(accept, kde=False)
accept.mean()<jupyter_output><empty_output><jupyter_text>Find the index of all diverging transitions:<jupyter_code>trace['diverging'].nonzero()<jupyter_output><empty_output><jupyter_text>It is often useful to compare the overall distribution of theenergy levels with the change of energy between successive samples.Ideally, they should be very similar:<jupyter_code>energy = trace['energy']
energy_diff = np.diff(energy)
sb.distplot(energy - energy.mean(), label='energy')
sb.distplot(energy_diff, label='energy diff')
plt.legend()<jupyter_output><empty_output><jupyter_text>If the overall distribution of energy levels has longer tails, the efficiency of the sampler will deteriorate quickly.  Multiple samplersIf multiple samplers are used for the same model (e.g. for continuous and discrete variables), the exported values are merged or stacked along a new axis.Note that for the `model_logp` sampler statistic, only the last column (i.e. `trace.get_sampler_stat('model_logp')[-1]`) will be the overall model logp.<jupyter_code>model = pm.Model()
with model:
    mu1 = pm.Bernoulli("mu1", p=0.8)
    mu2 = pm.Normal("mu2", mu=0, sigma=1, shape=10)
with model:
    step1 = pm.BinaryMetropolis([mu1])
    step2 = pm.Metropolis([mu2])
    trace = pm.sample(10000, init=None, step=[step1, step2], cores=2, tune=1000)
trace.stat_names<jupyter_output><empty_output><jupyter_text>Both samplers export `accept`, so we get one acceptance probability for each sampler:<jupyter_code>trace.get_sampler_stats('accept')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0804873546243308, 'fraction_numerical': 0.019198818534244048, 'mean_word_length': 5.615384615384615, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '13765', 'license': 'Apache-2.0', 'n_tokens_mistral': 1652, 'n_tokens_neox': 1535, 'n_words': 676, 'path': 'docs/source/notebooks/sampler-stats.ipynb', 'repo_name': 'JvParidon/pymc3', 'size': '5417'}