Information gains from Monte Carlo Markov Chains
  In this paper, we present a novel method for computing the relative entropy
as well as the expected relative entropy using an MCMC chain. The relative
entropy from information theory can be used to quantify differences in
posterior distributions of a pair of experiments. In cosmology, the relative
entropy has been proposed as an interesting tool for model selection,
experiment design, forecasting and measuring information gain from subsequent
experiments. In contrast to Gaussian distributions, these quantities are not
generally available analytically and one needs to use numerical methods to
estimate them which are certainly computationally expensive. We propose a
method and provide its python package to estimate the relative entropy as well
as expected relative entropy from a posterior sample. We consider the linear
Gaussian model to check the accuracy of our code. Our results indicate that the
relative error is below $0.2\%$ for sample size larger than $10^5$ in the
linear Gaussian model. In addition, we study the robustness of our code in
estimating the expected relative entropy in this model.
