When tension is just a fluctuation: How noisy data affect model
  comparison
  Summary statistics of the likelihood, such as the Bayesian evidence, offer a
principled way of comparing models and assessing tension between, or within,
the results of physical experiments. Noisy realisations of the data induce
scatter in these model comparison statistics. For a realistic case of
cosmological inference from large-scale structure we show that the logarithm of
the Bayes factor attains scatter of order unity, increasing significantly with
stronger tension between the models under comparison. We develop an approximate
procedure that quantifies the sampling distribution of the evidence at small
additional computational cost and apply it to real data to demonstrate the
impact of the scatter, which acts to reduce the significance of any model
discrepancies. Data compression is highlighted as a potential avenue to
suppressing noise in the evidence to negligible levels, with a proof of concept
on Planck cosmic microwave background data.
