We are not the first to highlight discrepancies between test log-likelihood and other analysis objectives.
For instance, [1]} and [2]} showed that when predicting discrete data with continuous distributions, test log-likelihood can be made arbitrarily large by concentrating probability into vanishingly small intervals.
[3]} observed that topic models with larger test log-predictive densities can be less interpretable.
[4]} highlighted the disconnect between test log-likelihood and posterior approximation error in the context of Bayesian neural networks.
Our examples, however, reveal more fundamental issues with test log-likelihood.
In particular, we show how comparisons based on test log-likelihood can contradict comparisons based on other objectives even in simple models like linear regression.
