Reliable prediction of future observations under minimal model assumptions is an important and fundamental problem in statistics and machine learning. There are two general approaches to quantifying uncertainty about the yet-to-be-observed value: one is based on prediction regions and the other is based on predictive distributions. Since predictive distributions can be used to construct prediction regions, they are believed to be more informative. However, the additional informativeness that comes with a full predictive distribution usually requires model assumptions, which creates several practical challenges. First, with model assumptions comes the risk of model misspecification biases that can negatively impact the quality of predictions; second, computation often requires Monte Carlo methods, and for sufficiently flexible predictive distributions based on, e.g., modern Bayesian nonparametric models [1]}, [2]}, this can be non-trivial. In a given application, the data analyst might be willing to take on these challenges, but not solely for the construction of prediction regions. Indeed, there are non-model-based prediction regions with provably exact/conservative coverage probability, while those derived from a model-based predictive distribution, at best, attain the target coverage probability asymptotically. Therefore, the motivation for developing a predictive distribution for quantifying uncertainty about the yet-to-be-observed value must be that there are other tasks, such as evaluating probabilities associated with general assertions about the to-be-observed value, which are of higher priority than the construction of prediction regions. This begs the question: how can the quality of a predictive distribution be assessed with respect to these other tasks?
