The intuition behind blackthe IPS method is that the propensity score \(p_{u,i}\)  is a balancing factor: blackconditioned on the propensity score, the blackobserved feedback distribution will be blackadjusted to blackthe examined model.
Unlike blackthe holdout evaluation \(\hat{R}_{\mathrm {holdout}}\) , the IPS evaluation method is theoretically unbiased (\(\mathbb {E}_{O}\left[\hat{R}_{\mathrm {IPS}}(\hat{Z})\right] = R(\hat{Z})\) ) if and only if we can sample open loop feedback from all possible items \(\mathcal {I}\)  uniformly at random [1]}. However, in various situations, it may be difficult or even impossible to expose users to random items due to blackobvious business constraints black(users are unlikely to tolerate that the system keeps showing them random non-relevant items). blackMoreover, the imbalance of the assigned propensities will allow for some feedback to be weighted disproportionately. As a result, a major difficulty in applying blackthe IPS evaluation method in practice is its high variance blackin the estimation of blacka given model's effectiveness [1]}.
In the following, we investigate the impact of blackour proposed propensity-based stratified evaluation in comparison to both blackthe holdout and counterfactual evaluation.
