blackNext, we consider the Kendall's \(\tau \)  correlation between the counterfactual evaluation (IPS) and the open loop evaluation methods. blackTable REF  reveals that the IPS evaluation method blackperforms slightly better than the holdout evaluation, i.e. the correlation values for IPS are greater than the holdout evaluation method for blackthe majority of nDCG cut-offs in both datasets. However, none of these blackcorrelation differences are statistically significant according to blackthe Steiger's test.
blackTherefore, we blackconclude that blackthe counterfactual (IPS) evaluation method blackdoes not better represent unbiased user preferences during blackthe evaluation than blackthe holdout evaluation blackfor the examined datasets. blackThis is because in the IPS evaluation method, each feedback is blackweighted individually based on its propensity. The feedback sampling process, i.e. the process blackfrom which the blackclosed loop feedback blackwas collected, is not a random process. As a result, the sampled feedback blackis skewed and the corresponding estimated propensity scores are imbalanced.
blackIn this situation, reweighting blackthe feedback based on blackits disproportionate propensities leads to high variance black in the blackmodels' performance estimation blackusing the IPS evaluation method [1]}, [2]}.
