The problem of evaluating recommendation models using historical feedback is similar to counterfactual evaluation in reinforcement learning. In this setting, the goal is to evaluate a new recommendation model based on a feedback dataset that blackis collected from a deployed recommendation model.
Inverse Propensity Scoring (IPS) leverages importance sampling to account for the fact that the feedback, blackwhich was collected from the deployed recommendation model blackis not blackuniformly random. blackTheoretically, the IPS evaluation method is able to evaluate a new model blackindependently from the deployed model (blackthe confounder). In the context of recommendation systems, blackthe IPS evaluation method blackwas proposed for both explicit [1]} and implicit [2]} feedback. The key idea blackof IPS is to reweight feedback based on blackthe propensities of the logged items using importance sampling. blackThe propensity \(p_{u,i}\)  in this context is the probability blackthat item \(i\)  blackwould blackhave blackbeen shown to the user \(u\)  blackby the time blackthe feedback data blackwas collected.
The effectiveness of a model (\(\hat{R}\) ) for the predicted item ranking \(\hat{Z}\)  and the provided propensity scores \(P\)  can be calculated as follows [2]}:
\(\hat{R}_{\mathrm {IPS}}(\hat{Z} | P)=\frac{1}{|\mathcal {U}|} \sum _{u \in \mathcal {U}} \frac{1}{\left|\mathcal {S}_{u}^{{black}{*}}\right|} \sum _{i \in \mathcal {S}_{u}^{*}} \frac{c(\hat{Z}_{u, i})}{p_{u, i}}\) 
