Figure REF  shows the blackinformation flow of a blacktypical recommendation system blackwhere the users' feedback is collected through the deployed system.
The deployed recommender system blackcomponent (denoted blackby RecSys) filters items for the target user (e.g. by suggesting a ranked list of items) depicted as blackthe exposure (\(e\) ).
On the other hand, the user's recorded preferences black(e.g. ratings blackor clicks) towards items (depicted as \(r\) ) are leveraged as interaction data to train blackand/or evaluate the next-generation of the recommendation model. blackHence, blacksince the blackusers' clicks are obtained on items exposed by the RecSysblackTheir clicks may also be affected by other confounding variables, such as temporal factors and other browsing interactions with the item catalogue, as illustrated in Figure REF ., the model itself influences the generation of data that is used to blackeither train or evaluate it [1]}.
The system presented in Figure REF  is a dynamic system where a simple associative reasoning about the system is difficult because each component influences the other. Figure REF  shows the corresponding causal diagram in such a closed loop feedback scenario. blackThe solid lines represent an explicit/observed relationship between blackthe causes and effects while the dashed line represents an implicit/unobserved relationship.
blackblackAs mentioned above, in the case of recommendation systems, blackthe blackmain confounding variable is the deployed blackmodel from which the interaction data were collected [2]}, [1]}, [4]}.
Our aim is to blackevaluate the effectiveness of a recommendation model (\(Y\) ) based on closed loop feedback (\(r\) ) collected from a deployed model while blackthe analysis is influenced by the blackmain confounder, i.e. the deployed model's blackcharacteristics. In this situation, it is difficult to blackdistinguish blackthose blackusers' interactions blackthat stem from blackthe users' true preferences blackfrom those that are influenced by the deployed recommendation model. blackAs a result, blackin the scenario where the users' feedback is typically collected through the deployed system blackwe postulate that the offline evaluation of recommendation models blackbased on closed loop feedback datasets is strongly influenced by the blackunderlying deployed recommendation model:
