For example, a feature of many recommender systems is that the user must provide feedback to the learner. In many of these applications, the user and the learner operate on vastly different time scales [1]}. In some cases, the learner can serve thousands of recommendations per second. Conversely, it is not uncommon for users to take several minutes to a couple of days to provide feedback, possibly avoiding giving feedback altogether. Alternatively, practitioners might want to optimise for a longer-term measure of success [2]}. In such cases, the reward is not observable or even defined immediately. Another setting where delayed feedback arises is in clinical trials, where it is rare that patients respond to their prescribed treatment immediately. On top of this, obtaining medical feedback is often a time-consuming task.
