Stochastically delayed feedback has received much attention in the simpler multi-armed bandit and contextual bandit problems [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. The results in the multi-armed bandit setting show delayed feedback causes an additive penalty in the regret that scales with the number of actions and the expected delay. Thus, we can expect a similar result in the harder episodic RL problem. However, dealing with delays in RL is more challenging than in the (contextual) bandit setting. Indeed in RL, since the environment is stochastic and changes depending on the actions taken, we cannot guarantee to observe a particular state nor the effects of taking a particular action. Consequently, this means that approaches such as the queuing technique of [2]} cannot easily be applied.
