Our results indeed show that value prediction error teacher is much more affected by stochastic returns than the value disagreement and policy disagreement teacher (Figure REF , left). The value prediction error teacher exclusively samples easy environments throughout training (Figure REF , 2nd from left). The reason is as follows: In easy environments, the agent spends the most time next to the goal box, because the agent does not need to overcome any obstacles to reach the goal. Each time step next to the goal box generates an irreducible contribution to the value prediction error, because the agent cannot predict whether the episode is rewarded or not. Easy environments therefore generate a larger value prediction error reward than hard environments, which incentivizes the value prediction error teacher to exclusively sample easy environments. The focus of this teacher on easy environments is analogous to the “Noisy-TV” phenomenon, where an agent that is rewarded for seeking unpredictable experience is attracted by unpredictable random noise [1]}.
