We also evaluate whether teacher rewards are susceptible to uncontrolled stochasticity in the environment. Just as intrinsically rewarded RL agents are sometimes attracted to “noisy TV” states with unpredictable transitions [1]}, [2]}, [3]}, a teacher may be incentivized to sample environments with unpredictable student returns, without any possibility of learning progress. We expect this problem to become more prevalent in more complex and harder-to-predict environments.
