Instead, we focus on teacher rewards that evaluate students' return relative to some dynamic prediction. In particular, actor-critic algorithms such as PPO [1]} train a value function to predict the return on each episode. A large difference between value prediction and value target (called value prediction error) might indicate an environment where students can learn something new [2]}. We also explore teacher rewards that measure the disagreement of an ensemble of student value functions with different initializations (value disagreement) [3]} and the disagreement between an agent's action distribution and the action distribution of a second, independently initialized policy that is trained to behaviorally clone the main again (policy disagreement). Large disagreement signals that a student is uncertain about the return or action distribution of an environment, which suggests that something new can be learned. Once a student has stopped learning in an environment, either disagreement measure is expected to converge to zero.
