To address these issues, deep reinforcement learning is often supplemented in practice with either unsupervised learning [1]}, or ‘self-supervised’ learning. In self-supervised learning the agent is trained to produce, in addition to an action, some auxiliary output which matches a training signal that is naturally available from the agent’s stream of experience, regardless of what specific RL task it is being trained on [2]}, [3]}. An example is prediction learning, where the agent is trained to predict, based on its current situation, what it will observe on future timesteps [4]}, [5]}. Unsupervised and self-supervised learning mitigate both problems associated with pure reinforcement learning, since they shape representations in a way that is not tied exclusively to the specific tasks confronted by the learner, thus yielding representations that have the potential to support transfer to other tasks when they arise. All of this is consistent with existing work in neuroscience, where unsupervised learning [6]}, [7]}, [8]} and prediction learning [9]}, [10]}, [11]} have been proposed to shape internal representations. Deep RL offers the opportunity to pursue these ideas in a setting where these forms of learning can mix with reward-driven learning [12]}, [13]} and where the representations they produce support adaptive behavior.
