Transformers are Meta-Reinforcement Learners
The transformer architecture and variants presented remarkable success across
many machine learning tasks in recent years. This success is intrinsically
related to the capability of handling long sequences and the presence of
context-dependent weights from the attention mechanism. We argue that these
capabilities suit the central role of a Meta-Reinforcement Learning algorithm.
Indeed, a meta-RL agent needs to infer the task from a sequence of
trajectories. Furthermore, it requires a fast adaptation strategy to adapt its
policy for a new task -- which can be achieved using the self-attention
mechanism. In this work, we present TrMRL (Transformers for Meta-Reinforcement
Learning), a meta-RL agent that mimics the memory reinstatement mechanism using
the transformer architecture. It associates the recent past of working memories
to build an episodic memory recursively through the transformer layers. We show
that the self-attention computes a consensus representation that minimizes the
Bayes Risk at each layer and provides meaningful features to compute the best
actions. We conducted experiments in high-dimensional continuous control
environments for locomotion and dexterous manipulation. Results show that TrMRL
presents comparable or superior asymptotic performance, sample efficiency, and
out-of-distribution generalization compared to the baselines in these
environments.