Large-Scale Retrieval for Reinforcement Learning
Effective decision making involves flexibly relating past experiences and
relevant contextual information to a novel situation. In deep reinforcement
learning, the dominant paradigm is for an agent to amortise information that
helps decision-making into its network weights via gradient descent on training
losses. Here, we pursue an alternative approach in which agents can utilise
large-scale context-sensitive database lookups to support their parametric
computations. This allows agents to directly learn in an end-to-end manner to
utilise relevant information to inform their outputs. In addition, new
information can be attended to by the agent, without retraining, by simply
augmenting the retrieval dataset. We study this approach in Go, a challenging
game for which the vast combinatorial state space privileges generalisation
over direct matching to past experiences. We leverage fast, approximate nearest
neighbor techniques in order to retrieve relevant data from a set of tens of
millions of expert demonstration states. Attending to this information provides
a significant boost to prediction accuracy and game-play performance over
simply using these demonstrations as training trajectories, providing a
compelling demonstration of the value of large-scale retrieval in reinforcement
learning agents.