How can reinforcement learning (RL) agents leverage relevant information to inform their decisions? Deep RL agents have typically been represented as a monolithic parametric function, trained to gradually amortise useful information from experience by gradient descent. This has been effective [1]}, [2]}, but is a slow means of integrating experience, with no straightforward way for an agent to incorporate new information without many additional gradient updates. In addition, this requires increasingly massive models (see e.g. [3]}) as environments become more complex – scaling is driven by the dual role of the parametric function, which must support both computation and memorisation. Finally, this approach has a further drawback of particular relevance in RL – the only way in which previously encountered information (that is not contained in working memory) can aid decision making in a novel situation is indirectly through weight changes mediated by network losses. There is no end-to-end means for an agent to attend to information outside of working memory to directly inform its actions. While there has been a significant amount of work focused on increasing the information available from previous experiences within an episode (e.g., recurrent networks, slot-based memory [4]}, [5]}), more extensive direct use of more general forms of experience or data has been limited, although some recent works have begun to explore utilising inter-episodic information from the same agent [6]}, [7]}, [8]}, [9]}, [10]}. We seek to drastically expand the scale of information that is accessible to an agent, allowing it to attend to tens of millions of pieces of information, while learning in an end-to-end manner how to use this information for decision making. We view this as a first step towards a vision in which an agent can flexibly draw on diverse and large-scale information sources, including its own (inter-episodic) experiences along with experiences from humans and other agents. In addition, given that retrieved information need not match the agent's observation format, retrieval could enable agents to integrate information sources that have not typically been utilised such as videos [11]}, text, and third-person demonstrations.
