Deep RL is built from components – deep learning and RL – that have already independently had a profound impact within neuroscience. Deep neural networks have proven to be an outstanding model of neural representation [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]} (Figure REF a). However, this research has for the most part utilized supervised training, and has therefore provided little direct leverage on the big-picture problem of understanding motivated, goal-directed behavior within a sensory-motor loop. At the same time, reinforcement learning has provided a powerful theory of the neural mechanisms of learning and decision making [9]}. This theory most famously explains activity of dopamine neurons as a reward prediction error [10]}, [11]}, [12]}, [13]} (Figure REF b), but also accounts for the role of a wide range of brain structures in reward-driven learning and decision making [14]}, [15]}, [16]}, [17]}, [18]}, [19]}. It has been integrated into small neural networks with handcrafted structure to provide models of how multiple brain regions may interact to guide learning and decision-making [16]}, [21]}. Just as in the machine learning context, however, RL itself has until recently offered neuroscience little guidance in thinking about the problem of representation [22]}, [19]}, [14]}, [25]}, [26]}.
