As we have stressed, phenomena arise within deep RL that do not arise in deep learning or RL considered separately. A pair of recent studies have focused on the neuroscientific implications of these emergent phenomena. In one, [1]} examined the behavior of recurrent deep RL systems, and described a novel meta-reinforcement learning effect: When trained on a series of interrelated tasks – for example, a series of forced-choice decision tasks with the same overall structure but different reward probabilities – recurrent deep RL networks develop the ability to adapt to new tasks of the same kind without weight changes. This is accompanied by correspondingly structured representations in the activity dynamics of the hidden units that emerge throughout training (Figure REF a). Slow RL-driven learning at the level of the network’s connection weights shape the network’s activation dynamics such that rapid behavioral adaptation can be driven by those activation dynamics alone, akin to the idea from neuroscience that RL can be supported, in some cases, by activity-based working memory [2]}. In short, slow RL spontaneously gives rise to a separate and faster RL algorithm. Wang and colleagues showed how this meta-reinforcement learning effect could be applied to explain a wide range of previously puzzling findings from neuroscientific studies of dopamine and prefrontal cortex function (Figure REF b).
