Perhaps most relevant is research that has adapted to deep RL ideas originating from the older field of hierarchical reinforcement learning. Here, RL operates at two levels, shaping a choice among high-level multi-step actions (e.g. `make coffee') and also among actions at a more atomic level [1]}. Deep RL research has adopted this hierarchical scheme in a number of ways [2]}, [3]}, [4]}, [5]}. In some of these, the low-level system can operate autonomously, and the higher-level system intervenes only at a cost which makes up part of the RL objective [6]}, [7]}, an arrangement that resonates with the notions in neuroscience of habit pathways and automatic versus controlled processing [8]}, [9]}, as well as the idea of a ‘cost of control’ [10]}. In deep RL, the notion of top-down control over lower-level habits has also been applied in motor control tasks, in architectures resonating with classical neuroscientific models of hierarchical control [11]}, [12]}.
