The advantages of the HQN, along with recent work by others [1]}, [2]}, [3]}, suggests that hierarchical structure is an effective catalyst for adaptive and generalizable learning in artificial agents. Indeed, substantial evidence from experimental and computational neuroscience suggests the same is true of biological brains [4]}, [5]}, [6]}, pointing towards the looped architecture of cortico-basal ganglia networks as an important feature for model-based and model-free learning systems [5]}, [6]}. The basal-ganglia (BG) is a subcortical network that receives widespread cortical input through the striatum, forming a channel-like architecture - each channel representing a particular action - that loops back up to motor cortex through the thalamus [9]}. Critically, each action channel in the BG contains a facilitation and suppression pathway, capable of exerting bidirectional control over the corresponding action channel in primary motor cortex. Schultz and colleagues [10]} famously showed that, during learning, the weights of these pathways are adjusted by phasic changes in striatal dopamine, encoding both the magnitude and sign of the prediction errors estimated from Q-learning models. This dopamine-dependent plasticity of cortico-striatal connections serves to reinforce the future selection of rewarding actions while also suppressing less desirable alternatives, serving a similar computational goal to that of Q-Networks [11]}, [12]}, [13]}. However, as previously mentioned, relying on feedback alone to drive learning 1) quickly becomes inefficient as task complexity increases, 2) limits the range of learned associations that can be simultaneously stored and exploited, and 3) fails to account for the robust and flexible nature of mammalian behavior.
