Working within the constraints of impartial game theory, we now consider the differences between RL agents with model-free and model-based learning strategies. Substantial evidence from cognitive psychology and neuroscience suggests that model-based learning is associated with hierarchical information processing, with action-value associations learned at lower levels of the hierarchy and abstract predictions about the environment at higher levels [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. One example of such a hierarchy is the prefrontal cortico-basal ganglia (BG) network [7]}, [8]} found in many mammalian species. Converging evidence from human and animal neurophysiological experiments shows that the prefrontal BG networks engage in model-free learning of action-values, driven by phasic dopaminergic signals from the midbrain that alter the weights of cortical inputs to the BG in accordance with environmental feedback [9]}, [10]}, [11]}. As a result of this plasticity, rewarded (or punished) actions become more (or less) likely to be executed in the future. This form of learning is analogous to that enacted by deep Q-Learning (DQL) agents that exhibit behavioral policies determined solely by the feedback of previous actions.
