An interesting question that has arisen in neuroscientific work is how the balance between model-free and model-based RL is arbitrated, that is, what are the mechanisms that decide, moment to moment, whether behavior is controlled by model-free or model-based mechanisms [1]}, [2]}. Related to this question, some deep RL work in AI has introduced mechanisms that learn through RL whether and how deeply to plan before committing to an action [3]}. The resulting architecture is reminiscent of work from neuroscience on cognitive control mechanisms implemented in the prefrontal cortex [4]}, a topic we discuss further below.
