article: Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to “model-free” and “model-based” strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.
  The fundamental problem in reinforcement learning is estimation of state-action values (cumulative future reward), which an agent then uses to choose actions. In the dual-system theory, the fast and automatic system corresponds to a “model-free” reinforcement learning strategy, which estimates state-action values from trial-and-error learning [36,37]. In essence, this strategy is an elaborated version of Thorndike’s Law of Effect: actions that previously led to reward are more likely to be taken in the future. The strategy is “model-free” because it has no representation of the environment’s causal structure (i. e. , the transition function between states and the reward function in each state). Instead, it incrementally constructs a look-up table or function approximation from which values can be quickly computed. However, this strategy can lead to errors if the environment changes, because the entire value function must be incrementally updated to accommodate changes. In addition, the strategy can produce sub-optimal credit assignment [8], a property we explore below. These forms of brittleness illustrate how model-free learning gives rise to “habits”—fast but inflexible response tendencies stamped in by repetition. The slow and deliberative system corresponds to a “model-based” learning strategy that possesses operating characteristics complementary to the model-free strategy. This strategy learns an explicit causal model of the environment, which it uses to construct plans (e. g. , by dynamic programming or tree search). In contrast to the habitual nature of the model-free strategy, the capacity to plan enables the model-based strategy to flexibly pursue goals. While more computationally expensive (hence slower and more effortful) than the model-free approach, it has the potential to be more accurate, because changes in the environment can be immediately incorporated into the model. The availability of a causal model also allows the model-based strategy to solve the credit-assignment problem optimally. This dual-system framework sketched above can account for important findings in the reinforcement learning literature, such as insensitivity to outcome devaluation following overtraining of an action-reward contingency [7,38]. Furthermore, the framework has spurred a wealth of new research on the neural [8,10–13,39,40] and behavioral implications of competition and cooperation between reinforcement learning strategies [19–25,29,41,42]. How might the brain arbitrate between model-free and model-based strategies? Since the model-based strategy attains more accurate performance through effortful computation, people can (up to a point) increase reward by engaging this system. However, in time-critical decision making settings, the model-based strategy may be too slow to be useful. Furthermore, if cognitive effort enters into the reward function [43–45], then it may be rational to prefer the model-free strategy in situations where the additional cognitive effort of model-based planning does not appreciably increase reward. It has been hypothesized that this trade-off between accuracy and demand plays a pivotal role in the arbitration between the two strategies [38,46–50], but so far direct evidence for arbitration has been sparse [51].
  Here, we will first describe in detail the design of the Daw two-step task, and the reinforcement-learning model of this task [8]. Next, we will show through computational simulations that model-based planning on this task does not yield increased performance accuracy. Finally, we will discuss several factors that contribute to this shortcoming in the current approach in this two-step task, and related paradigms. Since its conception, the design of the Daw two-step task has been used in many similar sequential decision making tasks. Given the surprising absence of the accuracy-demand trade-off in the original task, it is important to investigate whether related versions of this paradigm are subject to the same shortcoming. Despite the substantial differences between these variants of the two-step task, we found that none of them encompasses a motivational trade-off between planning and reward. This observation naturally raises a question: Why does planning not produce an increased reward rate in this task? What characteristics of the paradigm distort the accuracy-demand trade-off? We investigate five potential explanations. These are not mutually exclusive; rather, they may have a cumulative effect, and they may also interact with each other. First, we show that the sets of drifting reward probabilities that are most often employed are marked by relatively low distinguishability. Second, we show that the rate of change in this paradigm is slow and does not require fast online (model-based) flexibility. Third, we show that the rare transitions in the Daw two-step task diminish the reward-maximizing effect of a model-based choice. Fourth, we show that the presence of the choice at the second stage decreases the importance of the choice at the first stage, which is the only phase where the model-based system has an influence. Fifth, we show that the stochastic reward observations in this task do not carry enough information about the value of the associated stimuli. We use simulations of performance on novel tasks to demonstrate these five points and, as a result, develop a novel paradigm that embodies an accuracy-demand trade-off.
  For the participants who completed the Daw task, we found that a reward on the previous trial increased the probability of staying with the previous trial’s choice [t (196) = 7. 70, p < 0. 001; Fig 16A], but that this effect interacted with the type of transition on the previous trial [t (196) = 5. 38, p < 0. 001]. This result replicates the basic finding on the original two-step confirming that participants used both model-based and model-free strategies. For the participants who completed the new paradigm, we found that a positive reward on the previous trial significantly enhanced staying behavior from chance for both similar and different current start states, (p < 0. 001 for both effects), but this effect was larger for the same compared to the different start state condition [t (183) = 9. 64, p < 0. 001; Fig 16B]. This pattern of behavior suggests that the participants did not employ a pure model-based strategy (compare with Fig 15B). However, as described above, it does not allow us to assess the relative contributions of model-based and model-free strategies to control based on these raw stay probabilities: both a purely model-free agent and an agent with a mixture of model-based and model-free strategies choices are predicted to show an increased stay probability after a win in a different start state, since a reward is indicative of history of recently reward trials. The reinforcement learning models described above incorporates the (decayed) experience on all previous trials to choice and is better able to dissociate the contributions of the two strategies. This model consists of a model-free system that updates action values using temporal-difference learning and model-based system that learns the transition model of the task and uses this to compute action values online. The weighting parameter w determines the relative contribution between model-based and model-free control. The stickiness parameters π and ρ capture perseveration on either the response-level or the stimulus-choice. We first investigated whether the inclusion of either stickiness parameter (π and ρ) was justified by comparing both the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC), for models that included none, one, or both parameters for both tasks separately (see S1 Table). For the Daw task, we found that both goodness-of-fit measures favored a model that included both stickiness parameters. For the novel task, the BIC favored a model with response stickiness but not stimulus stickiness included, whereas the AIC favored a model that included both stickiness parameters. We decided to favor the more parsimonious model without stimulus stickiness, and parameter fits from this model will be reported in the following, but the results did not qualitatively change when the stimulus stickiness parameter was included. Second, we used model comparison with both goodness-of-fit measures to analyze whether the hybrid model including the w parameter fit the data better than either a pure model-based or model-free model (see S2 Table). For the Daw task, we found that the AIC favored the hybrid model, but that the BIC favored the pure model-free model. However, there have been many reports in the literature that justify the inclusion of the weighting parameter for this task [8], and so we adopt the hybrid model for consistency with prior work. (Note also that it would be impossible to assess the relationship between model-based control and reward without using the hybrid model). For the new task, we found that both BIC and AIC favored the hybrid model compared to the pure model-based and model-free models. This suggests that human performance in the new paradigm is characterized by a mixture of model-based and model-free strategies. In summary, the model fits presented below used all six free parameters for the participants that completed the Daw paradigm, but omitted the stimulus stickiness parameters for the participants that completed the novel paradigm. These parameter estimates and their quartiles are depicted in Table 1. Across participants, we found that the median weighting parameter w was 0. 27 for the Daw paradigm and 0. 48 for the novel paradigm, indicating that both strategies were mixed in the population for both tasks. However, we found that model-based control was significantly higher for participants in the novel paradigm compared to the Daw paradigm [Wilcoxon two-sample rank sum test, z = 3. 31, p < 0. 001], suggesting that the existence of the accuracy-demand trade-off in the novel paradigm induced a shift towards model-based control. Of greatest relevance to our present aims, we found that the weighting parameter was positively related to our measure of the reward rate that controlled for average chance performance for the novel task (r = 0. 55, p < 0. 001), but not for the Daw paradigm (r = 0. 10, p = 0. 15; Fig 17). A subsequent multiple regression showed that this relationship was significantly different between groups [t (377) = 4. 71, p < 0. 001]. Next, in order to quantify the average gain in points across the entire range of w for both tasks, we ran a set of linear regression analyses predicting the reward rate from the weighting parameters for both tasks. For the Daw task, we found a predicted reward rate of 0. 52 for w = 0 (i. e. , the intercept), and an increase of 0. 002 on top of this for w = 1 (i. e. , the slope), indicating a 0. 42% increase in points (to 0. 525). For the novel task, we found a predicted reward rate of 0. 58 for w = 0, and an increase of 0. 67 on top of this for reward rate w = 1, indicating a 215% increase in points (to 1. 25). When we computed these slopes using the corrected reward rates, subtracting the average value of each participant’s reward distribution from their reward rate, we found an average increase in reward rate of 0. 01 across the range of the weighting parameter for the Daw task, and an average increase in reward rate of 0. 51 for the novel task. These results validate the accuracy-demand trade-off of the novel two-step paradigm, and also demonstrate that the original Daw two-step paradigm does not embody such a trade-off.
  In recent years, the Daw two-step task has become the gold standard for describing the trade-off between accuracy (model-based control) and computational demand (model-free control) in sequential decision making. Our computational simulations of this task reveal that it does not embody such a trade-off. We have developed a novel version of this task that theoretically and empirically obtains a relationship between model-based control and reward (a proxy for the accuracy-demand trade-off). The current investigation reveals a critical role for computational simulation of predicted effects, even if these appear to be intuitive and straightforward. It also introduces a new experimental tool for behavioral and neural investigations of cost-benefit trade-offs in reinforcement learning. Finally, it opens new avenues for investigating the features of specific tasks, or domains of task, that favor model-based over model-free control.
summary: When you make a choice about what groceries to get for dinner, you can rely on two different strategies. You can make your choice by relying on habit, simply buying the items you need to make a meal that is second nature to you. However, you can also plan your actions in a more deliberative way, realizing that the friend who will join you is a vegetarian, and therefore you should not make the burgers that have become a staple in your cooking. These two strategies differ in how computationally demanding and accurate they are. While the habitual strategy is less computationally demanding (costs less effort and time), the deliberative strategy is more accurate. Scientists have been able to study the distinction between these strategies using a task that allows them to measure how much people rely on habit and planning strategies. Interestingly, we have discovered that in this task, the deliberative strategy does not increase performance accuracy, and hence does not induce a trade-off between accuracy and demand. We describe why this happens, and improve the task so that it embodies an accuracy-demand trade-off, providing evidence for theories of cost-based arbitration between cognitive strategies.
keywords: learning
  decision making
  astronomical sciences
  social sciences
  mathematical models
  neuroscience
  learning and memory
  simulation and modeling
  cognitive psychology
  systems science
  mathematics
  probability distribution
  cognition
  research and analysis methods
  random walk
  computer and information sciences
  human learning
  mathematical and statistical techniques
  agent-based modeling
  planets
  probability theory
  psychology
  celestial objects
  planetary sciences
  biology and life sciences
  physical sciences
  cognitive science
title: When Does Model-Based Control Pay Off?