It is often assumed that animals and people adjust their behavior to maximize reward acquisition. In visually cued reinforcement schedules, monkeys make errors in trials that are not immediately rewarded, despite having to repeat error trials. Here we show that error rates are typically smaller in trials equally distant from reward but belonging to longer schedules (referred to as “schedule length effect”). This violates the principles of reward maximization and invariance and cannot be predicted by the standard methods of Reinforcement Learning, such as the method of temporal differences. We develop a heuristic model that accounts for all of the properties of the behavior in the reinforcement schedule task but whose predictions are not different from those of the standard temporal difference model in choice tasks. In the modification of temporal difference learning introduced here, the effect of schedule length emerges spontaneously from the sensitivity to the immediately preceding trial. We also introduce a policy for general Markov Decision Processes, where the decision made at each node is conditioned on the motivation to perform an instrumental action, and show that the application of our model to the reinforcement schedule task and the choice task are special cases of this general theoretical framework. Within this framework, Reinforcement Learning can approach contextual learning with the mixture of empirical findings and principled assumptions that seem to coexist in the best descriptions of animal behavior. As examples, we discuss two phenomena observed in humans that often derive from the violation of the principle of invariance: “framing, ” wherein equivalent options are treated differently depending on the context in which they are presented, and the “sunk cost” effect, the greater tendency to continue an endeavor once an investment in money, effort, or time has been made. The schedule length effect might be a manifestation of these phenomena in monkeys.
In studying reward-seeking behavior it is often assumed that animals attempt to maximize long term returns. This postulate often forms the basis of normative models of decision making [1], choice behavior [2]–[4], and motivation [5], and plays a prominent role in the field of Reinforcement Learning (RL; see, e. g. , [6]). RL is a set of methods for learning to predict rewarding outcomes from their association with environmental cues, and to exploit these predictions to generate effective behavioral policies. These are policies that comply with principles of reward maximization [7], [8] and invariance [9], [10]. Applied to reward-seeking behavior, the principle of reward maximization states that subjects should maximize the reward/cost ratio, and the invariance principle that subjects should be equally motivated when facing situations with identical reward/cost ratios. The idea of maximizing reward over time or effort is general and has provided an effective basis for describing decision-making where the choice between available options is basically a matter of preference. RL methods such as the method of temporal differences (TD) constitute an efficient way of solving decision problems in tasks where a subject must choose between a larger vs. a smaller reward, or between a more probable vs. a less probable reward, and predict courses of actions comparable to the actual behavior observed in animals performing the same tasks [11]–[13]. RL methods have proven less successful, however, in situations where motivation, defined as the incentive to be engaged in a task at all, plays a strong role [14]–[16]. A case in point is the behavior of monkeys performing visually-cued reinforcement schedules [17], wherein a series of identical actions is required to obtain reward, and a visual cue indicates how many trials remain to be completed before a reward is delivered (“reward schedule task, ” see Figure 1). In this task, the error rate of the monkeys is proportional to the number of unrewarded trials remaining before reward, indicating that the value of the trial is modified by knowing the number of remaining trials. This violates the principle of reward-maximization: monkeys make errors in unrewarded trials that will have to be repeated, thus preventing optimal reward-harvesting behavior. Here we show that in trials equally far from reward, monkeys make fewer errors in longer schedules, when more trials have already been performed (“schedule length effect”). Thus, the value of the current trial is also modified by the number of trials already completed. This behavior violates the principle of invariance: monkeys perform differently in trials equally far from reward, depending on the number of trials already completed in the current schedule. Taken together, these results suggest that the behavior in the reward schedule task does not develop under the principles of invariance and reward-optimization, as commonly assumed when applying RL methods to understanding reward-seeking behavior. We present a RL rule which predicts the monkeys' behavior in the reward schedule task. Such a rule is a heuristic generalization of TD learning. When applied to the reward schedule task, it predicts all aspects of monkeys' behavior, including the sensitivity to the contextual effect due to schedule length leading to the violation of the invariance principle. When applied to a task involving choice preference, the new method predicts the same behavior as does the standard TD model. Thus, the behaviors in the reward schedule and in choice tasks can be the consequence of the same learning rule. Building on the special cases of the reward schedule and choice tasks, we then provide a general theory for Markov Decision Processes, wherein the transition to the next state is governed in a manner similar to a choice task, but is conditioned on whether the agent is sufficiently motivated to act at all, like in the reward schedule task. Finally, we link the schedule length effect to instances of “framing” [18], [19] and “sunk cost” effects [20], [21], which also emerge in conjunction with the violation of the principle of invariance.
In this work we collate the behavior of 24 monkeys tested in the reward schedule task [17], and analyze the entire set of data as a group (see Material and Methods). In this task, a series of trials had to be completed successfully to obtain reward at the end of the series. This series is defined to be a schedule, which is then characterized by its length measured in number of trials (Materials and Methods and Figure 1). The monkey starts each trial by holding a bar which causes a visual cue to appear on a computer screen, followed by the appearance of a red dot in the middle of the screen. The monkey must wait for the red dot to turn green (“GO” signal), at which point it must release the bar within a 200–1000 ms window. If the bar is released correctly, the monkey proceeds to the next trial of the schedule. Each trial must be repeated until performed correctly. In the presence of visual cues informing the monkey of the progress through the schedule (Valid Cue condition), the percentage of errors in all monkeys was directly related to the number of trials remaining to be completed in the schedule, i. e. , the largest number of errors occurred in the trials that are furthest from the reward (χ2 test, p<0. 05; Figure 2A and 2B, circles; each trial is labeled by the fraction τ/s, where τ stands for current trial and s stands for current schedule length). The performance in terminal trials was indistinguishable across schedules for each monkey, was above 94% correct in 14 out of 24 monkeys, and above 90% in 19 out of 24 monkeys. In the Random Cue condition the visual cues were selected at random and bore no relationship to schedule state. In such a condition, error rates were indistinguishable across all schedule states (or idiosyncratic; “x” in Figure 2A and 2B; χ2 test, p>0. 05 in 10 out of 15 monkeys tested in the Random Cue condition), and close to the error rates in terminal trials in the Valid Cue condition. Thus, performance in unrewarded trials in the Valid Cue condition was well below the ability of the monkeys. Since the individual trials of each schedule have the same perceptual and motor demands, we interpret the different error rates as being related to the different levels of motivation. This interpretation is also supported by the observation that, in most monkeys, the reaction times become faster as the end of the schedule is approached [17], [22]–[25]. In the penultimate trials of each schedule (i. e. , 1/2,2/3, and 3/4 when available) 20 of 24 monkeys made progressively fewer errors as the schedule became longer (sign test, p<0. 005). The error rate in state 1/2 was significantly larger than in state 2/3 in 12 out of 20 monkeys (Marascuilo procedure, p<0. 05, see Materials and Methods and Figure 2C and 2D). In two of three monkeys tested with 4 schedules, the error rate in state 2/3 was also significantly larger than in state 3/4. The third monkey tested with 4 schedules showed small error rates, and multiple comparisons between penultimate trials were not significant (monkeys often will not perform the task with 4 schedules [17]). In many of these studies the cues were distinguished by their brightness, where their brightness had been set according to the number of trials remaining in the schedule (Material and Methods), raising the possibility that performance was related to judging the brightness. However, this seems unlikely because the behavioral sensitivity was also seen when unique stimuli, e. g. , Walsh patterns, were used as cues (e. g. , Figure 2 of [26]), where no feature of the visual stimulus is a graded function of reward proximity or progress through the schedule. In conclusion, in a population of monkeys there was a significant tendency for motivation to increase with the number of trials already performed, at parity of proximity to reward. We refer this phenomenon to as the “schedule length effect. ” In the reward schedule task, all trials have the same cost because they all require the same action in response to the same trigger (the appearance of the green dot); trials differ only in their proximity to reward, which in turn does not depend on how many trials have already been performed. A standard reinforcement learning method can only learn to predict the proximity to reward correctly, and thus, unlike the behavior shown by the monkeys, is insensitive to the context introduced by the schedule length. We address this issue in detail in the remainder of this manuscript.
In our model, a single algorithm explains the differential behavior with valid and random cues. Assuming that the average value of the schedule states is a measure of overall motivation, the model predicts that the overall motivation is similar in the valid and random cue conditions. The difference in performance in the two paradigms is a consequence of the non-linear (sigmoidal) shape of the performance function Equation 1 (cf. Figure 4C). The finding that the same overall level of motivation leads to different patterns of error rates with valid and random cues is not built into the model but is an emergent property of the learning process. The context-sensitive model also predicts that, although the behavior appears to be the same in all terminal trials, terminal trials may acquire different values (see, e. g. , Equation 9). This difference is not reflected in the behavior since the latter depends on both the values (which might be different) and the performance function (Equation 1), which tends to remove value differences in the high value region (Figure 4C). In this region the performance function (or its complement) is almost flat and slight differences in value will be unlikely to produce observable differences in error rate. The context-sensitive behavior is also an emergent property of the model. The model does not change the definition of the schedule states to accommodate their contextual meaning. Valid cues come to “label” the schedule states via predictive learning. The basic model translates these labels into a pattern of motivational values and error rates which only depend on reward proximity, and thus are the same in penultimate trials. This symmetry is broken in the context-sensitive model as a consequence of generalizing the temporal difference so as to look backwards as well as forward, and not through a redefinition of the schedule states. It might seem at first that the model does not take into account the cost of performing a trial, i. e. , the cost of releasing the bar at the GO signal. In fact, this cost could be interpreted as the origin of the residual, non-zero error rate given by the performance function (Equation 1) when the values are maximal (approximately, the error rate in validly-cued rewarded trials). It is also possible to implement this cost so as to affect the values of each state, V (S) →V (S) −c, where c stands for cost. However, since the cost of the action is the same for all trials, it could not account for the differential error rates in different schedule states. Our analysis unveils the inadequacy of standard TD learning for the reward schedule task. The general statement can be proved that it is not possible to capture the schedule length effect with RL methods inspired to TD learning, including TD (λ) [27], if these only take into account the values of trials remaining in the current schedule (cfr. Equation 4; see Materials and Methods for details). Thus, for a method based on temporal differences to capture the schedule length effect, its learning prescription must have access either to the value of a past trial in the current schedule, as proposed in this manuscript, or to the value of a trial belonging to a different schedule, a method that is not clear how to generalize beyond the reward schedule task. The predictions of the context-sensitive model are the same as standard TD learning in a wide class of other tasks involving choice, where the values of states at decision nodes apply equally to whatever outcome of the decision. In simple choice tasks (cf. Figure. 5), both models predict a preference for more probable rewards, either always—under a greedy policy—or with occasional, temporary reversals of preference when the policy allows exploratory behavior—like the softmax function Equation 11. In the choice-schedule task of Figure 6A, the context-sensitive model predicts the same preference as the standard model. With schedules comprising more than two trials, choice preference of one model can be mapped into the choice preference of the other by readjusting the value of the discount rate γ appropriately. Thus the context-sensitive model, although heuristic in its derivation, appears to be a generalization of standard TD learning: it predicts the same behavior in tasks where human and animal subjects tprefer the choice leading to more probable or larger rewards; but it also predicts the violation of the principle of invariance occurring in the reward schedule task, not captured by the standard model; and it predicts the “procrastination-like” behavior of monkeys in the same task. The latter is to be generally expected in tasks requiring a step-wise approach to reward, where the willingness to act in each single trial exerts a powerful influence on the behavior. More work is required to characterize fully the mathematical properties of the model, and explore its possible derivation from well-defined principles as is customary in the fields of Machine Learning and RL, which is beyond the scope of this work. The reward schedule and choice tasks represent two particular cases of general MDPs where the problem of making a decision can be factorized into two sub-problems, the motivation to perform at all, and the selection of one among alternative choices given the motivation to act. We have used the strategy of dividing this general problem into two parts: we have analyzed the behavior as driven by motivational value using the reward schedule task, and the behavior as driven by choice preference using choice tasks. In both cases, we have compared the standard and the novel TD model using the same policy for both. These two components are simply multiplied in general MDPs, where by definition both the motivation to act and choice selection can occur. Our results indicate that only in the choice selection problem does the actor-critic architecture of RL [6], [37] potentially have a significant role. In the actor-critic architecture, the RL problem is solved by two related “structures, ” one responsible for performing the action (the actor), the other responsible for criticizing those actions based on evaluative feedback (the critic). Actor-critic architectures usually lead to policies that maximize the long-term return, and thus they seem to have only a small role in the reward schedule task. If an underlying actor-critic is present, its effectiveness in producing optimal control might be blunted by an opposing force deriving from the purely motivational nature of the problem encountered in this task, i. e. , whether or not to comply with its demands. Indeed, we have shown that it is sufficient for the critic to assess the value of the current trial and use it to direct the level of engagement in the task, without the need for a more specialized actor structure as would be required for action selection [38]. Instead, the process of valuation of several alternatives, potentially leading to different courses of actions and rewards as it typically occurs in general decision problems, could benefit more from an actor-critic organization of the behavior. The extension of RL to capture the fundamental role of motivation in reinforcement schedules is currently a major challenge for the field, and other authors have also considered how to include motivation in RL [14], [15]. These authors focused on incorporating overall drive (e. g. , such as degree of hunger or thirst) so as to describe how habitual responses can be modified by the current motivational level, which is, in turn, assumed to influence generalized drive through sensitivity to average reward levels [39]. In the reward schedule, however, we focused on how motivation orients behavior in a trial-specific, not generalized, manner. In such a case, an alternative solution to ascribing errors to a decreased level of motivation is Pavlovian-instrumental competition, which has been used to explain suboptimal behavior [16]. Applied to the reward schedule task, this solution would posit that error trials would result from the competition between the negative valence of the valid cue associated to an unrewarded trial (acquired through a Pavlovian-like mechanism), and the incentive to perform the same trial correctly to reach the end of the schedule and obtain reward. This interpretation is supported somewhat by the fact that the visual cues have no instrumental role in the reward schedule (they are neither triggers nor instructors of correct behavioral actions). The schedule length effect, however, escapes explanations in terms of Pavlovian-instrumental competition and would still have to be taken into account. Instead, the single motivational mechanism put forward in this work accounts for all the aspects of the behavior; has a natural interpretation in terms of learned motivation to act, however originated; and can be extended to general MDPs. A dependence on the value of the preceding state implemented in our learning rule suggests an explanation of the schedule length effect as a history effect. When environmental cues are not perfect predictors of the availability of resources, monkeys' decisions about where to forage depend on past information like the history of preceding reinforcements [40], or stored information about recent trends in weather [41]. Lau and Glimcher [28] have found that past choices, in addition to past reinforcements, must be taken into account to predict the trial-by-trial behavior of rhesus monkeys engaged in a choice task resulting in matching behavior. However, contrary to the statistical description of Lau and Glimcher [28], past information in our model bears an effect on the learning rule, not directly on the action selection process, and it does so through the value of the previous state, as opposed to past reinforcements or past choice history. Taken together, these findings point to some form of sensitivity to preceding actions and visited states (or their values) in primates' foraging behavior, and the schedule length effect might be a side effect of such a mechanism, perhaps also present in other forms of reinforcement learning. Current theories of reinforcement learning posit that dopaminergic neurons code for a prediction error signal analogous to δ in our model and in TD learning in general [30]. Data from dopaminergic neurons of monkeys performing a reward schedule task, however, are not in sufficient accord with the predictions of such theories [24]. For example, one prediction is that δ, and therefore dopamine neurons, after sufficient training should cease to respond to predicted reward, and this was not observed. Recent developments [42], [43] rule out that this could be the consequence of the small temporal jitter around reward delivery. Despite the incongruence with the assumed role of dopamine neurons as signaling some form of prediction error, there is clear evidence of the involvement of dopamine in learning. In the reward schedule, the importance of dopamine D2 receptors for learning the meaning of new valid cues has been demonstrated in perirhinal cortex [26], and Ravel and Richmond [24] have argued that salient events may drive dopaminergic neurons, whose activity may be required for enhancing the connection of the stimulus with its prediction in perirhinal cortex. The contextual impact of the organization of the task in schedules has been found in the event-related responses of neurons in all neural structures investigated thus far in the reward schedule task, except perhaps for neurons of the area TE [22]. The brain area where the neural modulation with schedule state is most apparent is the anterior cingulate cortex [44]. One third of the neurons recorded in this area keep track of the progress through the schedule in the Valid Cue condition, and could reflect the (motivational) value of the schedule states and their being linked to one another in a chain of states culminating in the rewarded trial. Another candidate structure for the representation of the schedule states is the perirhinal cortex, whose neurons become selective for the meaning of the visual cues, as opposed to, e. g. , TE neurons' responses that are locked to their physical identity [22]. In some brain regions, neuronal responses are different in trials of different schedules that might be regarded as homologous, particularly last trials of different schedules. Dopamine neurons [24], perirhinal neurons [22] and ventral striatum neurons [23] respond differently to valid cues in last trials (predicting the same reward, but in different schedules). This is reminiscent of the phenomenon that the context-sensitive model assigns different values to terminal trials belonging to different schedules. Neurons of the basolateral complex of the amygdala often have differential post-cue activity in first trials [25]. In these neurons another, different effect related to the organization in schedules has also been observed: these neurons increase their activity in the pre-cue period before the beginning of each schedule. No pre-cue activity was observed in the Random Cue condition, supporting the hypothesis that pre-cue activity is related to the contextual imprint of the task' s organization in schedules [25]. This activity could be related to a context-sensitive representation of the values of the states, either in the amygdala itself, or in areas connected to the amygdala like perirhinal cortex [22], anterior cingulate cortex [44] and ventral striatum [23], where the schedule state meaning of valid cues is strongly represented. The possibility of a more specific role of the amygdala for the emergence of the schedule length effect will be considered later when discussing the analogous phenomenon of “framing” in humans. Finally, there is evidence for the role of the primate striatum in learned action selection, with some authors [45] proposing for its ventral part coding for the values of states (reminiscent of the critic in actor-critic RL methods), and its dorsal part coding for the values of actions and for action selection (reminiscent of the actor [11], [13], [45]; but see [38]). In the reward schedule, the largest population of ventral striatum neurons which are responsive around the time of bar release, do so in rewarded trials, with the second larger population being responsive in all trials [23]. Comparison of latency and periods of peak activity between these neurons and neurons of the orbitofrontal cortex suggest that the latter are better positioned for representing the reward contingency and thus for guiding action, whereas the former are more related to executing the action [46]. This role is usually ascribed to more dorsal regions of the striatum, but the involvement of the ventral striatum is conceivable in the reward schedule, given the simple action selection required (it amounts to the timely execution of the bar release in all contingencies), and it is compatible with our model, where the probability of a correct bar release is based solely on the value of the current state and not on action values. In the context-sensitive model, the mechanism responsible for the schedule length effect leads to the violation of invariance. The violation of this principle was invoked by Tversky and Kahneman in their description of “framing” [18], [19]. Framing describes the process whereby the choice made is influenced by the manner or context in which the choice is presented. Thaler [47] and Tversky and Kahneman [18] showed that humans often act as if they kept separate accounts for gains and losses, rather than estimate the total value. A consequence of keeping separate accounts is that the manner in which a problem is cast, in terms of gains, of losses, or of total value influences choices. For example, people purchasing two items, costing respectively $15 and $125, are more willing to put an effort (for example by driving to another store) to save $5 when this is presented as a discount on the $15 item, than when presented as a discount on the $125 item, even though the total saving is the same [18], [47], [48]. Similarly, monkeys are willing to put more effort in a trial if the total effort to get there had been larger, even though this does not affect the upcoming reward. A “minimal account” would consider only the proximity to reward, whereas the behavior of the monkeys shows that a combination of minimal (reward proximity) and topical (workload) accounts affects their motivation when facing a reward schedule. From this point of view, reward proximity could be seen as a property defining the state (in accord with Equation 7), much like the $5 discount defines the saving in the example above, independently of the item to which it is nominally attached. In both cases, it is the comparison with some truly contextual attribute that assigns a different motivational value to the same action. Thus, especially on penultimate trials, the length of the schedule seems to exert a contextual effect on the monkeys' motivation analogous to framing. A more direct, preliminary example of framing in monkeys has been reported recently [49] using a task similar to one previously used with starlings [50]. The schedule length effect is also reminiscent of the so-called “sunk cost” effect [20], [21], [51], [52], “a maladaptive behavior that is manifested in a greater tendency to continue an endeavor once an investment in money, effort or time has been made” [21]. The sunk cost phenomenon comes in different varieties and with different interpretations (to the point of having different names, like “Concorde effect, ” “cognitive dissonance, ” “work ethics, ” see [20] for a review), some of which come close to framing. In one interpretation, sunk cost derives from the violation of the principle that “a prior investment should not influence one' s consideration of current options; only the incremental costs and benefits of the current options should influence one' s decision” [20]. The similarity with the schedule length effect and with the previous discussion about its interpretation in terms of framing seems obvious. A relevant example is Experiment 2 of Arkes and Blumer [21]. In this experiment, three groups of patrons were sold season tickets for the Ohio University Theater at three different prices, and those who purchased tickets at either of the discounted prices attended fewer plays during the season. In this case, the money spent at the beginning of the season influenced the patrons' choice to attend the plays. It could be argued that, in the reward schedule task, the cost of performing trials is not strictly a “sunk” (wasted) cost, as it would be if the monkeys had to start the schedule anew after each error trial. However, this would only be a minor difference with other instantiations of sunk cost effects; and it could similarly be argued that the money spent in Experiment 2 of Arkes and Blumer [21] is not a wasted cost, since it is necessary to attend the plays. Various explanations of sunk cost and framing have been proposed. Arkes and Ayton [20] explain the sunk cost fallacy as an overgeneralization of the “don' t waste” rule, since based on their review of the literature, the effect is not unambiguously present in lower animals, and is not found in children [20]. Even if the schedule length effect can legitimately be interpreted in terms of sunk cost or framing, we think that this is unlikely to be the correct explanation. A better explanation may be linked to emotional factors. A functional imaging study [53] points to an important emotional component in the susceptibility to frames in humans. This study found the susceptibility to framing linked to amygdala activations, with the ability to resist the frame linked to activation of the orbital and medial frontal cortex. Similarly, we believe that there is a strong emotional component responsible for the monkey' s reaction to unrewarded cues (leading to larger error rates), and possibly for the schedule length effect. Thus, a connection between this emotional component and parameter σ, which quantifies the schedule length effect in our model, could be speculated on the basis that a larger σ implies a larger schedule length effect, in the same way as a larger emotional component would imply a stronger susceptibility to framing [53]. We do not reject this idea as a possibility, but our data are not sufficient evidence for it. Our model does make a clear prediction in one case where framing has been found, i. e. , in the increase in preference due to training with a larger cost [51], a case of state-dependent learned valuation. In this experiment, starlings preferred to choose stimuli which had previously associated with a larger effort (16 1-m flights vs. four 1-m flights) to obtain an otherwise identical reward. Since this paradigm pitted two reinforcement schedules of different length against each other, there are obvious similarities with our reward schedule task. Indeed, it would be possible to run a similar test in monkeys by associating different cues to terminal trials in different schedules (e. g. , cue H for the longer schedule and cue L for the shorter), and then test the monkeys' preference in a choice task where there is no cost (or equal cost) to obtain the same reward from two sources, one cued with H, the other with L. Would the monkey prefer the cue associated during training with the longer schedule, as found in starlings [51]? Our model predicts exactly this. Because of the accumulation of previous values, the values of terminal trials are larger in longer schedules in the model with σ>0. Assuming that in the choice task preference depends on the same learned values, the source of reward cued by H (previously associated with the longer schedule) would be preferred. This also means that our model implies state-dependent learned valuation when the state of the animal is defined by the cumulative effort expended to obtain the reward. We stress, however, that our learning model is not meant to be a general model of the effects that frames, or sunk costs, have on humans and animals. For example, Pompilio et al [52] offer additional evidence of state-dependent valuation in an invertebrate (the grasshopper), but in their case the state of the animals at the time of learning is defined by their nutritional state (e. g. , more or less hungry) as opposed to their expended cost. They found that the grasshoppers, in a later choice task with equal cost, prefer the food experienced when in a lower nutritional state during learning. We do not see a connection between this finding and the schedule length effect, or the role of the parameter σ. This should not be surprising. As Pompilio et al. [52] point out, there may be more than a single mechanism responsible for state-dependent valuation, depending on the animal and, in the same animal, depending on the paradigm used for training. In the heuristic modification of TD learning introduced in this work, the schedule length effect emerges spontaneously from the sensitivity to the immediately preceding trial, leading to the violation of the invariance principle. Since this principle is violated in instances of framing and sunk cost effects, we have interpreted the monkeys' behavior using the framing and sunk cost analogies, even though monkeys might not be susceptible to framing or sunk cost the way humans are. We are not aware of alternative RL models predicting the violation of the principle of invariance.
In this work we collate the behavioral data from earlier studies on monkeys (Macaca mulatta) tested in the reward schedule task [22]–[26], [32], [44]. In all of these studies, randomly interleaved schedules of one, two or three trials must be completed to obtain a reward. In n = 3 monkeys, schedules with 4 trials were also used. A trial begins when the monkey touches a bar (Figure 1A), causing the appearance of a visual cue. Four hundred milliseconds later a red dot (WAIT signal) appears in the center of the cue. After a random interval of 500–1500 ms the dot turns green (GO signal). The monkey is required to release the touch-bar between 200 and 800 ms after the green dot appeared, in which case the dot turns blue (OK signal), and a drop of liquid reward is delivered 250 to 350 ms later. If the monkey releases the bar outside the 200–800 ms interval after the GO signal, an error is registered, and no reward is delivered. To start, monkeys are trained on this simple color discrimination task, with or without the presence of a cue, and are rewarded for every correct trial. When performance reaches criterion (at least 75% correct), reward schedules start. Each reward schedule is a sequence of 1, or 2, or 3, …, or Ns trials, where Ns is the maximal schedule length for that session (3 or 4; see Figure 1B for a 2-trial schedule). All schedules are selected with equal probability, and within a schedule error trials must be repeated until performed correctly. Only correct terminal trials are rewarded. After a correct terminal trial, a new schedule is selected pseudo-randomly. Each schedule state is labeled by the pair {τ, s}, where τ = 1,2, …, s stands for trial and s = 1,2, …, Ns stands for schedule. Terminal trials have τ = s. Trials of different schedules representing the same schedule fraction (e. g. , 1/2 and 2/4) are considered different schedule states, even though they might have been associated to the same visual cue (Valid Cue condition, see below). Different cue sets have been used in different studies [17], [22]–[26], [32], [44], [54], producing similar behavioral results. For the data shown in Figure 2, collected by Sugase-Miyamoto and Richmond [25] (panel A) and Shidara and Richmond [44] (panel B), horizontal bars with different brightness were used as cues, and the cues were brighter as the schedule progressed. Other cue sets have also been used. Some, still based on cue brightness, had the opposite relationship between brightness and proximity to reward, e. g. , cues were darker towards the end of the schedule, as, e. g. , in Figure 1 [22], [24], [32], [54], to ensure that the behavior of the monkeys was not biased by the direction of brightness. Other cue sets were based on bar length [26], [32]; still others consisted of unique stimuli like, e. g. , Walsh patterns [26], to establish that the behavior was not a consequence of having a sensory attribute (like length or brightness) increasing or decreasing with proximity to reward. The typical behavioral patterns that are the main focus of this work were similar across individual experiments and cue sets. In the paradigm with random cues, the same visual stimuli are present, but each stimulus is selected pseudo-randomly with equal probability in each trial (Random Cue condition). In such a case, there is no relationship between cues and schedule states, although the schedules are still in effect. The monkeys were not taught the “rules” of the reward schedule task but were simply exposed to it. The behavior reported in Figure 2 emerges spontaneously, typically within a week of the first exposure, depending on the monkey (in some cases, it emerges on the very first day), and it generalizes rapidly (in less than 3 days) to different cue sets. For each monkey, the error rates were calculated as the ratio of the total number of incorrect trials (in all sessions) to the total number of trials for each schedule state. Differences in error rates across schedule states were tested with a χ2 test of the contingency table obtained from the numbers of correct and incorrect trials (confidence was taken at the 5% level). Pair-wise comparisons of the error rates in different schedule states were tested with the Marascuilo procedure after a significant χ2 test [55]. If the χ2 test is significant at the α level, the Marascuilo procedure [55], [56] provides a confidence interval of 100 (1−α) % for each pair-wise difference of error rates |pi−pj|, where pi = ei/ni is the error rate in schedule state i, and ei, ni are, respectively, the number of error trials and total trials in schedule state i. The Marasquilo confidence interval on |pi−pj| is given by. In this formula, is the critical value of χ2 with N−1 degrees of freedom at α level of significance (the point of the distribution which leaves an area of α in the upper tail of the distribution). N is the number of different schedule states. Schedule states with |pi−pj|>pˆij are significantly different at the α level. A sign test [57] was run on the number (n+) of monkeys showing better performance in penultimate trials belonging to longer schedules, as compared to the number (n−) of monkeys where either the inverted pattern, or no difference, was observed. The “exact” binomial probability for n+ successes in n++n− trials was used. Reaction times were defined as the time elapsed since the appearance of the GO signal and the bar release, and, as reported previously, were generally shorter in trials more proximal to reward [17], [22]–[25]. Reaction times had a similar relationship to schedule states as did error rates. Since they provide no new qualitative interpretation, they were not analyzed further. For each monkey, the theoretical error rates (pth) were fitted to the experimental error rates (pex) by minimizing a weighted sum of squares, , where the sum goes over all schedule states in both the Valid and Random Cue conditions, and [58]. The reason for this choice is that the interval is approximately a 68% confidence interval around pex, i based on Wilson' s “score” equation [59], [60], and (Li, +−Li, −) /2 = Δpi. The theoretical error rates were given by Equations 2,9, and 10. The minimization of χ2 was accomplished with a full factorial search of the best-fit values for parameters β, γ, and σ of Equations 2,9, and 10. The formula Equation 6 of the main text for the equilibrium values of the basic model is exact only in the absence of errors, otherwise the values are smaller and are given by the self-consistent, recursion formula: (13) Here, S′ is the next state in the schedule, Pc|V (S) ≡P (c|V (S) ) is the probability of correct performance in (current) state S, conditioned on the value of that state, V (S). V (S) appears also on the left hand side, and for this reason the formula defines V (S) only implicitly. If S is a terminal trial, γV (S′) must be replaced by r in Equation 13. By iteration, Equation 13 gives (14) where to simplify the notation. This set of equations must be solved self-consistently for Vτs as the Pτs depend on Vτs. Under the optimal policy of not making any errors, i. e. , with each Pc|V≡1 independently of V instead of Equation 1, Equation 14 becomes the explicit solution given by Equation 6 reported in the main text. Equation 13 can be derived as follows: at equilibrium, V (S) is the average of the value obtained after an error (Ve, occurring with probability Pe|V≡1−Pc|V) and the value obtained after a correct trial (Vc, probability Pc|V), conditioned on current average value being V, i. e. (15) with Vc (V) = V+α (γV′−V) and Ve = V+α (γV−V). The last two equations are simply the update equation for V after a correct and an incorrect trial respectively; V′≡V (S′) is the value of the next schedule state after a correct trial (γV′ must be replaced by r in terminal trials). Solving Equation 15 for V gives Equation 13. The same procedure, though more involved algebraically, gives the values in the context-sensitive model: where Prs is defined as for the basic model. This system of equations must be solved self-consistently for the values Vrs. In the absence of errors, each Prs = 1 and Equations 9 of the main text follow. We have checked with simulations that the approximate solution given by Equations 9 gives a good approximation to the correct values on our dataset of monkeys' data. For this reason, Equations 2 and 9 were used to estimate the theoretical values when fitting the theoretical error rates to the experimental error rates. In the Random Cue condition, the cues define the states of the model. The model learns the values of the cues using the same algorithm specified by Equations 1,3, and 8, with St≡cuet. The next cue is selected at random with equal probability for all cues if the trial is performed correctly, otherwise the current cue remains as the next. We set δ = rt+σV (cuet−1) −V (cuet) in terminal trials, in keeping with the rule adopted with valid cues. The average value of random cues can be obtained by averaging the update equation over all trial types that produce a different temporal difference δ, obtaining (16) i. e. , ∑i fiδi (V) = 0, where V is the sought average value, fi is the average frequency with which trial i occurs, and δi is the temporal difference in trial i. In the basic model, it is sufficient to distinguish three trial types: correct terminal trials, incorrect terminal trials, and non terminal trials. The frequency (f) of correct terminal trials is N+Pc|V, where N+ is the average fraction of rewarded trials, equal to the number of schedules divided by the number of schedule states. In correct terminal trials the temporal difference is δ = r−V. Incorrect terminal trials occur with frequency N+ (1−Pc|V) and have δ = −V; non-terminal trials occur with frequency 1−N+ and generate a temporal difference δ = (γ−1) V, whether the trial is correct or not. Replacing these values in Equation 16 and solving for V gives (17) Equation 17 defines V only implicitly and must be solved self-consistently to give the exact value of V. For the small error rates usually encountered with random cues, Equation 17 is well approximated by its version in the absence of errors (Pc|V = 1 for any V), i. e. . Note how V increases with γ and is constrained between the average collected reward N+r (for γ = 0) and r (for γ = 1). Setting γ = 0 (value at which V is minimal) is the same as assuming that the next cue is always unknown and its value is zero (cfr. Equation 4). This implies that having some expectation about the next state, even a random expectation as for the random cues, increases the values and hence the motivation to perform correctly. The context-sensitive model can be solved in a similar way, with in addition non-first trials to be taken into account. The final result is, from which Equation 10 of the main text follows under the approximation of small error rates, i. e. , Pc|V≈1. Similar results are obtained in the case of post-reward expectation, where the value of the next state after a rewarded trial is not set to zero, as shown in a later subsection. Since it is required that V>0, this result requires (γ+σ) (1−N+) <1, or (γ+σ) < (Ns+1) (Ns−1) −1. This inequality is never violated in the basic model (where σ = 0), but it might be, and must be imposed, in the context-sensitive model, especially for long maximal schedule lengths. Similar restrictions coming from the values of valid cues also apply (e. g. , σ<1/2γ from Equation 9). Here we show that it is not possible to obtain values dependent on schedule length (like in the context-sensitive model) by using a standard TD learning rule, which considers only future trials within the current schedule. The most general such rule can be written as, where the coefficients {ai}i = 1,2, …, T may depend on pre-reward number (i. e. , the number of trials remaining before reward), but not on schedule length. t+T is the time at which the terminal trial is reached: when St is a terminal trial, the states St+i are not defined and their values are set to zero. It is more convenient to express the values as a function of the number, n, of trials remaining before reward (“0” being the terminal trial), conditioned on schedule length being s, V (n|s), as in Equation 7 of the main text. At equilibrium (δt = 0) one has V (1|s) = a1V (0|s). Since V (0|s) = r does not depend on s, V (1|s) does not depend on s, which in turn implies that V (2|s) = a1V (1|s) +a2V (0|s) does not depend on s, and so on. It follows by induction that V (n|s) does not depend on s for all pre-reward numbers n, and for any value of the coefficients an (some of which may vanish). This result holds also in the case of post-reward expectation, where the value of the next state after a rewarded trial is not set to zero and the forward terms in the series ∑ aiV (St+i) are taken to the (T+1) th term. As shown later, all values are re-scaled by a constant factor which does not depend on s, leaving the above argument unchanged. It follows from this argument that, to obtain the schedule-length effect, it is necessary either to look backwards at the values of previous trials in the same schedule (as in the context-sensitive model of the main text), or to take into account trials belonging to different schedules [61]. The notable TD (λ) rule (see, e. g. , [6]), that has been suggested to be implemented by dopamine neurons of rats [62], considers only forward trials within the current schedule, and therefore cannot produce the contextual effect due to schedule length. In fact, here we show that for the reward schedule, the equilibrium values in the TD (λ) rule are the same as those obtained with the basic model. In TD (λ), all forward trials within a schedule are considered, weighted by imminence. Formally, when in state St at time t, the TD term is evaluated as (18) where the sum is over all states remaining until the terminal one (reached after T steps). λ is a parameter between zero and one; NT≡1+λ+λ2+…+λT−1 is a normalization factor; and is the i-steps-ahead prediction starting from St. (If λ = 0, Equation 18 reduces to δt = rt+γV (St+1) −V (St), the basic model of the main text.) The values are updated in the usual way: Vt+1 = αδt. In the reward schedule it is (only the terminal trial is rewarded), and Equation 18 reads (19) where t+T is the time at which the terminal trial is reached. The solution to δt = 0, with δt given by Equation 19, is the same as for the basic rule (Equations 3 and 4 of the main text), i. e. , V (St+i) = γT−ir, or V (τ, s) = γs−τr if St≡{τ, s}, as can be proved, e. g. , by direct substitution. This was confirmed in simulations of TD (λ) -learning of the reward schedule implemented through the use of eligibility traces, an alternative approach to TD (λ) (see [6] for details). So far, the value of the next state at the end of each schedule had been to set to zero. In other words, the learning rule following a rewarded trial is δt = rt+σV (St−1) −V (St), which written in this form applies to all cases, including the case of random cues and the basic model (where σ = 0). As said in the main text, this is the common choice in RL [6]. Here we show that behavior predicted by the model does not change if we assign a positive value to the next state (“post-reward expectation”). The reason is that, in a terminal trial, the next trial is not known and thus the same value must be assumed independently of current schedule. The actual value is immaterial, but for the sake of argument we shall make a choice. In the Random Cue condition, the current value of any cue chosen at random will do; in the Valid Cue condition, since the only available information is that the next state will be one of the initial trials {1, s}, the average value of all first trials will be taken, i. e. , . It can be shown that the average value of each state is increased by a constant factor (1−γϑ) −1, where ϑ is the ratio of the value of the state post-reward to the value of rewarded trials. For the value chosen above (average of first trials), (note that this choice gives γϑ<1). Similarly, the value of random cues changes from to. Thus, there is no qualitative difference with respect to the case of no post-reward expectation of the main text. A similar argument also shows that the qualitative behavior does not change if σ>0 in first trials of each schedule in the context-sensitive model.
Theories of rational behavior are built on a number of principles, including the assumption that subjects adjust their behavior to maximize their long-term returns and that they should work equally hard to obtain a reward in situations where the effort to obtain reward is the same (called the invariance principle). Humans, however, are sensitive to the manner in which equivalent choices are presented, or “framed, ” and often have a greater tendency to continue an endeavor once an investment in money, effort, or time has been made, a phenomenon known as “sunk cost” effect. In a similar manner, when monkeys must perform different numbers of trials to obtain a reward, they work harder as the number of trials already performed increases, even though both the work remaining and the forthcoming reward are the same in all situations. Methods from the theory of Reinforcement Learning, which usually provide learning strategies aimed at maximizing returns, cannot model this violation of invariance. Here we generalize a prominent method of Reinforcement Learning so as to explain the violation of invariance, without losing the ability to model behaviors explained by standard Reinforcement Learning models. This generalization extends our understanding of how animals and humans learn and behave.
neuroscience/behavioral neuroscience
neuroscience/animal cognition
neuroscience/theoretical neuroscience
neuroscience/cognitive neuroscience
Modeling the Violation of Reward Maximization and Invariance in Reinforcement Schedules