Hard-wired, Pavlovian, responses elicited by predictions of rewards and punishments exert significant benevolent and malevolent influences over instrumentally-appropriate actions. These influences come in two main groups, defined along anatomical, pharmacological, behavioural and functional lines. Investigations of the influences have so far concentrated on the groups as a whole; here we take the critical step of looking inside each group, using a detailed reinforcement learning model to distinguish effects to do with value, specific actions, and general activation or inhibition. We show a high degree of sophistication in Pavlovian influences, with appetitive Pavlovian stimuli specifically promoting approach and inhibiting withdrawal, and aversive Pavlovian stimuli promoting withdrawal and inhibiting approach. These influences account for differences in the instrumental performance of approach and withdrawal behaviours. Finally, although losses are as informative as gains, we find that subjects neglect losses in their instrumental learning. Our findings argue for a view of the Pavlovian system as a constraint or prior, facilitating learning by alleviating computational costs that come with increased flexibility.
The functional architecture of responding involves two fundamental components that are behaviourally [1] and computationally [2] separable: Pavlovian and instrumental. The instrumental component respects the stimulus-dependent contingency between responses and their outcomes (stimulus-response and action-outcome learning) [3]. By contrast, preparatory Pavlovian responses, chiefly involving approach and withdrawal, are elicited by the appetitive or aversive valence associated with predictive stimuli in a manner that is not dependent on the consequences of those responses [3]–[5]. The interactions between the two systems are most evident when automatically-elicited Pavlovian responses interfere with contingent instrumental responding [1], [6]–[9]. For instance, pigeons will strikingly continue to peck at a light predictive of food (a preparatory approach elicited by the appetitive prediction), even if the food is withheld every time they peck the light (the instrumental contingency) [10], [11]. Pavlovian interference likely contributes to many quirks of behaviour such as impulsivity [12], framing and [13], endowment effects [14] and many other “anomalies” [15], including neurological [16]–[19] and psychiatric diseases [20]–[26]. Further, puzzling facets of seemingly purely instrumental behaviour such as the difficulties in learning ‘go’ responses to avoid punishments; or ‘nogo’ to obtain rewards (unpublished data) and even the restrictions in associations evident in ‘evolutionarily preparedness’ [27], [28] might be traced to Pavlovian principles. However, instrumental and Pavlovian systems share overlapping neural hardware. Their bidirectional interaction is characterised by two key triads: rewards are tied to approach and vigour; and punishments to withdrawal and behavioural inhibition. The neuromodulator dopamine (DA) responds predominantly to rewards [22], [29]–[31], induces behavioural activation and enhances approach [32]–[35]. Each aspect of this triad confounds the role of the phasic DA bursts in the flexible acquisition of instrumental values [36]–[42]. Serotonin appears to lie at the heart of the aversive triad, having been linked to punishments [43]–[45], behavioural inhibition and withdrawal [25], [32], [46]–[52], although dopamine acting via D2 receptors likely also plays a role in linking absence of rewards to nogo [17], [53], [54]. Signatures of both triads are also evident in neural circuits involved in response and choice. In the dorsal striatum, there are interdigitated pathways for ‘go’ and ‘nogo’, with the go pathways again linked positively to rewards via dopamine [16], [18], [55], [56]. The ventral striatum is primarily organized along an appetitive/aversive axis with direct links to approach and withdrawal behaviours [57], [58]. The aversive triad is also tightly linked to the dorsal raphé and the periaquaeductal gray [59], [60]. The main routes to the scientific investigation of these interactions consists of tasks in which Pavlovian stimuli are presented during ongoing instrumental tasks. However, these have as yet not explored the full set of interactions characterising the overlap between the two systems. Two critical confounds remain: The first confound concerns the precise nature of the effect of Pavlovian stimuli on instrumental behaviours. The instrumental behaviours studied have largely been appetitively motivated approach behaviours (in Pavlovian-Instrumental Transfer (PIT) and conditioned suppression tasks, [1], [6]–[8], [61]–[63]), and one instance of aversively motivated withdrawal behaviour [64]. The relative role of the appetitive-aversive motivation axis versus that of the approach-withdrawal axis is unknown. This in turn obscures the nature of the interaction: whether Pavlovian stimuli interact with the value of the instrumental behaviour, or by promoting specific responses [1], or even simply by modulating behavioural activation [5]. Second, the extent to which the separation of reward and punishment processing into opponent motivational structures applies to instrumental as well as Pavlovian learning is incompletely explored [1], [27], [28], [65]. All these issues can simultaneously be addressed in a combined PIT and conditioned suppression task with both approach and withdrawal actions in which the overall motivational component of approach and withdrawal are matched (Figure 1 and Table 1). The task separates the contributions of approach and withdrawal by using two counterbalanced blocks, one involving approach go versus nogo, and the other withdrawal go versus nogo. The comparison between go and nogo controls for effects of behavioural activation or inhibition. In each block, subjects first underwent brief instrumental training (Figure 1A), learning from positive and negative feedback (monetary gains and losses of €0. 20) whether to produce a go or a nogo response associated with sorting mushrooms. In the approach block (Figure 1A, top, all 46 subjects), go responses involved moving the cursor onto a mushroom (to collect it), while nogo involved doing nothing, thus not collecting the mushroom. To test for the effect of low-level motor variables, subjects performed one of two types of withdrawal actions. In “throwaway” (24 subjects, Figure 1A, middle), go involved moving the cursor physically away from the mushroom and clicking into an empty blue box; nogo involved doing nothing, and thus keeping the mushroom. Importantly, both approach to and withdrawal from the instrumental stimulus were orthogonal to any approach and withdrawal that might be directed at the Pavlovian background stimulus. In “release” (22 subjects, Figure 1A, bottom), the subjects had to start by pressing the mouse button. Go involved releasing the button to avoid collecting the mushroom; nogo involved continuing to press the button and thereby receiving the mushroom. In order to orthogonalise the approach-withdrawal and appetitive-aversive axes, the learned instrumental values in approach and withdrawal blocks needed to be matched. To achieve this, both go and nogo responses were, if correct, rewarded. Additionally, to avoid the confound of activation, in each block (i. e. in both approach and withdrawal blocks) the go action was designated as the correct response to half the instrumental stimuli, and the nogo action to the other half (see Table 1). Incorrect responses had opposite outcome contingencies to correct responses, yielding more punishments than rewards. This ensured that go, nogo, approach and withdrawal overall had the same learned association with rewards and punishments. We tested both deterministic and probabilistic outcomes but found no differences. In the second part of each block, subjects passively viewed unrelated, fractal, stimuli paired with separate auditory tones (Figure 1B). Each compound Pavlovian stimulus was deterministically associated with a monetary gain or loss, i. e. its Pavlovian value was equal to that monetary outcome. Every fifth trial in the Pavlovian block was a query trial (Figure 1C), in which subjects chose the better of two fractal visual stimuli without being informed about the outcome. Finally, in the PIT stage, the instrumental stimuli were presented on a background of fractal Pavlovian stimuli together with the auditory tones, and again without outcome information. Our task addressed the key confounds described above. With respect to the triads, we found that the Pavlovian influence is action specific: appetitive Pavlovian cues boosted go approach responses and suppressed withdrawal go responses; aversive Pavlovian cues did the opposite. Additionally, subjects were substantially biased against withdrawal, but we found no evidence that the instrumental learning component itself differed between the approach and withdrawal condition.
There was no difference between the results for probabilistic and deterministic feedback, and we therefore present the combined data. Analysis of the components of the experiment indicate robust, yet moderate, instrumental conditioning that was stable during the PIT period, combined with highly robust Pavlovian conditioning. Figure 2A shows the instrumental probability of choosing the more rewarded (“correct”) stimulus over time. Subjects rapidly came to prefer the more rewarded action. Preference was weaker for go withdrawal, against which there was a consistent bias. We intended the instrumental preference to be weak to avoid ceiling effects when assessing PIT. Subjects also exhibited predictable variability on a shorter time-scale: Figure 2B shows the immediate consequences of rewards and punishments on subsequent behaviour. It is notable that punishments did not reduce the repeat probability below chance level (mean is not, one-tailed t-test). The same was found when analysing go and nogo choices separately: in both cases, was not significantly different from 0. 5 (both, two-tailed t-test), and was significantly smaller than (both, paired t-test). Whether this really does represent an insensitivity to punishments depends, however, on the average stay probability, and on how this average stay probability is related to past reinforcements. Subjects were instructed that the outcomes of responses in the PIT block would be counted as in the instrumental block. Figure 2C shows that this led to stable maintenance of the instrumental response tendencies throughout the PIT block. Figure 2D shows that all but one (excluded) subject showed extremely good performance on the Pavlovian query trials interleaved with the Pavlovian training (mean correct). Given the success of instrumental and Pavlovian training, we next analysed the raw effect of Pavlovian stimuli on approach and withdrawal choices. Figure 2E shows a highly significant interaction between block and Pavlovian stimulus valence. Relative to neutral stimuli, positive Pavlovian stimuli enhanced approach and inhibited withdrawal go over nogo. Conversely, negative Pavlovian stimuli enhanced withdrawal and inhibited approach go over nogo. A similar analysis looking at the probability of responding incorrectly (outside the blue box) showed no effect of the Pavlovian stimuli in either approach or withdrawal condition and no interaction (respectively, ANOVA), suggesting that these results were not due to response competition. Note that the withdrawal go probabilities were lower than the approach ones, again reflecting the overall bias against go withdrawal. Average reaction times for go approach and go withdrawal actions did not differ (, 2-tailed t-test). Against our expectations, Pavlovian stimuli of both positive and negative valence shortened reaction times in a parametric manner relative to neutral Pavlovian stimuli (Figure 2F, p = 0. 0310, ANOVA), although this effect was not present in either block separately (p = 0. 5502 and p = 0. 0781 respectively, ANOVA). The size of the PIT effect may have been affected by the extent of instrumental learning (and thus the actual learned action values), by response biases, and by generalization from the instrumental to the PIT stage. In addition, there may have been differences in the instrumental learning of approach and withdrawal actions (Figure 2A). We decomposed and analysed all such factors using a detailed reinforcement learning model. This contained explicit parameters capturing all the instrumental and Pavlovian effects in the task, and was fit to the choice data of all subjects. We used group-level Bayesian model comparison [66] to choose amongst a variety of model formulations (reporting scores relative to the final model), and ensured that inference yielded correct parameter estimates when run on surrogate data generated from the assumed underlying decision process. The final model included parameters associated directly with the instrumental requirements of the task. These comprise one learning rate; two parameters and representing the bias towards go in the approach and withdrawal blocks; and two separate free parameters and, representing the effective strengths of rewards and punishments. At a group level, subjects were biased against active withdrawal, but showed no bias for or against approach (and respectively, two-tailed t-test), the difference being significant (, ANOVA, Figure 4A). Withdrawal biases in the release and throw away experimental subgroups did not differ (, ANOVA), controlling for motor effects. The withdrawal bias accounts for the lower performance on go withdrawal in Figure 2A. One concern is that differences in the biases might have masked differences in learning (i. e. the reward sensitivities) in the approach and withdrawal conditions. We tested this by allowing for separate reward and punishment sensitivities in the two conditions (Model 6) or separate learning rates (Model 7). The use of these extra parameters was structurally rejected by the model selection process (respectively for the purely instrumental trials); and the freedom to choose different parameter values in these conditions was duly not used (Figure 5). The absence of any difference in the learning parameters for approach and withdrawal suggests that the instrumental system treated approach and withdrawal entirely equally. We will see below that this was not true for the Pavlovian system. Although, by design, rewards and punishments were equally informative, subjects chose to rely more on rewards than punishments (Figure 4B). Rewards had a stronger effect than punishments both at a group level and for all individual subjects, the difference being significant (, ANOVA). Indeed, the average punishment sensitivity was not distinguishable from zero (, two-tailed t-test). This remained true when we separately tested subjects who were given deterministic (, two-tailed t-test) and probabilistic (, two-tailed t-test) feedback. Supplementary analyses (Text S1) excluded two further explanations for the punishment insensitivity: first, that it is due to choice perseverance (Figure S1 Text S1); and second that it is due to an emerging maximisation behaviour (Figure S2 in Text S1). Thus, it appears that the pattern seen in Figure 2B is indeed due to a differential sensitivity to rewards and punishments. We next analysed the generalization of instrumental values from the instrumental to the PIT blocks. Generalization could be imperfect in two ways - the starting values in the PIT block could differ from the ending values in the preceding instrumental block, and the values could then decay over time or trials during the PIT block given the lack of information about the outcomes. We constructed models including such effects, and tested whether their excess complexity was outweighed by their fit to the data. As expected from the stable raw probabilities of choosing the correct (i. e. , more rewarded) option (Figure 2C), a model in which the instrumental values decayed exponentially over time during the PIT block (mimicking extinction) did not provide a good account of the data (Model 9, compared to Model 10). Rather, the final model allowed for the addition of random generalization noise to each. These factors were drawn independently from the same normal distribution for all stimulus-action pairs, and the mean and variance of this distribution were both inferred without constraints (see Methods). Figure 4C visualizes the resulting changes; each dot represents the preference for the go action () for all subjects and all stimuli. The abscissa shows this at the end of the instrumental stage, the ordinate after addition of the noise for the PIT stage. Importantly, there was no systematic difference in mean correct action values either in the instrumental or PIT stage (Figure 4D). We were mainly interested in the effect of the Pavlovian values on instrumental performance. We therefore fitted unconstrained parameters to separately capture the influence of each of the five Pavlovian stimuli on instrumental go actions in both the approach and withdrawal condition. All models accounted for performance in the PIT part by adding up instrumental and Pavlovian influences prior to taking a softmax [67], [68]. This amounts to treating instrumental and the Pavlovian controllers as separate experts, each of which ‘voted’ for its preferred action. The model captured in detail, and thereby controlled for, variability in instrumental learning and generalization. The final model predicted the choices of every individual subject better than chance (binomial probability, for every subject, overall predictive probability 0. 7544). The maximum a posteriori (MAP) estimates of this model' s parameters painted a picture very similar to that seen in the raw data. Figure 4E shows the parameters of the model related to the influence of each Pavlovian stimulus. The pattern mirrored that seen in the raw data: there are highly significant, and opposite, effects in the approach and withdrawal blocks, with appetitive stimuli (++ and +) promoting approach but inhibiting withdrawal; and aversive stimuli (-- and -) promoting withdrawal but inhibiting approach. At a single subject level, the effect in the approach block was seen in 45/46 subjects (98%), while it was seen in 30 subjects (65%) in the withdrawal block. Since there was no difference in the learned value of go or nogo actions in either approach or withdrawal blocks, and in either the instrumental learning or the PIT stages (Figure 4D), any PIT effects are unlikely to be due to a preferential association of a Pavlovian stimulus with the learned value of an action. Rather, they reflect the approach or a withdrawal nature of the action. We included two separate groups of subjects who either performed a throwaway withdrawal action, or a release withdrawal action. This was both to test the contribution of an approach/withdrawal component aimed at the Pavlovian stimuli tiling the background, and in recognition of the sophistication of defensive reactions [27]. Figure 4F shows that Pavlovian stimulus value had a significant, linear effect on both withdrawal action types, and that this overall linear effect did not differ between the two action types. At an individual level, linear correlations were positive for 16 (72%) and 14 (58%) subject in the release and throwaway condition, respectively. No psychometric measure of anxiety or depression correlated with any of the parameters in the main model.
One of the central motivations for our investigation was the observation that the neural substrate does not respect the logical independence of reward/punishment and approach/withdrawal. Rather, as we have discussed, these are tied together, via the structure of the striatum and also specific neuromodulators. While the neural basis for the promotion of approach responses by appetitive stimuli is known to involve both amygdala and striatum [62], [63], [75], the neural bases for the effects of aversive Pavlovian stimuli are less clear. There are no data on withdrawal responses per se, i. e. with positive expectations. Nevertheless, animal models, genetic studies and pharmacological manipulations suggest that serotonin plays a crucial role in the inhibition of active behaviours by aversive expectations [25], [47], [48], [50], [73], [76]–[78]. In humans, there is evidence for the serotonergic mediation of the inhibition of active approach by aversive predictions [51], and of approach responses to stimuli that are predictive of negative reinforcement [73]. It should be noted, though, that, acting via the indirect path and D2 receptors, dopamine itself has also been suggested to be important in mediating ‘nogo’ behaviour due to punishments [18], [53], [79]. Aversive Pavlovian stimuli can also potentiate behaviour [1], [64], [80], [81], with both serotonin and dopamine involved. Dopamine may have a dominant influence in this: it is both known to be released, and influential, in some aversive settings [82]–[85] and has a more evident relationship to vigour [33], [34]. This observation has led to a re-interpretation of previous notions [43] of the opponency between dopamine and serotonin, putting an axis spanning invigoration and inhibition together with spanning reward and punishment [52]. Thus, the literature suggests three predictions for genetic correlates of the Pavlovian influences we observe. When considering these, the caveats concerning the interaction of genetic variation with psychopathology (e. g. anxiety or depression), and with development need to be kept in mind. Nevertheless, the conditioned suppression effect of aversive Pavlovian stimuli on approach should be enhanced by D2 receptors, and hence be positively related to D2 striatal receptor density thought to be modulated by C975T (rs6277; [17]). Second, conditioned suppression should be increased in subjects with higher serotonin levels, i. e. as might be the case with the less efficient (s) allelic variation of the serotonin reuptake transporter (5HTTLPR SLC6A4 [86]). Third, given dopamine' s established positive correlation with approach and PIT [87], [88], we expect genetic polymorphisms that boost DA levels, such as the SLC6A3 polymorphism of the dopamine transporter [89], to increase the impact of appetitive Pavlovian stimuli on approach. A similar effect may be expected from DARPP-32, although its closer relationship to synaptic plasticity would also suggest effects on instrumental learning [90]–[92]. Although the learning parameters associated with instrumental approach and withdrawal did not differ, the impact of rewards and punishments on the acquisition of responding was highly asymmetric. In general, subjects neglected punishments, whilst maintaining a fixed sensitivity to reward. This was gratuitous as, in our setting, rewards and punishments were equally informative. It is, however, the case that the optimal strategy can be arrived at by concentrating on either. Subjects were not globally insensitive to punishments, as their choice behaviour in the Pavlovian learning was highly accurate both for rewards and punishments. Furthermore, it should be emphasized that ascribing punishments a value of zero outcome would still effectively behave as a punishment because a zero outcome is well below the average expectation of correct actions (Figure 4D) and as such would reduce the tendency to emit the action that caused it. The asymmetry has been noted before. Others have fitted models with separate learning rates for rewards and punishments and reported significantly slower learning rates for punishments than rewards [93], [94]. In some restricted regimes, learning rates and inverse temperature parameters can trade off, and we explicitly tested both types of models to address this. One potential confound is the emergence of determinism. Subject were instructed to perform choices relative to mushrooms. Real world mushrooms are either edible or poisonous, and this dichotomy may have predisposed subjects towards a deterministic, rather than a matching, strategy. (For instance, subjects may have chosen responses based on a classification of the mushrooms into ‘good’ and ‘bad’ ones, rather than on the particular value of a response for a mushroom.) Indeed, in RL settings it is typically optimal to start with a low, exploratory, sensitivity to outcomes, but to increase this over time to encourage exploitation, culminating in a deterministic strategy [2]. However, subjects did not behave deterministically at any point (Figure 2A) and supplementary analyses showed that the time-varying pattern of reinforcement sensitivities this would predict is not observed in the data (Text S1). A further potential confound is the average stay probability. If this were precisely half-way between the stay probabilities after rewards and punishments in Figure 2B, then rewards and punishments would have the same effect relative to the baseline, and hence arguably be equally informative. However, this argument would neglect the fact that the mean stay probability itself must be a function of the reinforcement history; and that this must be included in making inferences about the reinforcement sensitivity. We have previously made the argument on theoretical grounds that part of the asymmetry observed in appetitive and aversive systems might be due to the inherent difference in how informative rewards and punishments are processed, enshrined again in the architecture of the striatum and neuromodulation [50]. Rewards tell us what to do; punishments tell us what not to do. The former is more informative in naturalistic settings where many options are available but only few are good. The fact that subjects gratuitously rely on rewards rather than on punishments in the present setting may reflect an implicit appreciation of this fact, although our findings are certainly in no way conclusive evidence. Interestingly, it is known that stronger optimality results can be shown for a stochastic learning automata rule called linear reward-inaction, which does not change propensities in the light of punishments but only rewards ([95], [96]; also known as a benevolent automaton [97]), than for a rule that changes propensities for both. The computational model served several central roles. First, it encapsulated the manifold aspects of behaviour and learning jointly, thereby controlling for them: the bias against withdrawals is not a due to a difference in learning; and variations in learning or generalization do not account for the PIT effects we saw. Secondly, its close fit to the behaviour argues that the PIT effects can be accounted for by a simple superposition of an instrumental and a Pavlovian controller: the action propensities due to both controllers were simply multiplied (as additive factors in an exponential), rather than being allowed to interact in more complex ways. The simplicity of this interaction eschews questions about peripheral versus central response competition, whether appetitive and aversive systems compete centrally [7], and whether Pavlovian learning is involved in instrumental learning [1]. It takes the view of multiple, separate controllers contributing in parallel [98], and weighting the ultimate choice by the reward expected from that choice. One alternative would be to weigh contributions by different controllers according to their certainty [99], although it is unclear how to compute the Pavlovian controller' s certainty. There are various pressing directions for future studies. First, despite the role the architecture of decision-making has played in the argument, our work does not directly address the neural mechanisms concerned. These could be examined using imaging and pharmacological manipulations. Second, our task was not designed to distinguish between outcome-specific and general mechanisms [63], [75] as we relied on one, monetary, outcome throughout. Studying different outcomes is important, given evidence for partly parallel pathways through different nuclei of the amygdala and different targets in the nucleus accumbens [100], [101]. Third, we are missing one crucial further orthogonalization to do with the overall framing of the instrumental task. It is important to consider the case in which subjects can at best avoid losing money by doing the correct action [51]. We would expect punishment to maintain its instrumental force in this case; but there could also be a systematic difference in the nature of the Pavlovian influences. Pavlovian responses are believed to be hard-wired to reflect evolutionarily appropriate attitudes to predictions, being highly adaptive and sensitive to environmental structures [102]. Here, we showed that Pavlovian influences on instrumental behaviour depend on the intrinsic affective label of an action, independent of its learned reward expectation. It has long been known that prepared or compatible [27], [69] behaviours are easier targets for instrumental conditioning. These intrinsic biases, or priors, may serve a crucial function both by reducing the need for collecting data (i. e. sample complexity) about the effects of actions, and by reducing the need for executing complex processing necessary to work out optimal actions (i. e. computational complexity). Both of these can be expensive or dangerous, particularly in an aversive context. Our findings sharpen the understanding of the relative contribution of Pavlovian and instrumental contingencies in general tasks. We showed clearly that the interaction of Pavlovian and instrumental behaviours is organized along the lines of appetitive and aversive motivational systems, and that a critical contributor to this is the affective nature of actions.
54 healthy subjects of central European origin were recruited from the Berlin area. Subjects were screened for a personal history of neurological, endocrine, cardiac and psychiatric disorders (SCID-I screening questionnaire), and for use of drugs and psychotropic medication in the past 6 months. Subjects received performance-dependent compensation (5–32 Euro) for participation. Three subjects did not meet inclusion criteria and one subject did not complete the task; the data for three further subjects were lost due to a programming error. One further subject was excluded from the analysis because the instrumental task was not satisfactorily performed. The 46 remaining subjects were years old. 59% were female (). The study was approved by the local Ethics Committee and was in accord with the Declaration of Helsinki 2008. Subjects were given detailed information and gave written consent. They were seated comfortably at a table in front of a laptop with headphones and used a mouse with their dominant hand to indicate their choices. The amount earned was indicated by the computer, and the sum paid in cash at the end of the session. The computer task was followed by completion of self-rating scales. The task was written using Matlab and Psychtoolbox (http: //psychtoolbox. org). It consisted of one approach and one withdrawal block separated by a 2 minute break. Each block was in turn divided into a instrumental training, a Pavlovian training and a PIT part. Table 1 illustrates this. We modified a standard reinforcement learning model to capture the behavioural choices in the experiment. We first describe the main model, and then the alternative control models. Considering first the instrumental part, let be the instrumental stimulus (out of up to 12; i. e. the subscript now designates time rather than identity as in Table 1) presented at trial, and the action (choice) on that trial. An action can be one of four types: go withdrawal and nogo withdrawal in the withdrawal block, and go approach and nogo approach in the approach block. Let also be the reinforcement obtained, either for a punishment, or for a reward. We write the probability of action in the presence of stimulus as a standard probabilistic function of i) the reinforcement expectations associated with that pair on that trial, and ii) a time-invariant, fixed, response bias: (1) (2) where is the instrumental weight of action, and where the variable can take on value for withdrawal go actions, or for the approach go actions. It is always zero for the nogo action. There was no delayed outcome in the instrumental task, and the expectations were thus constructed by a Rescorla-Wagner-like rule with a fixed learning rate. The immediate, intrinsic, value of the reinforcements delivered in the experiment may have different meaning for different subjects. To measure this effect, we added two further parameters: the reward sensitivity and the punishment sensitivity, yielding an update equation for the expectations: This is model 5 in Table 2, which has the lowest score (see below). Alternative models tested on the instrumental data only are as follows: Model 1 assumes that, and that. Model 2 allows only for separate reward and punishment sensitivities and model 4 for separate biases. Model 3 again assumes, and that, but allows for two separate learning rates, i. e. in Equation 3 is replaced by on trials where, and by on trials where. Model 6 and 7 are expansions of the final model, allowing for separate reward and punishment sensitivities (model 6) and for separate learning rates (model 7) in the approach and withdrawal conditions. Our main measure of interest is the effect of Pavlovian stimuli on the approach and withdrawal actions. Let additionally be the Pavlovian stimulus on trial. We can then write an equation similar to equation 2 for the trials where both instrumental and Pavlovian stimuli were present, but including a term that quantifies the effect of the particular Pavlovian stimulus on the action. This means that the action weights due to the instrumental and Pavlovian controllers are added inside the exponent of equation 2, and that thus the probabilities each controller attaches to a particular action are multiplied and renormalized. The two controllers are therefore treated as two distinct entities, each separately voting for a particular action to be emitted. The influence of each system on action choice is relative to the strength with which the other enhances one particular action. We write the PIT weight of action as: (3) Here we force at all times. The go values can take on 10 separate, inferred, values, meaning that there is one separate parameter for each of the five Pavlovian stimuli in each of the two blocks. Each of these parameters captures how much boosts the go over the nogo action (if) or the inverse (if). Note that because these are separately inferred, independent, parameters, this formulation does not impose any assumptions about the effect of the value of the stimulus, or about the relative effect of different stimuli with different values. Hence, this controls for variation in learning during the Pavlovian training block (though the query trials indicate that learning was very robust). Equation 3 (Model 8 in Table 2) assumes that the stimulus-action values at the end of the instrumental block are perfectly and exactly generalized to the PIT block. We first tested an alternative model (Model 9 in Table 2) that included an exponential extinction factor, letting the values decay on each PIT trial by with. Next, we tested the model described in the main text (Model 10 in Table 2), which allowed for a fixed, Gaussian random offset between the effective values in the instrumental and PIT stages, i. e. we wrote: The noise factor took on value for the nogo action (akin to the bias and variables). It took on a separate value—which was inferred as a separate parameter—for each subject and each stimulus. However, all stimuli shared the same prior distribution for this noise variable. That is, in the E step of our EM procedure, we fitted one Gaussian mean and variance to all the' s that had been inferred for all stimuli for all subjects. In this sense, the generalization factors were drawn from one Gaussian prior whose mean and variance were fitted just like the mean and variance of the other parameters. For each subject, each model specifies a vector of parameters. Assuming Gaussian prior distributions, we find the maximum a posteriori estimate of the parameters for each subject: where are all actions by the subject. We assume that actions are independent (given the stimuli, which we omit for notational clarity), and thus factorize over trials. The prior distribution on the parameters mainly serves to regularise the inference and prevent parameters that are not well-constrained from taking on extreme values. We set the parameters of the prior distribution to the maximum likelihood given all the data by all the subjects: where. This maximisation is straightforwardly achieved by Expectation-Maximisation [109]. We use a Laplacian approximation for the E-step at the iteration: where denotes a normal distribution over with mean and is the second moment around, which approximates the variance, and thus the inverse of the certainty with which the parameter can be estimated. Finally, the hyperparameters are estimated by setting the mean and the (factorized) variance of the prior distribution to: ansformed before inference to enforce constraints. Unconstrained parameters are inferred in their native space. These model fitting procedures were verified on surrogate data generated from a known decision process. We fitted a large number of different models to the data, and some of these models differ in their flexibility. For instance, Model 8, which assumes that the instrumental values are generalized exactly to the PIT stage is much less flexible than models 9–10, which allow for an offset. It is important to choose that model which is flexible enough to explain the data, but not so flexible that it would also fit very different data equally well [109]. Ideally, this is achieved by computing the posterior log likelihood of each model given all the data. As we have no prior on the models themselves (testing only models we believe are equally likely a priori), we instead examine the model log likelihood directly. This quantity can be approximated in two steps. First, the integral over [110]: Importantly, however, is not the sum of individual likelihoods, but in turn an integral over the parameters of each individual subject: The second line shows that we approximated the integrals by (importance) sampling times from the empirical prior distribution [109]. These samples were then also used to derive the error bars as the second moments around the maximum: where is a vector of zeros of the same dimension as with only entry set to one. The shifted likelihoods can be easily computed by re-weighting the samples drawn before: Note that while this model comparison procedure does give a good comparative measure of model fit, we still need an absolute measure to ensure that the best model does indeed provide a model fit that is adequate (even the best might be bad). Given each subject' s MAP parameter estimate, we compute the total “predictive probability”: (4) where we suppressed the dependence on stimuli on the LHS for clarity. We note that depends on the parameters, which have been fitted to the data. We term it a predictive probability in the sense that it predicts a subject' s choice at time given that subject' s past behaviour. We emphasize however, that this does depend on the MAP parameters fitted to that subjects' entire choice dataset. Finally, we test whether the expected number of choices predicted correctly exceeds that expected by chance (using a binomial test). The overall predictive probability is given by the geometric mean over all choices and subjects: .
Beautiful background music in a shop may well tempt us to buy something we neither need nor want. Valenced stimuli have broad and profound influences on ongoing choice behaviour. After replicating known findings whereby approach is enhanced by appetitive Pavlovian stimuli and inhibited by aversive ones, we extend this to withdrawal behaviours, but critically controlling for the valence of the withdrawal behaviours themselves. We find that even when withdrawal is appetitively motivated, it is still inhibited by appetitive Pavlovian stimuli and enhanced by aversive ones. This shows, for the first time, that the effect of background Pavlovian stimuli depends critically on the intrinsic valence of behaviours, and differs between approach and withdrawal.
neurobiology of disease and regeneration
behavioral neuroscience
cognitive neuroscience
computational neuroscience
biology
neuroscience
animal cognition
Disentangling the Roles of Approach, Activation and Valence in Instrumental and Pavlovian Responding