Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference.
Neural computations mediating instrumental conditioning are suggested to depend on two distinct mechanisms: a model-based reinforcement learning system, in which the value of actions are computed on the basis of a rich knowledge of the states of the world and the nature of the transitions between states, and a “model-free” reinforcement learning system in which action-values are updated incrementally via a reward prediction error without using a rich representation of the structure of the decision problem [1]–[6]. Accumulating evidence supports the existence of model-based representations during instrumental conditioning in a number of brain regions, including the ventromedial prefrontal cortex, striatum and parietal cortex [7]–[9]. However, instrumental conditioning is not the only associative learning mechanism in which model-based computations might play a role. Pavlovian conditioning can also be framed as a model-based learning process, in which the animal begins with a model of the possible structure of the world: the stimuli within it, and sets of possible contingencies that could exist between conditioned stimuli and unconditioned stimuli, as well as assumptions about how these contingencies might change over time. In essence, learning within such a system corresponds to determining the statistical evidence for which structure out of the set of possible causal structures best describes the environment, as well as determining whether or when the relevant causal processes have changed as a function of time. Model-based approaches to classical conditioning to date have used Bayesian methods to yield inference over structure space [10]. Very little is known about the extent to which such model-based algorithms are implemented in the brain during Pavlovian conditioning. The aim of the present study was to address this question using computational fMRI. Human participants were scanned while undergoing a Pavlovian conditioning procedure with a sufficiently complex structure to enable the predictions of model-based and model-free algorithms to be compared and contrasted (see Figure 1). We then constructed a Bayesian algorithm incorporating a model of the structure of the learning problem and compared the predictions of this algorithm against two widely adopted prediction-error driven “model-free” algorithms for Pavlovian conditioning: the Rescorla-Wagner (RW) learning rule [11] and the Pearce-Hall (PH) learning rule [12] as well as a recently developed model which combines the two: the Hybrid model [13]. In order to test for model-based signals in the brain we focused on the amygdala, a structure heavily implicated in Pavlovian conditioning in both animal and human studies [14]–[17]. To obtain signals from this region with sufficient fidelity, we used a high-resolution fMRI protocol in which we acquired images with more than four times the resolution of a standard 3 mm isotropic scan, alongside an amygdala specific normalization procedure [18]. We hypothesized that the model-based algorithm would account better for both behavioral and fMRI data acquired during both the appetitive and aversive conditioning phases than would the models of Pavlovian conditioning which do not contain such structured knowledge.
We report results from our analyses based on our model-based learning algorithm (the HMM model) within the amygdala using a height threshold of p<0. 005, with an extent threshold significant at p<0. 05 corrected for multiple comparisons. We first report signals correlating with signals generated by our model-based HMM, and then we compare the performance of our model-based algorithm against its model-free counterparts in terms of the capacity of these models to account for BOLD activity in the amygdala.
In this study, we used a Pavlovian conditioning task with a rudimentary higher-order structure in both appetitive and aversive domains to investigate whether neural activity in the human amygdala reflects learning that requires access to model-based representations. By comparing neural activity correlating with expected value signals generated by model-based versus model-free learning algorithms using a Bayesian model selection (BMS) procedure, we have been able to show that in at least some parts of the human amygdala activity during Pavlovian conditioning is better accounted for by a model-based algorithm rather than by prediction error driven model-free algorithms. One of the critical distinctions between the prediction error driven model-free and model-based learning algorithms in the present study is that while the expected value of a stimulus previously paired with the unpleasant outcome is still low following reversal of contingencies because that was the value it had before reversal in a model-free system, the expected value of this stimulus will become high in a model-based system because it incorporates the knowledge that after a reversal, stimulus values switch (i. e. there is full resolution of uncertainty when a reversal occurs). We have captured model-based representations in formal terms using an elementary Bayesian hidden Markov computational model that incorporates the task structure (by encoding the inverse relationship between the cues and featuring a known probability that the contingencies will reverse). Our behavioral analysis demonstrated that participants showed evidence of conditioned responses to the conditioned stimuli and thus successfully learnt the associations between the different cues and outcomes. In a trial-by-trial analysis in which we correlated reaction times against the model predictions, we found that the HMM model predicted changes in reaction times over time as a function of learning better than the prediction-error driven model-free alternatives, and that the prediction error model-free algorithms did not predict variation in reaction times significantly better than chance. In the neuroimaging data, we found trial-by-trial positive correlations of model-based expected values in an area consistent with the basolateral complex of the amygdala according to the Mai atlas in the appetitive session, and in areas in the likely vicinity of the centromedial complex in the aversive session [25]. It is interesting to note that activity in these same areas (i. e. basolateral versus centromedial complex) has been found to correlate with expected value signals generated by a simple RW model in a recent reward versus avoidance instrumental learning task (in an appetitive versus aversive context respectively) [18]. Using a BMS procedure, we found that amygdala activity correlating with expected value was best explained by the model-based than by the prediction error driven model-free learning algorithms. Whereas the model-free system has received considerable attention in the past [26], the more sophisticated and flexible model-based system, has been more sparsely studied particularly in relation to its role in Pavlovian learning. Thus, our results point to the need for integrating model-based representations and their rich adaptability into our understanding of Pavlovian conditioning in general, and of the role of the amygdala in implementing this learning process in particular. Another important feature of the model-based algorithm featured in this study, is that as well as keeping track of expected value, this model also keeps track of the degree of precision in the prediction of expected value over the course of learning. This precision starts off low at the beginning of a learning session with a new stimulus because the expected value computation is very uncertain at this juncture, but once outcomes are experienced in response to specific cues, the precision in the estimate quickly increases. However, this precision lessens again as the trial progresses because a reversal in the contingencies is increasingly expected to occur (hence the expected value becomes more and more uncertain). Signals correlating with precision were found to be located in the vicinity of the centromedial complex in both the appetitive and aversive sessions. Precision signals might play an important role in the directing of attentional resources toward stimuli in the environment. The presence of a precision signal in the centromedial amygdala in the present paradigm could be a key computational signal underpinning the putative role of this structure in directing attention and orienting toward affectively significant stimuli. The presence of a precision-related signal in the amygdala during Pavlovian conditioning may relate to other findings in which the amygdala has been suggested to play a role in “associability” as implemented in a model-free algorithm such as the Pearce-Hall learning rule [13], [27]. Associability as defined in such a model is essentially a model-free computation of uncertainty, the inverse of precision: associability is maximal when the absolute value difference between expected and actual rewards is greatest. However, in our case, an associability signal is clearly distinct from the signal we observe in the amygdala in the centromedial complex (even leaving aside the fact the signal we found is negatively as opposed to positively correlated with uncertainty). First of all, because the signal in our HMM is model-based, it changes to reflect anticipated changes in task structure (such as a reversal), whereas Pearce-Hall associability does not change to reflect anticipated changes in task structure, both rather changes only reflexively once contingencies have reversed. Further evidence that the amygdala is involved in model-based computations came from an additional analysis in which we compared the signals generated by our model-based HMM against signals generated by a reduced version of our HMM in which knowledge of when contingencies are expected to reverse was not incorporated. Although this reduced model still generated very similar expected value signals as the model-based HMM and thus made similar predictions about behavior, the precision signals generated by these two algorithms are quite distinct and can therefore be compared against neural activity in the amygdala. In a direct comparison, activity in the amygdala was best accounted for by the precision signal generated by the full HMM. It is interesting to note that evidence for model-based processing in the amygdala was more robust in the aversive case given the traditional view of the amygdala as being associated especially with aversive processing. However, it is unlikely that this pattern of results reflects a qualitative difference in the way that appetitive and aversive learning is mediated by the amygdala, particularly in the light of considerable evidence implicating this structure in both reward-related as well as aversive-learning [28], [29]. Finally, we checked the correlation between the precision signal we found here and an associability signal generated by the Pearce-Hall learning rule, and we found the correlation between these signals to be essentially negligible (with r ranging from −0. 06 to −0. 14), as opposed to being strongly negatively or positively correlated as would be anticipated were these signals to tap similar underlying processes. The fact that in the present study we found model-based signals in the amygdala does indicate that this structure is capable of performing model-based inference even during Pavlovian conditioning. However, it is important to note that the findings of the present study do not rule out a role for this structure in prediction error driven model-free computations during Pavlovian conditioning. Indeed, while the prediction error driven model-free learning rules we used did not work very well in accounting for behavior on the task (as indexed by changes in reaction times), we did find some evidence (albeit weakly) of model-free value signals in the amygdala as generated by either a Rescorla-Wagner, a Pearce-Hall or a Hybrid learning rule. Indeed, while using our HMM model we did not find evidence for aversive-going expected value signals in the aversive session (i. e. by showing an increase in activity the more the unpleasant tasting liquid was expected), we did find such a signal correlating with expected value as computed by a Pearce-Hall learning rule. As a consequence, we cannot rule out a contribution for the amygdala in model-free computations. It is important to note however, that in many tasks in which neuronal activity was found in the amygdala to correlate with the predictions of model-free learning algorithms [18], [30]–[32], such tasks were either not set up to discriminate the predictions of model-free versus model-based learning rules, or else the relevant model comparisons were not performed. Thus, it is entirely feasible that many of the computations found in the amygdala in previous studies correspond more closely to model-based as opposed to model-free learning signals. More generally, if indeed, both model-based and model-free signals are present in the amygdala during Pavlovian conditioning, then an important question for future research will be to address how and when these signals interact with each other. To conclude, we have found in the present study evidence for the existence of model-based learning signals in the human amygdala during performance of a Pavlovian conditioning task with a simple task structure. These findings provide an important new perspective into the functions of the amygdala by suggesting that this structure may participate in model-based computations in which abstract knowledge of the structure of the world is taken into account when computing signals leading to the elicitation of Pavlovian conditioned responses. The findings also resonate with an emerging theme in the neurobiology of reinforcement learning whereby value signals are suggested to be computed via two mechanisms: a model-based and a model-free approach [1], [3]. Whereas up to now, theoretical and experimental work on this distinction has tended to be focused on the domain of instrumental conditioning [4], [7], [8], the present study illustrates how similar principles may well apply even at the level of Pavlovian conditioning. Thus the distinction between model-based and model-free learning systems may apply at a much more general level across multiple types of associative learning in the brain. Furthermore, the present results provide evidence that model-based computations may be present not only in prefrontal cortex and striatum, but also in other brain structures such as the amygdala.
Nineteen right-handed subjects (8 females) with a mean age of 22. 21±3. 47 participated in the study. All subjects were free of neurological or psychiatric disorders and had normal or correct-to-normal vision. Written informed consent was obtained from all subjects, and the study was approved by the Trinity College School of Psychology research ethics committee. Subjects participated in a Pavlovian task where they had to learn associations between different cues (fractal images) and a pleasant (blackcurrant juice [Ribena, Glaxo-Smithkline, UK]), affectively neutral (artificial saliva made of 25 mM KCl and 2. 5 mM NaHCO3) or unpleasant (salty tea made of 2 black tea bags and 29 g of salt per liter) flavor liquid. The task consisted of two sessions lasting approximately 22 minutes each. Each session was composed of 120 trials, leading to a total of 240 trials. In one of the sessions, subjects underwent an appetitive Pavlovian conditioning procedure whereby they were presented with cues leading to the subsequent delivery of either the pleasant flavor, or the affectively neutral one, while in the other aversive conditioning session, subjects underwent an aversive conditioning procedure whereby they were presented with cues leading to the subsequent delivery of either the unpleasant flavor stimulus, or else the affectively neutral stimulus. The rationale for including the appetitive and aversive conditioning procedures in separate sessions as opposed to including both conditions intermixed within the same sessions was to avoid contrast effects observed in prior behavioral piloting whereby cues signaling the aversive outcome tended to overwhelm cues signaling the pleasant one such that both the pleasant and the neutral cue stimuli were viewed as relief stimuli (contrasted against the aversive outcome) [33]. Performing the appetitive and aversive conditioning procedures in separate sessions ensured robust behavioral conditioning in both the appetitive and aversive cases and largely avoided contrast effects between the appetitive and aversive conditions. For both sessions, on each trial, a cue was displayed randomly on either the left or right side of a fixation cross for 4 seconds. Following a well-established Pavlovian conditioning protocol [34]–[36], subjects were also instructed to indicate on which side of the screen the cue was presented by means of pressing the laterally corresponding button on a response box, yet they were also instructed that the subsequent outcomes were not contingent on their responses. This serves two purposes: it allows one to monitor the extent to which participants are paying attention to the cues on each trial, as well as offering a response time measure which can serve as an index of conditioning. The offset of the cue (after 4 seconds) was followed by delivery of one of the liquid flavor stimuli with a probability of 0. 6, or else no liquid stimulus was delivered. The next trial was triggered following a variable 2–11 secs inter-trial interval. At the beginning of each session, subjects were presented with two novel fractal cues (not seen before in the course of the experiment): which we will denote as cue 1 and cue 2. In the appetitive session, cue 1 predicted the subsequent presentation of the pleasant liquid 60% of the time (or no liquid delivery 40% of the time), while in the aversive session cue 1 predicted the delivery of the aversive liquid 60% of the time (or no liquid delivery 40% of the time). Cue 1 and cue 2 trials were presented in a randomly intermixed order. After 16 trials (8 trials of each type), a reversal of the cue-outcome associations was set to occur with a probability of 0. 25 on each subsequent trial. The probabilistic triggering of the reversal after the 16th trial ensured that the onset of the reversal was not fully predictable by subjects. Once a reversal was triggered, cue 1 no longer predicted the appetitive or aversive outcome but instead was associated with delivery of the neutral outcome, while cue 2 now predicted the appetitive or aversive outcome. After another 16 trials (8 trials of each type) following the onset of the reversal, another event was triggered to occur with probability 0. 25 on one of the subsequent trials: this time instead of a reversal, a completely novel pair of stimuli was introduced. One of these, cue 3, was now paired with the appetitive or aversive outcome, while cue 4 was now paired with the neutral outcome. These new cues were presented for a further 16 trials, and followed again after a probabilistic trigger of p = 0. 25 on each subsequent trial with a reversal of the associations. After the reversal, a new set of cues were introduced according to the same probabilistic rule and this was followed again by a reversal. Thus in total, 3 unique pairs of stimuli were used in each session and each of these pairs underwent a single reversal (Figure 1a, b). A completely different set of cues were used for each session, so that subjects experienced a total of 6 pairs of fractal stimuli throughout the whole experiment. Within each session, the presentation order of the affective and neutral cue presentations was randomized throughout, with the one constraint that the cue predicting the neutral tasting liquid delivery had to be delivered twice every four trials. This ensured that the appetitive and neutral cues, and aversive and neutral cues were approximately evenly distributed in their presentation throughout the appetitive and aversive sessions respectively. All fractal images were matched for luminance. The order of the sessions was counterbalanced across subjects so that half of the subjects started the experiment with the appetitive session and half of the subjects with the aversive session. Before the conditioning session, subjects received the following task instructions: “In each trial, an image will appear on the screen and may be followed by some liquid delivery. There are six different images per session. Each image will lead to either a pleasant, neutral or unpleasant tasting liquid. You will have to learn these associations. However, during the experiment, this may change (or reverse), making image 1 associated with the liquid of image 2 and image 2 associated with the liquid of image 1. This reversal may actually happen more than once during the experiment and you have to fully pay attention and realize that it has happened. These cues may change during the experiment, so that you will have to learn these associations again with these new cues (which may also reverse). At the beginning of each trial, the image will either appear on the left or right side of the screen. You will have to press the left button of the response pad if the image appears on the left side, or the right button if it appears on the right side. It is important that you press the button because we need to record your response times, although the trial will carry on if you don' t press any button. At the beginning and end of each session, we will ask you to rate different images and liquids. You will also have to rate these images in the middle of each session. ” The pleasant, neutral and unpleasant tasting liquids were delivered by means of three separate electronic syringe pumps positioned in the scanner control room. These pumps pushed 1 mL of liquid to the subject' s mouth via ∼10 m long polyethylene plastic tubes, the other end of which were held between the subject' s lips like a straw, while they lay supine in the scanner. Functional imaging was performed on a 3T Philips scanner equipped with an 8-channel SENSE (sensitivity encoding) head coil. Since the focus of our study was on the amygdala, we only acquired partial T2*-weighted images centered to include the amygdala while subjects were performing the task. These images also encompassed the ventral part of the prefrontal cortex, the ventral striatum, the insula, the hippocampus, the ventral part of the occipital lobe and the upper part of the cerebellum (amongst other regions). Nineteen contiguous sequential ascending slices of echo-planar T2*-weighted images were acquired in each volume, with a slice thickness of 2. 2 mm and a 0. 3 mm gap between slices (in-plane resolution: 1. 58×1. 63 mm; repetition time (TR): 2000 ms; echo time (TE): 30 ms; field of view: 196×196×47. 2 mm; matrix: 128×128). A whole-brain high-resolution T1-weighted structural scan (voxel size: 0. 9×0. 9×0. 9 mm) and three whole-brain T2*-weighted images were also acquired for each subject. To address the problem of spatial EPI distortions which are particularly prominent in the medial temporal lobe (MTL) and especially in the amygdala, we also acquired gradient field maps. To provide a measure of swallowing motion, a motion-sensitive inductive coil was attached to the subjects' throat using a Velcro strap. The time course derived from this measure was used as a regressor of no interest in the fMRI data analysis. Finally, to account for the effects of physiological noise in the fMRI data, subjects' cardiac and respiratory signals were recorded with a pulse oximeter and a pressure sensor placed on the umbilical region and further removed from time-series images. We discarded the first 3 volumes before data processing and statistical analysis to compensate for the T1 saturation effects. All EPI volumes (‘partial’ scans acquired while subjects were performing the task and the three whole-brain functional scans acquired prior to the experiment) were corrected for differences in slice acquisition and spatially realigned. The mean whole-brain EPI was co-registered with the T1-weighted structural image, and subsequently, all the partial volumes were co-registered with the registered mean whole-brain EPI image. Partial volumes were then unwarped using the gradient field maps. After the structural scan was normalized to a standard T1 template, the same transformation was applied to all the partial volumes with a resampled voxel size of 0. 9×0. 9×0. 9 mm. In order to maximize the spatial resolution of our data, no spatial smoothing kernel was applied to the data. These preprocessing steps were performed using the statistical parametric mapping software SPM5 (Wellcome Department of Imaging Neuroscience, London, UK). To test whether amygdala activity was better explained by model-based or model-free learning algorithms, we correlated brain activity in this region with expected value signals estimated by a number of different computational models. In model-free learning algorithms, the agent is surprised when a reversal occurs and starts learning again after it happens, whereas in model-based learning algorithms, the agent expects the reversal and considers it as resolution of uncertainty and does not need to relearn. The two modes of learning are diametrically opposed in the current task, therefore allowing us to test whether amygdala is tracking model-based or model-free computations. To perform a formal model comparison on the behavioral conditioning data, we used the trial-by-trial reaction time data (measuring the length of time taken on each trial for participants to press a button to indicate which side of the screen the Pavlovian cue stimulus had been presented). Many previous studies have shown that changes in RTs to a Pavlovian cue are correlated with changes in associative encoding between cues and behaviorally significant outcomes [13], [34], [36], [42]. For each session separately, we log transformed and adjusted the RT data to account for a linear trend in RTs over time independently of trial type, as well as to remove the effects of changes in reaction time related to switching responses from one side of the screen to the other. This was done by regressing the log transformed RTs against a matrix containing a column of ones, a column accounting for the linear trend over time and a column indicating whether participants switched their response from left to right or vice versa between the current and previous trial using the function regress in Matlab. Using the same function, we then regressed these adjusted response times against the expected values generated by our model-based HMM our model-free RW, PH and Hybrid algorithms and our baseline model (for the baseline model, a small amount of noise was added to each expected value in order to compute the regression; without any noise the regression would not be calculable). This second regression analysis was run for each of these models, and cycled through all the possible learning rate parameters for the RW model and CS intensity parameters for the Pearce-Hall and hybrid models between 0 and 1, with increments of 0. 001. This method returned Sum Squared Error (SSE) values for each of these parameter values thereby allowing us to obtain the best fitting value for the free parameter for the appetitive and aversive sessions (i. e. the free parameter associated with the lowest SSE value). In order to compare the model goodness between these four different algorithms, we converted the best SSE value of each session (appetitive and aversive) and each model into a Bayesian information criterion (BIC) value. The BIC adds a penalty proportional to the number of additional free parameters to the SSE value of each model, depending also on the number of degrees of freedom which in this case, is the total number of trials per session across all subjects [43]. Using this procedure, we found that in both the appetitive and aversive sessions, the model-based HMM outperformed the prediction-error driven model-free algorithms (Table 1). In the model validation analyses, where we compared the prediction-error driven models against a random baseline model, only the model-based HMM fit our behavioral data significantly better than the baseline model (Table 2). Hence, unlike RW, PH and the Hybrid model, the model-based HMM predicted RTs better than chance performance. Note that we did not regress the expected values generated by our reduced HMM since they were highly correlated with that of our model-based HMM. The event-related fMRI data were analyzed by constructing sets of δ (stick) functions at the time of cue presentation and at the time of outcome for the appetitive and aversive sessions. For our main GLM (illustrated in Figures 4 and 5), additional regressors were constructed by using the expected values and the precision values generated by the model-based HMM as modulating parameters at the time of cue presentation. In order to compare model-based versus model-free learning algorithms in the amygdala, we ran three additional GLMs. For RW, the regressors were similar to our model-based HMM except that we did not have a regressor for precision which is not estimated by RW, and we added a modulating parameter for prediction error at the time of outcome. The regressors used in the GLM computed using PH model were the same as the ones used in our model-based HMM, except that the precision modulating parameter was replaced with an associability modulating parameter at the time of cue presentation (note that similar regressors were used for the Hybrid model). Finally, we ran an analysis using our reduced model HMM using the same regressors as for our model-based HMM. All of these regressors were convolved with a canonical hemodynamic response function (HRF). The six scan-to-scan motion parameters derived from the affine part of the realignment procedure were included as regressors of no interest to account for residual motion effects. To account for motion of the subjects' throat during swallowing, we added a regressor of no interest for swallowing motion. Finally, we also included thirteen additional regressors to account for physiological fluctuations (4 related to heart rate, 9 related to respiration) which were estimated using the RETROICOR algorithm [44]. Six of the 38 (2 sessions×19 subjects) log files could not be used to estimate these regressors due to a technical problem during data collection, and the missing physiological regressors were simply omitted for those sessions. All of these regressors were entered into a general linear model and fitted to each subject individually using SPM5. The resulting parameter estimates for regressors of interest were then entered into second-level one sample t-tests to generate the random-effects level statistics used to obtain the results shown in figures 4 and 5. All reported fMRI statistics and p values arise from group random-effects analyses. We present our statistical maps at a threshold of p<0. 005, corrected for multiple comparisons at p<0. 05. To correct for multiple comparisons, we first used the 3dFWHMx function in AFNI to estimate the intrinsic smoothness of our data, within the area defined by a mask corresponding to our amygdala template. We then used the AlphaSim function in AFNI to estimate via Monte Carlo simulation an extent threshold for statistical significance that was corrected for multiple comparisons at p<0. 05 for a height threshold of p<0. 005 within the amygdala ROI. In order to test whether amygdala activity was better accounted for by the model-based than model-free learning algorithms, we used a Bayesian model selection procedure (BMS) [45]. For both the appetitive and aversive sessions, we included in this model comparison individual betas averaged across voxels within a 4 mm sphere centered on the peak voxels of the amygdalar activities correlating with either expected value signals for the HMM or the model-free algorithms using the leave-one out method, thereby avoiding a non-independence bias in the voxel selection [46]. Using the spm_BMS function in SPM8, we compared expected value signals across all model-based (HMM) and model-free models separately for the appetitive and aversive sessions. We used a similar approach to compare neural activity pertaining to precision signals estimated by our model-based and reduced model HMMs. The difference between these two HMMs is that the model-based HMM does not allow for a reversal without moving from a “non-reversal state” to a “possible reversal state”. As a consequence, the precision values generated by these models are clearly distinguishable and thus easily comparable using a BMS (whereas the estimated expected rewards are strongly correlated). Again, we included in this model comparison voxels within a 4 mm sphere centered on the peak voxels of the amygdalar activities correlating with precision signals for either the model-based HMM or the reduced model HMM using the leave-one out method. Here, we compared activity correlating with precision signals between the model-based and reduced HMM separately for the appetitive and aversive sessions (see Results section for the exceedance probabilities). Functional regions of interest (ROIs) were defined using the MarsBaR toolbox (http: //marsbar. sourceforge. net/). Beta estimates were extracted for each subject from the functional clusters of interest as they appeared on the statistical maps of a given contrast using the leave-one out method to avoid a non-independence bias. They were then averaged across subjects to plot expected reward (Figure 4b) and precision (Figure 5b) according to 3 categories (category 1 corresponding to the lowest values and category 3 corresponding to the highest values).
A hot topic in the neurobiology of learning is the idea that there may be two distinct mechanisms for learning in the brain: a model-based learning system in which predictions are made with respect to a rich internal model of the learning environment, versus a “model-free” mechanism in which trial-and-error learning occurs without any rich internal representation of the world. While the focus in the literature to date has been on the role of these mechanisms in instrumental conditioning, almost nothing is known about whether more fundamental kinds of learning such as Pavlovian conditioning also involve model-based processes. Furthermore, nothing is known about the extent to which the amygdala, which is known to be a core structure for Pavlovian learning, contains neural signals consistent with a model-based mechanism. To address this question, we used a novel Pavlovian conditioning task and scanned human volunteers with a special high-resolution fMRI sequence that enabled us to obtain signals within the amygdala with over four times the resolution of conventional imaging protocols. Using this approach in combination with sophisticated computational analyses, we find evidence to suggest that the human amygdala is involved in model-based computations during Pavlovian conditioning.
cognitive neuroscience
decision making
computational neuroscience
biology
neuroscience
Evidence for Model-based Computations in the Human Amygdala during Pavlovian Conditioning