article: Successful interaction with the environment requires flexible updating of our beliefs about the world. By estimating the likelihood of future events, it is possible to prepare appropriate actions in advance and execute fast, accurate motor responses. According to theoretical proposals, agents track the variability arising from changing environments by computing various forms of uncertainty. Several neuromodulators have been linked to uncertainty signalling, but comprehensive empirical characterisation of their relative contributions to perceptual belief updating, and to the selection of motor responses, is lacking. Here we assess the roles of noradrenaline, acetylcholine, and dopamine within a single, unified computational framework of uncertainty. Using pharmacological interventions in a sample of 128 healthy human volunteers and a hierarchical Bayesian learning model, we characterise the influences of noradrenergic, cholinergic, and dopaminergic receptor antagonism on individual computations of uncertainty during a probabilistic serial reaction time task. We propose that noradrenaline influences learning of uncertain events arising from unexpected changes in the environment. In contrast, acetylcholine balances attribution of uncertainty to chance fluctuations within an environmental context, defined by a stable set of probabilistic associations, or to gross environmental violations following a contextual switch. Dopamine supports the use of uncertainty representations to engender fast, adaptive responses.
  An assortment of theoretical, behavioural, and neurobiological evidence suggests that the brain computes uncertainty estimates relating to the environment’s sensory events, contextual associations, and their changes over time [6–17]. Uncertainty about the validity of one’s own perceptual beliefs about the world should have the effect of suppressing top-down prior expectations relative to new bottom-up sensory evidence, promoting learning about the current environmental context [18]. With their broad distribution and extensive connectivity, neuromodulatory networks are well placed to facilitate the widespread changes in gain required to support such a function [19,20]. In particular, acetylcholine (ACh) and noradrenaline (NA) are known to enhance bottom-up, feedforward thalamocortical transmission of sensory information relative to top-down, intracortical, and feedback processing [4,21–30]. Thus, uncertainty representations existing within and between environmental contexts are crucial for optimal predictions about the probability of future events. While good predictions facilitate anticipatory preparation of appropriate behavioural responses [5], they are not sufficient for adaptive performance in dynamic environments. An additional mechanism is required to modify action selection based on one’s own beliefs about the latent changes in the environment and/or the occurrence of unexpected events. Indeed, when an unexpected event occurs, humans are capable of engaging resources to inhibit a prepared response and replace it with an alternative [56,57], albeit at the expense of a prolonged reaction time (RT) [58,59]. In sum, while physiological, pharmacological, behavioural, and theoretical work has suggested separable neuromodulatory involvement in different uncertainty computations, attempts to characterise the relative contributions of NA, ACh, and DA within a single computational scheme are lacking (but see noticeable exceptions from Varazzani et al. [68] contrasting the roles of NA and DA in motivation and Brown et al. [69] assessing NA and ACh involvement in orienting responses to environmental novelty). Here we employ a Hierarchical Gaussian Ffilter (HGF) model [9,10] to track human learning and response modulation in dynamic, probabilistic environments under pharmacological NA, ACh, and DA interventions. The HGF has been successfully applied in several recent studies of learning under volatility [15,17,70–74]. We developed a novel instantiation of the HGF with two components: a three-level perceptual model of an agent’s mapping from environmental causes to sensory inputs and a response model that maps those inferred environmental causes to observed RT responses [75]. Thus, we sought to disentangle the effects of pharmacological interventions of the NA, ACh, and DA neuromodulatory systems on participant-specific perceptual belief updating from those on the sensitivity of motor responses to perceptual estimates.
  We first conducted a series of conventional, model-agnostic analyses of behaviour, to assess whether participants learned about the underlying stimulus transition contingencies and whether learning was influenced by our pharmacological interventions (Fig 2). First, we conducted a repeated-measures analysis of variance (RM-ANOVA) of the log (RTs) for correct responses on trials binned according to the five true conditional probabilities that existed in each of the TMs, grouped into High (0. 85 and 0. 70), Mid (0. 25 and 0. 20), and Low (0. 05) transition probabilities, with drug as a between-subject factor. This revealed a significant decrease in log (RTs) with increasing transition probability (main effect of probability: F1. 27,150. 10 = 28. 32, p < 0. 001, effect size ηp2 = 0. 19), which was modulated by drug type (probability x drug interaction: F3. 82,150. 10 = 12. 33, p < 0. 001, ηp2 = 0. 24). Moreover, across the course of a contextual block (Fig 2B), participants became faster at responding to High and Mid probability stimuli and slower at responding to the Low probability stimuli (significant main effects of probability: F1. 28,151. 41 = 27. 80, p < 0. 001, ηp2 = 0. 19, and time: F1,118 = 12. 01, p = 0. 001, ηp2 = 0. 09; probability x time interaction: F2,236 = 6. 55, p = 0. 002, ηp2 = 0. 05). The effect was modulated by drug type (probability x time x drug interaction: F6,236 = 3. 16, p = 0. 005, ηp2 = 0. 07) but again not systematically related to differences in Δalertness or body weight between drug groups (all p > 0. 05). Post hoc (Benjamini-Hochberg-corrected) pairwise comparisons indicated that the impact of drug was driven by the ACh- group, which showed significant log (RT) slowing compared to Placebo (t57 = 2. 98, p = 0. 003, effect size Cohen’s d = 0. 79). Together, these results indicate that participants learned about the true stimulus transition contingencies and that this learning was modulated by our pharmacological manipulations. As is common in such behavioural tasks [80–83], participants showed evidence of post-error slowing on correct trials following those on which they made an error (F1,118 = 10. 92, p = 0. 001, ηp2 = 0. 09; Fig 2C). This effect was not modulated by drug group (trial-type x drug interaction: p = 0. 933) or by Δalertness or body weight (both p > 0. 29). Participants also demonstrated significant log (RT) slowing on correct post-infrequent trials (true transition probability = 0. 05) compared to all other correct trials (F1,118 = 36. 00, p < 0. 001, ηp2 = 0. 23; Fig 2D), which was modulated by drug group (F3,118 = 4. 58, p = 0. 005, ηp2 = 0. 10), but not by Δalertness or body weight (both p > 0. 15). This effect was driven by the ACh- group, with pairwise comparisons revealing significant slowing compared to Placebo (t57 = 3. 38, p = 0. 001, d = 0. 90). Error rates significantly decreased with increasing transition probability (main effect of probability: F1. 57,182. 11 = 5. 04, p = 0. 013, ηp2 = 0. 04). The effect was again modulated by drug type (probability x drug interaction: F4. 71,182. 11 = 4. 72, p = 0. 001, ηp2 = 0. 11), but not by Δalertness or body weight (both p > 0. 78). There was no between-subject effect of drug group (p = 0. 768). For additional model-agnostic analyses, please refer to S1 Text. Raw RT data can be found at https: //dx. doi. org/10. 6084/m9. figshare. 3793410. v2. The HGF model (Fig 3) allows us to map an individual’s beliefs about stimulus transitions, transition contingencies, and volatility—and the respective irreducible, estimation, and volatility uncertainty about these beliefs—onto his/her observed RT responses. The HGF is hierarchical not only in that learning occurs simultaneously at multiple levels, but also in that belief updating at one level is constrained by beliefs at the level above. This provides a generic framework for implementing dynamic learning rates, which underlie learning in volatile environments [1,84]. We predicted that antagonising NA and ACh would impact on participants’ computation of volatility uncertainty and estimation uncertainty respectively, while DA antagonism would impede motor adaptation to uncertain outcomes without perturbing the course of learning. In S1 Text, we provide details of the physiological and subjective control measures. We also provide additional assessments of the HGF model and its ability to provide a good fit to the behavioural data by examining model parameter correlations, residuals and simulations (S2–S6 Figs). Finally, we report the results of additional permutation tests (S5 Table), which allow us to make distribution-free comparisons of the effects of our drug manipulations on the model parameters.
  We found considerable overlap in the influence of NA and ACh antagonism on perceptual belief updating but also quantitative differences between drug conditions. While part-synergistic, part-antagonistic interactions between the two neuromodulators during uncertainty processing have been theorised previously [4], to our knowledge this is the first study to directly assess these putative computational roles, and to distinguish them from dopaminergic effects, under three pharmacological manipulations and within the same computational framework. We propose that ACh guides probabilistic learning within environmental contexts, while NA has a more circumscribed role in modulating the rate at which an agent learns about the volatility latent in the environment. As per its construction, our computational model allowed us not only to characterise perceptual belief updating under our three pharmacological manipulations but also to assess how each intervention influenced the deployment of motor responses in response to individual estimates of uncertainty. Pharmacologically manipulating DA and ACh altered the degree to which participants’ perceptual beliefs modulated the preparation of their speeded responses to uncertain stimuli. In contrast, NA antagonism had no significant impact on the sensitivity of participants’ motor responses to their current perceptual beliefs relative to placebo. We had originally predicted that an individual’s capacity to modulate response selection following a sensory PE would be dependent on DA. Indeed, it has previously been shown that pharmacological DA depletion impedes adaptive reactions to unexpected events occurring within predictable contexts [58]. However, in the present study, we found no evidence that DA receptor antagonism influenced participants’ reactions to low-level sensory PE (δ1). Rather, we observed that suppressing DA significantly reduced β3, which we interpret as a reduction in the sensitivity of participants’ motor responses to their higher-level volatility estimates (μ3). It is important to note that some key differences distinguish our present experimental design from previous paradigms. In earlier work, participants were pre-trained to respond to stimuli presented within one predictable context, defined by one TM. Furthermore, switches from predictable to unpredictable contexts, consisting of random presentations of stimuli, were explicitly signalled [58]. Therefore, any probabilistic learning and higher-level perceptual uncertainty were removed. In this earlier setting, dopaminergic antagonism under haloperidol selectively impaired participants’ reactions to unexpected events that elicited large sensory PEs. In contrast, our present task created a more complex, and arguably more ecologically valid, scenario in which individuals had to infer the current context for themselves and adapt to any contextual changes. Here, uncertainty representations had to be acquired through direct sampling from a distribution of observations. To our knowledge, our study is the first attempt to interrogate the impact of DA, ACh, and NA on nonrewarded probabilistic learning within a single behavioural paradigm and a unified Bayesian framework. By estimating beliefs about various forms of uncertainty, we sought to identify neuromodulatory contributions specifically related to particular forms of uncertainty, as opposed to any confounding variables. Our finding that haloperidol reduced the sensitivity of participants’ responses to their phasic volatility estimates does sit well with an alternative line of work highlighting the importance of DA in cognitive switching [61,62]. For instance, Parkinson’s disease patients with DA dysfunction have an impaired capacity to switch from naming digits to letters when both types of stimuli are presented simultaneously, even when the task shift is explicitly cued [60]. In summary, we propose that our DA manipulation suppressed response modulation by impeding cognitive switching following complex contextual rule changes. Muscarinic ACh receptor antagonism under biperiden also led to decreased response modulation by parameters at all three hierarchical levels: sensory PE (δ1), precision-weighted contingency PE (ε3), and phasic volatility estimates (μ3) compared to Placebo. We propose that ACh receptor antagonism impeded participants’ abilities to learn the statistical structure of the behavioural task, which in turn impaired their capacities to respond accordingly. Although both ACh and DA had effects on response modulation, in light of previous work, we suggest that DA’s role is to modulate motor responses according to the widespread perceptual effects of ACh. One of the main constraints of our study is that although prazosin, biperiden, and haloperidol are rather selective for NA, ACh, and DA receptors, respectively, there are complex interactions and dependencies between noradrenergic, cholinergic, and dopaminergic systems. Such interactions are a main reason why direct quantitative comparison between drug groups would not have provided direct comparisons between the action of different neuromodulators and therefore why our study was designed to detect changes relative to placebo instead. While our results highlight qualitative differences in how NA, ACh, and DA influence perceptual belief updating, future work will have to conduct direct quantitative comparisons of their roles. Further, it is the receptors rather than the neuromodulators themselves that bring about psychophysiological effects, and there are dissociable roles of different receptor subtypes. For instance, the functions of nicotinic versus muscarinic cholinergic receptors in uncertainty signalling have yet to be directly compared. Distinctions have also been made between D1 and D2 dopaminergic receptor subtypes in regulating adaptive responses to unexpected stimuli [58]. Thus, future work could usefully be extended with genetic profiling and a range of selective agonists and antagonists for different receptor subtypes. Finally, it is likely that all the neuromodulators operate over multiple timescales. For instance, separate, even competing, tonic and phasic effects have been a special target of investigation for NA. Teasing these timescales apart more fully is an ambition for the future, requiring a temporally richer design. Nevertheless, our findings emphasise the necessity of studying the systems conjointly, as tasks associated with uncertainty will tend to involve them all. In summary, our results offer novel and direct insight into the complex and intricate effects of NA, ACh, and DA during a probabilistic serial RT task. Employing a hierarchical Bayesian learning model that allowed us to assess various forms of uncertainty and PEs, we provide interventional evidence linking ACh and NA to uncertainty computations within and between behavioural contexts. In contrast, DA appears to be involved in sensitising motor responses to perceptual volatility estimates. While pharmacological manipulations do not selectively target particular neuromodulatory systems, our results offer a fresh perspective on the effects of noradrenergic, cholinergic, and dopaminergic neurotransmission on the computational mechanics of perceptual belief updating according to Bayesian principles. Future studies will verify the generality of our observed effects to different behavioural paradigms with and without learning, reward, prediction, and action. By characterising uncertainty computations and response modulation, our methodology can also be used to offer fresh insight into the numerous neurological and psychiatric disorders in which there is dysregulation of processes dependent on NA, ACh, and DA.
  A total of 128 healthy participants (56 male, aged 18–38 y, 119 right-handed) with normal or corrected-to-normal vision undertook this study. Participants gave written informed consent in accordance with the Declaration of Helsinki. The experimental protocol was approved by the UCL Research Ethics Committee (Project ID: 3491/001). The following exclusion criteria applied: history of neurological or psychiatric disease, intake of medication (other than contraceptives), smoking, recreational drug use, and current participation in other pharmacological studies. Following a screening interview to rule out intolerances or contraindications, the study clinician (DR) assigned participants pseudorandomly (i. e. , ensuring a balanced distribution of gender, age, and body weight) to receive a NA, ACh, or DA antagonist or a placebo. All other authors, including the experimenter (LM), were blind to the drug conditions. We employed a double-blind, between-subject design. Each participant attended one experimental session during which they received a single oral dose of one of the following: 1 mg prazosin (α1-receptor antagonist; NA- group), 6 mg biperiden (M1-receptor antagonist; ACh- group), 2. 5 mg haloperidol (D1/D2-receptor antagonist; DA- group), or a placebo. We selected doses that were in line with previous studies showing clear behavioural and neurophysiological effects [58,96–98]. On arrival, participants completed computerised versions of the Digit Span test, Barratt Impulsiveness Scale (BIS-11) [99], Doman-Specific Risk-Taking (DOSPERT) Scale [100], and Cognitive Failures Questionnaire (CFQ) [101]. Participants also self-reported their baseline mood (alertness, calmness, and contentedness) with visual analogue scales (VASs) [102], and we measured their baseline heart rate (HR) and blood pressure (BP). To assess any subjective and/or physiological drug effects, the VAS, HR, and BP measurements were repeated before participants started the RT task and again once they completed it. Two different drug administration times were used to match peak plasma concentration across drugs, based on previous pharmacokinetic data. To ensure that participants undertook the RT task when the drug was at its most active, haloperidol was administered 2 h in advance, while prazosin and biperiden were administered 1. 5 h before the main experimental session [96–98]. A random 50% of participants from the Placebo group were administered a placebo tablet at the first time point, and the other 50% at the second time point. The study clinician administered the drug or placebo while the experimenter was away from the testing room. Participants were asked not to eat for at least 1 h before the first drug administration time. Participants sat facing a computer screen positioned approximately 60 cm away. They were instructed to rest their left and right index and middle fingers on the four buttons of a custom-made button box placed in front of them and to maintain this position throughout the task. On each trial, participants were required to respond to the presentation of one of four visual stimuli by pressing an appropriate button as quickly as possible. Each stimulus was associated with one particular button. The stimulus-response mappings remained consistent within an experimental session but were counterbalanced across participants. Each participant acquired the stimulus-response mappings for their session during a training block in which they received visual error feedback after each trial. The training session comprised at least 100 trials and did not finish until the participant had reached a minimum performance criterion of 85% accuracy on the last 20 trials. Participants were then given 40 practice trials, in which the stimuli were presented in a random order and without error feedback, to familiarise them with the timings of the main experiment. An additional refresher block, consisting of at least 26 trials with error feedback, was completed immediately before the main experiment. Again, participants had to achieve 85% accuracy in the last 20 trials to proceed. On average, participants reached this criterion in 28. 1 ± 1. 1 trials, indicating adequate learning and retention of the mappings. There was no difference in the number of refresher trials required between groups (F3,120 = 1. 17, p = 0. 324). Each participant performed 1,200 trials of the probabilistic RT task. Fig 1A shows an example trial sequence. Anticipatory responses (<80 ms RT) were recorded as incorrect. At any given time, the trial sequence was generated by one of eight TMs, which changed every 50 trials. In each case, there were 16 combinations that determined the probabilistic relationship between the stimuli presented on the current trial, t, and the previous trial, t-1. Three types of TM were utilised: first-order sequences, alternating sequences, and zeroth-order sequences (see Fig 1B and S1 Fig for further details). The order of TMs was pseudorandom, with no consecutive repeats. Importantly, the overall probability of each stimulus was equal across the 1,200 trials. The pseudorandom order of TMs was used to generate one stimulus sequence that was used for all participants to ensure comparable learning processes and model parameter estimates. Rest periods occurred every 185 trials, orthogonal to TM switches. The importance of fast responses was stressed. Participants were told that by paying attention to any patterns in the order in which stimuli were presented and to any switches in these patterns, it might be possible to respond faster. No further information about the nature of the experiment was provided. Combining our behavioural paradigm with three pharmacological manipulations allowed us to assess directly any separable roles for NA, ACh, and DA in belief updating under irreducible uncertainty, estimation uncertainty and volatility uncertainty and in sensitising the motor system to participants’ individual perceptual beliefs. At the end of the experimental session, participants were debriefed, indicated whether they thought they had taken an active drug or placebo, and reported the quality and quantity of their sleep on the previous night [103]. Trialwise RT was calculated as the time between stimulus onset and the subsequent button press. The RT data were log transformed [58]. To assess the interaction between stimulus transition probability and drug, trials were binned according to three probability levels corresponding to the presented stimuli’s true conditional probabilities as existed in the TMs (High: 0. 85 and 0. 70; Mid: 0. 25 and 0. 20; Low: 0. 05) [58,59]. Mean log (RTs) for each participant’s correct responses were compared across these three probability levels and between drug groups. To obtain a model-agnostic indication of learning across the course of the probabilistic contexts, we performed a median split on each 50-trial contextual block and compared correct mean log (RTs) on Early (1–25) and Late (26–50) trials at each probability level and between drug groups. In reporting statistical differences, a significance threshold of α = 0. 05 was used. Where assumptions of sphericity were violated (Mauchly’s test p < 0. 05), the Greenhouse-Geisser correction was applied. For comparisons across the four groups, partial eta-squared (ηp2) is reported as the effect size. For planned pairwise comparisons between each drug group and Placebo, we applied a Benjamini-Hochberg correction for three comparisons to the significance threshold. For pairwise comparisons, Cohen’s d is reported as the effect size. Model-based analyses permit quantification of the inferences participants made during the task and decomposition of the impact of different forms of uncertainty. Hierarchical Bayesian models have proven powerful for explaining the adaptation of behaviour to probabilistic contexts in volatile environments [1,15,17,70,72–74,84,104,105]. Here we developed a novel instantiation of the HGF model [9] with a focus on TMs and two components: a three-level perceptual model and a response model (Fig 3A). We implemented the model using the HGF Toolbox (http: //www. translationalneuromodeling. org/tapas/). Matlab (Mathworks, United States) code can be downloaded directly from http: //www. tnu. ethz. ch/de/software/tapas/download. html. Note that, for the present study, we used the tapas_logrt_linear_whatworld family of scripts. The HGF is unconstrained by normative optimality assumptions of “ideal Bayesian” models that enshrine the true characterisation of the task (or alternatively Bayesian hierarchical uncertainty about this true characterisation) [76,106,107]; instead, it allows for interindividual variability in learning by fitting parameters that determine a participant-specific approximation to Bayes optimality. We validated our model parameter results using permutation tests. Moreover, to further illustrate the implications of different model parameters and to demonstrate that the HGF provided a good fit to the behavioural data, we assessed the correlations between model parameters, analysed residuals, and used the HGF to generate simulated log (RT) data. Please see S1 Text for details.
summary: Interacting with dynamic and ever-changing environments requires frequent updating of our beliefs about the world. By learning the relationships that link events in the current environmental context, it is possible to prepare and execute fast, accurate responses to those events that are predictable. However, the world’s complex dynamics give rise to uncertainty about the relationships that exist between events and uncertainty about how these relationships might change over time. Several neuromodulators have been proposed to signal these different forms of uncertainty, but their relative contributions to updating beliefs and modulating responses have remained elusive. Here we combine a probabilistic reaction time task, pharmacological interventions, and a hierarchical Bayesian learning model to identify the roles of noradrenaline, acetylcholine, and dopamine in individual computations of uncertainty. We propose that noradrenaline modulates learning about the instability of the relationships that link environmental events. Acetylcholine balances the attribution of uncertainty to unexpected events occurring within an environmental context or to gross violations of our expectations following a context change. In contrast, dopamine sensitises our actions to our beliefs about uncertainty.
keywords: cognitive science
  learning
  body weight
  medicine and health sciences
  neuropharmacology
  neurochemistry
  dopaminergics
  social sciences
  neuroscience
  learning and memory
  cognitive psychology
  physiological parameters
  pharmacology
  neurotransmitters
  cholinergics
  neurochemicals
  biochemistry
  psychology
  neuromodulation
  physiology
  neurology
  biology and life sciences
  sensory perception
  drug interactions
title: Pharmacological Fingerprints of Contextual Uncertainty