The brain constantly generates predictions about the environment to guide action. Unexpected events lead to surprise and can necessitate the modification of ongoing behavior. Surprise can occur for any sensory domain, but it is not clear how these separate surprise signals are integrated to affect motor output. By applying a trial-to-trial Bayesian surprise model to human electroencephalography data recorded during a cross-modal oddball task, we tested whether there are separate predictive models for different sensory modalities (visual, auditory), or whether expectations are integrated across modalities such that surprise in one modality decreases surprise for a subsequent unexpected event in the other modality. We found that while surprise was represented in a common frontal signature across sensory modalities (the fronto-central P3 event-related potential), the single-trial amplitudes of this signature more closely conformed to a model with separate surprise terms for each sensory domain. We then investigated whether surprise-related fronto-central P3 activity indexes the rapid inhibitory control of ongoing behavior after surprise, as suggested by recent theories. Confirming this prediction, the fronto-central P3 amplitude after both auditory and visual unexpected events was highly correlated with the fronto-central P3 found after stop-signals (measured in a separate stop-signal task). Moreover, surprise-related and stopping-related activity loaded onto the same component in a cross-task independent components analysis. Together, these findings suggest that medial frontal cortex maintains separate predictive models for different sensory domains, but engages a common mechanism for inhibitory control of behavior regardless of the source of surprise.
Surprise occurs when expectations about the multi-sensory environment are violated. It provides an elementary cognitive and physiological process that forms the backbone of many influential theories of cognitive processing and control [1–5]. The rapid modification of ongoing actions after surprise is critical for effective goal-directed behaviors [6,7]. For example, while eating berries, one needs to rapidly stop ongoing actions when encountering a berry that looks, smells, or feels surprising, lest one eats a rotten berry. However, the manner in which the brain tracks surprise across different sensory domains is not fully understood. Prior imaging work has shown that unexpected events, regardless of their sensory modality, activate similar brain networks [8–11]. In line with this, scalp-electroencephalography (EEG) shows that unexpected events are followed by a modality-independent fronto-central P3 event-related potential [12, ERP, 13]. The canonical neural response to surprise across modalities could indicate that the brain integrates environmental information across modalities and generates global predictions that form the basis of surprise-processing. Alternatively, surprise might result from separate, independent predictions for each sensory domain. In this latter case, the modality-independent surprise response could index a common set of downstream mechanisms triggered by surprise, regardless of sensory domain. In the current study, we tested these two alternatives against each other. While performing a cross-modal oddball task [CMO, 14], human subjects were presented with visual or auditory unexpected events. Using the statistics of the trial sequence, we constructed two models of Bayesian surprise [5]. In one model, surprise-values were separately coded for each sensory domain (i. e. , an unexpected sound did not reduce surprise of a subsequent unexpected visual event). In the alternative model, surprise was coded in a common term across modalities (i. e. , an unexpected sound reduced surprise for a subsequent unexpected visual event). We fit both models to the trial-to-trial electroencephalographic response to unexpected events at each of 64 scalp-sites to determine which model better represents the neural surprise response. As mentioned above, in case this trial-to-trial modeling of the neural surprise response suggests that surprise-terms are computed separately for each sensory domain (i. e. , surprise is not integrated into a common model), the expected cross-modal overlap in neural response may be explained by a common, supra-modal control mechanism that is triggered by surprise, regardless of modality. Therefore, in a second step, we aimed to test the hypothesis that the fronto-central P3 after unexpected events indexes the modality-independent activation of a cognitive control mechanism aimed at inhibiting ongoing behavior. This hypothesis was recently proposed in a theoretical framework claiming that surprise automatically engages the same motor inhibition mechanism that is recruited when ongoing actions have to be stopped [15]. The activity of this mechanism can be measured in the stop-signal task [SST, 16], where fronto-central P3 activity following (non-surprising) stop-signals indexes the speed of motor inhibition [17,18]. To determine whether the fronto-central P3 after unexpected events in the CMO task and the P3 after stop-signals in the SST reflect the same process, we first correlated their amplitudes across tasks and subjects. We hypothesized that if they indeed reflect the same process, their amplitudes should be positively correlated. Additionally, we used independent component analysis to determine if both fronto-central waveforms load onto a common independent component [19,20]. In doing so, we aimed to provide converging support for the proposal that surprise-signals in frontal cortex lead to the automatic activation of a control process that aims to inhibit ongoing behavior, independent of the modality of the unexpected event.
The procedure was approved by the University of Iowa Institutional Review Board (#201612707). Fifty-five healthy young adult volunteers from the Iowa City community were recruited via a research-dedicated email list, as well as through the University of Iowa Department of Psychological and Brain Sciences’ online subject recruitment tool. The sample consisted of thirty-one females and twenty-four males (mean age: 20. 9 y, SEM: 0. 05, range 18–31), eight of them left-handed. Participants were compensated with course credit or an hourly payment of $15. Stimuli for both tasks were presented using the Psychophysics toolbox [21] (RRID: SCR_002881) under MATLAB 2015b (TheMathWorks, Natick, MA; RRID: SCR_001622) on an IBM-compatible computer running Fedora Linux. Visual stimuli were presented on an ASUS VG278Q low-latency flat screen monitor (144 Hz), while sounds were played at conversational volume (~70dB) through speakers positioned on either side of the monitor. Responses were made using a standard QWERTY USB-keyboard. Each trial began with a central white fixation cross on black background (500ms), which was followed by an audio-visual cue (Fig 1). Participants were instructed that this cue would be informative regarding the timing of a subsequent target stimulus (a left- or rightward white arrow) that they would have to respond to. Participants were instructed that the cue would consist of a green circle presented in place of the fixation cross for 200ms, accompanied by a 600Hz sine wave tone of 200ms duration. There was no latency difference between the onset of the tone and the visual stimulus. After cue presentation, the fixation cross reappeared for 300ms, followed by the target (i. e. , the target appeared exactly 500ms after cue onset). Participants were instructed to respond to the target as fast as possible. Target responses were collected through the keyboard (q for leftward and p for rightward arrows) with the index finger of the respective hand. Participants had 1,000ms to respond to the target, after which the fixation cross reappeared and the inter-trial interval began. The duration of the inter-trial interval lasted until 2,500ms from the initial onset of the fixation cross (beginning of the trial) was reached. Furthermore, to prevent predictable trial initiation timing, a variable-length jitter was added to the ITI (100 – 500ms in 100ms increments, uniform distribution), resulting in an overall trial duration ranging from 2,600ms to 3,000ms. After 10 practice trials without any unexpected cues, participants performed 240 trials, spread across 4 blocks. During these experimental trials, 80% of trials contained cues that were as described above (hereafter referred to as standard cues). On 10% of trials, the sine-wave tone was replaced with one of 120 unique birdsong segments, which were matched in amplitude and duration to the sine-wave tone (unexpected auditory cue; mean rise time: 3. 56ms, SEM: 1. 45ms). For these auditory unexpected cues, the visual part of the cue remained the same as for standard trials. On the remaining 10% of trials, the green circle was replaced by one of seven different geometric shapes (upwards/downwards triangle, square, diamond, cross, hexagon, or a serifed “I”-shape) in one of 15 different non-green colors spread across the RGB spectrum (unexpected visual cue, cf. Fig 1). This produced 105 possible unique visual cue combinations. For these visual unexpected cues, the auditory part of the cue remained the same as for standard trials. Trials were presented in pseudorandom order, with the following constraints: the three first trials of each block had to contain standard cues; no two consecutive unexpected-cue trials were allowed to occur; and each block had to have the same number of unexpected auditory and visual cues. Trials began with a white fixation cross on a gray background (500ms duration), followed by a white leftward- or rightward-pointing arrow (go-signal). Participants had to respond as fast and accurately as possible to the arrow by using their left or right index finger as indicated by the direction of the arrow (the respective response-buttons were q and p on the QWERTY keyboard). On 33% of trials, a stop-signal occurred (the arrow turned from white to red) at a delay after the go-stimulus (stop-signal delay, SSD). The SSD, which was initially set to 200ms, was dynamically adjusted in 50ms increments to achieve a p (stop) of. 5: after successful stops, the SSD was increased; after failed stops, it was decreased. This was done independently for leftward and rightward go-stimuli: SSD started at 200ms for both left- and right-arrow trials. Then, if a stop-trial with a leftward arrow lead to a failed stop, the SSD for the next leftward arrow was decreased by 50ms, whereas the SSD for the next rightward response remained unchanged. This way, the SSD was allowed to vary independently for each arrow/response direction. Trial duration was fixed at 3000ms. Six blocks of 50 trials were performed (200 go, 100 stop). Before the main experiment, subjects practiced the task for 24 trials (16 go, 8 stop). All analysis code, as well as the task code, can be downloaded alongside the raw data at the following URL: https: //osf. io/p7s32/. For the CMO task, we quantified mean reaction time (RT), mean error rate (wrong button pressed), and mean miss rate (no response made within 1,000ms after target onset) for each of the three trial types (standard cue, unexpected auditory cue, unexpected visual cue). We analyzed these dependent variables using a 3 x 4 repeated-measures ANOVA with the factors TRIAL TYPE (1–3) and BLOCK (1–4). In case of a significant interaction, we performed follow-up paired-samples t-tests that compared each of the two unexpected cue conditions to the standard cue condition separately for each of the four blocks, resulting in eight total tests. The alpha-level for these comparisons was corrected using the Bonferroni correction to a corrected alpha of. 0063 (i. e. , p =. 05 / 8). For the stop-signal task, we examined the following measures: mean Go-trial RT, mean failed-stop trial RT, and mean stop-signal RT (SSRT; computed using the integration method, Verbruggen & Logan, 2009; Boehler et al. , 2014). EEG was recorded using a 62-channel electrode cap connected to two BrainVision MRplus amplifiers (BrainProducts, Garching, Germany). Two additional electrodes were placed on the left canthus (over the lateral part of the orbital bone of the left eye) and over the part of the orbital bone directly below the left eye. The ground was placed at electrode Fz, and the reference was placed at electrode Pz. EEG was digitized at a sampling rate of 500 Hz. The CMO and SST datasets were preprocessed separately, using custom routines in MATLAB, incorporating functions from the EEGLAB toolbox [22]. The channel * time-series matrices for each task were imported into MATLAB and then filtered using symmetric two-way least-squares finite impulse response filters (high-pass cutoff: . 3 Hz, low-pass cutoff: 30 Hz). Non-stereotyped artifacts were automatically removed from further analysis using segment statistics applied to consecutive one-second segments of data [joint probability and joint kurtosis, with both cutoffs set to 5 SD, cf. , 23]. After removal of non-stereotypic artifacts, the data were then re-referenced to common average and subjected to a temporal infomax ICA decomposition algorithm [24], with extension to subgaussian sources [25]. The resulting component matrix was screened for components representing eye-movement and electrode artifacts using outlier statistics and non-dipolar components [residual variance cutoff at 15%, 26], which were removed from the data. The remaining components (an average of 17. 1 per subject) were subjected to further analyses. To investigate whether surprise is represented separately for each sensory domain (Hypothesis 1: Cross-modal representation of surprise), we constructed two Bayesian surprise terms on a trial-by-trial basis, based on the trial sequences for each subject (cf. Fig 2). For both terms, the surprise value associated with an unexpected cue on a particular trial was based on the following equation [5]: Surprisei=log2 (punexpected_cue (1…n) punexpected_cue (1…i−1) ) This equation corresponds to the trial-wise Kullback-Leibler divergence between the prior probability of an unexpected cue (denominator) and the posterior probability of an unexpected cue (numerator). In other words, the denominator is the proportion of unexpected cues on all trials prior to the current trial, while the numerator is the proportion of unexpected cues on all trials, including the current trial. For instance, if there have been 10 prior trials before the current trial, and if those trials included 1 unexpected cue, the prior probability in the denominator would be 1/10 =. 1. After an unexpected cue on the 11th trial, the posterior probability would be 2/11 =. 1818, resulting in a ratio of. 1818/. 1 = 1. 818, and a surprise value of log2 (1. 818) =. 8624. Because the ratio can’t be less than 1 or greater than 2, the Surprise parameter is bounded between 0 (posterior = prior -> no surprise) and 1 (maximum surprise). The one exception is the first occurrence of an unexpected cue (where the prior is zero, leading to a division by 0 inside the log). To address this, the surprise value for that trial was set to 1 (maximum surprise). Based on this equation, we generated two different models. In Model 1 (separate surprise terms, Fig 2 top), values for each sensory domain were calculated separately. In other words, the first time the subject encountered an unexpected auditory cue in the trial sequence, the surprise for that trial was 1. Subsequent unexpected auditory cues then produced lower surprise values as the posterior and prior probabilities of unexpected auditory cues converge on the same value (i. e. , as the ratio approaches 1, the log approaches 0) with increasing numbers of previous unexpected auditory cues. Critically, these prior and posterior probabilities for auditory cues are calculated without reference to the number of prior unexpected visual cues. Thus, once a subject encounters the first unexpected visual cue, the surprise value for that trial is again 1 (maximum surprise). Hence, the prior for each sensory domain is unaffected by the occurrence of unexpected cues in the other sensory domain. Note that this formulation of Model 1 assumes statistical independence in calculating these probabilities. However, the two kinds of unexpected cues were not statistically independent in the experimental design, as no trial included unexpected cues for both sensory domains. To address this, we also investigated an alternative formulation of Model 1 that respected this mutual exclusivity inherent in the experimental design. For example, upon realizing that the current trial contained an expected visual cue, this increases the prior probability for an unexpected auditory cue. It is not clear whether subjects could have reasonably learned this mutual exclusivity. Regardless of the formulation of Model 1, the first unexpected event for either modality is maximally surprising. As a result, the model fits from this alternative formulation of Model 1 were nearly identical to the reported version, which assumed statistical independence. Therefore, we focus the report on the version that assumed statistical independence. In contrast to Model 1, Model 2 (common surprise term; Fig 2, bottom) extracted a combined surprise value, calculated without reference to sensory domain. In other words, the prior and posterior probabilities are based on the number of unexpected cues, regardless of whether those cues were visual or auditory. Note that an argument could be made that the standard-cue trials should be included in the model construction. More specifically, if the surprise term generally reflects changes in the base rates of occurrence for any event, then standard trials should be very slightly ‘surprising’ after there have been a few unexpected cue trials. However, the lack of P3 adaptation to stop-signals in the stop-signal experiment (see Discussion) suggests that the surprise term should only apply to events that are overtly surprising in light of task instructions, rather than any event that is novel in terms of base rates. Furthermore, including the standard trials leads to a much compromised (i. e. , severely reduced) fit compared to models that only consider the unexpected cues (see S1 Fig). This is a result of the fact that introducing a large number of near-zero surprise trials (i. e. , trials with standard cues) heavily skews the distribution of the model terms towards zero. Therefore, our reported modeling results in the main manuscript only include the unexpected-cue trials. These values were then used to model the whole-brain event-related single-trial EEG response on all trials that contained unexpected cues. This was done using procedures reported by Fischer and Ullsperger [27]. For each subject, sixty-four matrices (one for each EEG channel) were generated that contained the event-related EEG response for each individual trial with an unexpected cue (24 auditory, 24 visual = 48), measured in 10 consecutive time windows covering the entire cue-target interval (500ms, S2 Fig). The time windows were centered around time points ranging from 50 to 500ms and were 48ms long (24ms before and after the exact time point). EEG activity within each time window was averaged for each trial (prior to averaging, the single-trial data were baseline-corrected by subtracting the activity ranging from 100ms– 0ms relative to the cue). Hence, this resulted in a matrix of 48 (trials) * 10 (time points) for each channel (unless trials were excluded because of artifacts); cf. the blue matrix in S2 Fig. Both of the two candidate surprise models constructed from the Bayesian equation were then applied to these EEG matrices (i. e. , the average voltage data in each time window after cue-onset was modeled separately). In applying the models, both the surprise terms and EEG response were z-scored (to standardize the resulting beta weights) and the model terms were regressed onto each time-window vector of the trial by time window EEG response matrix. This was done using MATLAB’s robustfit () function, which performs a linear regression that is robust to outliers. The resulting matrix of beta values was tested against 0 (using paired-samples t-tests for the beta values, with subject as the random factor) at each channel and time point separately. This identified channels and time periods at which the respective model surprise terms reliably captured variability in the EEG signal. This resulted in two sets of 64 (channels) * 10 (time points) = 640 individual tests (one set for each model). To test which model provided a superior fit of the neural data at each channel and time-point, the resulting beta weights from each model also tested against each other, producing a third set of 640 paired-samples t-test (again with subject as the random factor). To correct for multiple comparisons across these three sets of 640 t-tests, we adjusted the alpha-level using the false discovery rate correction procedure [FDR, 28] based on a family-wise alpha-level of. 01. This resulted in an adjusted alpha-level of p =. 00044. A detailed graphical illustration of this overall analysis strategy can be found in S2 Fig. In a separate exploratory analysis of the trial-to-trial reaction times, we similarly regressed the surprise terms from each model onto the response latencies for each target stimulus to assess whether surprise, according to each model, predicted slower responses. In addition to our above-described test of whether surprise is represented in the brain separately for each sensory domain (Hypothesis 1), we also tested whether the predicted fronto-central neural response to unexpected cues (i. e. , the P3) reflects an inhibitory control signal aimed at inhibiting ongoing behavior during surprise (Hypothesis 2 –Surprise-related frontal cortex activity reflects inhibitory control). To this end, we employed cross-task comparisons between the fronto-central P3 extracted for each subject from the CMO task and a separate ‘functional localizer’ task–the stop-signal task–which all subjects performed after the CMO task (subjects performed the SST after the CMO task so they were not biased to use inhibitory control in the CMO task). We used two different approaches to compare activity across tasks: amplitude correlations and indepdendent component analysis (ICA).
Stop-signal behavior was as expected for a sample of healthy young adults. Mean Go-RT was 520ms (SEM: 15. 2), mean failed-stop RT was 444. 3ms (SEM: 13. 2). Mean SSRT was 252. 4ms (SEM: 8). Mean error and miss rates were low (1% and 2. 6%, respectively). Mean stopping success was 51. 4% (SEM: . 45, range: 46–59%), demonstrating the effectiveness of the adaptive stop-signal delay algorithm. In the cross-modal oddball task, correct trial RTs showed the expected pattern as well: There was a main effect of TRIAL TYPE (F (2/108) = 25. 3, p = 9. 74*10−10, partial-eta^2 =. 32), a main effect of BLOCK (F (3/162) = 7. 64, p = 8. 2567*10−5, p-eta^2 =. 12), and a significant INTERACTION (F (6/324) = 9. 78, p = 6. 51*10−10, p-eta^2 =. 15). Individual comparisons revealed that in Block 1, both visual and auditory unexpected-cue RTs were significantly longer compared to standard-cue RTs (t (54) = 9. 41, p = 5. 48*10−13, d =. 75 for visual and t (54) = 3. 14, p =. 0028, d =. 29 for auditory, respectively). Furthermore, in Blocks 2 and 4, visual unexpected-cue RTs were also longer compared to standard-cue RTs (t (54) = 4. 45, p = 4. 3*10−5, d =. 33 and 3. 5, p =. 00094, d =. 26, respectively). No other comparisons survived corrections for multiple comparisons. Taken together, the data indicate the presence of an initial slowing of reaction times following unexpected cues in both modalities, which wore off over the course of the experiment (Fig 3). With regards to error rates, there was a significant main effect of TRIAL TYPE (F (2/108) = 3. 89, p =. 023, p-eta^2 =. 067), with no main effect of BLOCK (F (3/162) =. 4096, p =. 74631, p-eta^2 =. 0075), and no INTERACTION (F (6/324) =. 7, p =. 65, p-eta^2 =. 013). The main effect was accounted for by lower error rates on both types of unexpected-cue trials compared to the standard-cue trials, which persisted throughout the task. With regards to miss rates, there was no significant main effect or interaction (all p >. 14).
In the current study, we tested two hypotheses about the nature of surprise processing in human frontal cortex. First, we found that fronto-central event-related activity at roughly 275-375ms following the appearance of unexpected cues tracks surprise for each sensory domain separately. Rather than incorporating surprise into a common cross-modal term, the neural response was better characterized by a model in which surprise was tracked for each domain separately. The time range and topographical extent of this activity overlaps with the well-characterized P3 trial-average ERP, which is in line with classic averaging-based ERP studies of surprise [1,12,34]. Our single-trial approach was able to disentangle two competing explanations for the common activity found for unexpected events across sensory domains, thereby providing novel insights into how frontal cortex constructs and updates models of the multi-sensory environment. We then tested whether the modality-independent fronto-central neural activity during surprise indexes a rapid inhibition of ongoing motor activity–i. e. , whether the convergence between neural signals following unexpected events, regardless of sensory domain, can be explained by a common control mechanism that is downstream from surprise. This hypothesis is relatively new [15,35–37], as most previous studies of surprise focused on its cognitive effects [12,14,38,39]. The comparatively large sample size of our study allowed us to take the novel approach of correlating electrophysiological signal amplitudes across different tasks, revealing that the P3 amplitude following stop-signals in the stop-signal task reliably correlated with the fronto-central P3 found during multi-modal surprise. Our control analyses indicated that this correlation reflects a common process rather nuisance variables (such as non-specific correlations of ERP amplitudes within or across tasks). Moreover, both ERPs reflected the same component when submitted to a joint independent components analysis. We conclude that the same process that is reflected in the stop-signal P3 is also active during cross-modal surprise. However, what is that process? The most parsimonious explanation is that this signal reflects cognitive control within frontal cortex aimed at inhibiting ongoing motor activity. In the case of stop-signals, this stops the planned motor action, whereas in response to surprise, it produces a ‘pause’, which purchases time for the cognitive system to update the model of the environment without continuing an action that may have been rendered inappropriate by the unexpected change in environmental demand. This pause can also be observed in the reaction time times to the subsequent target. Alternatively, the common process might reflect model updating or surprise (as operationalized in the CMO). However, in the SST, stop-signals are explicitly part of the task (and are introduced during pre-task practice). In other words, participants are expecting and planning for stop-signals, and their occurrence should not produce surprise. Indeed, if stop-signals were surprising, one would expect the amplitude of the stop-signal P3 to decrease as the task progressed (i. e. , as the priors become stable and the surprise terms become smaller and smaller, which is what occurred for the fronto-central P3 in the CMO task). However, as the auxiliary plot in Fig 10 shows, the amplitude of the stop-signal P3, unlike the P3 to unexpected cues in the CMO task, remained constant throughout the experiment. Therefore, surprise is unlikely to be the unifying factor that explains the commonality in brain activity across both tasks. If not surprise, perhaps the common factor is stimulus (in) frequency: in both the CMO task and the SST, the events in question are relatively infrequent (20% of trials in the CMO task include unexpected cues; 33% of trials in the SST include stop-signals). However, the same results that rule out surprise as the common factor rule out (in) frequency. Although the base rates of these (in) frequent events are constant in both tasks, the amplitude of fronto-central neural activity in the CMO task diminishes across the experiment, whereas it remains constant in the SST. Finally, we consider the manner in which novelty relates to surprise. There is a subtle but important distinction between surprise and novelty [40]: novelty is thought to describe a specific stimulus that is not already present in the memory system [41], whereas surprise is a violation of prediction. Novelty and surprise often go hand-in-hand, but can be clearly dissociated. For instance, in situations with concrete priors for a specific outcome, a previously unexperienced ‘novel’ stimulus for which there is no comparison in memory can be within the realm of expectation–e. g. , an encounter of a previously unknown exotic animal at the zoo is a high-novelty, low-surprise situation. Conversely, a surprising event can be entirely non-novel. For example, finding a familiar shirt upon opening the fridge is a high-surprise, low-novelty situation. In light of this distinction, it is worth considering whether novelty might be the common factor that explains our results. Indeed, the unexpected cues in the CMO were specifically designed to be unique and novel, with the unexpected visual cue sampled from 105 unique visual cues and the unexpected auditory cue sampled from 120 unique auditory cues. In contrast, the (in) frequent stop-signals were anything but novel, being sampled from just two possibilities: a leftward pointing red arrow or a rightward pointing red arrow. For the SST, these two possibilities were demonstrated prior to the experiment, and participants experienced dozens of instances of the two stop-signals in the first half of the SST. Nevertheless, for the second half of the experiment, stop-signals consistently produce a large positive deflection (cf. Fig 10). Thus, novelty does not appear to be the common factor. Our preferred interpretation of the common process in terms of motor control is supported by recent studies, which found that unexpected perceptual events lead to a broad, reactive suppression of the motor system, as measured using transcranial magnetic stimulation [37,42]. Additionally, measurements of isometrically exerted force have shown that unexpected events lead to a rapid, reactive reduction of such steadily exerted motor activity [36]. Furthermore, unexpected events have been found to interrupt ongoing finger-tapping [43]. Finally, studies using optogenetics have shown that when regions of the subcortical network that cause inhibition of motor activity are experimentally inactivated, unexpected events no longer yield interruptive effects on motor behavior [44]. All these studies show that surprise, in addition to its prominent cognitive effects, also lead to interruption of ongoing motor activity. The interpretation that the common process between the stop-signal and CMO tasks is motor control is also supported by some features of our data. Specifically, our behavioral data indicated an incidental slowing of reaction times to the target in the CMO task when that target was preceded by unexpected cues, which is in line with prior behavioral studies [45–47]. Our exploratory analysis showed that during the task period in which this RT effect was present, the surprise model (specifically, the separate-term model that also provided the best fit to the neural data) was positively related to the RT data: trials with more surprising cues, according to the Bayesian model, yielded longer reaction times to the subsequent target. We propose that this extra time reflects a momentary suppression of the motor system produced by the unexpected event. Supporting this claim that this ‘pause’ is an adaptive process, accuracy was also increased following unexpected cues (i. e. , a speed-accuracy tradeoff was enacted after unexpected cues, which may be enabled by the transient pause in the motor system that we purport to be reflected in the fronto-central P3). In that vein, one notable observation is that while the surprise term fit the neural data for both domains to similar degrees (Fig 6), the trial-average response to unexpected auditory cues in our current study appeared to be larger in amplitude compared to unexpected visual cues (Fig 5). Interestingly, the reverse was the case in the reaction time pattern, where visual unexpected cues seemed to have larger effects (Fig 3). While we are hesitant to make strong conclusions based on the trial-average data, it is notable that the timing of the P3 to the different stimuli also differs in latency, which likely reflects the fact that early auditory processing is faster than visual processing [48]. Since the increase in trial-averaged P3 response to unexpected visual cues extends to a time period much closer to target presentation (compared to the P3 to auditory unexpected cues, cf. Fig 5), it is tempting to assume that this may explain the difference in RT effects. However, further studies are necessary to explicitly test this hypothesis. Finally, it is worth mentioning that motor inhibition is not the only process that is triggered by unexpected events: such events are known to trigger a cascade of (inhibitory and excitatory) processes, including a reorienting of attention [3,49], autonomic arousal [50], a shift towards exploratory behaviors [51,52], a re-evaluation of learned associations [53] and a potential interruption of ongoing cognitive processes [54,55]. From the current study, as well as other past studies [36,42,43], it is now becoming increasingly obvious that motor inhibition is a part of this multi-faceted cascade of processing, which–in concert–enables the cognitive system to flexibly react to unexpected changes in the environment. There is some debate in the literature about the interpretation of the surprise term used in our model comparison analysis. We followed the nomenclature of Itti and Baldi [5], who termed the calculation of the Kullback-Leibler divergence of the posterior and prior probability distributions (Equation 1) ‘Bayesian surprise’. However, other authors have interpreted this term as ‘model updating’, rather than surprise [56]. Instead of KL divergence, they favor Shannon-based information theoretical quantifications of surprise [i. e. , surprise is quantified as the inverse of the log-scaled prior expectation of a given stimulus, 57]. In past EEG studies, Shannon-based surprise has been associated with the amplitude of the centro-parietal P3 ERP [58,59], rather than the fronto-central P3 examined here. This is in line with BOLD activation of parietal cortex, which tracks such Shannon-surprise in fMRI [56]. Conversely, trial-by-trial indices of Bayesian surprise are associated with the fronto-central P3 [59], which is in line with the current study, as well as with fMRI work showing that BOLD activity in medial frontal cortex tracks Bayesian surprise [56]. Collectively, these results underscore that Shannon-surprise and Bayesian surprise are not only different computational terms but that they may be related to different neural signals. However, in terms of the theoretical distinction between Bayesian surprise and Shannon surprise, it is important to note that both concepts are closely related in most circumstances–i. e. , whenever there is surprise, it will lead to the updating of internal models of the environment. This is also reflected in a high correlation between Shannon- and Bayesian surprise that is present in most experimental circumstances (including the current one). Under some circumstances, it is possible to untangle surprise and model updating by introducing different degrees of volatility into the environment [60] or by explicitly instructing participants that certain surprising cues should not be used to update the internal model of the task [56]. However, in studies like the current one, the two terms are largely identical, with the exception being trials in which in an unexpected cue follows a prolonged sequence of expected cues. (Such trials introduce non-monotonous upticks in the Shannon surprise term, whereas the Bayesian surprise / model updating term is always monotonically decreasing). Perhaps most relevant is the question which term better reflects the commonplace meaning of ‘surprise’ in the everyday world, outside of the laboratory, and which term better reflects the participants’ approach to the experiment. If subjects place strong emphasis on the recent trial sequence and dynamically adapt to the changing local probabilities of unexpected cues, then the Shannon term may provide a better characterization of surprise. This would be the case if participants assume that the current environment constantly changes (i. e. , high volatility). However, if subjects approach the experimental task as a specific, unchanging environment that they need to adapt to by learning the base rates of occurrence, then the Bayesian surprise term may provide a better characterization of surprise. In the current study we assumed that the latter is the case (indeed, the experimental design involved a stable procedure for each task), and as such, ‘surprise’ and ‘model updating’ are essentially synonymous in our study. Taken together, our study suggests that when an environmental model is updated because of an unexpected cue, this leads to surprise, which is accompanied by inhibitory control of the motor system. From a real-world perspective, it makes sense for the cognitive apparatus to operate this way. Because we interact with the environment by executing motor commands, it is important that we interrupt ongoing motor behavior while the model of the environment is updated; ongoing actions need to be re-evaluated in light of changing environmental contingencies. We hypothesize that motor inhibition prevents the execution of actions that were appropriate under the old, now outdated model, and may also free up resources to rapidly initiate appropriate new actions. This interpretation of the medial frontal cortex is in line with prior findings regarding its role in the control of behavior [2,61,62]. Here, we propose a specific neural mechanism by which such control of behavior is achieved during surprise. In conclusion, we found that surprise-based model updating in frontal cortex occurs separately for each sensory domain, but shares a supra-model control mechanism that likely involves the inhibitory control of behavior. These results suggest a specific control mechanism that is rapidly deployed when the model of the environment unexpectedly changes.
Surprise is an elementary cognitive computation that the brain performs to guide behavior. We investigated how the brain tracks surprise across different senses: Do unexpected sounds make subsequent unexpected visual stimuli less surprising? Or does the brain maintain separate expectations of environmental regularities for different senses? We found that the latter is the case. However, even though surprise was separately tracked for auditory and visual events, it elicited a common signature over frontal cortex in both sensory domains. Importantly, we observed the same neural signature when actions had to be stopped after non-surprising stop-signals in a motor inhibition task. This suggests that this signature reflects a rapid interruption of ongoing behavior when our surroundings do not conform to our expectations.
medicine and health sciences
reaction time
brain electrophysiology
social sciences
electrophysiology
brain
neuroscience
cognitive neuroscience
clinical medicine
cognition
brain mapping
bioassays and physiological analysis
vision
event-related potentials
neuroimaging
electroencephalography
research and analysis methods
frontal lobe
imaging techniques
cerebral cortex
clinical neurophysiology
behavior
electrophysiological techniques
psychology
sensory cues
anatomy
physiology
biology and life sciences
sensory perception
cognitive science
neurophysiology
Frontal cortex tracks surprise separately for different sensory modalities but engages a common inhibitory control mechanism