Neuroimaging has identified many correlates of emotion but has not yet yielded brain representations predictive of the intensity of emotional experiences in individuals. We used machine learning to identify a sensitive and specific signature of emotional responses to aversive images. This signature predicted the intensity of negative emotion in individual participants in cross validation (n =121) and test (n = 61) samples (high–low emotion = 93. 5% accuracy). It was unresponsive to physical pain (emotion–pain = 92% discriminative accuracy), demonstrating that it is not a representation of generalized arousal or salience. The signature was comprised of mesoscale patterns spanning multiple cortical and subcortical systems, with no single system necessary or sufficient for predicting experience. Furthermore, it was not reducible to activity in traditional “emotion-related” regions (e. g. , amygdala, insula) or resting-state networks (e. g. , “salience, ” “default mode”). Overall, this work identifies differentiable neural components of negative emotion and pain, providing a basis for new, brain-based taxonomies of affective processes.
Emotions are a class of psychological states comprised of physiological responses, expressive behavior, and subjective experiences that are central to our daily lives and to multiple forms of psychopathology [1] and chronic medical diseases [2]. Emotional information organizes physiological, cognitive, and motor systems into adaptive [3], organism-wide responses to events and situations relevant for survival and well-being [4–6]. These responses allow us to pursue resources and avoid harm [7], translate cognitive goals into motivated behavior [8], and navigate the social world [9,10]. Conversely, emotional dysregulation is at the heart of many brain- and body-related disorders (e. g. , mood, anxiety, personality, cardiovascular, and substance use disorders) and likely cuts across traditional diagnostic boundaries [11]. Thus, understanding the neurobiological mechanisms that generate and mitigate negative emotional experience is paramount to understanding both human flourishing and dysfunction. The importance of understanding the “emotional brain” has motivated hundreds of neuroimaging studies in healthy humans [12,13] and those suffering from psychopathology [14–16]. The promise of these studies for basic research is that they will permit a brain-based taxonomy of emotional processes, avoiding the sole reliance on psychological categories [17,18], while the hope for clinical development is to provide transdiagnostic markers for psychopathology that can identify functional brain dysregulation [19] and physical health risk [2,20], predict treatment response [21,22], and guide new, brain-based treatments [23,24]. In spite of this promise, fundamental requirements must be met before neuroimaging findings can be considered brain representations of emotion that are useful for translational purposes [25]. Previous work has identified many brain correlates of emotional versus nonemotional stimuli [12] and physiological responses [26,27] but has yet to uncover brain signatures diagnostic of an individual’s emotional experience. For example, the amygdala, dorsal anterior cingulate (dACC), anterior insula (aINS), and other regions reliably respond to aversive stimuli [28], and functional alterations in these regions are considered prominent features of anxiety disorders [14,29]. However, activation in these regions does not imply an emotional experience. Amygdala activation can occur in the absence of emotional experience [30] and does not appear to be involved in all aversive experiences [31]. In addition, the dACC and aINS are among the most frequently activated regions in the brain across all types of emotional and nonemotional states [28] and have recently been conceptualized as network “hubs” that may be integrating cognitive, emotional, and motivational information [32,33]. One factor that contributes to this limitation is that the vast majority of studies focus on comparing types of stimuli [12], e. g. , “negative” versus “neutral” images, rather than finer grained differences in reported experience [34]. While these emotion-related comparisons are assumed to reflect “affective processing, ” confounds with attention, salience, and other processes may render many findings superfluous to emotional experience. Thus, there is a pressing need for neural signatures that are optimized to predict emotional experiences and functional outcomes. These indicators should: (1) specify a precise set of brain voxels that can be tested in new individuals and prospectively applied to new samples and (2) be sensitive and specific to a class of affective experiences (e. g. , negative emotion and not other states such as attention or arousal) [35]. Machine learning provides a new toolbox of algorithms suited for developing sensitive and specific signatures of psychological processes [36–39], particularly when those signatures involve measures across multiple neural systems, as is likely to be the case with emotional experience [12,18,40]. Standard neuroimaging methods generally preclude estimation and optimization of the strength of the brain experience correspondence [28,41–43], but cross validated machine learning analyses can identify whether brain effects are of sufficient magnitude (e. g. , sensitive enough) and specific enough to have translational utility. These techniques have recently shown great promise in identifying patterns that discriminate among types of affective experiences from brain [35,44–46] and physiology [47], discriminating patient from control groups [19,48], and predicting treatment response [49]. Here, we use machine learning in a large sample (n = 183) to identify the brain systems that predict the intensity of negative affective experiences elicited by viewing images from the International Affective Picture System (IAPS) [50], which is among the most robust methods of eliciting brief affective experiences (d = 0. 81) [51]. In spite of the widespread use of IAPS images in basic and clinical research (e. g. , it is the primary affective task in the human connectome project [52]), the brain mechanisms that underlie the genesis of the negative experiences they evoke have not been clearly identified. In addition, it is unclear (a) whether it is possible to identify a pattern that strongly predicts emotional experience prospectively in out-of-sample individuals, (b) which brain systems are involved (cortical, subcortical, or both), and (c) whether brain activity that tracks negative affect is specific for negative affect, or whether it codes for “salience, ” arousal, or more general features of stimulus processing. Answers to all of these questions are critical for continued progress in both basic affective and clinical sciences. We address each of these questions by developing a multivariate pattern that predicts negative emotion and assess its sensitivity and specificity relative to pain—another type of arousing, salient, negative experience. Finally, to examine the distributed versus localized nature of the signature, we examined the subsystems necessary and sufficient for accurately predicting negative emotional experience.
We used Least Absolute Shrinkage and Selection Operator and Principle Components Regression (LASSO-PCR) [35,53] to identify a distributed Picture Induced Negative Emotion Signature (PINES) that monotonically increased with increasing affective ratings in leave-one-subject-out cross validated analyses (n = 121). To apply the model to data from individual test subjects in both cross validation (n = 121) and separate hold-out test datasets (n = 61), we calculated the pattern response—the dot product of the PINES weight map and the test image—for individual subjects’ activation maps for each of 5 levels of reported negative emotion (see Fig 1). The resulting continuous values reflect the predicted intensity of negative emotion for a given activation map. We used these values to classify which of two conditions elicited a stronger negative emotion for an individual (a “forced-choice” test) [35], providing accuracy estimates (Fig 1E). We also used similar classification tests, described below, to evaluate the sensitivity and specificity of PINES responses to negative emotion versus pain. We focus primarily on results for the test sample, as it was completely independent of all model-training procedures and provides the strongest evidence for generalizability [54]. The PINES accurately predicted ratings of negative emotional experience in both cross validation and hold-out test datasets (Fig 2). For individual participants in the cross validation sample, the average root mean squared error (RMSE) was 1. 23 ± 0. 06 (standard error; SE) rating units, and the average within-subject correlation between predicted and actual ratings was r = 0. 85 ± 0. 02). Accuracy was comparable in the test sample (RMSE = 0. 99 ± 0. 07, r = 0. 92 ± 0. 01). The PINES accurately classified highly aversive (rating 5) versus nonaversive (rating 1) pictures with 100% forced-choice accuracy in both cross validation and test samples (Fig 2B). Classification accuracy was also high in both the highly aversive range (rating of 5 versus 3: forced-choice = 91%; test sample) and the moderately aversive range (rating of 3 versus 1: 100%; test sample) (See S1 Table). We also assessed single-interval classification based on a single image rather than a relative comparison (Table 1), which were only slightly less accurate (Table 1). Comparisons with Support Vector Regression (SVR), another popular algorithm, indicate that these results appear to be robust to the choice of algorithm and, to a large extent, the amount of data used in the training procedure (see S1 Methods). The PINES pattern included reliable predictive weights across a number of cortical and subcortical regions (Fig 2A). Positive weights (greater activity predicts more negative emotion) were found in many regions typically associated with negative emotion [12,40], including the amygdala, periaqueductal gray (PAG), aINS, dorsomedial prefrontal cortex (dmPFC), ventral occipital cortex, presupplementary motor area (preSMA), ventromedial temporal lobe (mTL), and posterior cingulate cortex (PCC). Negative weights were found in the bilateral parahippocampal gyrus, right superior temporal gyrus, left temporal parietal junction (TPJ), right caudate, and occipital and somatomotor cortices. These regions likely comprise multiple functional systems, as we describe in more detail below. Though the PINES comprises nonzero predictive weights across the brain (see S1 Fig), supplementary analyses indicated that a sparse pattern thresholded at p <. 001, as shown in Fig 2 (1. 6% of in-brain voxels), was sufficient to predict emotional experience with comparable sensitivity to the full model (see S1 Methods and S5 Fig). Affect systems may be organized by valence so that a brain signature for negative affect may be found across stimulus modalities and contexts, or in a modality-specific manner, such that there is not one “negative affect system” but many. Testing these hypotheses requires comparing multiple types of negative affect across modalities. Here, we assessed the generalizability and specificity of the PINES response across IAPS pictures and somatic pain, which is a negative, arousing experience arising from a different modality. We employed two types of analyses to examine the PINES specificity. First, we compared the spatial topography of the PINES to another pattern map, the Neurologic Pain Signature (NPS), which shows high sensitivity and specificity to somatic pain across multiple studies [35]. The PINES and NPS maps were almost completely uncorrelated (robust ranked spatial correlation, ρ^ = −0. 01; Fig 4). Several regions showed positive weights in both maps, including the anterior cingulate (ACC), insula, and amygdala. As shown in Fig 5C, however, the weight patterns within these regions were also uncorrelated (bilateral ACC, ρ^ = 0. 04, insula, ρ^ = −0. 05), though weights in the amygdala were modestly correlated (ρ^ = 0. 21). Second, we assessed the specificity of the pattern responses in the test IAPS (n = 61) and thermal pain (n = 28) [56] datasets. The PINES accurately predicted negative affect in the IAPS dataset (n = 61) but showed no response to increasing pain intensity in the pain dataset (Fig 4). Conversely, the NPS responded robustly to increasing pain but showed no response to increasing negative affect in the IAPS dataset. To further assess sensitivity and specificity, we examined how well responses in each pattern could discriminate (a) high pain versus high negative affect, (b) high versus low pain, and (c) high versus low negative affect (Table 1). Because this involves comparing responses from two separate, imbalanced test sets (n = 61 versus n = 28), the analyses described below employ single interval classification, in which individual images are tested for suprathreshold responses independently (as compared to relative within-subject differences in forced-choice classification). The threshold was determined by finding the point that minimized signal detection response bias (see Methods for details), and we report balanced emotion classification accuracy (chance = 50%), sensitivity, and specificity (See S2 Table for equivalent forced-choice analyses). Another question is whether the precise pattern of activity specified in the PINES uniquely captures negative affect, or whether regions and networks previously used in the literature are sufficient. In order to fully appreciate the sensitivity and specificity of the PINES, it is necessary to compare it to the standard univariate approach, which typically examines average activation within ROIs compared to baseline activity. In this analysis, we examined the average response to emotion and pain stimuli within anatomical ROIs and canonical networks defined in large-scale resting-state studies [57]. Defining a brain pattern sensitive and specific to a type of negative emotion is a critical first step towards developing meaningful models of brain representations of emotion. Here, the development of the PINES affords the opportunity to characterize the basis of this pattern representation within and across brain networks. Constructionist theories of emotion [12,18] predict that negative affect is created by interactions among discrete subnetworks that span multiple brain systems, whereas more traditional modular views predict that one system may be sufficient. We tested whether the PINES might be composed of multiple distinct subnetworks and whether responses in multiple subnetworks are necessary for predicting emotional responses. If so, the negative affect captured by the PINES might be considered a truly multisystem distributed process. For this analysis, we calculated pattern responses within each of the largest regions in the PINES (p <. 001, k = 10 voxels; see S1 Methods) for every individual trial within each participant and used a robust clustering algorithm to group the PINES regions into separate networks based on similar patterns of trial-by-trial covariation (see Methods). The best solution contained nine separate clusters, which provides a descriptive characterization of the subnetworks that comprise the PINES (Fig 6, S3 Table) that is broadly consistent with constructionist accounts of emotion [12] and previous meta-analyses of emotion-related networks [17]. These subnetworks included (a) two networks encompassing different parts of the visual cortex (e. g. , lateral occipital cortex [LOC] and occipital pole) consistent with the visual modality of the stimuli, (b) a left amygdala-right aINS-right putamen network, which has been implicated in multiple forms of arousal and salience, (c) a network that includes bilateral posterior parahippocampi and the precuneus, which are broadly involved in memory and other forms of contextual processing, and (d) a network that includes parts of the dmPFC and PCC that are likely involved in social cognition but are distinct from more executive processes [64,65]. An additional network that includes the right somatosensory cortex and contralateral cerebellum may be involved in preparing for the rating action but may also play a more fundamental role in the emotion generation process [66].
These results have theoretical implications for the neurobiology of emotion in terms of both the diversity of processes underlying affective experiences and how they are represented in the brain. Emotions are often defined as a composite of multiple intrinsically inter-related processes (e. g. , autonomic arousal, expressive behavior, action tendencies, interoception, and conscious experiences). Theories differ widely on how these processes combine to give rise to emotional experience [1], but most major theories suggest that cognitive, sensory, motor, motivational, and interoceptive processes are critical ingredients of emotional experience. For example, appraisal theories view emotion as a dynamically unfolding process and emphasize the role of appraisals [8,68,69], embodied affect theories emphasize interoceptive and somatomotor representations [70], and constructionist theories view emotions as being constructed from all of these component processes [12,18]. In spite of this richness, since MacLean [71], theories of the emotional brain have treated emotion as a singular faculty that is localizable to a specific system. Often, this view has translated into “structure-centric” theories of emotional experience; e. g. , the amygdala is critical for fear [72], the ACC for pain affect [73], and the insula for disgust [74]. In other cases, this view translates into circumscribed pathways or networks for “core affect” [17] and emotional awareness [75]. It remains unclear how far the structure-centric view can take us in understanding the brain bases of emotional experience. The regions most strongly identified with emotion are also intimately involved in a wide array of cognitive functions such as attention, error monitoring, associative learning, and executive control [33]. Recent connectivity [63] and functional diversity analyses [32] suggest that these regions are not solely processing affective signals but rather represent functional “hubs” for integrating many types of information. As the limitations of the structure-centric view are increasingly widely recognized [12,33], researchers have moved towards the identification of intrinsically connected networks conserved both at rest and during active tasks [76]. These networks have been labeled with process-general names including the “salience network” [63], “default mode” network [77], and others, and a modern incarnation of the “emotional brain” theory suggests that the basis of emotional experience is encapsulated in one or a few of these networks such as the “limbic” network named after MacLean’s original formulation. Our results corroborate the view that structure-centric—and even network-centric—models of emotion are limited and provide an alternative model for the brain representation of emotional experience. In this study, we targeted the conscious experience component, which is the defining feature of subjective distress and suffering. None of the anatomical regions identified in previous literature (e. g. , amygdala, ACC, insula) predicted the intensity of emotional experience or discriminated emotion from pain in this study. This suggests that the effects identified in previous work using traditional statistical parametric mapping approaches are small and unlikely to serve as effective signatures of the type or magnitude of an emotional experience in an individual person. Furthermore, activity in predefined networks was insufficient to capture negative emotion ratings, demonstrating that the pattern we identified using targeted machine-learning analysis is not reducible to these more process-general networks. The fact that networks and regions defined a priori, even from very large resting-state samples [57], were insufficient to capture emotional experience here has broad implications for the study of emotion and attempts to identify biomarkers for mental health disorders going forward [21,25,49]. Finally, our clustering analysis of the PINES map indicated that multiple, separable subnetworks distributed widely throughout the brain made independent contributions to predicting emotional experience. Importantly, no single subnetwork appeared to be necessary or sufficient in characterizing the emotional experience, as the accuracy in predicting the magnitude or type of experience did not significantly decrease when any given network was omitted. This pattern is consistent with both appraisal [68,69] and constructionist theories of emotion [12,78], which posit that emotional experiences result from interactions between core affect, sensory, memory, motor, and cognitive systems [40]. Overall, these results provide an important step towards identifying emotion-related patterns that can serve as indicators for components of emotional experience. Such signatures can be used as intermediate phenotypes for genetic or risk-stratification studies, and they may provide objective neurobiological measures that can supplement self-report. The identification of intermediate brain-based phenotypes is critical, as self-reported emotion can be affected by many independent processes [8,18,68]—e. g. , core experience, self-reflection, decision-making heuristics, and communicative intentions—which have different implications for understanding what exactly treatments that modulate emotion are measuring and which processes are affected by interventions. We close with several key points and future directions. Importantly, the PINES is not necessarily a biomarker of negative emotion in general. We have demonstrated that it is a signature for the type of affect induced by aversive IAPS images, but its transferability to other emotional states (e. g. , emotion induced by recall, rejection, positive emotion, or stress) remains to be tested. Such tests are a long-term program of future research that must span many studies and papers. We still know very little about the underlying structure of affect and which types of emotional responses can be cross predicted by the same brain markers. It is possible that the PINES captures some types of negative emotion and not others, and findings to this effect will help us move beyond the categories proscribed in our language to develop a more nuanced, brain-based view of affective processes [7,17]. In addition, testing the specificity and transfer of the PINES across many different kinds of affect is a key to developing more robust and specific markers. The PINES can undoubtedly be improved. For example, with further development and testing, it may be differentiated into markers for more specific types of emotional experiences (e. g. , emotion categories like fear, disgust, etc. or canonical affect-inducing appraisals). In addition to types of affect, the PINES can be tested for responses across patient groups (e. g. , schizophrenia, depression, or anxiety) and treatments thought to affect emotion (e. g. , self-regulation, drug treatment, psychotherapy, etc.). This study provides a foundation and a benchmark for such future developments.
Emotions are an important aspect of human experience and behavior; yet, we do not have a clear understanding of how they are processed in the brain. We have identified a neural signature of negative emotion—a neural activation pattern distributed across the brain that accurately predicts how negative a person will feel after viewing an aversive image. This pattern encompasses multiple brain subnetworks in the cortex and subcortex. This neural activation pattern dramatically outperforms other brain indicators of emotion based on activation in individual regions (e. g. , amygdala, insula, and anterior cingulate) as well as networks of regions (e. g. , limbic and “salience” networks). In addition, no single subnetwork is necessary or sufficient for accurately determining the intensity and type of affective response. Finally, this pattern appears to be specific to picture-induced negative affect, as it did not respond to at least one other aversive experience: painful heat. Together, these results provide a neurophysiological marker for feelings induced by a widely used probe of negative affect and suggest that brain imaging has the potential to accurately uncover how someone is feeling based purely on measures of brain activity.
A Sensitive and Specific Neural Signature for Picture-Induced Negative Affect