A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment  Flexibility in behavior relies critically on an animal’s ability to alter its choices based on past experiences. In particular, the behavior of an animal is greatly shaped by the consequences of specific actions – whether a previous action led to positive or negative experiences. One of the fundamental questions in neuroscience is how animals learn from rewards and punishments.A neurotransmitter dopamine, is thought to be a key regulator of learning from rewards and punishments . NeuronsA large body of experimental and theoretical studies have suggested that dopamine neurons signal reward prediction errors (RPEs) – the discrepancy between actual and predicted rewards . In partPrevious studies have shown that different parts of the striatum control distinct types of reward-oriented behaviors . First, Instrumental behaviors are shaped by reward, and it is generally thought that dopamine is involved in their acquisition . HoweverUntil recently an implicit assumption across many studies was that dopamine neurons broadcast the same teaching signals throughout the striatum to support different kinds of learning . HoweverIn the present study, we sought to characterize dopamine signals in different striatal regions  during instrumental behaviors. We used a task involving both perceptual and value-based decisions in freely-moving mice – a task that is similar to those previously used to probe various important variables in the brain such as values, biases , confidechanges in the expected future reward . Notably, although we observed correlates of accuracy and confidence in dopamine signals, consistent with previous studies . Compared to the block with equal-sized water, the psychometric curve was shifted laterally to the BIG side . The fitPrevious studies have shown that animals shift their decision boundary even without reward amount manipulations in perceptual decision tasks . These sAlthough we imposed a minimum time required to stay in the odor port, the mice showed different reaction times (the duration between odor onset and odor port exit) across different trial types . First, Animals were required to stay in a water port for 1 s to obtain water reward. However, in rare cases, they exited a water port early, within 1 s after water port entry. We examined the effects of choice accuracy (correct or error) on the premature exit . We founTo monitor the activity of dopamine neurons in a projection-specific manner, we recorded the dopamine axon activity in the striatum using a calcium indicator, GCaMP7f  with fibThe main analysis was performed using the calcium signals obtained in the presence of water amount manipulations. To isolate responses that are time-locked to specific task events but with potentially overlapping temporal dynamics, we first fitted dopamine axon activity in each animal with a linear regression model using multiple temporal kernels  with LasThe constructed model captured modulations of dopamine axon activity time-locked to different events . On averIn previous studies, RPE-related signals have typically been characterized by phasic responses to reward-predictive cues and a delivery or omission of reward. Overall, the above results demonstrate that observed populations contain the basic response characteristics of RPEs. First, dopamine axons were excited by reward-predicting odor cues, and the magnitude of the response was stronger for odors that instructed the animal to go to the side associated with a higher value (i.e. BIG side). Responses to water were modulated by reward amounts, and the water responses were suppressed by higher reward expectation. These characteristics were also confirmed by using the actual responses, instead of the fitted kernels . Finally−4 ± 1.5×10−4 of frames , and nose tracking stayed within 2 cm when a mouse poked its nose into a water port for >1 s in 96.0% ± 0.3 of trials   . We examessions) . We did essions) .Although excitation to unpredicted reward is one of the signatures of dopamine RPE, recent studies found that the dopamine axon response to water is small or undetectable in some parts of the dorsal striatum . TherefoWe noticed greatly diminished water responses when the reward amount was not manipulated, that is, when dopamine axon signals were monitored during training sessions before introducing the reward amount manipulations . In thesThe above results emphasized the overall similarity of reward responses across areas, but some important differences were also observed. Most notably, although a delivery of a small reward caused an inhibition of dopamine axons below baseline in VS and DMS, the activity remained non-negative in DLS. The overall responses tended to be higher in DLS.In order to understand the diversity of dopamine responses to reward, we examined modulation of dopamine axon activity by different parameters . First, Next, we sought to characterize these differences between areas in simpler terms by fitting response curves (response functions). Previous studies that quantified responses of dopamine neurons to varied amounts of reward under different levels of expectation indicated that their reward responses can be approximated by a common function, with different levels of expectation just shifting the resulting curves up and down while preserving the shape . We, theon (α=1) . Similaron (α=1) .Since the recording locations varied across animals, we next examined the relationship between recording locations and the zero-crossing points . The zer−5, analysis of variance [ANOVA]). These results indicate that the elevated or decreased responses, characterized by different zero-crossing points, were not due to a difference in ‘baseline’ but were related to the difference that manifests specifically in responses to reward.We next examined whether the difference in zero-crossing points manifested specifically during reward responses or whether it might be explained by recording artifacts; upward and downward shifts in the response function can be caused by a difference in baseline activity before trial start (odor onset), and/or lingering activity of pre-reward activity owing to the relatively slow dynamics of the calcium signals . To examine these possibilities, the same analysis was performed after subtracting the pre-reward signals . We obseConsiderably small zero-crossing points in dopamine axons in DLS were not due to a poor sensitivity to reward amounts nor a poor modulation by expected reward . DiffereTaken together, these results demonstrate that dopamine reward responses in all three areas exhibited characteristics of RPEs. However, relative to canonical responses in VS, the responses were shifted more positively in the DLS and more negatively in the DMS.The analyses presented so far mainly focused on phasic dopamine responses time-locked to cues and reward. However, dopamine axon activity also exhibited richer dynamics between these events, which need to be explained. For instance, the signals diverged between correct and error trials even before the actual outcome was revealed (a reward delivery versus a lack thereof)  Choice. choice)) . A previchoice)) , althougchoice)) . Additiochoice)) . The preOur task design included two delay periods, imposed before choice movement and water delivery, to improve our ability to separate neuronal activity associated with different processes . The preWe first examined dopamine axon activity after water port entry (0–1 s after water port entry). In this period, the animals have committed to a choice and are waiting for the outcome to be revealed. Responses following different odor cues were plotted separately for trials in which the animal chose the BIG or SMALL side. The vevaiometric curve (a plot of responses against sensory evidence) followed the expected ‘X-pattern’ with a modulation by reward size , which m,choice) . The cro,choice) .When this analysis was applied to the time period before choice movement (0–1 s before odor port exit), the pattern was not as clear; the activity was monotonically modulated by the strength of sensory evidence (%Odor BIG) only for the BIG choice trials, but not for the SMALL choice trials . This reThe aforementioned analyses, using either the kernel regression or actual activity showed that cue responses were modulated by whether the cue instructed a choice toward the BIG or SMALL side . These rAs discussed above, a neural correlate of ‘confidence’ appears at a specific time point (after choice commitment and before reward delivery) or in a specific trial type  before choice. We, therefore, next examined whether a simple model can account for dopamine axon activity more inclusively . To examThe variability and errors in choice in psychophysical performance are thought to originate in the variability in the process of estimating sensory inputs  or in the process of selecting an action (decision noise). We first considered a simple case where the model contains only perceptual noise . In thisWe first simulated the dynamics of state values and TD errors when the model made a correct choice in easy trials, choosing either the BIG or SMALL side . NotablyIn the first model, we assumed that the model picks the best option given the available information in every trial . In thisInterestingly, we were able to explain various peculiar features of dopamine axon signals described above simply by adding some stochasticity in action selection . Note thThe observed activity pattern in each time window is potentially caused by physical movement. For example, the qualitative similarity of reaction time and dopamine axon activity before choice  and 7H,Ionly at that point), the state value becomes equivalent to the reward size multiplied with the confidence, that is the probability of reward given the stimulus and the choice. This means that, as long as the animal has appropriate representations of states, a representation of ‘confidence’ can be acquired through a simple associative process or model-free reinforcement learning without assuming other cognitive abilities such as belief states or self-monitoring (meta-cognition). In total, not only the phasic responses but also some of the previously unexplained dynamic changes can be explained by TD errors computed over the state value, provided that the model contains some stochasticity in action selection in addition to perceptual noise. Similar dynamics across striatal areas , can capture various aspects of dynamics in dopamine axon activity observed in the data, including the changes that occur before and after choice commitment, and the detailed pattern of cue-evoked responses. These results were obtained as long as we introduced some stochasticity in action selection (decision noise), regardless of how we did it. The state value dynamically changes during the performance of the task because the expected value changes according to an odor cue  and the changes in potential choice options. A drop of the state value and TD error at the time of choice commitment occurs merely because the state value drops when the model chose an option that was more likely to be an error. Further, a correlate of ‘confidence’ appears after committing a choice, merely because at that point  while mice performed instrumental behaviors involving perceptual and value-based decisions. In addition to phasic responses associated with reward-predictive cues and reward, we also analyzed more detailed temporal dynamics of the activity within a trial. We present three main conclusions. First, contrary to the current emphasis on diversity in dopamine signals , we found that dopamine axon activity in all of the three areas exhibited similar dynamics. Overall, dopamine axon dynamics can be explained approximately by the TD error which calculates moment-by-moment ‘changes’ in the expected future reward  in our choice paradigm. Second, although previous studies propose confidence as an additional variable in dopamine signals , correlaThe positively or negatively biased reward responses in DLS and DMS can be regarded as important departures from the original TD errors, as it was originally formulated . HoweverRecent studies reported that dopamine neurons are modulated by various variables . One of Whereas our model takes a primitive strategy to estimate state value, state value can be also estimated with different methods. The observed dopamine axon activity resembled TD errors in our model if agent's choice strategy is not deterministic (i.e. there is decision noise). However, confidence measurements in previous models Accumulating evidence indicates that dopamine neurons are diverse in many respects including anatomy, physiological properties, and activity . Our stuOur observation of similarity across striatal areas  would giOur results in DMS are consistent with previous studies that reported small and somewhat mysterious responses to reward . We notiIn contrast to DMS, we observed reliable excitation to water reward in dopamine axons in DLS. However, because we only recorded population activity of dopamine axons, our results do not exclude the possibility that some dopamine neurons that do not respond to reward also project to DLS. Alternatively, the previous observation that some dopamine neurons in the substantia nigra show small or no excitation to reward  may mainIt has long been observed that the activity of many dopamine neurons exhibits a phasic inhibition when an expected reward is omitted or when the reward received is smaller than expected . This inDopamine ‘dips’ are thought to act as aversive stimuli and/or can facilitate extinction of previously learned behaviors (weakening) . The lac proposedThe clear deviation from TD error in dopamine signals in DLS, the lack of inhibitory dip with negative prediction error, revises existing reinforcement learning models in the basal ganglia that assume the same teaching signals across the striatum. On the other hand, recent studies pointed out that the basic reinforcement learning models do not explain the function of DLS, proposing different mechanisms such as value-less teaching signals to support the law of exercise . Here, wThe observation that DLS dopamine signals lack inhibitory responses raises the question what is actually learned by the system. Learning of values depends on the balance between positive and negative prediction errors: the learned value converges to the point at which positive and negative prediction errors form an equilibrium. If a reinforcement signal lacks negative prediction errors, this learning would no longer work as it was originally conceptualized. In reinforcement learning theories, an alternative approach is policy-based reinforcement learning, learning of ‘preference’  rather tIn summary, we propose that the learning of habits and skills are a natural consequence of reinforcement learning using a specialized reinforcement signal (positively shifted response to outcomes) and the unique anatomical property  of the DLS. Future experiments using tasks involving sequence of actions  can testWe found that, across the striatum, dopamine signals overall resemble TD errors, with positive or negative bias in a subregion-specific manner . How sucAlternatively, DLS-projecting dopamine neurons may add ‘success premium’ at each feedback. Signals of success feedback were observed in multiple cortical areas , which iMechanistically, biases in dopamine signals may stem from a difference in the excitation-inhibition balance at the circuit level. In addition to dopamine neurons, there are multiple brain areas where activity of some neurons resembles RPE . Among tThis study is one of the first systematic comparisons of dopamine axon activity across the striatum using parametric decision-making task. Although we tried to target various locations along anterior-posterior, dorsal-ventral, and medial-lateral axes , we did Our task incorporated a typical perceptual task using different levels of sensory evidence  into a vIn this study, we modeled dynamical representation patterns of dopamine neurons in a steady state, but did not examine relationship between dopamine activity and actual learning. Especially, while our model used a discrete single stimulus state in each trial, it is naturalistic that animals use the experience from a single trial to update value in other stimulus states , and/or Multiple studies suggested a close relationship between dopamine signaling and movement . While wTaken together, our results showed that dopamine axon signals in the striatum approximate TD error dynamics. We propose that dopamine in different striatal areas conveys TD errors in a biased manner. One compelling idea is that the lack of negative teaching signals in DLS plays a role in skill/habit, although further examination is needed to establish its functions. We also observed some deviation from TD errors such as contra-lateral turn-related activity in DMS and slight inhibition with locomotion. It is important to test these other parameters in the future in order to understand the meaning of the diversity of dopamine neurons and organization of dopamine-striatum systems.IMSR JAX:006660) (IMSR JAX:007914) (17 dopamine transporter (DAT)-cre (B6.SJL-Slc6a3tm1.1(cre)Bkmn/J, Jackson Laboratory; RRID::006660)  heterozy:007914)  double h13 particles/ml, UNC Vector Core, NC) and AAV5-CAG-FLEX-tdTomato  into both the VTA and SNc  in the DAT-cre mice. Only AAV5-CAG-FLEX-GCaMP7f  was used for DAT;Ai14 double transgenic mice. Virus injection lasted around 20 min, and then the injection pipette was slowly removed over the course of several minutes to prevent damage to the tissue. We also implanted optic fibers  into the VS, DMS, or DLS (one fiber per mouse). To do this, we first slowly lowered optical fibers into the striatum. Once fibers were lowered, we first attached them to the skull with UV-curing epoxy , and then a layer of rapid-curing epoxy to attach the fiber cannulas even more firmly to the underlying glue. After waiting 15 min for this to dry, we applied a black dental adhesive . We used magnetic fiber cannulas  and the corresponding patch cords to allow for recordings in freely moving animals. After waiting 15 min for the dental adhesive to dry, the surgery was complete. We used the following coordinates to target our injections and implants.All surgeries were performed under aseptic conditions with animals anesthetized with isoflurane (1–2% at 0.5–1.0 l/min). Analgesia was administered pre  and postoperatively . To express GCaMP7f  specific(VTA) Bregma: −3.0 mm, Lateral: 0.6 mm, Depth: between 4.5 mm and 4.3 mm(SNc) Bregma: −3.0 mm, Lateral: 1.6 mm, Depth: between 4.3 mm and 4.1 mm(VS) Bregma: between 1.5 and 1.0 mm, Lateral: 1.8 mm, Depth: 3.8 mm, angle 10°(DMS) Bregma: between 1.5 and 0 mm, Lateral: 1.3 mm, Depth: 2.3 mm(DLS) Bregma: between 1.3 and −0.8 mm, Lateral: 3.0 mm, Depth: 2.3 mmThe behavioral apparatus consisted of a custom-built behavioral box (32 × 19 × 30 cm)  containiAll behavioral experiments were controlled by a NIDAQ board  and Labview , similar to a previous study . Mice weMice were first trained only with pure odors and with the same amounts of water reward (~6 µl). After mice achieved greater than 90% accuracy, mice received a surgery for viral injection and fiber implantation. Following a 1 week recovery period, mice received re-training and then, mixtures of odor A and B  were gradually introduced. After the accuracy of all the mixture odors achieved more than 50%, neuronal recording with fiber fluorometry was performed for five sessions. Subsequently, a task with different amounts of water was introduced. Mixtures of odor A and B  but no odor C were used in this task. Each recording session started with 88–120 trials with an equal amount of water  in the first block to calibrate any potential bias on the day. In the second block, different amounts of reward were delivered in each water port. In order to make the water amounts unpredictable, one water port delivered big or medium size of water  in a pseudo-random order, and another water port delivered medium or small size of water  in a pseudo-random order. Block 2 continued for 200 trials or until the end of recording sessions, whichever came earlier. A mouse performed 134.3 ± 3.4 (mean ± SEM) trials in block 2. The water condition (BIG or SMALL) was assigned to a left or right water port in a pseudo-random order across sessions. Recording was conducted for 40 min every other day to avoid potential bleaching. On days with no recording, animals were trained with pure odors A and B with the standard amount of water.Fiber fluorometry (photometry) was performed as previously reported  with a fCalcium transients may not reflect spike counts, because of autofluorescence, bleaching, motion artifacts and inevitable normalization. We recorded tdTomato signals to monitor motion artifacts because a previous study showed that the red signals reflect motion artifacts reliably . AlthougMice were perfused using 4% paraformaldehyde and then brains were sliced into 100 µm thick coronal sections using a vibratome and stored in PBS. Slices were then mounted in anti-fade solution  and imaged using a Zeiss Axio Scan Z1 slide scanner fluorescence microscope .We fitted % of odor mixture (X) to % of choice left or choice BIG (μ) using generalized linear model with logit link function in each animal as previously reported .log and a block type  to choice.We first fitted a control block (block 1) and a reward-manipulation block (block 2) separately to examine difference of a slope, b0 + b2)/ b1, which is a lateral shift compared to no bias, and b0/b1 - (b0 + b2)/ b1, which is a lateral shift compared to choice in block 1.Choice bias in block two was quantified choice bias as a lateral shift of the psychometric curve equivalent to % mixture of odors, 50 - , fitted tdTom signals were subtracted from green signals.Responses were calculated by subtracting the average baseline activity from the average activity of the target window. Unless specified otherwise, odor responses were calculated by averaging activity from 1 to 0 s before odor port out (before choice) minus the average activity from the baseline period (1–0.2 s before odor onset). Responses after choice were calculated by averaging activity from 0 to 1 s after water port in minus the same baseline. Outcome responses were calculated by averaging activity from 0 to 1 s after water onset minus the same baseline. When comparing activity before and after water onset, average activity in 1–0.2 s before water onset was used as baseline. To normalize GCaMP signals across sessions within an animal, GCaMP signals were divided by average of peak responses during 1 s after odor onset in all the successful trials in the session. Z-scores of the signals were obtained using mean and standard deviation of signals in all the choice trials (from 2 s before odor onset to 6 s after odor onset) in each animal.We built a regularized linear regression to fit cosine kernels   and noise in the process of selecting an action (decision noise). The first model contained only perceptual noise , and theS subjective odors (NS = 60 or four discrete states), choice (BIG versus SMALL), and different timing  (circles in These models had different ‘states’ considering NS possible subjective odor states (O’) which comprise SubOdor1 and SubOdor2 states. We assumed that, in each trial, an internal estimate of the stimulus or a ‘subjective odor’ (O’) was obtained by adding a noise to the presented odor stimulus (O)   . In the As in the behavioral paradigm, whether the model receives a reward or not was determined solely by whether the presented odor (O) instructed the BIG side or SMALL side. Each subjective odor state contains cases when the presented odor (O) is consistent or congruent with the subjective odor (O’). For each subjective odor state, the probability of receiving a reward after choosing the BIG side, For each subjective odor, the model chose either the BIG or the SMALL side based on the value of choosing the BIG or SMALL side (ibution) . In the We also tested other stochastic decision rules such as Herrnstein’s matching law  or ε-gre trials) . In HerrThe perceptual noise and a set of decision rule determine the behavioral performance of the model. The first model has only one free parameter, For the first model, the best fit The state value for each state was obtained as the weighted sum of expected values of available options which was computed by multiplying expected values of the option with probability of an option in the next step.Outcome2 state represents the timing when the animal recognizes the amount of water. The state value is given by the amount of water that the model received ,try data . Using αOurcome1 state, or Water/No-water states  represent when the animal noticed the presence or absence of reward, respectively, but not the amount of reward. The value of a W (Water) state was defined by the average value of the next states. At the BIG side,The values of N (No-water) states at the BIG and SMALL side are zero,WaterPort1 and WaterPort2 states represent when the animal entered and stayed in the water port, respectively. The state value was obtained separately for the BIG and SMALL side. The value of choosing the BIG and SMALL sides is given by weighted sum of the values of the next states  and before making a choice. The model chooses the BIG or SMALL side with the probability of The two successive states had the same state values.OdorOn state represents when the animal recognized the presentation of an odor but before recognizing the identity of that odor. The state value of the OdorOn state is defined by the weighted sum of the values of the next states (SubOdor1).ITI state represents when the animal is in the inter-trial interval (i.e. before odor presentation). The value of ITI state was set to zero.After obtaining state values at each state, we then computed TD errors using a standard definition of TD error which is the difference between the state values at consecutive time points plus received rewards at each time step . For simWe also tested the effect of including invalid trials. At water acquisition, we included failures  where a mouse did not fulfil the requirement of odor poke duration (short odor poke), but did indicate a choice. At an odor port, failures resulted from multiple pokes of odor port , and a short odor poke . Values for these failures were set to 0. Existence or omission of these failures in models did not change the conclusion.To examine which model explains actual data better, we examined Pearson's correlation between model values and actual recording signals using bootstrapping. We tested models with deterministic choice and softmax choice, focusing on activity before and after choice. First, we randomly sampled 22 actual data  in each trial type  before and after choice. Because not all the animals have all the trial types, this sampling rule weighted to reflect rare trials equally well. Next, correlation of the sampled data (22 × 8 × 2 data) with each model was examined by Pearson's correlation, and difference of the correlation was calculated. The same procedure was repeated 500 times, and probability that correlation with a softmax choice model was equal to or smaller than a deterministic choice model was used as p-value.To examine effects of different ratios of inputs whose efficacy is slightly different each other, we constructed a simplest model with only two inputs. Both inputs (excitatory and inhibitory) encode intact TD errors, but they mainly send information with excitation (10 times efficacy than inhibition). To vary ratios of inputs, we examined postsynaptic neurons that receive excitatory and inhibitory inputs at the ratio of 1:1 , 2:1 (more excitation) or 1:2 (more inhibition). Original TD errors in inputs were computed similar to A camera  was set on the ceiling of the behavioral box. We used infrared (IR) light  to illuminate the arena and recorded video at 60 to 84 frames per s (fps) with H.264 video compression and streaming recording mode. The video was captured using the FlyCap2 software that accompanies the camera and processed using DeepLabCut (DLC) . Frames We processed 43 videos . Video was synchronized with task events and fluorometry signals by sending TTL signals every 10 s from a computer that controlled the task, recorded mouse behavioral events to a fluorometry recording channel and at the same time turned on a 20 ms flash of infrared LED light . To ensure that the LED light was invisible to a mouse, the light was set at the top edge of the behavioral box and covered by black light-shielding tape except for a small hole which was faced toward the camera. TTL-controlled light was detected in each video frame by thresholding the maximum intensity in the illuminated area.−2% and 1.8 × 10−3% respectively in an example video) and were filled by the average of tracking points in adjacent frames. To confirm tracking quality in the task, nose, head and body positions were examined during the time window when a mouse waited for water for 1 s. Locations of water ports were estimated at median of nose positions at 17 ms after water port entry in all the trials excluding trials with premature exit. Trials were excluded if nose positions were >2 cm, head positions were >2.5 cm or body positions were >8 cm from median of nose positions at 17 ms after water port entry in all the trials with >1 s in a water port. Excluded trials decreased over training .Head position was obtained by averaging the positions of 3 body parts: the nose, left ear and right ear, and body position was obtained by averaging the positions of the head and tail stem. The optic fiber occasionally occluded or was mistaken for a body part, especially tail midpoint or tip. We therefore did not use locations of tail midpoint and tip in this study. To verify the tracking, tracked points were first visually examined by observing merged video with tracked points. Next, tracked points in adjacent frames were compared to detect biologically impossible gaps between frames. The warped points of >5.5 cm in a single frame decreased over training  for >0.5 s to a moving phase (>3 cm/s body speed) for >0.5 s, and from a moving phase for >0.5 s to a quiet phase for >0.5 s, while a nose is >11 cm from an odor port side. To compare movement speed and fluorometry signals, 2000 frames were randomly picked in each video, and Pearson's correlation coefficient was obtained in each video.To compare gross movement and fluorometry signals while animals were waiting for water in a water port, we calculated head and body distance traveled during the periods  in each trial, and examined Pearson's correlation coefficient with average fluorometry signals during 0–1 s after water port entry. To examine whether movement is responsible for dopamine activity modulation by accuracy after choice, fluorometry signals were linearly regressed with body speed and accuracy (correct or error) with elastic net regularization (α = 0.1) with 5-fold cross validation.Photometry dataset was deposited at Dryad. No formal power analysis was carried out to determine the total animal number. We aimed for a sample size large enough to cover wide areas of the striatum. Chemicals were randomly assigned to an odor cue. Trial types (odors) were pseudorandomized in a block. Session types were pseudorandomized in a recording schedule. Animals were randomly assigned to a recording location. The experimenter did not know location of recording until the recording schedule was completed. No animals were excluded from the study: all analysis includes data from all animals. No trials were excluded from statistical analyses. To visualize average activity pattern in a stretched time window, outlier trials  were excluded.Data analysis was performed using custom software written in MATLAB . All statistical tests were two-sided. For statistical comparisons of the mean, we used one-way ANOVA and two-sample Student’s t tests, unless otherwise noted. Paired t tests were conducted when the same mouse’s neural activity was being compared across different conditions or different time windows. The significance level was corrected for multiple comparisons using Holm–Sidak’s tests unless otherwise indicated. All error bars in the figures are SEM. In boxplots, the edges of the boxes are the 25th and 75th percentiles , and the whiskers extend to the most extreme data points not considered outliers. Points are drawn as outliers if they are larger than q3+1.5×(q3-q1) or q1-1.5×(q3-q1). Individual data points were overlaid on boxplots to compare striatal areas. In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.Acceptance summary:In this paper, fiber photometry and a perceptual choice task are utilized to provide a very interesting and important window onto the activity of dopamine axons in three different regions of the striatum. Unexpectedly, activity in all three regions was found to be quite similar, in contrast to accepted function of dopamine actions in different striatal. In addition, a model is provided that captures some features of the behavior and may explain this homogeneity.Decision letter after peer review:eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Kate Wassum as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Melissa J Sharpe (Reviewer #4).Thank you for submitting your article "Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task" for consideration by The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.eLife". Please let us know if you would like to pursue this option. As the editors have judged that your manuscript is of interest, but as described below that additional analyses and major text revisions are required before it is published, we would like to draw your attention to changes in our revision policy that we have made in response to COVID-19 (https://elifesciences.org/articles/57162). First, because many researchers have temporarily lost access to the labs, we will give authors as much time as they need to submit revised manuscripts. We are also offering, if you choose, to post the manuscript to bioRxiv  along with this decision letter and a formal designation that the manuscript is "in revision at Summary:In this paper, the authors use fiber photometry coupled to a perceptual choice task to provide a functional window onto the activity of dopamine axons in three different regions of the striatum . Unexpectedly, fluorescence signals in all regions are exhibit only relatively subtle differences of baseline (DLS has an elevated baseline) and movement sensitivity (largely in the DMS). The authors lay out a picture of the various signals, and also provide a model that captures some aspects of the behavior.Essential revisions:1) While the models used to explain the data are quite elegant, the overall results remain correlative. There are no experimental manipulations to test the primary conclusions. Most notably, the lack of "dopamine dip" in the DLS following a smaller than expected reward may support a role for this region in habitual behaviors. However, this remains speculative, as there is no operationally defined metric that animals are unambiguously showing habits.2+ imaging  or voltammetry . For example, Oyama et al., 2010, record single-unit activity in DLS and find that these neurons do not show negative errors in response to unexpected omission of reward in a similar task used here. Please provide a potential and detailed explanation for why such dramatic differences are observed in the current work, that cannot be solely related to differences in the technique used.2) The homogenous pattern of dopamine neuronal signaling across striatal regions during reward-directed behavior in the current study contrasts starkly with extensive heterogeneity observed in prior studies using Ca3) The idea that the DLS might be engaged in something like the actor learning part of the actor-critic learning rule  has been relatively widely suggested – the authors might comment on the observation in REINFORCE rules (that the paper cites) that it is possible to add an "arbitrary" action-independent baseline to the equivalent of the prediction error and derive the same expected change in the weights (only the variance is affected). Does this not imply that the baseline shift in the DLS is unlikely to be associated with a different functional property?4) It would be expected that the model would make a much larger distinction between the medium reward in the BIG than the SMALL condition than was apparent in the data. Indeed, one of the puzzles for me about the paper was the tiny effect of the BIG vs SMALL side  – why might that be? Would the authors' previous results not predict a larger difference?5) The FP data shows three peaks in DA in correct trials, whereas the model only really shows two. The second peak, associated with leaving the sampling well is important – particularly for the DMS – could the authors comment on its absence in the model?6) Further analysis is required from the transition between block 1 and block 2 and the speed of adaptation could provide useful cues about the employment of goal-directed or habitual values, and distinguish DMS and DLS.7) It is unclear why the data is combined across regions in showing responses to the odor or choice. Is the only time these signal diverge during reward delivery? I think it would significantly increase the novelty and impact of the findings if the authors could focus on representing the differences between these regions in their figures , and conduct the analyses in relation to the different regions . What are the parameters that change when data is sorted out this way? The claim that these data all line up a TD error simply cannot be the case, because DLS does not show the hallmark dip in activity to an unexpected reward that would be expected of a TD error.8) The claim is made that DLS activity is consistent with the "law of exercise" proposed by Thorndike, encoding how to perform motor sequences. However, these data are also consistent with this region encoding the reward outcome as a stimulus . Alternatively, it could be encoding the "long run history" of rewards (and impervious to single reductions in reward magnitude), which is consistent with current theories of habitual learning. Therefore, suggesting changes to existing theories on the basis of a single characteristic of neural activity in a limited data set, without a behavioral test that would facilitate such an understanding is not advisable. In fact, this can be removed with no detrimental effect to the paper.9) The fact that these data can be modeled in a TD framework does not mean dopamine terminals are acting as a model-free TD error. This is because there is no test outside this hypothesis. There is nothing other than reward "value" that is manipulated. And dopamine neurons are capable of acting outside a model-free TD error. This has been demonstrated many times by many different labs in the past 3-5 years; all of this should be noted and discussed. Indeed, dopamine neurons in VTA respond to unexpected changes in reward identity , and stimulation of dopamine neurons drives learning between neutral cues without endowing them with "model-free" value . This is not predicted by a model-free TD signal. Indeed, similar "model-based" findings have been found in ventral striatum .10) Please rephrase your interpretation as follows: " our modeling approach is consistent with activity at DA terminals functioning within the lines of a TD error framework in this particular reward-learning task, while we acknowledge that it is likely these signals are also acting outside of this framework in other tasks".11) It is crucial to report whether animals engage in licking in the water (or sample) port before the reward is provided? Does this covary with model 'value' and confidence? Does it covary with FP signals?eLife. Your revised article has been evaluated by Kate Wassum (Senior Editor) and a Reviewing Editor.Thank you for resubmitting your work entitled "Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task" for further consideration by The manuscript has been improved but there are some remaining issues that need to be addressed before acceptance, as outlined below.In subsection “Similarity of dopamine axon signals across the striatum” paragraph three, optimism in Q learning should not be described without qualification. The fact that Q learning uses max_b Q(s(t+1),b) at the next state (s(t+1)) is not thought of being borne of optimism – this in fact reflects what the subject thinks it could do at that state . Confusion may arise in that optimistic Q values are associated with the motivation to explore. If the authors wish to include the assertion about Q learning, they should expand on their exact point.The" dip" in the signal is important for the TD error. And if the proxy for the DLS signals is "more excitation" represented in Figure 8, this should still predict a dip in the DLS . However, signals in DLS still show an increase in activity from baseline during water omission. This should be clarified.The authors propose that the observed signals look like the scalar difference between the expected an observed value (with the exception of the DLS), but are not necessarily used to attach this scalar value product to the antecedent cue or action. This should be specifically and explicitly stated in the text as it is not necessarily scalar value that attaches to antecedent events, which is alluded to by referencing and modeling using the model-free TD algorithm.Please include summary statistics in additional to P values where appropriate . Essential revisions:1) While the models used to explain the data are quite elegant, the overall results remain correlative. There are no experimental manipulations to test the primary conclusions. Most notably, the lack of "dopamine dip" in the DLS following a smaller than expected reward may support a role for this region in habitual behaviors. However, this remains speculative, as there is no operationally defined metric that animals are unambiguously showing habits.Thank you for your comment. It is important to distinguish what we can conclude directly from the data and speculations. The idea that the lack of dopamine dip in the DLS supports a role in habit is our speculation, given the literature pointing to the role of DLS in habitual behaviors and our present results. Before addressing the concern, let us explain the main goal of the study and the main conclusions that we can draw directly from the data.We designed our task to quantitatively examine effects of reward value, prediction error and confidence separately on dopamine axon activity. From observation in these experiments, we drew several conclusions.Our primary conclusions consist of three points;1) Dopamine axon activities in VS, DMS and DLS are very similar and consistent with RPE (TD error) during value- and perception-based decision task.2) Excitation thresholds of dopamine reward responses are different across areas.3) Representation of confidence- or accuracy-like signals can naturally emerge as dynamics of TD errors, providing a unified account on dynamic dopamine signals in the task.We believe that each of these findings is important, which trigger many types of future studies. One of future directions we propose is the relationship between positively-biased RPE signals in dopamine axons in DLS that we found, and function of DLS in habit/skill that had been proposed in previous many studies. This is what we discussed in Discussion, but never in Results. In order to distinguish our conclusions in the Results and Discussion, we added sentences to summarize our main conclusions.“We present three main conclusions. First, contrary to the current emphasis on diversity in dopamine signals , we found that dopamine axon activity in all of the three areas exhibited similar dynamics. […] Nonetheless, dopamine axon signals overall exhibited temporal dynamics that are predicted by TD errors, yet, the representation of TD errors was biased depending on striatal areas.”Our task is not designed specifically to test habitual behaviors, and the ability of the mice to bias their choices according to the reward amounts suggests that the mice were not behaving completely in a habitual mode . The stochastic variation of reward amounts might have facilitated this flexible behavior. Nonetheless, this result does not necessarily indicate that the brain area related to habit learning and its reinforcement signals are completely shut down. It is likely that whether the animal exhibits goal-directed versus habitual behaviors is determined by sensitive balance between various factors , and dopamine input to DLS might still be providing “permissive” reinforcement signals. Furthermore, the choice behavior (decision as to left versus right) is just one aspect of behaviors that the animal has to learn in our paradigm. The animal learns various task-related, sequence of movements through learning in the task (e.g. smooth movement from the center to a reward port). In this sense, we believe that it is still remarkable that dopamine signals in DLS possess properties that could readily explain aspects of habit or skill learning in the current task. Our focus is to extract a rule that governs dopamine axon activity in each striatal area by comparing activity in a single task. In any case, whether the behavior in our task are habitual or goal-directed would not affect the validity of our three main conclusions listed above.Nonetheless, to reflect the limitation that the reviewers pointed out, we have deleted the word "habit" from Abstract, and replaced the concluding sentence with a more general conclusion as follows:“The differences in dopamine signals in these regions put specific constraints on the properties of behaviors controlled by dopamine in these regions.”We also agree that the Discussion in our previous version of the manuscript focused too much on this one point (habit/skill). Following the recommendations in other comments below, we have shortened our discussion on habit/skills, and more evenly discuss the three points in the revised manuscript.2+ imaging  or voltammetry . For example, Oyama et al., 2010, record single-unit activity in DLS and find that these neurons do not show negative errors in response to unexpected omission of reward in a similar task used here. Please provide a potential and detailed explanation for why such dramatic differences are observed in the current work, that cannot be solely related to differences in the technique used.2) The homogenous pattern of dopamine neuronal signaling across striatal regions during reward-directed behavior in the current study contrasts starkly with extensive heterogeneity observed in prior studies using Ca”This is a very important point, and we expanded Discussion related to previous studies in subsection “Similarity of dopamine axon signals across the striatumBriefly, our results were consistent with many previous studies. The most important difference between these and the present study is that many of the previous studies used PREDICTED reward to examine dopamine responses to reward. In contrast, we varied the reward amounts parametrically, and introduced some stochasticity in reward. We speculate that more homogeneous pattern that we observed is, at least in part, due to this task design.To illustrate, we made an additional figure presenting the information in the way similar to the previous studies . As is clear in this new figure, if we presented dopamine axon activity in a similar manner as previous studies, the results are similar to the previous studies (these plots emphasize difference). In other words, if we only conducted the standard experiments with fixed amounts of water, our conclusion would have been the same as previous studies, emphasizing diversity rather than similarity. However, we performed additional experiments, and found that if the water amounts are manipulated in a probabilistic manner to cause quantitative RPE, RPE signal patterns looked similar with one another . Thus, some of the diversity observed in previous studies might be caused by predictability of reward amounts, and could be explained by differences in excitation thresholds. We added a comment on similarity of results more clearly :“Our observation of similarity across striatal areas  would give an impression that these results are different from previous reports. […] Taken together, although we cannot exclude the possibility that dopamine activity in DLS is modulated by a specific movement in particular conditions, our results do not support that TD error-like activity in DLS is generated by a completely different mechanism or based on other types of information than other dopamine neuron populations.”Our results in DMS are consistent with previous studies that reported low and somewhat mysterious responses to reward . We noticed that while animals were trained with fixed amounts of water, some of dopamine axon signals in DMS did not exhibit clear responses to water, and in average water responses were smaller than other areas. Once reward amounts became dynamic, dopamine axons in DMS showed clear responses according to RPE, similar to the previous observation that dopamine responses to reward in DMS emerged after contingency change . We state:“Notably, the study  also used predictable reward (fixed amounts of water with 100% contingency) to examine dopamine responses to reward.”Why dopamine in DMS shows movement-related activity, and why it sometimes does not show responses to water are not fully addressed in this study. We added more discussion about each previous finding.“As pointed out in a previous study , this movement-related activity in DMS is unlikely to be a part of RPE signals.”“Why are reward responses in DMS sometimes observed and sometimes not? We found that the response function for water delivery in dopamine axons in different striatal areas showed different zero-crossing points, the boundary between excitatory and inhibitory responses . […] It will be important to further examine in what conditions these dopamine neurons lose responses to water, or whether there are dopamine neurons which do not respond to reward in any circumstances.”We also emphasize the limitations of our photometry method in our study.“We also have to point out that because fluorometry in our study only recorded average activity of dopamine axons, we likely missed diversity within dopamine axons in a given area.”“However, because we only recorded population activity of dopamine axons, our results do not exclude the possibility that some dopamine neurons that do not respond to reward also project to DLS.”Further, to compare our recording data with previous studies, we conducted video analyses using DeepLabCut, and compared movement and dopamine activity. Consistent with previous observation , we observed slight modulation of dopamine axon signals with movement. Now the new results are presented in Figure 2—figure supplement 1 and Figure 7—figure supplement 6.3) The idea that the DLS might be engaged in something like the actor learning part of the actor-critic learning rule  has been relatively widely suggested – the authors might comment on the observation in REINFORCE rules (that the paper cites) that it is possible to add an "arbitrary" action-independent baseline to the equivalent of the prediction error and derive the same expected change in the weights (only the variance is affected). Does this not imply that the baseline shift in the DLS is unlikely to be associated with a different functional property?Thank you for providing us with the opportunity to address this important and interesting point. We have expanded our Discussion on this. Briefly, we proposed preference learning in DLS  instead of value learning, as proposed in actor-critic models, but we do not propose that DLS does policy gradient updates such as in REINFORCE because the underlying assumption that behaviors (or policy) converge to the optimal solution might not be consistent with the concept of habit. Previous studies indicated that DLS is responsible for habit and skill, even when it is not optimal. Instead, reinforcing rules could be (but are not limited to) direct updates as proposed in an original actor-critic model . This is different from the REINFORCE rules, which is inversely proportional to action probability.“We propose that policy learning may be a better way to conceptualize the function of the DLS, as was proposed in previous studies . Instead of finding an optimal solution seen in policy gradient methods , positively-biased TD errors in DLS may directly reinforce taken actions as proposed originally , thus preserving the law of effect but also emphasize the law of exercise.”4) It would be expected that the model would make a much larger distinction between the medium reward in the BIG than the SMALL condition than was apparent in the data. Indeed, one of the puzzles for me about the paper was the tiny effect of the BIG vs SMALL side  – why might that be? Would the authors' previous results not predict a larger difference?This is a good point. In retrospect, this might be due to suboptimal conditions of the current design with respect to our ability to observe the effect of BIG and SMALL expectations on medium-sized reward. The range of reward is 0 to 2.2 L, whereas the expectations differ by less than the half of the range (1.0 L). Relative to the expectation, the expected positive and negative RPEs are -0.7 (medium reward in BIG) and +0.3 (medium reward in SMALL). In our typical experiments, reward responses are reduced by approximately the half of the response of the full reward response even when the reward is fully expected , likely due to temporal uncertainty of reward timing . Therefore, the relatively small variation  into a value-based learning with probabilistic reward manipulation using 2 sets of different sizes of water. Although the task was demanding to mice, we were able to detect RPE natures in dopamine axon activity without over-training. However, the difference of prediction BIG versus SMALL sides was still small. Further, most analyses relied on pooled data across sessions because of the limited number of trials in each trial type, especially error trials. Further improvement of the task will facilitate more quantitative analyses over learning.”5) The FP data shows three peaks in DA in correct trials, whereas the model only really shows two. The second peak, associated with leaving the sampling well is important – particularly for the DMS – could the authors comment on its absence in the model?eLife paper to address this question, and concluded that movement-related activity in DMS-projecting dopamine neurons are not a part of RPE. We did not see movement-related activity in TD errors in our models, consistent with this idea. We added a citation of this paper.The second peak corresponds to the peak we identified in Figure 2C “Movement”. Nathaniel Daw devoted a whole “Third, interestingly, however, our results showed consistent deviation from what TD model predicts. As reported previously , during choice movements, contra-lateral orienting movements caused a transient activation in the DMS, whereas this response was negligible in VS and DLS. As pointed out in a previous study , this movement-related activity in DMS is unlikely to be a part of RPE signals.”6) Further analysis is required from the transition between block 1 and block 2 and the speed of adaptation could provide useful cues about the employment of goal-directed or habitual values, and distinguish DMS and DLS.In this manuscript, we are proposing that recipients of dopamine signals (neurons in the striatum) potentially learn in a slightly different way, because excitation threshold of TD error is slightly different across striatal areas. Thus, neurons in DLS may learn habit, not DLS-projecting dopamine neurons, themselves. To distinguish roles of dopamine and the striatum in our proposals more clearly, we added an illustration in a new figure, Figure 8D.The difference in learning speed in dopamine neurons is an interesting question and we should design tasks to parametrically test their speed. Our task has limitations as we addressed/discussed in the answer to question 4. We explicitly added a sentence about the learning:“Further improvement of the task will facilitate more quantitative analyses over learning.”Nevertheless, we compared effects of prediction BIG versus SMALL in dopamine axon responses to water as in Figure 4D, but separated trials in early and late sessions . We did not observe clear differences across striatal areas .7) It is unclear why the data is combined across regions in showing responses to the odor or choice. Is the only time these signal diverge during reward delivery? I think it would significantly increase the novelty and impact of the findings if the authors could focus on representing the differences between these regions in their figures , and conduct the analyses in relation to the different regions . What are the parameters that change when data is sorted out this way? The claim that these data all line up a TD error simply cannot be the case, because DLS does not show the hallmark dip in activity to an unexpected reward that would be expected of a TD error.We observed different dopamine axon activities across the striatum not only during reward delivery but also at cue responses . We apologize that we did not have average traces at different time windows as in Figure 4A in the previous version. We have added average traces in each area for both before choice and after choice in Figure 6—figure supplement 1.Our main point is that other than the lack of “dip”, other responses are consistent with TD errors which correspond to changes in state values. Following the original definition of TD errors, the dip is just one aspect. To produce different zero-crossing points across the striatal areas, we propose 3 different possibilities. First, the difference might come from the same mechanism as the distributional RL . Some dopamine neurons might be optimistic and some are pessimistic. The model is fully described in the published paper, and how to combine this idea and TD is a hot topic in the field. Second, DLS-projecting dopamine neurons may receive something additional whenever animals obtain a positive outcome as "success premium", although we do not have an anatomical clue for this mechanism. Third, which is not exclusive with the first idea, dopamine subpopulations with different projection targets may receive different ratios of inputs, some of which represent already partially RPE . In this previous study, we observed many presynaptic neurons with RPE, but RMTg uniquely showed a flipped version of RPE (the sign is opposite to dopamine RPE). Interestingly, inactivation of RMTg only affects negative RPE in dopamine neurons although RMTg show both negative and positive RPE , suggesting that excitation of these neurons is more important. Further, RMTg shows a topographic projection pattern to dopamine neurons . From these results, we propose that different ratios of inputs to dopamine neurons cause different zero-crossing points in dopamine neurons. To illustrate, we added a simple model with only two inputs in Figure 8A-C.“Mechanistically, biases in dopamine signals may stem from a difference in the excitation-inhibition balance at the circuit level. […] It will be fascinating if we can connect all these levels of studies into functional meaning in the future.”8) The claim is made that DLS activity is consistent with the "law of exercise" proposed by Thorndike, encoding how to perform motor sequences. However, these data are also consistent with this region encoding the reward outcome as a stimulus . Alternatively, it could be encoding the "long run history" of rewards (and impervious to single reductions in reward magnitude), which is consistent with current theories of habitual learning. Therefore, suggesting changes to existing theories on the basis of a single characteristic of neural activity in a limited data set, without a behavioral test that would facilitate such an understanding is not advisable. In fact, this can be removed with no detrimental effect to the paper.There are several points we have to clarify here. First, although "a limited data set" was blamed here, our data is the first dataset which examined potential differences of teaching signals by dopamine in different striatal areas, utilizing parametric tasks. In other words, existing theories are not based on actual dopamine activity in DLS, nor did they propose that the lack of “dip” may be related to habit learning. We reiterate that although the current task is not designed specifically to test habitual behaviors, dopamine signals in DLS may still be providing permissive reinforcement signals. Second, the idea that DLS plays an important role in habit/skill is not our new proposal, but already supported by substantial evidence in the past, most notably by studies involving lesions and pharmacological manipulations in DLS. Our speculation that the lack of dip in DLS dopamine signals may underlie habit learning builds on this rich literature pointing the importance of DLS in habit and skill and the present result. We clarified these two points in Discussion.We appreciate that the reviewers agree that our data is valid even without the discussion that we made. As we answered to question 1, we more emphasized our findings themselves, and deleted most of the discussion about sequential behaviors. We would like to respectfully argue, however, that our discussion on habit learning through DLS dopamine signals is a natural logical speculation that one can make based on the observed results and the existing literature. Our goal is to contextualize our observations in existing literature and make proposals for predictions, future directions and future experiments, which are one of the most important goals of Discussion section in original research paper, and our discussions are well-based on the data.That habit learning follows "law of exercise" or repetition/training is not a new idea and well accepted in wide fields. In spite of this well-known phenomenon, current popular computational theories often use value-based, model-free reinforcement learning to model habit. To challenge these theories, several models were proposed recently, and one of these  uses action history as a teaching signal, which sometimes approximates long run history of rewards because animals tend to repeat rewarded actions. As Miller et al. emphasized, this way of computation allowed habit learning to follow law of exercise, although this computational model was purely theoretical, with no underlying neuronal mechanism. Our proposal fits well with the idea of learning from repetition. While Miller et al. totally removed effects of reward, our proposal is a fusion, based on observation of dopamine activity. Dopamine axon activity in DLS was sensitive to water amounts, and did not show excitation with mere action with no outcome. However, it showed strong excitation even with small reward. Thus, dopamine in DLS can function as a teaching signal emphasizing law of exercise, but not ignoring reward, or value-less.Existing computational models have problems and that is why these recent studies are trying to revise models for habit. However, these new models do not have a neuronal basis. On the other hand, there is a history of manipulation studies that showed the importance of DLS and dopamine in habit/skill. The present study builds on these literatures and our finding in DLS dopamine signals bridges the gap between these behavioral and computation studies.There are many existing theories to explain DLS function. One of them is reinforcement learning theory, assuming that dopamine activity is uniform across the striatum including DLS. Here, we designed a task to propose the potential function of dopamine in DLS in reinforcement learning, based on actual recording data. We discuss what positively-biased teaching signals can affect learning in downstream areas. An important point here is that learning with balanced-TD error can converge to value, whereas learning with biased TD error does not, with simple reinforcement learning. Specifically, positively-biased TD error would cause reinforcement by repetition. We believe that this is a natural logical speculation that one can draw based on both the observed data, and well-established reinforcement learning theory.Now a totally different question is what can produce positively-biased TD error-like signals. Because this question pertains to the next step, we have a different section in Discussion subtitled “Potential mechanisms underlying diverse TD error signals”As we discussed at question 7, we discuss more than three possibilities. Now, the possibility of sensory prediction error is additionally raised here. We have not seen any data showing that dopamine in DLS encodes sensory prediction error, nor an existing theory to connect sensory prediction error and habit. We showed previously that activity of TS-projecting dopamine neurons uniquely co-varies with stimulus intensity , but TS is a distinct area from DLS. Another proposal here is long run history of reward in dopamine neurons. Single unit recording data indicated that dopamine neurons in the VTA encoded longer time-scale of reward value, compared to lateral SNc , in contrast to the reviewers' suggestion.9) The fact that these data can be modeled in a TD framework does not mean dopamine terminals are acting as a model-free TD error. This is because there is no test outside this hypothesis. There is nothing other than reward "value" that is manipulated. And dopamine neurons are capable of acting outside a model-free TD error. This has been demonstrated many times by many different labs in the past 3-5 years; all of this should be noted and discussed. Indeed, dopamine neurons in VTA respond to unexpected changes in reward identity , and stimulation of dopamine neurons drives learning between neutral cues without endowing them with "model-free" value . This is not predicted by a model-free TD signal. Indeed, similar "model-based" findings have been found in ventral striatum .Here we do not see any disagreement and we were somewhat puzzled why the reviewers raised this as a major issue. We completely agree with the reviewers that “the fact that these data can be modeled in a TD framework does not mean dopamine terminals are acting as a model-free TD error”. In the manuscript, we do not make any claim as to whether the observed dopamine signals are model-free or model-based. If our manuscript gave even slight impression that we are claiming that dopamine neurons only take into account model-free information, that would be our mistake and we would like to correct it. However, we would need to know which of our statements invoked such a concern before responding to it.It is clear that, based on previous studies , dopamine neurons have access to model-based information or some form of inference using a model (or task states), and these conclusions were drawn by experiments designed to address this specific question. The present experiments are not designed for that, but to distinguish specific variables previously proposed to co-vary with dopamine signals such as RPE (TD error), sensory evidence, and reward values. Because computation of any of these variables may involve model-free or model-based mechanisms, our experiments have no power for distinguishing model-free versus model-based computations. Generally, we are reluctant to over-simplify dopamine activity by generalizing dopamine coding as always model-free versus always model-based. We never did that. Indeed, distinguishing model-based and model-free is generally very difficult even when a task is designed to do so. The logic is the other way around. Reviewers raised the point that dopamine activity following a primitive rule does not rule out dopamine's access to higher information. We totally agree. However, it is important to seek for a most parsimonious way to explain data, let alone a predominant theory or model that has long existed in the field. The significance of the present study is that we provide a parsimonious and unified explanation of dopamine signals in a task, with which some previous studies proposed more complex interpretations of dopamine signals. In the same logic, our results do not completely dispute the previous explanation. However, we believe that it does raise a question in the validity of the previous explanations.To avoid reader's confusion, we added a new discussion in subsection “Confidence and TD errors”As we emphasize in this section, our aim is not to dissociate model-based versus model-free function. We added a potential interaction in the last paragraph:“Whereas our model takes a primitive strategy to estimate state value, state value can be also estimated with different methods. […] Together, our results and these models underscore the importance of considering moment-by-moment dynamics, and underlying computation.”10) Please rephrase your interpretation as follows: " our modeling approach is consistent with activity at DA terminals functioning within the lines of a TD error framework in this particular reward-learning task, while we acknowledge that it is likely these signals are also acting outside of this framework in other tasks".Thank you for the suggestion. We believe that in the previous manuscript, we clearly stated that there are some movement-related activity consistent with a previous study . This activity is clearly not a TD error, as analyzed in their subsequent study . In this sense, we did more than stating “these signals are also acting outside of this framework in other tasks”. Our data showed non-TD error signals in the present task.Furthermore, we believe that “acting” is a misleading term because the present study primarily addresses the nature of signals or the information conveyed by these signals (i.e. the question of what is represented), rather than what they do (which may be implied by “acting”). It is our position to distinguish the question of representation (i.e. what information they convey) from functions (i.e. what they do). As we have discussed before , TD errors can function as reinforcement as well as a regulator of motivation  or a regulator of excitability of downstream neurons (as opposed to plasticity). We do not make any claim that dopamine signals do not function outside the traditional reinforcement learning framework.Reviewers raised a question about generality of our findings. As we repeated, our main findings are three points.1) Similarity of dopamine RPE signals in decision-task2) Difference of baseline for the RPE3) Correlates of confidence can naturally emerge in primitive TD error without additional variables "confidence"We strengthened our discussion of each part to avoid reader's confusion. Related to the answers above, we emphasized the distinction between our conclusions and discussion, repeating our aims several times. We also added more results and discussion about movement-related activity.11) It is crucial to report whether animals engage in licking in the water (or sample) port before the reward is provided? Does this covary with model 'value' and confidence? Does it covary with FP signals?We agree on the importance of this point for two reasons. First, such measurements would confirm that mouse behaviors actually co-vary with confidence in our tasks. Second, more behavioral measurements can clarify if dopamine signals might be driving actions or vice-versa. Because we did not have a lick sensor in our freely-moving setup yet, we did not detect licks. To address these two questions, we first measured a leave frequency across different confidence levels to examine whether mice show behaviors modulated by confidence. In fact, we confirmed that a leave frequency was slightly, but significantly increased when mice made a choice error compared to when they made a correct choice, consistent with a previous report . Because the leave frequency was low overall, and we used a fixed water delay, we could not breakdown trial types further. We added this results in Figure 1H.“Animals were required to stay in a water port for 1s to obtain water reward. However, in rare cases, they exited a water port early, within 1s after water port entry. We examined the effects of choice accuracy (correct or error) on the premature exit . We found that while animals seldom exited a water port in correct trials, they occasionally exited prematurely in error trials, consistent with a previous study .”“Since animals occasionally exit the water port prematurely in error trials , we performed the same analyses as Figure 6C excluding trials where animals exited the water port prematurely. The results, however, did not change .”To address potential movement signals, we also compared reaction time (odor port in ~ odor port out) or movement time (odor port out ~ water port in) and dopamine axon activity. First, reaction time was not correlated with dopamine axon signals. The result is shown in Figure 7—figure supplement 3. Second, while we found slight modulation of dopamine axon signals by movement time, movement time was not correlated with sensory evidence, and cannot explain dopamine axon activity patterns. We present this new result in Figure 7—figure supplement 4. Finally, to examine effects of any gross movement on dopamine axon signals, we performed video analyses using DeepLabCut, and examined relationship between GCaMP signals, body movement and control fluorescence (tdTom) signals. We found that body movement and tdTom signals cannot explain dopamine axon activity patterns. These results are shown in Figure 7—figure supplement 6.The manuscript has been improved but there are some remaining issues that need to be addressed before acceptance, as outlined below:In subsection “Similarity of dopamine axon signals across the striatum” paragraph three, optimism in Q learning should not be described without qualification. The fact that Q learning uses max_b Q(s(t+1),b) at the next state (s(t+1)) is not thought of being borne of optimism – this in fact reflects what the subject thinks it could do at that state . Confusion may arise in that optimistic Q values are associated with the motivation to explore. If the authors wish to include the assertion about Q learning, they should expand on their exact point.Thank you for indicating the potential confusion that may arise from our phrasing. Although we did not explore action values in this study, higher criteria in Q-learning and in dopamine axon signals in DMS potentially suggest a common feature in learning such as function to push toward optimal actions (or state transitions). Since Q-learning and optimistic dopamine neurons have never been connected, we hoped that our data may encourage this new idea. At this moment, however, we totally agree with the reviewer’s suggestion, and we now explicitly mention that we have not found connection yet because the meaning of optimism may not be exactly the same.“Optimistic expectation echoes with the idea of Watkin's Q-learning algorithm  where an agent uses the maximum value among values of potential actions to compute RPEs, although we did not explore action values explicitly in this study. Future studies are needed to find the functional meaning of optimism in dopamine neurons and to examine whether the optimism is responsible for a specific learning strategy in DMS.”The" dip" in the signal is important for the TD error. And if the proxy for the DLS signals is "more excitation" represented in Figure 8, this should still predict a dip in the DLS . However, signals in DLS still show an increase in activity from baseline during water omission. This should be clarified.As the reviewer points out, the difference in “baseline” can be regarded as a departure from the original TD errors. However, the overall dynamics are similar among the three regions of the striatum. Our conclusion that dopamine signals in all three areas is similar to TD errors is based on a more holistic approach, considering temporal dynamics associated with various events in a trial. Although the overall positive or negative reward responses of DLS and DMS dopamine axons, respectively are important deviations from TD errors, it would be important to point out similarity as well as the difference. We believe that our manuscript unambiguously demonstrates the difference . To clarify these points, we added the following paragraph in the first section of Discussion.“The positively or negatively biased reward responses in DLS and DMS can be regarded as important departures from the original TD errors, as it was originally formulated . […] In any case, the different baselines in TD error-like signals that we observed in instrumental behaviors can provide specific constraints on the behaviors learned through dopamine-mediated reinforcement in these striatal regions.”We did not observe clear excitation or inhibition time-locked at water timing in error trials in dopamine axons in DLS. Whether they never show detectable inhibition with actual water omission is important to address in future.“It is important to examine whether these dopamine neurons show detectable inhibition with large negative prediction error such as actual reward omission in an easy task, as the model predicts.”The authors propose that the observed signals look like the scalar difference between the expected an observed value (with the exception of the DLS), but are not necessarily used to attach this scalar value product to the antecedent cue or action. This should be specifically and explicitly stated in the text as it is not necessarily scalar value that attaches to antecedent events, which is alluded to by referencing and modeling using the model-free TD algorithm.Our main conclusion is that dopamine signals differ across different populations. Thus, our results clearly indicate that dopamine signals are not a scaler value, and add to the increasing literature indicating the heterogeneity of dopamine signals. It is, therefore, unclear why the reviewer thought that we are proposing a scalar dopamine signal. That is clearly not what we are proposing.The idea that TD error equals a scalar signal has to be abandoned and this study is not the first to indicate that. It is also important to notice that there has been increasing appreciation that dopamine signals can vary while these signals are still consistent with TD errors. Therefore, the observation that dopamine signals are on average consistent with TD errors does not indicate that dopamine signals are a scaler value. Recent studies have started exploring the significance of variability in TD errors. That is, vector-type TD errors can provide computational advantages, for example, through distributional reinforcement learning . In these ideas, the important point is that the variation can exist within TD errors, and the variation in TD errors can garner certain computational powers. We believe that the present results indeed extend these ideas by showing that a type of variation can enable learning of different types of behavior.Temporal difference error is defined by “To examine how the value and RPE may change within a trial, we employed a Monte-Carlo approach to simulate an animal’s choices assuming that the animal has already learned the task. We used a Monte-Carlo method to obtain the ground truth landscape of the state values over different task states, without assuming a specific learning algorithm.”In other words, we are agnostic to how values :“The positively or negatively biased reward responses in DLS and DMS can be regarded as important departures from the original TD errors, as it was originally formulated . […] In any case, the different baselines in TD error-like signals that we observed in instrumental behaviors can provide specific constraints on the behaviors learned through dopamine-mediated reinforcement in these striatal regions.”We also expanded future directions related to learning in Discussion.“In this study, we modeled dynamical representation patterns of dopamine neurons in a steady state, but did not examine relationship between dopamine activity and actual learning. Especially, while our model used a discrete single stimulus state in each trial, it is naturalistic that animals use the experience from a single trial to update value in other stimulus states , and/or animals represent states in a more continuous manner . It is important in the future to examine how dynamical and diverse dopamine signals are used during learning and/or performance.”Please include summary statistics in additional to P values where appropriate .We added a supplementary source file in each figure to show summary statistics. We noticed that we had used dF/F for some of statistics in previous versions. To be consistent throughout the paper, now we use z-score in all the statistics and clearly state it in the summary statistics.


Unless the molecular axis is fixed in the laboratory frame, intrinsic structural information of molecules can be averaged out over the various rotational states. The macroscopic directional properties of polar molecules have been controlled by two fs pulses with an optimized delay. In the method, the first one-color laser pulse provokes molecular alignment. Subsequently, the molecular sample is irradiated with the second two-color laser pulse, when the initial even—J states are aligned, and the odd—J states are anti-aligned in the thermal ensemble. The second pulse selectively orients only the aligned even—J states in the same direction, which results in significant enhancement of the net degree of orientation. This paper reports the results of simulations showing that the two-pulse technique can be even more powerful when the second pulse is cross-polarized. This study shows that the alignment and orientation can be very well synchronized temporally because the crossed field does not disturb the preformed alignment modulation significantly, suggesting that the molecules are very well confined in the laboratory frame. This cross-polarization method will serve as a promising technique for studying ultrafast molecular spectroscopy in a molecule-fixed frame. The most widely-adopted technique is a transient alignment achieved by irradiating a fs laser pulse, so-called nonadiabatic (impulsive) alignment technique6. Because a fs laser pulse is short enough compared to the typical rotational speed of a molecule  field, was theoretically proposed32 and experimentally realized36. While the state-selection has provided the highest degree of orientation thus far, the all-optical two-pulse technique is more suitable for the high-order harmonic generation experiment because of the available high target density35.These methods themselves, however, require additional specific efforts to increase the degree of orientation measured from a molecular ensemble. This is because some of the molecules are oriented in one direction, and the others are oriented in the opposite direction in the presence of the orientation potential. Such a limit to the achievable macroscopic orientation has been addressed in many reports. In recent studies36, all the laser fields were polarized linearly in the same directions, and more complex polarizations have not been studied. To orient general polyatomic (asymmetric-top) molecules, it is necessary to confine two molecular axes in the laboratory frame. Therefore, elliptical37 and crossed polarization38 have been employed for the three-dimensional orientation of asymmetric-top molecules. Recently, the interaction Hamiltonian induced by the crossed polarization of a two-color field was derived, and it was demonstrated that it could enhance the degree of orientation of linear molecules in the quasi-adiabatic condition39.In previous reports on the orientation of linear moleculesIn this work, we show that a two-color field with crossed polarization can improve the field-free orientation of linear molecules, by solving the time-dependent Schrödinger equation (TDSE) numerically in the short-pulsed crossed polarization regime for the first time. A series of simulations reveal that the two-pulse orientation technique based on the linear polarization, in fact, deteriorates the degree of alignment of the molecular axis in the laboratory frame, even though the oriented molecular sample has a large up-down asymmetry (high degree of orientation). This is because the preformed molecular alignment by the first one-color pulse can be disturbed significantly by the second pulse. On the other hand, this problem can be circumvented by employing the crossed polarization. In the case of the second pulse with crossed polarization, the molecular orientation is generated without significantly destroying the preformed alignment modulation. As a result, modulations of the alignment and orientation are temporally well synchronized, ensuring the tight angular confinement of the molecular ensemble.B and 39:Z-aixs, andX-axis and the Z-axis. Here, z-axis and the laboratory-fixed Z-axis are represented by Z-axis. Meanwhile, the crossed polarization can orient the molecule in the Z-axis, and the alignment direction can be either Z- or X-axis depending on which field component is stronger than the other one39. We have solved the TDSE using the Spherical harmonics as a basis set. While under the parallel field condition the quantum number M is preserved, but it is not a good quantum number any-more under the crossed field condition. Since the reflection symmetry in the ZX-plane remains, it is possible to separate the total Hamiltonian by two irreducible representations. The symmetry issue and the numerical method for solving the TDSE are provided elsewhere40. In this work, the OCS molecule has been benchmared as a prototype, which has constants of 41. The rotational temperature of the molecular ensemble is set to 3 K. The molecules are irradiated with a two-color laser with the wavelengths of 800 nm and 400 nm. In this condition, the laser frequencies are far from any molecular resonance and much higher than the rotational frequency of molecule.We have used the popular rigid rotor model to describe the rotational dynamics, in which the Hamiltonian of a linear molecule Z-axis is irradiated with the molecular ensemble. The peak intensity and the pulse width are Z-axis. In Fig. A two-color pulse collinearly-polarized in the mentclass2pt{minim42. Therefore, the even or odd states can be oriented selectively by controlling the delay of the two-color pulse. As shown in Fig. 32 and experimentally35After irradiation with the first one-color pulse having the peak intensity and pulse width of entclass1pt{minimaentclass1pt{minima43. In the orientation methods, however, the two-color pulse counteracts the alignment method. That is, the second pulse orients the even-J states selectively, and at the same time, it also partially cancels and disturbs the preformed alignment of the even–J states. The high degree of orientation, On the other hand, by irradiating the two-color pulse, the preformed well-aligned status can be remarkably disturbed, as can be confirmed in Fig. entclass1pt{minimaentclass1pt{minimaZX-plane, which is equivalent to the anti-alignment against the Y-axis. At the same time, the two-color pulse can generate the orientation of the molecule in the Z-direction. We note that the orientation potential in the Z-direction, which is much weaker than the alignment potential, can play a role only when the molecules are aligned in the Z-axis39. Therefore, without destroying the preformed alignment in the Z-axis, the orientation can be generated in the temporally synchronized way with respect to the alignment. Furthermore, the odd–J states have much smaller degrees or alignment and negligible degrees of orientation in the Z-direction. As a consequence, at Figure 44, we evalulate the up-down asymmetry of the ionization probability. For the wavefunctions shown in Figs. Z-axis, the effective up-down asymmetry and the anisotropic response of the molecular sample can be more enhanced, which will depend on the target molecule.The two different polarization configurations provide similar molecular orientation in terms of the achieved maximum macroscopic degree of orientation entclass1pt{minimaentclass1pt{minimaentclass1pt{minimaZ-direction, and the odd–J states are less oriented in the opposite direction at Z- and X- axes respectively, and only the even–J states are well oriented in the Z-direction. Most linear molecules have anisotropic polarizability and ionization probability, which are maximized in the molecular axis. Therefore, the X-aligned odd—J states would not be probed by a linearly polarized laser pulse and most of the observation can be made from the tightly aligned and oriented even—J states. This study confirms the advantages of the crossed-polarization technique over the collinearly-polarization technique: temporally synchronized higher degree of orientation and alignment. Because the maximum laser intensity for the molecular orientation is limited by the ionization threshold of the target molecule, a much higher laser intensity can be used because the instantaneous field strength 46, to the unidirectional rotation of the oriented molecules. The experimental studies on the molecular structure and many molecular-anisotropy-based phenomena will benefit considerably from this method.A high degree of orientation (the plus-minus inversion asymmetry) does not necessarily ensure that the molecules are tightly aligned. In the collinear polarization configuration, even–J states are well oriented in the