A second such study comes from [1]}. This leveraged a deep RL technique developed in recent AI work, and referred to as distributional RL [2]}. Earlier, in discussing the history of deep RL, we mentioned the reward-prediction error or RPE. In conventional RL, this signal is a simple scalar, with positive numbers indicating a positive surprise and negative ones indicating disappointment. More recent neuroscientifically inspired models have suggested that accounting for the distribution and uncertainty of reward is important for decision-making under risk [3]}. In distributional RL, the RPE is expanded to a vector, with different elements signaling RPE signals based on different a priori forecasts, ranging from highly optimistic to highly pessimistic predictions (Figure REF a,b). This modification had been observed in AI work to dramatically enhance both the pace and outcome of RL across a variety of tasks, something – importantly – which is observed in deep RL, but not simpler forms such as tabular or linear RL (due in part to the impact of distributional coding on representation learning [4]}). Carrying this finding into the domain of neuroscience, Dabney and colleagues studied electrophysiological data from mice to test whether the dopamine system might employ the kind of vector code involved in distributional RL. As noted earlier, dopamine has been proposed to transmit an RPE-like signal. Dabney and colleagues obtained strong evidence that this dopaminergic signal is distributional, conveying a spectrum of RPE signals ranging from pessimistic to optimistic (Figure REF c).
<FIGURE>