@inproceedings{
masset2023multitimescale,
title={Multi-timescale reinforcement learning in the brain},
author={Paul Masset and Pablo Tano and HyungGoo Kim and Athar N. Malik and Alexandre Pouget and Naoshige Uchida},
booktitle={UniReps:  the First Workshop on Unifying Representations in Neural Models},
year={2023},
url={https://openreview.net/forum?id=2ApmouuUOC}
}
To thrive in complex environments, animals and artificial agents must learn to act adaptively to maximize fitness and rewards. Such adaptive behavior can be learned through reinforcement learning1, a class of algorithms that has been successful at training artificial agents and at characterizing the firing of dopamine neurons in the midbrain. In classical reinforcement learning, agents discount future rewards exponentially according to a single time scale, known as the discount factor. This strategy is at the odds  with the empirical observation that humans and animals use non-exponential discounts in many situations. Here, we explore the presence of multiple timescales in biological reinforcement learning. We first show that reinforcement agents learning at a multitude of timescales possess distinct computational benefits. Next, we report that dopamine neurons in mice performing two behavioral tasks encode reward prediction error with a diversity of discount time constants. Our model explains the heterogeneity of temporal discounting in both cue-evoked transient responses and slower timescale fluctuations known as dopamine ramps. Crucially, the measured discount factor of individual neurons is correlated across the two tasks suggesting that it is a cell-specific property. Together, our results provide a new paradigm to understand functional heterogeneity in dopamine neurons, and open new avenues for the design of more efficient reinforcement learning algorithms.
Multi-timescale reinforcement learning in the brain
Paul Masset Pablo Tano HyungGoo Kim Athar N. Malik Alexandre Pouget Naoshige Uchida