Reservoir computing is a powerful tool to explain how the brain learns temporal sequences, such as movements, but existing learning schemes are either biologically implausible or too inefficient to explain animal performance. We show that a network can learn complicated sequences with a reward-modulated Hebbian learning rule if the network of reservoir neurons is combined with a second network that serves as a dynamic working memory and provides a spatio-temporal backbone signal to the reservoir. In combination with the working memory, reward-modulated Hebbian learning of the readout neurons performs as well as FORCE learning, but with the advantage of a biologically plausible interpretation of both the learning rule and the learning paradigm. Learning complex temporal sequences that extend over a few seconds -such as a movement to grab a bottle or to write a number on the blackboard -looks easy to us but is challenging for computational brain models. A common framework for learning temporal sequences is reservoir computing (alternatively called liquid computing or echo-state networks) [1, 2, 3] . It combines a reservoir, a recurrent network of rate units with strong, but random connections [4] , with a linear readout that feeds back to the reservoir. Training of the readout weights with FORCE, a recursive least-squares estimator [1] , leads to excellent performance on many tasks such as motor movements. The FORCE rule is, however, biologically implausible: update steps of synapses are rapid and large, and require an immediate and precisely timed feedback signal. A more realistic alternative to FORCE is the family of reward-modulated Hebbian learning rules [5, 6, 7] , but plausibility comes at a price: when the feedback (reward minus expected reward) is given only after a long delay, reward-modulated Hebbian plasticity is not powerful enough to learn complex tasks. Here we combine the reservoir network with a second, more structured network that stores and updates a two-dimension continuous variable as a "bump" in an attractor [8, 9] . The activity of the attractor network acts as a dynamic working memory and serves as input to the reservoir network ( fig. 1 ). Our approach is related to that of feeding an abstract oscillatory input [10] or a "temporal backbone signal" [11] into the reservoir in order to overcome structural weaknesses of reservoir computing that arise if large time spans need to be covered. In computational experiments, we show that a dynamic working memory that serves as an input to a reservoir network facilitates reward-modulated Hebbian learning in multiple ways: it makes a biologically plausible three-factor rule as efficient as FORCE; it admits a delay in the feedback signal; and it allows a single reservoir network to learn and perform multiple tasks. We showed that a dynamic working memory can facilitate learning of complex tasks with biologically plausible three-factor learning rules. Our results indicate that, when combined with a bump attractor, reservoir computing with reward-modulated learning can be as efficient as FORCE [1] , a widely used but biologically unrealistic rule. The proposed network relies on a limited number of trajectories in the attractor network. To increase its capacity, a possible future direction would be to combine input from the attractor network with another, also input-specific, but transient input that would bring the reservoir into a different initial state. In this case the attractor network would work as a time variable (as in [9] ), and the other input as the control signal (as in [1] ). Apart from the biological relevance, the proposed method might be used for real-world applications of reservoir computing (e.g. wind forecasting [13] ) as it is computationally less expensive than FORCE. It might also be an interesting alternative for learning in neuromorphic devices.
We show that a working memory input to a reservoir network makes a local reward-modulated Hebbian rule perform as well as recursive least-squares (aka FORCE)
{"src": "scitldr-AIC_train", "paper_id": "B1g0QmtIIS"}