Supervised learning, more specifically Convolutional Neural Networks (CNN), has surpassed human ability in some visual recognition tasks such as detection of traffic signs, faces and handwritten numbers. On the other hand, even state-of-the-art reinforcement learning (RL) methods have difficulties in environments with sparse and binary rewards. They requires manually shaping reward functions, which might be challenging to come up with. These tasks, however, are trivial to human. One of the reasons that human are better learners in these tasks is that we are embedded with much prior knowledge of the world. These knowledge might be either embedded in our genes or learned from imitation - a type of supervised learning. For that reason, the best way to narrow the gap between machine and human learning ability should be to mimic how we learn so well in various tasks by a combination of RL and supervised learning. Our method, which integrates Deep Deterministic Policy Gradients and Hindsight Experience Replay (RL method specifically dealing with sparse rewards) with an experience ranking CNN, provides a significant speedup over the learning curve on simulated robotics tasks. Experience ranking allows high-reward transitions to be replayed more frequently, and therefore help learn more efficiently. Our proposed approach can also speed up learning in any other tasks that provide additional information for experience ranking.
In biological systems, experience replay has played an important role for learning behaviors. Various studies show that there is frequent experience replay in the hippocampus of rodents of both awake and sleeping animals. Research also shows that disrupting experience replay might lead to impairing spatial memory @cite_11 @cite_9 , and some show that experience replay can contribute to better memory consolidation and retrieval @cite_13 , both are instrumental in boosting learning ability.
@cite_9 @cite_13 @cite_11
1974367810 1900041959 2963477884 2062205907
Replay is the sequential reactivation of hippocampal place cells that represent previously experienced behavioral trajectories. Although first studied during sleep, recent work suggests that replay occurs frequently in the awake state and could be a potential substrate for memory consolidation and retrieval. The consolidation of memories for places and events is thought to rely, at the network level, on the replay of spatially tuned neuronal firing patterns representing discrete places and spatial trajectories. This occurs in the hippocampal-entorhinal circuit during sharp wave ripple events (SWRs) that occur during sleep or rest. Here, we review theoretical models of lingering place cell excitability and behaviorally induced synaptic plasticity within cell assemblies to explain which sequences or places are replayed. We further provide new insights into how fluctuations in cholinergic tone during different behavioral states might shape the direction of replay and how dopaminergic release in response to novelty or reward can modulate which cell assemblies are replayed. Abstract: Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games. The hippocampus plays a key role in the acquisition of new memories for places and events. Evidence suggests that the consolidation of these memories is enhanced during sleep. At the neuronal level, reactivation of awake experience in the hippocampus during sharp-wave ripple events, characteristic of slow-wave sleep, has been proposed as a neural mechanism for sleep-dependent memory consolidation. However, a causal relation between sleep reactivation and memory consolidation has not been established. Here we show that disrupting neuronal activity during ripple events impairs spatial learning. We trained rats daily in two identical spatial navigation tasks followed each by a one-hour rest period. After one of the tasks, stimulation of hippocampal afferents selectively disrupted neuronal activity associated with ripple events without changing the sleep-wake structure. Rats learned the control task significantly faster than the task followed by rest stimulation, indicating that interfering with hippocampal processing during sleep led to decreased learning.