Memory-efficient Reinforcement Learning with Knowledge Consolidation
Artificial neural networks are promising as general function approximators
but challenging to train on non-independent and identically distributed data
due to catastrophic forgetting. Experience replay, a standard component in deep
reinforcement learning, is often used to reduce forgetting and improve sample
efficiency by storing experiences in a large buffer and using them for training
later. However, a large replay buffer results in a heavy memory burden,
especially for onboard and edge devices with limited memory capacities. We
propose memory-efficient reinforcement learning algorithms based on the deep
Q-network algorithm to alleviate this problem. Our algorithms reduce forgetting
and maintain high sample efficiency by consolidating knowledge from the target
Q-network to the current Q-network. Compared to baseline methods, our
algorithms achieve comparable or better performance on both feature-based and
image-based tasks while easing the burden of large experience replay buffers.