Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
In deep reinforcement learning (RL), data augmentation is widely considered
as a tool to induce a set of useful priors about semantic consistency and
improve sample efficiency and generalization performance. However, even when
the prior is useful for generalization, distilling it to RL agent often
interferes with RL training and degenerates sample efficiency. Meanwhile, the
agent is forgetful of the prior due to the non-stationary nature of RL. These
observations suggest two extreme schedules of distillation: (i) over the entire
training; or (ii) only at the end. Hence, we devise a stand-alone network
distillation method to inject the consistency prior at any time (even after
RL), and a simple yet efficient framework to automatically schedule the
distillation. Specifically, the proposed framework first focuses on mastering
train environments regardless of generalization by adaptively deciding which
{\it or no} augmentation to be used for the training. After this, we add the
distillation to extract the remaining benefits for generalization from all the
augmentations, which requires no additional new samples. In our experiments, we
demonstrate the utility of the proposed framework, in particular, that
considers postponing the augmentation to the end of RL training.