Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum
Despite considerable advances in deep reinforcement learning, it has been
shown to be highly vulnerable to adversarial perturbations to state
observations. Recent efforts that have attempted to improve adversarial
robustness of reinforcement learning can nevertheless tolerate only very small
perturbations, and remain fragile as perturbation size increases. We propose
Bootstrapped Opportunistic Adversarial Curriculum Learning (BCL), a novel
flexible adversarial curriculum learning framework for robust reinforcement
learning. Our framework combines two ideas: conservatively bootstrapping each
curriculum phase with highest quality solutions obtained from multiple runs of
the previous phase, and opportunistically skipping forward in the curriculum.
In our experiments we show that the proposed BCL framework enables dramatic
improvements in robustness of learned policies to adversarial perturbations.
The greatest improvement is for Pong, where our framework yields robustness to
perturbations of up to 25/255; in contrast, the best existing approach can only
tolerate adversarial noise up to 5/255. Our code is available at:
https://github.com/jlwu002/BCL.