Attacking and Defending Deep Reinforcement Learning Policies
Recent studies have shown that deep reinforcement learning (DRL) policies are
vulnerable to adversarial attacks, which raise concerns about applications of
DRL to safety-critical systems. In this work, we adopt a principled way and
study the robustness of DRL policies to adversarial attacks from the
perspective of robust optimization. Within the framework of robust
optimization, optimal adversarial attacks are given by minimizing the expected
return of the policy, and correspondingly a good defense mechanism should be
realized by improving the worst-case performance of the policy. Considering
that attackers generally have no access to the training environment, we propose
a greedy attack algorithm, which tries to minimize the expected return of the
policy without interacting with the environment, and a defense algorithm, which
performs adversarial training in a max-min form. Experiments on Atari game
environments show that our attack algorithm is more effective and leads to
worse return of the policy than existing attack algorithms, and our defense
algorithm yields policies more robust than existing defense methods to a range
of adversarial attacks (including our proposed attack algorithm).