On the Robustness of Safe Reinforcement Learning under Observational Perturbations
Safe reinforcement learning (RL) trains a policy to maximize the task reward
while satisfying safety constraints. While prior works focus on the performance
optimality, we find that the optimal solutions of many safe RL problems are not
robust and safe against carefully designed observational perturbations. We
formally analyze the unique properties of designing effective state adversarial
attackers in the safe RL setting. We show that baseline adversarial attack
techniques for standard RL tasks are not always effective for safe RL and
proposed two new approaches - one maximizes the cost and the other maximizes
the reward. One interesting and counter-intuitive finding is that the maximum
reward attack is strong, as it can both induce unsafe behaviors and make the
attack stealthy by maintaining the reward. We further propose a more effective
adversarial training framework for safe RL and evaluate it via comprehensive
experiments. This work sheds light on the inherited connection between
observational robustness and safety in RL and provides a pioneer work for
future safe RL studies.