Enhancing Safe Exploration Using Safety State Augmentation
Safe exploration is a challenging and important problem in model-free
reinforcement learning (RL). Often the safety cost is sparse and unknown, which
unavoidably leads to constraint violations -- a phenomenon ideally to be
avoided in safety-critical applications. We tackle this problem by augmenting
the state-space with a safety state, which is nonnegative if and only if the
constraint is satisfied. The value of this state also serves as a distance
toward constraint violation, while its initial value indicates the available
safety budget. This idea allows us to derive policies for scheduling the safety
budget during training. We call our approach Simmer (Safe policy IMproveMEnt
for RL) to reflect the careful nature of these schedules. We apply this idea to
two safe RL problems: RL with constraints imposed on an average cost, and RL
with constraints imposed on a cost with probability one. Our experiments
suggest that simmering a safe algorithm can improve safety during training for
both settings. We further show that Simmer can stabilize training and improve
the performance of safe RL with average constraints.