Safe RL optimizes policies under episodic or instantaneous constraints.
The most common approach to solving the episodic constraint is Lagrangian relaxation (Chow et al. 2017; Tessler at al. 2017; Stooke et al. 2020). Other works [1]}, [2]} approximate the constrained policy iteration with a quadratic constrained optimization. Recently, first-order methods [3]}, [4]}, [5]} start to gain attractions as the objective is efficient to optimize and easy to handle multiple constraints.
For the instantaneous constraint, Lagrangian relaxation is also a feasible solution [6]}. [7]} perform quadratic programming to project actions back to the safe set. Other works model the instantaneous cost as Gaussian Processes and plan in the safety-proven neighboring states [8]}, [9]}.
In this paper, we formulate safe RL following state-wise safety constraints, which are slightly different from the above genres.
