Orthogonal to existing algorithms, we propose a novel approach referred to as Unrolling Safety Layer (USL) in this paper, which is inspired by the complementary of safety projection and safety optimization. For projection-based approaches, the correction (even if tractable) only enforces the feasibility but lacks the equivalence to the optimal maximum return. For optimization-based approaches, most of them tend to find the optimal solution to the constrained problem with the help of neural networks. However, the forward computing lacks explicit restrictions on the output actions, and thus the “soft" loss function often fails to fully satisfy “hard" constraints.
Recently, Deep Constraint Completion and Correction (DC3) [1]} shows potential to achieve optimal objective values while preserving feasibility, which is of independent interest to general constrained problems. As illustrated in Figure REF , we employ a similar joint architecture that combines safety optimization (serves as the first stage's approximate solver) and safety projection (serves as the second stage's iterative correction). To the best of our knowledge, this is the first study to introduce deep unrolling optimization into safe RL.
<FIGURE>