Off-policy Lagrangian applies stochastic primal-dual optimization [1]} to update primal and dual variables alternatively, which follows as
\(\small {\left\lbrace \begin{array}{ll}\theta \leftarrow \theta + \eta _\theta \nabla _\theta \mathbb {E}_{\mathcal {D}}\big ( Q^\pi (s,\pi _\theta (s)) - \lambda Q^\pi _{c}(s,\pi _\theta (s)) \big )\\\lambda \leftarrow \big [ \lambda + \eta _\lambda \mathbb {E}_{\mathcal {D}}\big (Q^\pi _{c}(s,\pi _\theta (s)) - \epsilon \big ) \big ]^+\end{array}\right.}\) 
