In the seminal work of [1]}, it is shown that the gradient of (REF ) with respect to the parameters \( \phi \)  is given by
\(\nabla _1 \Gamma (\phi ) = \mathbb {E}_{x^+ \sim p_x}[\nabla _2 R(x^+, \phi )] - \mathbb {E}_{x^-\sim p_\phi }[\nabla _2 R(x^-, \phi )],\) 
