We use \((s, a)\sim d_{\pi }\)  to denote \(s\sim d_{\pi }, a\sim \pi \) .
Moreover, the following useful identity expresses the expected return of another policy \(\tilde{\pi }\) 
in terms of the advantage over \(\pi \)  [1]}:
\(J(\tilde{\pi }) = J(\pi ) + \frac{1}{1-\gamma } \sum _{s}d_{\tilde{\pi }}(s)\sum _{a}\tilde{\pi }(a|s) A_{\pi }(s, a),\) 
