This proof relies on the same strategy used in [1]}, [2]}.
According to (REF ),
for any two policies \(\tilde{\pi }\)  and \(\pi \) , we have the following
\(& J(\tilde{\pi }) - J(\pi ) = \frac{1}{1-\gamma } \mathbb {E}_{\leavevmode {\color {red}s\sim d_{\tilde{\pi }}}, a\sim \tilde{\pi }}[A_{\pi }(s, a)] \\= & \frac{1}{1-\gamma } \Big [\mathbb {E}_{\leavevmode {\color {red}s\sim d_{\tilde{\pi }}}, a\sim \tilde{\pi }}[A_{\pi }(s, a)] - \mathbb {E}_{\leavevmode {\color {blue}s\sim d_{\pi }}, a\sim \tilde{\pi }}[A_{\pi }(s, a)]+ \mathbb {E}_{\leavevmode {\color {blue}s\sim d_{\pi }}, a\sim \tilde{\pi }}[A_{\pi }(s, a)] \Big ] \\= & \frac{1}{1-\gamma } \Big [ \underbrace{\mathbb {E}_{\leavevmode {\color {red}s\sim d_{\tilde{\pi }}}, a\sim \tilde{\pi }}[A_{\pi }(s, a)] - \mathbb {E}_{\leavevmode {\color {blue}s\sim d_{\pi }}, a\sim \tilde{\pi }}[A_{\pi }(s, a)]}_{\text{Correction term}}+ \underbrace{\mathbb {E}_{\leavevmode {\color {blue}s\sim d_{\pi }}, a\sim \pi }\big [ \frac{\tilde{\pi }(a|s)}{\pi (a|s)} A_{\pi }(s, a) \big ]}_{\text{Surrogate term}} \Big ].\) 
