[Derivarion of Eq. (REF )]
We follow the literature [1]}, [2]} to assume that \(\forall \, s,a,\, \omega (s,a) \le \omega _{\max }\) 
for some unknown finite constant \(\omega _{\max }\) .
Using Remark REF  and this assumption, we have
\(\leavevmode \xbox {resizebox}{\XMLaddatt {width}{427.0pt}\begin{aligned}D_{\pi } (P^{*}, \widehat{P}) =&~ \mathbb {E}_{(s,a) \sim d_{\pi _b,\gamma }^{P^{*}}}\left[\omega (s,a) \mathrm {KL}\left(P^*(s^{\prime }\,|\,s,a) \pi _b(a^{\prime }\,|\,s^{\prime }) \,||\, \widehat{P}(s^{\prime } \,|\,s,a)\pi (a^{\prime }\,|\,s^{\prime }) \right)\right]\\\le &~ \omega _{\max } \cdot \mathbb {E}_{(s,a) \sim d_{\pi _b,\gamma }^{P^{*}}}\left[\mathrm {KL}\left(P^*(s^{\prime }\,|\,s,a) \pi _b(a^{\prime }\,|\,s^{\prime }) \,||\, \widehat{P}(s^{\prime } \,|\,s,a)\pi (a^{\prime }\,|\,s^{\prime }) \right)\right]\\\le &~\omega _{\max }\cdot \mathrm {KL}\left(P^*(s^{\prime }\,|\,s,a) \pi _b(a^{\prime }\,|\,s^{\prime })d_{\pi _b,\gamma }^{P^{*}}(s,a) \,||\, \widehat{P}(s^{\prime } \,|\,s,a)\pi (a^{\prime }\,|\,s^{\prime }) d_{\pi _b,\gamma }^{P^{*}}(s) \pi (a\,|\,s)\right)\\\approx &~\omega _{\max }\cdot \mathrm {JSD}\left(P^*(s^{\prime }\,|\,s,a) \pi _b(a^{\prime }\,|\,s^{\prime })d_{\pi _b,\gamma }^{P^{*}}(s,a) \,||\, \widehat{P}(s^{\prime } \,|\,s,a)\pi (a^{\prime }\,|\,s^{\prime }) d_{\pi _b,\gamma }^{P^{*}}(s) \pi (a\,|\,s)\right)\\\triangleq &~ \omega _{\max } \cdot \widetilde{D}_{\pi }(P^{*}, \widehat{P}),\end{aligned}}\) 
