Unfortunately, we can not directly estimate the KL term in \(D_{\pi }(P^{*}, \widehat{P})\)  if \(\pi \)  is an implicit policy, since we can only draw samples from \(\pi \)  but the density is unknown.
A potential solution is to use the dual representation of KL divergence \(\mathrm {KL}(p \,||\, q) = \sup _{T} \mathbb {E}_{p}[T] - \log (\mathbb {E}_q[e^T])\)  [1]}, which can be estimated with samples from the distributions \(p\)  and \(q\) .
However, the exponential function therein makes the estimation unstable in practice [2]}.
We instead use the dual representation of the Jensen–Shannon divergence (JSD) to approximate \(D_{\pi }(P^{*}, \widehat{P})\) , which can be approximately minimized using the GAN structure [3]}, [4]}.
Our framework can thus utilize the many stabilization techniques developed in the GAN community (Appendix REF ).
