Unfortunately, the RHS of Eq. (REF ) is not estimable since we do not know the density values therein.
We therefore multiply both sides of Eq. (REF ) by some test function and subsequently sum over \((s^{\prime },a^{\prime })\)  on both sides to get a tractable objective that only requires samples from the offline dataset and the initial state-distribution \(\mu _0\) .
It is desired to choose a test function that can better distinguish the difference between the left-hand side (LHS) and the RHS of Eq. (REF ).
A potential choice is the action-value function of the policy \(\pi \) , due to some primal-dual relationship between the stationary state-action density-(ratio) (\(d_{\pi ,\gamma }^{P^{*}}(s,a)\)  or \(\omega ^{*}(s,a)\) ) and the action-value function [1]}, [2]}, [3]}.
A detailed discussion on the choice of the test function is provided in Appendix .
