Here \(L(s, s_t)\)  is a loss function that is optimized by model \(P\) . In practice, \(P\)  may be any model that allows to sample from conditional distribution and estimate its density. Also, as we now have estimation \(P(s|s_t)\)  of distribution \(q_\pi (s | s_t)\) , we will replace \(H(q_\pi (s | s_t))\)  in equation REF  with \(H(P(s|s_t))\)  with the assumption that \(H(q_\pi (s | s_t)) \approx H(P(s|s_t))\)  if distributions are close enough. However, even for distribution estimation \(P\)  we may not be able to compute the entropy analytically and, moreover, not able to compute its gradient over actor network parameters. The solution is to estimate it by using importance sampling as in equation REF  followed by applying Monte Carlo sampling. As in SAC [1]} or DDPG [2]}, such estimation is theoretically consistent for any actor architecture that either deterministic or uses a reparametrization trick.
