
where \(R_\psi (u_t, a_t) = \log p_\psi (u_t|a_t) - \log p_\psi (u_t)\) .
To compute this estimate we need to know the transition dynamics of the MDP, \(p(x_t, u_t|a_{t-1}, x_{t-1}, u_{t-1})\) .
Although the model-based requirement may seem stringent, it is plausible that a model will be available in higher-stakes settings where privacy is a consideration.
Furthermore, model-based techniques are increasingly popular due to empirical [1]} and theoretical [2]} sample-efficiency improvements over model-free techniques.
With the increasing prevalence of model-based techniques in addition to sim2real [3]}-based training pipelines, it is not unreasonable to assume the presence of some sort of dynamics model if required.
Note that since we can compute \(q_\phi (x_t, u_t|x_{t-1}, u_{t-1})\)  as \(\int _{a_{t-1}}p(x_t, u_t|a_{t-1}, x_{t-1}, u_{t-1})q_\phi (a_{t-1}|x_{t-1}, u_{t-1})da_{t-1}\) , we do not need to learn a separate \(q_\phi (x_t, u_t|x_{t-1}, u_{t-1})\)  distribution.
