In order to solve the inner maximization problem with gradient descent, we require estimators
for \(\nabla _\phi \mathbb {E}_{\tau \sim {q_\phi }}\left[R(\tau )\right]\)  and \(\nabla _\phi I_{q_\phi }(a_t;u_t)\) .
Since even evaluating the mutual information in high dimensions is challenging [1]}, this
is not trivial. Our main contribution is a method to approximate the objective \(I_{q_\phi }(a_t;u_t)\) ,
and a method for estimating \(\nabla _\phi I_{q_\phi }(a_t;u_t)\)  where a dynamics model is available.
We also give a model-free gradient estimator for \(I_{q_\phi }(\tau _a, \tau _x;\tau _u)\) , which we can use as an upper
bound to \(I_{q_\phi }(a_t;u_t)\)  or as the relevant constraint in an alternative threat model as described
above.
