REINFORCE suffers from high variance in practice, as its gradient estimate depends directly on the sum of rewards along the entire trajectory (dialog in our case) [1]}.
Its variance may be reduced by using an estimated value function of the current policy, often referred to as the critic in actor-critic algorithms [2]}, [3]}:
\(\nabla _\theta J(\theta ) = \sum _{t=1}^{T-1} \gamma ^{t-1} \left(\nabla _\theta \log \pi (a_t|s_t;\theta ) \tilde{Q}(s_t,a_t,h) \right)\,,\) 
