where \(\tilde{Q}(s,a,h)\)  is an estimated value function for the current policy \(\pi (s;\theta )\)  that is used to approximate \(\sum _{h=t}^{T} \gamma ^{h-t} r_h\)  in Equation REF . \(\tilde{Q}(s,a,h)\)  measures the average discounted long-term reward by first select \(a\)  and then following policy \(\pi \)  thereafter. The value can be learned by standard temporal difference methods [1]}.
There have been many studies on the methods of computing the gradient \(\nabla _\theta J\)  more effectively than
Equation REF .
Interested readers can refer to a few related works and the references therein for further details
 [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}.
