Such equivalent update rule is established in [1]} but only under the condition that \(w\)  is the solution of an appropriate optimization problem, which forbids [1]} from using the equivalent update equation (REF ) in the analysis of function approximation. Here we establish such equivalence in the case where \(w\)  is arbitrary. On the one hand, this seemingly simple but important extension enables us to use the lower dimensional update equation \(\theta _{t+1}=\theta _t+\beta w_t\)  in the algorithm. On the other hand, we can use the equivalent update equation (REF ) for the analysis to obtain better convergence rate than [1]}. Using Lemma REF , we have the following performance bound for the actor. See Appendix REF  for the proof.
