Off-policy AC. Off-policy AC, was first proposed in [1]}. After that, there has been numerous extensions to that work such as DPG [2]}, DDPG [3]}, ACER [4]}, TD3 [5]}, IMPALA [6]}, ACE [7]}, etc. The asymptotic convergence of off-policy AC was established for Gradient-AC in [8]}, and for AC with emphasis in [9]}. The first finite-sample bound of off-policy NAC was established in [10]}. However, in [10]} only tabular setting was studied. In the function approximation setting, [12]} provided the finite sample analysis of a doubly robust off-policy AC. [13]} also provided a convergence bound for off-policy AC, however their convergence bound does not involve a bound for the critic. A detailed comparison between our results and the related literature on off-policy AC-type algorithms with function approximation is presented in Table REF .
