In practice, AC algorithms are usually used along with function approximation to overcome the curse of dimensionality in RL [1]}. However, it has been observed that the combination of function approximation, off-policy sampling, and bootstrapping (also known as the deadly triad [2]}) can result in instability or even divergence [2]}, [4]}. In this work, we develop a variant of off-policy NAC with function approximation, and we establish its finite-sample convergence guarantee in the presence of the deadly triad.
