Novelty in the Critic.
Off-policy TD with function approximation is famously [1]} known to diverge due to deadly triad. To overcome this difficulty, we employ \(n\) -step TD-learning, and show that a proper choice of \(n\)  naturally achieves convergence, and we present finite-sample bounds under both constant and diminishing stepsizes. To the best of our knowledge, we are the first to design a single time-scale off-policy TD with function approximation with provable finite-sample bounds.
