TD-Learning. The policy evaluation problem within the critic is usually solved with TD-learning.
In the on-policy setting, the asymptotic convergence of TD-learning was established in [1]}, [2]}, [3]}, and the finite-sample bounds were studied in [4]}, [5]}, [6]}, [7]}, [8]}, [9]}. When TD-learning is used with off-policy sampling and function approximation, all the three elements of the deadly triad are present [10]}. As a result, the algorithm can diverge. In order to overcome the divergence issue, numerous variants of TD-learning algorithms, such as GTD [11]}, TDC [12]}, and emphatic TD-learning [13]}, are proposed in the literature. However, all these algorithms require to maintain two iterates and hence are two time-scale algorithms, while our proposed algorithm is a single time-scale algorithm.
<TABLE>