In general, the issue of high variance due to the importance sampling ratio (cf. \(\zeta _{\max }\) ) is a fundamental problem in multi-step off-policy TD-learning [1]}. In order to reduce such high variance, several variants of off-policy RL such as Retrace\((\lambda )\)  [2]} and \(Q\) -trace [3]} have been proposed. These algorithms use truncated importance sampling ratios to reduce \(\zeta _{\max }\) , thus reducing the variance. However, none of them are shown to converge in the function approximation setting. Designing efficient algorithms to control the high variance in multi-step off-policy TD-learning with function approximation is one of our future directions.
