In this section, we present the \(n\) -step off-policy TD-learning algorithm under linear function approximation [1]}, which is used for solving the policy evaluation (critic) sub-problem in our AC framework. Let \(\pi \)  be the target policy we aim to evaluate, and let \(\pi _b\)  be the behavior policy we used to collect samples. For any state-action pairs \((s,a)\) , let \(\rho (s,a)=\frac{\pi (a|s)}{\pi _b(a|s)}\) , which is called the importance sampling ratio between \(\pi \)  and \(\pi _b\)  at \((s,a)\) . For any positive integer \(n\) , Algorithm REF  presents the off-policy \(n\) -step TD-learning algorithm for estimating \(Q^\pi \) .
