Novelty in the Actor.
NAC under function approximation was developed in [1]} by projecting the \(Q\) -values (gradients) to the lower dimensional space, and this involves the use of the discounted state visitation distribution, which is hard to estimate. We develop a new NAC algorithm for the function approximation setting that is instead based on the solution of a projected Bellman equation [2]}, which our critic is designed to solve.
