Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization
Policy-gradient methods in Reinforcement Learning(RL) are very universal and
widely applied in practice but their performance suffers from the high variance
of the gradient estimate. Several procedures were proposed to reduce it
including actor-critic(AC) and advantage actor-critic(A2C) methods. Recently
the approaches have got new perspective due to the introduction of Deep RL:
both new control variates(CV) and new sub-sampling procedures became available
in the setting of complex models like neural networks. The vital part of
CV-based methods is the goal functional for the training of the CV, the most
popular one is the least-squares criterion of A2C. Despite its practical
success, the criterion is not the only one possible. In this paper we for the
first time investigate the performance of the one called Empirical
Variance(EV). We observe in the experiments that not only EV-criterion performs
not worse than A2C but sometimes can be considerably better. Apart from that,
we also prove some theoretical guarantees of the actual variance reduction
under very general assumptions and show that A2C least-squares goal functional
is an upper bound for EV goal. Our experiments indicate that in terms of
variance reduction EV-based methods are much better than A2C and allow stronger
variance reduction.