We then compare our scheme with the popular schemes, including: The standard A2C [1]}: only using the vanilla gradient update at the actor without policy optimization improvement; the standard actor-critic [2]}: without using advantage function in the policy estimation; and the baseline deep Q-network [3]}: using action sampling approximation. From Fig. REF (b), we find that our proposed parameterized A2C method is superior to the existing schemes in terms of system reward and convergence rate and stability. This is due to the efficient action sampling and improved policy optimization based on trust region enforcement which helps avoiding possible gradient divergence. Thus, a faster and more reliable policy search is achieved toward an optimal solution. Although the standard A2C scheme achieves a relatively stable reward performance, its reward gain is lower due to the natural gradient descent with greedy policy update. Its convergence is also slower (after 4500 episodes compared to 2500 episodes in our proposed scheme).
