The Sufficiency of Off-policyness: PPO is insufficient according to an Off-policy Measure
One of the major difficulties of reinforcement learning is learning from {\em
off-policy} samples, which are collected by a different policy (behavior
policy) from what the algorithm evaluates (the target policy). Off-policy
learning needs to correct the distribution of the samples from the behavior
policy towards that of the target policy. Unfortunately, important sampling has
an inherent high variance issue which leads to poor gradient estimation in
policy gradient methods. We focus on an off-policy Actor-Critic architecture,
and propose a novel method, called Preconditioned Proximal Policy Optimization
(P3O), which can control the high variance of importance sampling by applying a
preconditioner to the Conservative Policy Iteration (CPI) objective. {\em This
preconditioning uses the sigmoid function in a special way that when there is
no policy change, the gradient is maximal and hence policy gradient will drive
a big parameter update for efficient exploration of the parameter space}. This
is a novel exploration method that has not been studied before given that
existing exploration methods are based on the novelty of states and actions. We
compare with several best-performing algorithms on both discrete and continuous
tasks and the results showed that {\em PPO is insufficient in off-policyness},
and our P3O is {\em more off-policy} than PPO according to the "off-policyness"
measured by the DEON metric, and P3O explores in a larger policy space than
PPO. Results also show that our P3O maximizes the CPI objective better than PPO
during the training process.