Efficient Reinforcement Learning from Demonstration Using Local Ensemble and Reparameterization with Split and Merge of Expert Policies
The current work on reinforcement learning (RL) from demonstrations often
assumes the demonstrations are samples from an optimal policy, an unrealistic
assumption in practice. When demonstrations are generated by sub-optimal
policies or have sparse state-action pairs, policy learned from sub-optimal
demonstrations may mislead an agent with incorrect or non-local action
decisions. We propose a new method called Local Ensemble and Reparameterization
with Split and Merge of expert policies (LEARN-SAM) to improve efficiency and
make better use of the sub-optimal demonstrations. First, LEARN-SAM employs a
new concept, the lambda-function, based on a discrepancy measure between the
current state to demonstrated states to "localize" the weights of the expert
policies during learning. Second, LEARN-SAM employs a split-and-merge (SAM)
mechanism by separating the helpful parts in each expert demonstration and
regrouping them into new expert policies to use the demonstrations selectively.
Both the lambda-function and SAM mechanism help boost the learning speed.
Theoretically, we prove the invariant property of reparameterized policy before
and after the SAM mechanism, providing theoretical guarantees for the
convergence of the employed policy gradient method. We demonstrate the
superiority of the LEARN-SAM method and its robustness with varying
demonstration quality and sparsity in six experiments on complex continuous
control problems of low to high dimensions, compared to existing methods on RL
from demonstration.