Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
Conveying complex objectives to reinforcement learning (RL) agents often
requires meticulous reward engineering. Preference-based RL methods are able to
learn a more flexible reward model based on human preferences by actively
incorporating human feedback, i.e. teacher's preferences between two clips of
behaviors. However, poor feedback-efficiency still remains a problem in current
preference-based RL algorithms, as tailored human feedback is very expensive.
To handle this issue, previous methods have mainly focused on improving query
selection and policy initialization. At the same time, recent exploration
methods have proven to be a recipe for improving sample-efficiency in RL. We
present an exploration method specifically for preference-based RL algorithms.
Our main idea is to design an intrinsic reward by measuring the novelty based
on learned reward. Specifically, we utilize disagreement across ensemble of
learned reward models. Our intuition is that disagreement in learned reward
model reflects uncertainty in tailored human feedback and could be useful for
exploration. Our experiments show that exploration bonus from uncertainty in
learned reward improves both feedback- and sample-efficiency of
preference-based RL algorithms on complex robot manipulation tasks from
MetaWorld benchmarks, compared with other existing exploration methods that
measure the novelty of state visitation.