SEREN: Knowing When to Explore and When to Exploit
Efficient reinforcement learning (RL) involves a trade-off between
"exploitative" actions that maximise expected reward and "explorative'" ones
that sample unvisited states. To encourage exploration, recent approaches
proposed adding stochasticity to actions, separating exploration and
exploitation phases, or equating reduction in uncertainty with reward. However,
these techniques do not necessarily offer entirely systematic approaches making
this trade-off. Here we introduce SElective Reinforcement Exploration Network
(SEREN) that poses the exploration-exploitation trade-off as a game between an
RL agent -- \exploiter, which purely exploits known rewards, and another RL
agent -- \switcher, which chooses at which states to activate a pure
exploration policy that is trained to minimise system uncertainty and override
Exploiter. Using a form of policies known as impulse control, \switcher is able
to determine the best set of states to switch to the exploration policy while
Exploiter is free to execute its actions everywhere else. We prove that SEREN
converges quickly and induces a natural schedule towards pure exploitation.
Through extensive empirical studies in both discrete (MiniGrid) and continuous
(MuJoCo) control benchmarks, we show that SEREN can be readily combined with
existing RL algorithms to yield significant improvement in performance relative
to state-of-the-art algorithms.