Denoised MDPs: Learning World Models Better Than the World Itself
The ability to separate signal from noise, and reason with clean
abstractions, is critical to intelligence. With this ability, humans can
efficiently perform real world tasks without considering all possible nuisance
factors.How can artificial agents do the same? What kind of information can
agents safely discard as noises?
  In this work, we categorize information out in the wild into four types based
on controllability and relation with reward, and formulate useful information
as that which is both controllable and reward-relevant. This framework
clarifies the kinds information removed by various prior work on representation
learning in reinforcement learning (RL), and leads to our proposed approach of
learning a Denoised MDP that explicitly factors out certain noise distractors.
Extensive experiments on variants of DeepMind Control Suite and RoboDesk
demonstrate superior performance of our denoised world model over using raw
observations alone, and over prior works, across policy optimization control
tasks as well as the non-control task of joint position regression.