Between Rate-Distortion Theory & Value Equivalence in Model-Based Reinforcement Learning
The quintessential model-based reinforcement-learning agent iteratively
refines its estimates or prior beliefs about the true underlying model of the
environment. Recent empirical successes in model-based reinforcement learning
with function approximation, however, eschew the true model in favor of a
surrogate that, while ignoring various facets of the environment, still
facilitates effective planning over behaviors. Recently formalized as the value
equivalence principle, this algorithmic technique is perhaps unavoidable as
real-world reinforcement learning demands consideration of a simple,
computationally-bounded agent interacting with an overwhelmingly complex
environment. In this work, we entertain an extreme scenario wherein some
combination of immense environment complexity and limited agent capacity
entirely precludes identifying an exactly value-equivalent model. In light of
this, we embrace a notion of approximate value equivalence and introduce an
algorithm for incrementally synthesizing simple and useful approximations of
the environment from which an agent might still recover near-optimal behavior.
Crucially, we recognize the information-theoretic nature of this lossy
environment compression problem and use the appropriate tools of
rate-distortion theory to make mathematically precise how value equivalence can
lend tractability to otherwise intractable sequential decision-making problems.