Non-Markovian policies occupancy measures
A central object of study in Reinforcement Learning (RL) is the Markovian
policy, in which an agent's actions are chosen from a memoryless probability
distribution, conditioned only on its current state. The family of Markovian
policies is broad enough to be interesting, yet simple enough to be amenable to
analysis. However, RL often involves more complex policies: ensembles of
policies, policies over options, policies updated online, etc. Our main
contribution is to prove that the occupancy measure of any non-Markovian
policy, i.e., the distribution of transition samples collected with it, can be
equivalently generated by a Markovian policy.
  This result allows theorems about the Markovian policy class to be directly
extended to its non-Markovian counterpart, greatly simplifying proofs, in
particular those involving replay buffers and datasets. We provide various
examples of such applications to the field of Reinforcement Learning.