While related, this challenge is subtly different from the well-known divergence issues of off-policy learning with function approximation,
demonstrated by Baird's famous counterexample [1]} (see also [2]}) and conceptualized as the Deadly Triad [3]}, [4]}.
While these depend on bootstrapping as a mechanism to cause a feedback-loop resulting in value
divergence, our results show that the offline learning challenge persists even without bootstrapping,
as small differences in behavior cause a drift in the `test distribution' itself.
Instead of a training-time output drift caused by bootstrapping, the central role is taken by a test-time drift of the state distribution caused by the interplay of function approximation and a fixed data distribution (as opposed to dynamically self-generated data).
