The strength of the last two experiments lies in the observation that, since active and passive networks have identical parameter values at the beginning of passive training,
their divergence cannot be attributed to small initial differences
getting amplified by training on an inadequate data distribution.
With so many factors held fixed, the collapse of passive performance when trained on the very data distribution produced by its own initial policy begs the question whether off-policy Q-learning itself is to blame
for this failure mode, e.g. via statistical over-estimation bias introduced by the \(\max \)  operator [1]}.
Here we provide a negative answer, by performing on-policy evaluation with SARSA [2]} (Fig. REF ), and even purely supervised regression on the Monte-Carlo returns (Appendix Fig. REF ), in the forked tandem setup. While evaluation succeeds, in the sense of minimizing evaluation error on the given behavior distribution, atypical action values under the behavior policy suffer substantial estimation error, resulting in occasional over-estimation. The resulting \(\varepsilon \) -greedy control policy under-performs the initial policy at forking time as catastrophically as in the other forked tandem experiments (more details in Appendix REF ).
Strengthening the roles of REF  and REF  while further weakening that of  REF, these observations point to an inherent instability of offline learning, different from that of Baird's famous example [3]} or the `Deadly Triad' [4]}, [5]}; an instability that results purely from erroneous extrapolation by the function approximator, when the utilized data distribution does not provide adequate coverage of relevant state-action pairs.
<FIGURE><FIGURE>