Our results contrast with the strong offline RL results in [1]}.
We hypothesize that the difference is due to the vastly different dataset size (full training of 200M transitions vs. replay buffer of 1M).
Interpolating between the tandem and the offline RL setting from [1]},
we increase the replay buffer size, thereby giving the passive agent access to somewhat larger data diversity
and state-action coverage (this does not affect the active agent's training as the active agent is constrained to only sample from the
most recent 1M replay samples, as in the baseline variant). A larger
replay buffer somewhat mitigates the passive agent's under-performance (Fig. REF ), though it appears to mostly slow down
rather than prevent the passive agent from eventually under-performing its active counterpart substantially.
As we suspect that a sufficient replay buffer size may depend on the effective state-space size of an environment,
we also perform analogous experiments on the (much smaller) classic control domains; results
(Appendix Fig. REF ) remain qualitatively the same.
<FIGURE>