Here we provide an example application for the Tandem RL setting as an analytic tool for the study of learning algorithms
in isolation from the confounding factor of behavior. As observed in [1]}, the QR-DQN algorithm [2]} can be preferable to DQN in the offline setting, motivating our attempt to use
it as a passive agent, coupled with a regular Double-DQN active agent. As shown in Fig. REF , QR-DQN indeed provides a somewhat different passive performance profile when compared to the regular Double-DQN tandem, albeit not a clearly better one. While perfectly matching active performance in one game (Space Invaders) and even out-performing the active agent in another (Breakout), it also shows exacerbated under-performance or instability in the other two domains. A fine-grained diagnosis of the causes of this are left to future work.
