The context-conditioned supervised learning strategy used by CALM provides for reward maximization, but is in general not optimal for arbitrary reinforcement learning problems: in general RL settings, learning a value function with dynamic programming in general can attain significantly better returns than imitating high-performing trajectories, by recombining good parts of multiple different trajectories (which might individually be suboptimal) [1]}, [2]}. The simple supervised learning strategy works well in the domain we tested, but extending CALM to use value-based reinforcement learning methods is a promising direction for future work. Indeed, the improvement obtained from planning on top of the CALM model likely indicates that the supervised learning approach we employ has room for improvement. Additionally, the auxiliary objectives and relabeling strategies we employ require some amount of domain-specific design, and more general strategies could be developed in future.
