We propose a transformer-based network architecture that can learn the action-state correlation and generate planning sequences. The overview of this architecture is shown in Figure REF . We introduce some critical design choices that assist in learning cross-modal correspondence and, more importantly, improve the accuracy of generated planning sequences. These choices include cross-modal transformer architecture, attention mechanism-causal attention [1]} vs. full
attention [2]} for each transformer and the supervision
scheme. Our design choices are explained in detail below.
