We introduce two cross-modal transformers: the action transformer \(h(\hat{a}_t\vert \hat{s}_t, \hat{a}_{t-1}, \hat{s}_T)\) , which learns the correspondence between previous action feature \(\hat{a}_{t-1}\)  and state feature \(\hat{s}_t\) , \(\hat{s}_T\)  and generates the action prediction \(\hat{a}_t\) ; the state transformer \(g(\hat{s}_{t+1}\vert \hat{s}_{t}, \hat{a}_{t}, \hat{s}_T)\) , which learns the correspondence between the state feature \(\hat{s}_{t}\) , \(\hat{s}_T\)  and action feature \(\hat{a}_{t}\)  and generates the future state prediction \(\hat{s}_{t+1}\) .
Attention is the core of transformer network [1]}, [2]}.
We set up four different settings: causal attention [3]}, and full-attention [4]} with
future-\(\lbrace 1, 2, 3\rbrace \)  supervision.
Specifically, the output of the attention layer, the context vector \(\mathbf {C}\)  is computed using the query vector \(\mathbf {Q}\)  and the key \(\mathbf {K}\)  value \(\mathbf {V}\)  pair
from the input with a mask \(\mathbf {M}\)  via
\(\begin{split}\mathbf {C} & = {\rm Attn}(\mathbf {Q, K, V, M}) \\& = {\rm softmax} \left(\frac{\mathbf {QK}^T + \mathbf {M}}{\sqrt{D}}\right)\mathbf {V},\end{split}\) 
