Time Series Processing. We use either an LSTM [1]} or a Transformer network [2]}, masked such that it only attends to the past, and feed a sequence of visual frame, plan, and dialogue embeddings as aforementioned to produce a latent representation of game interactions for every step.
