We establish baseline performance for the EDH and TfD tasks using the Episodic Transformer (E.T.) model [1]}, designed for the ALFRED benchmark.
The original E.T. model trains a transformer language encoder and uses a ResNet-50 backbone to encode visual observations.
Two multimodal transformer layers are used to fuse information from the language, image, and action embeddings, followed by a fully connected layer to predict the next action and target object category for interaction actions.
E.T. uses a MaskRCNN [2]} model pretrained on ALFRED images to predict a segmentation of the egocentric image for interactive actions, matching the predicted mask to the predicted object category.
We convert the centroid of this mask to a relative coordinate specified to the TEACh API wrapper for AI2-THOR.
