Datasets.
We evaluate PlaTe on an instructional video dataset CrossTask [1]} and an interactive dataset ActioNet [2]}, which is based on AI2-THOR [3]}. For real-world UR-5 experiments, we collect a UR-5 Reaching Dataset which consists of 100 trajectories (2150 first-person-view RGB image and corresponding action pairs) as a training set and evaluate on a real UR-5 platform. We perform an ablation study on CrossTask dataset.
