Agents that engage in dialogue instead of simply following natural language instructions can be learned by combining individual rule-based or learned components [1]}, [2]}, [3]}.
Simulated clarification can also improve end-to-end VLN models [4]}, [5]}.
Models are also able to take advantage of conversational history in human-human dialogues to perform better navigation [6]}, [7]}, learn agent-agent policies for navigating and speaking [8]}, [9]}, and deploy individual agent policies for human-in-the-loop evaluation [10]}.
However, such models and underlying datasets are limited to navigation actions and turn-taking conversation.
In contrast, TEACh involves Follower navigation and object interaction, as well as freeform dialogue acts with the Commander.
The Minecraft Dialogue Corpus (MDC) [11]} gives full dialogues between two humans for assembly tasks.
MDC is similar in spirit to TEACh; we introduce a larger action space and resulting object state changes, such as slicing and toasting bread, as well as collecting many more human-human dialogues.
