Despite recent advances, we are still far away from fully enabling these desired agent behaviors. One key challenge is in an agent's ability to establish and maintain common ground in tandem with human partners, especially in a setting where beliefs about the world and of each other may change on the fly [1]}, [2]}. It is important to understand how changes in a dynamic, physical world affect agents' beliefs of each other—i.e., theory of mind [3]}—and how such beliefs influence teamwork and communication in collaborative tasks.
As a first step to address this question, this paper explores theory of mind modeling [4]}, [5]}, [6]} in situated language communication [7]}, [8]} for collaborative tasks within the 3D virtual blocks world of Minecraft.
Through a novel experimental setup, we collect a situated dialogue dataset that demonstrates how collaborative partners with a set of asymmetric knowledge and skills are able to collaborate to achieve joint goals, and how, in particular, their beliefs of each other evolve and converge over time. Based on this dataset, we further build several baseline computational models to explicitly predict key elements of a collaboration partner's mental state from the viewpoint of an agent as a task unfolds. Our empirical results demonstrate that while language is certainly important in this inference, the shared physical environment and the perceived activities play a greater role in shaping a partner's understanding of each other in order to come to a common ground.
