Theory of mind as a subject, especially in computation [1]}, has gained increased attention in areas including agent-agent reinforcement learning [2]}, dialogue systems [3]}, human-computer interaction [4]}, agent-agent collaborative dialogue [5]}, and explainable AI [6]}. Worthy of note is the type of mental state recording we employ: specifically, we ask players to record their own mental states during interaction. Unlike prior work that has largely utilized external annotators for post-hoc mental state attribution
, we expand on [7]} and [4]} by specifically bringing user self-reported mental states from that of only the linguistic domain to multimodal situated dialogue. Specifically, the novelty in our work exists in studying and bringing explicit theory of mind modeling to 3D situated collaborative interactions.
