Theory of mind(ToM)  evaluations currently focus on testing models using passive narratives that inherently lack interactivity.
We introduceFANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering.
Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs) .
In particular, we formulate multiple types of questions that demand the same underlying reasoning to identifyillusoryor false sense of ToM capabilities in LLMs.
We show thatFANToMis challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.111https://hyunw.kim/fantom