Know your audience: specializing grounded language models with the game of Dixit
Effective communication requires adapting to the idiosyncratic common ground
shared with each communicative partner. We study a particularly challenging
instantiation of this problem: the popular game Dixit. We formulate a round of
Dixit as a multi-agent image reference game where a (trained) speaker model is
rewarded for describing a target image such that one (pretrained) listener
model can correctly identify it from a pool of distractors, but another
listener cannot. To adapt to this setting, the speaker must exploit differences
in the common ground it shares with the different listeners. We show that
finetuning an attention-based adapter between a CLIP vision encoder and a large
language model in this contrastive, multi-agent setting gives rise to
context-dependent natural language specialization from rewards only, without
direct supervision. In a series of controlled experiments, we show that the
speaker can adapt according to the idiosyncratic strengths and weaknesses of
various pairs of different listeners. Furthermore, we show zero-shot transfer
of the speaker's specialization to unseen real-world data. Our experiments
offer a step towards adaptive communication in complex multi-partner settings
and highlight the interesting research challenges posed by games like Dixit. We
hope that our work will inspire creative new approaches to adapting pretrained
models.