The color reference game from<cite> Monroe et al. (2017)</cite> consists of rounds played between a speaker and a listener. Each round has a context of two distractors and a target color (Figure 1a ). Only the speaker knows the target, and must communicate it to the listener-who must pick out the target based on the speaker's English utterance. Similarly, each round of our new color-grid reference game contains target and distractor color-grid objects, and the speaker must communicate the target grid to the listener (Figure 1b) . We train neural network models to play the listener role in these games.