<commit_before>\documentclass[9pt,twocolumn,twoside]{pnas-new}
% Use the lineno option to display guide line numbers if required.
% Note that the use of elements such as single-column equations
% may affect the guide line number alignment. 

\templatetype{pnasresearcharticle} % Choose template 
% {pnasresearcharticle} = Template for a two-column research article
% {pnasmathematics} = Template for a one-column mathematics article
% {pnasinvited} = Template for a PNAS invited submission

\usepackage{pslatex}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{color}
\usepackage{todonotes}
\usepackage{dsfont}
\usepackage{array}
\usepackage{textcomp}
\usepackage{multirow}
\usepackage{subfig}
\usepackage{trackchanges}
% trackchanges commands:
%   \note[editor]{The note} 
% \annote[editor]{Text to annotate}{The note} 
%    \add[editor]{Text to add} 
% \remove[editor]{Text to remove} 
% \change[editor]{Text to remove}{Text to add}


\title{Contextual flexibility in visual communication}

% Use letters for affiliations, numbers to show equal authorship (if applicable) and to indicate the corresponding author
% \author[Author Names]
% {<NAME> \affil{1,*}, <NAME> \affil{1,*}, <NAME> \affil{2}, \and <NAME> \affil{1,2}}

% \affiliation{1}{Department of Psychology, Stanford University, Stanford, CA 94305}
% \affiliation{2}{Department of Computer Science, Stanford University, Stanford, CA 94305}
% \affiliation{*}{These authors contributed equally to this work.}

% \correspondingauthor{<NAME>}{<EMAIL>}

\author[a,1,*]{<NAME>}
\author[a,*]{<NAME>} 
\author[b]{<NAME>}
\author[a,b]{<NAME>}

\affil[a]{Department of Psychology, Stanford University}
\affil[b]{Department of Computer Science, Stanford University}
\affil[*]{These authors contributed equally to this work.}

% Please give the surname of the lead author for the running footer
\leadauthor{Fan} 

% TODO: Currently a bit too long (and probably too in the weeds...) 
\significancestatement{}

% Please include corresponding author, author contribution and author declaration information
\authorcontributions{Please provide details of author contributions here.}
\authordeclaration{Please declare any conflict of interest here.}
\correspondingauthor{\textsuperscript{1}To whom correspondence should be addressed. E-mail: <EMAIL>}

% Keywords are not mandatory, but authors are strongly encouraged to provide them. If provided, please include two to five keywords, separated by the pipe symbol, e.g:
\keywords{drawing $|$ deep learning $|$ pragmatics $|$ computational modeling $|$ Rational Speech Act framework}

\begin{abstract}
Communication is central to the success of our species: it allows us to learn from each other, coordinate our actions, and express otherwise hidden thoughts. Critically, human communication goes beyond language production --- humans also express their ideas in visual form. Visual communication lies at the heart of key innovations, and forms the foundation for the cultural transmission of knowledge and higher-level reasoning. This paper examines drawing, the most basic form of visual communication. Communicative uses of drawing pose a core challenge for theories of vision and communication alike: they require a detailed understanding of how sensory information is encoded and how social context guides what information is relevant to communicate. Our computational modeling approach addresses this challenge by combining a high-performing computational model of vision with a formal Bayesian model of social reasoning during communication in order to explain how people flexibly adapt their drawings according to the current context. We employ drawing-based reference games throughout our modeling and experimental work. These reference games involve two players: a \textit{sketcher} who aims to help a \textit{viewer} pick out a target object from an array of alternative, distractor objects by representing it in a drawing. We found that people exploit information in common ground with their partner to efficiently communicate, and that this contextual flexibility was well explained by our computational model. In the long run, understanding the computational basis of visual communication may shed light on the nature of human visual abstraction and the emergence of graphical conventions.
\end{abstract}

\dates{This manuscript was compiled on \today}
\doi{\url{www.pnas.org/cgi/doi/10.1073/pnas.XXXXXXXXXX}}

\begin{document}

% Communication is not limited to verbal language; humans can make use of many different tools to convey meaning to a partner. While theories of communication often rely upon the existence of modality-general pragmatic reasoning mechanisms, these predictions have rarely been tested outside the verbal modality. In this experiment, we aim to test how context -- the set of potential referents in common ground -- affects visual communication: using drawings to convey the identity of rich naturalistic images. Past work in the verbal modality has shown a strong influence of speaker informativity: when the intended referent is surrounded by very similar objects, speakers send more detailed and verbose messages to help the listener distinguish it. We expect analogous results to hold in the visual modality. For instance, when referring to a car in a context containing dogs and birds, speakers should use simpler, more abstract drawings than when referring to the same object in the context of other cars.


% Optional adjustment to line up main text (after abstract) of first page with line numbers, when using both lineno and twocolumn options.
% You should only change this length when you've finalised the article contents.
\verticaladjustment{-2pt}

\maketitle
\thispagestyle{firststyle}
\ifthenelse{\boolean{shortarticle}}{\ifthenelse{\boolean{singlecolumn}}{\abscontentformatted}{\abscontent}}{}

% high level
Communication is central to the success of our species: it allows us to learn from each other, coordinate our actions, and express otherwise hidden thoughts. Critically, human communication goes beyond language production --- humans also express their ideas in visual form. Visual communication lies at the heart of key innovations, and forms the foundation for the cultural transmission of knowledge and higher-level reasoning. Drawing is perhaps the most direct and basic form of visual communication. We perceive drawings of objects as resembling physical objects in spite of the fact that drawings and objects are profoundly different in composition.

In the appropriate context, even a few strokes can express the identity of a face \cite[]{bergmann2013impact}, a suggested route \cite[]{agrawala2001rendering}, or an intention to act \cite[]{Galantucci:2005uh}. Conventionalized representations such as maps, graphs, and diagrams can transmit high-dimensional data and convey complex ideas by recombining relatively few geometric primitives, such as boxes, lines, arrows, ellipses \cite[]{tversky2000lines}. Despite the ubiquity and importance of such visualizations, we lack a unified computational theory of how human perception, action, and social inference are integrated to support flexible visual communication.

Communicative uses of drawing pose a core challenge for theories of vision and communication alike: they require a detailed understanding of how sensory information is encoded, how social context guides what information is relevant to communicate, and estimating the costs of visual production. In this paper, we meet this challenge by combining high-performing models of sensory representation from deep learning with insights from Bayesian cognitive models of social reasoning in language. 

There is now abundant and converging evidence that deep convolutional neural networks (DCNNs) optimized to recognize objects in natural photographs can learn general-purpose feature representations that support a wide variety of visual tasks, including but not limited to object categorization, segmentation, and pose/size estimation \cite[]{hong2016explicit,he2017mask}.

% recent successes learning general visual features, but these inherently acontextual
In particular, despite the fact that drawings are highly abstracted away from natural visual inputs, a relatively modest amount of `fine-tuning' of models previously trained only on photos has been shown to lead to human-level performance on sketch categorization \cite[]{yu2015sketch}, as well as vast improvements in discerning image-level correspondences between sketches and photos \cite[]{sangkloy2016sketchy}. Even without fine-tuning, prior work has shown a striking isomorphism in the representations of object categories in drawings and photos at higher layers in these models \cite[]{fan2015common}. As such, models from this architecture class are a strong candidate for an encoder that can support the level visual abstraction required for representing abstract information in drawings.

However, these visual feature representations are inherently acontextual, and cannot explain why people might use one drawing to communicate an idea in one context (e.g., XX), and a different drawing in another context (e.g., XX). Rational Speech Act (RSA) models provide a probabilistic framework for deriving communicative flexibility from general principles of social cognition \cite{GoodmanFrank16_RSATiCS}.  

These models draw on ideas and insights from decision theory, probabilistic models of cognition, bounded rationality, and linguistics, taking particular inspiration from the insights of Paul Grice (\citeyear{grice1975syntax}), who provided a philosophical framework for understanding how natural language use reflects social reasoning in context. Gricean listeners assume speakers are cooperative, choosing appropriately informative utterances to convey particular meanings. Under this assumption, listeners attempt to infer the speaker's intended communicative goal, working backwards from the form of the utterance. RSA models formalizing this core idea in terms of Bayesian inference have provided unified explanations for a variety of complex linguistic patterns, achieved good quantitative fits with experimental data, and been incorporated into artificial agents that communicate with each other to solve real-world tasks.

% Cite old context-manipulation stuff like Krauss & Weinheimer (1967) or 

Previous work has shown that RSA models account for context sensitivity in human speakers \cite[]{GrafEtAl16_BasicLevel}. 

% For instance, if a speaker needs to refer to a Dalmatian that appears alongside a poodle and a bear, the model predicts that this speaker will prefer the more specific utterance (``Dalmatian'') to the shorter and lower-cost basic-level utterance (``dog''). This is because the literal meaning of the word ``dog'' fits both the Dalmatian and poodle, whereas the word ``Dalmatian'' only fits the Dalmatian. Consequently, the speaker model reasons that ``Dalmatian'' is more informative. By using the word ``Dalmatian'', the speaker encourages the listener to infer the single correct world state (i.e., the target is the Dalmatian) instead of allowing two world states to seem probable, one of which is incorrect (i.e., the target is the poodle). In a context where there is only one dog, however, the speaker will prefer to say ``dog'', which is as informative as ``Dalmatian'' but is less costly to produce. 

\section*{Methods}

\subsection*{Participants}

A total of 192 unique participants, who were recruited via Amazon Mechanical Turk (AMT) and grouped into pairs, completed the experiment. 

\subsection*{Stimuli}

Stimuli were 32 3D mesh models of objects belonging to 4 categories (i.e., birds, chairs, cars, dogs), containing eight objects each. 40 color images of each object were produced by rendering it from a 10$^{\circ}$ viewing angle (i.e., slightly above) at a fixed distance on a gray background, each rotated by an additional 9$^{\circ}$ about the vertical axis. 

\subsection*{Task}

Drawings were collected in the context of an online, sketching-based reference game (``Guess My Sketch!''). The game involved two players: a \textit{sketcher} who aims to help a \textit{viewer} pick out a target object from a set of distractor objects by representing it in a sketch. On each trial, both participants were shown an array of the same four objects; however, the positions of these objects were randomized for each participant. On each trial, one of the four objects was highlighted on the sketcher's screen to designate it as the target. 

Sketchers drew using black ink on digital canvas (pen width = 5 pixels; 300 x 300 pixels) embedded in a web browser window using Paper.js (http://paperjs.org/). Participants drew using the mouse cursor, and were not able to delete previous strokes. Each stroke of which was rendered on the viewer's screen immediately upon the completion of each stroke. There were no restrictions on how long participants could take to make their drawings. After clicking a submit button, the viewer guessed the identity of the drawn object by clicking one of the four objects in the array. Otherwise, the viewer had no other means of communicating with the sketcher. Both participants received immediate task-related feedback: the sketcher learned which object the viewer had clicked, and the viewer learned the identity of the target. Both participants earned bonus points for each correct response. 

For each pair, objects were grouped into eight quartets: Four of these quartets contained objects from the same category (``close''); the other four of these quartets contained objects from different categories (``far'' condition). Each quartet was presented four times, such that each object in the quartet served as the target exactly once. The assignment of objects to quartet and condition was randomized across pairs. 

\subsection*{Modeling}

\subsubsection*{Visual encoder module}

Features were extracted using VGG-19 \cite{simonyan2014very}, a high-performing deep convolutional neural network model architecture that had been pretrained to categorize objects on the Imagenet database, which contains millions of photographs from hundreds of object categories \cite{deng2009imagenet}. Despite the fact that drawings are highly abstracted away from natural visual inputs, prior work has shown a striking isomorphism in the similarity structure \cite{kriegeskorte2008matching} of object categories in drawings and photos at higher layers in these models \cite[]{fan2015common}, without any additional training.

While these models excel at (and were indeed optimized for) categorization of objects, two additional competencies are required to succeed at the visual communication task described above.

First, an observer must be able to represent fine-grained relationships between different images of objects from the same category; second, this observer must also be able to discern image-level correspondences between drawings and photos of the same object. 

In order to better approximate the granularity with which human observers can distinguish different images, we augment VGG19 with a smaller \textit{adaptor} network that learns a common feature representation for drawings and photos. 

This approach has been successfully used in prior work to better predict human semantic categorization judgments \cite{peterson2016adapting}, and to improve sketch-based image retrieval \cite[]{sangkloy2016sketchy}. 


\subsubsection*{Social inference module}

To instantiate communicative flexibility in context, we generalize the Rational Speech Act (RSA) modeling framework, originally developed for linguistic communication. At the core of the RSA framework, as applied to linguistic communication, is the Gricean proposal that speakers aim to produce utterances that are concise yet informative. We adapt this proposal to visual communication by assuming that sketchers also aim to produce drawings that are both simple and informative. 

Formally, the sketcher, $\mathcal{S}$, is a decision-theoretic agent who produces drawings, $d$, by (soft) maximizing their utility function, $U$, given a particular object referent, $o$: 
<commit_msg>Minor update to social inference intro<commit_after>\documentclass[9pt,twocolumn,twoside]{pnas-new}
% Use the lineno option to display guide line numbers if required.
% Note that the use of elements such as single-column equations
% may affect the guide line number alignment. 

\templatetype{pnasresearcharticle} % Choose template 
% {pnasresearcharticle} = Template for a two-column research article
% {pnasmathematics} = Template for a one-column mathematics article
% {pnasinvited} = Template for a PNAS invited submission

\usepackage{pslatex}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{color}
\usepackage{todonotes}
\usepackage{dsfont}
\usepackage{array}
\usepackage{textcomp}
\usepackage{multirow}
\usepackage{subfig}
\usepackage{trackchanges}
% trackchanges commands:
%   \note[editor]{The note} 
% \annote[editor]{Text to annotate}{The note} 
%    \add[editor]{Text to add} 
% \remove[editor]{Text to remove} 
% \change[editor]{Text to remove}{Text to add}


\title{Contextual flexibility in visual communication}

% Use letters for affiliations, numbers to show equal authorship (if applicable) and to indicate the corresponding author
% \author[Author Names]
% {<NAME> \affil{1,*}, <NAME> \affil{1,*}, <NAME> \affil{2}, \and <NAME> \affil{1,2}}

% \affiliation{1}{Department of Psychology, Stanford University, Stanford, CA 94305}
% \affiliation{2}{Department of Computer Science, Stanford University, Stanford, CA 94305}
% \affiliation{*}{These authors contributed equally to this work.}

% \correspondingauthor{<NAME>}{<EMAIL>}

\author[a,1,*]{<NAME>}
\author[a,*]{<NAME>} 
\author[b]{<NAME>}
\author[a,b]{<NAME>}

\affil[a]{Department of Psychology, Stanford University}
\affil[b]{Department of Computer Science, Stanford University}
\affil[*]{These authors contributed equally to this work.}

% Please give the surname of the lead author for the running footer
\leadauthor{Fan} 

% TODO: Currently a bit too long (and probably too in the weeds...) 
\significancestatement{}

% Please include corresponding author, author contribution and author declaration information
\authorcontributions{Please provide details of author contributions here.}
\authordeclaration{Please declare any conflict of interest here.}
\correspondingauthor{\textsuperscript{1}To whom correspondence should be addressed. E-mail: <EMAIL>}

% Keywords are not mandatory, but authors are strongly encouraged to provide them. If provided, please include two to five keywords, separated by the pipe symbol, e.g:
\keywords{drawing $|$ deep learning $|$ pragmatics $|$ computational modeling $|$ Rational Speech Act framework}

\begin{abstract}
Communication is central to the success of our species: it allows us to learn from each other, coordinate our actions, and express otherwise hidden thoughts. Critically, human communication goes beyond language production --- humans also express their ideas in visual form. Visual communication lies at the heart of key innovations, and forms the foundation for the cultural transmission of knowledge and higher-level reasoning. This paper examines drawing, the most basic form of visual communication. Communicative uses of drawing pose a core challenge for theories of vision and communication alike: they require a detailed understanding of how sensory information is encoded and how social context guides what information is relevant to communicate. Our computational modeling approach addresses this challenge by combining a high-performing computational model of vision with a formal Bayesian model of social reasoning during communication in order to explain how people flexibly adapt their drawings according to the current context. We employ drawing-based reference games throughout our modeling and experimental work. These reference games involve two players: a \textit{sketcher} who aims to help a \textit{viewer} pick out a target object from an array of alternative, distractor objects by representing it in a drawing. We found that people exploit information in common ground with their partner to efficiently communicate, and that this contextual flexibility was well explained by our computational model. In the long run, understanding the computational basis of visual communication may shed light on the nature of human visual abstraction and the emergence of graphical conventions.
\end{abstract}

\dates{This manuscript was compiled on \today}
\doi{\url{www.pnas.org/cgi/doi/10.1073/pnas.XXXXXXXXXX}}

\begin{document}

% Communication is not limited to verbal language; humans can make use of many different tools to convey meaning to a partner. While theories of communication often rely upon the existence of modality-general pragmatic reasoning mechanisms, these predictions have rarely been tested outside the verbal modality. In this experiment, we aim to test how context -- the set of potential referents in common ground -- affects visual communication: using drawings to convey the identity of rich naturalistic images. Past work in the verbal modality has shown a strong influence of speaker informativity: when the intended referent is surrounded by very similar objects, speakers send more detailed and verbose messages to help the listener distinguish it. We expect analogous results to hold in the visual modality. For instance, when referring to a car in a context containing dogs and birds, speakers should use simpler, more abstract drawings than when referring to the same object in the context of other cars.


% Optional adjustment to line up main text (after abstract) of first page with line numbers, when using both lineno and twocolumn options.
% You should only change this length when you've finalised the article contents.
\verticaladjustment{-2pt}

\maketitle
\thispagestyle{firststyle}
\ifthenelse{\boolean{shortarticle}}{\ifthenelse{\boolean{singlecolumn}}{\abscontentformatted}{\abscontent}}{}

% high level
Communication is central to the success of our species: it allows us to learn from each other, coordinate our actions, and express otherwise hidden thoughts. Critically, human communication goes beyond language production --- humans also express their ideas in visual form. Visual communication lies at the heart of key innovations, and forms the foundation for the cultural transmission of knowledge and higher-level reasoning. Drawing is perhaps the most direct and basic form of visual communication. We perceive drawings of objects as resembling physical objects in spite of the fact that drawings and objects are profoundly different in composition.

In the appropriate context, even a few strokes can express the identity of a face \cite[]{bergmann2013impact}, a suggested route \cite[]{agrawala2001rendering}, or an intention to act \cite[]{Galantucci:2005uh}. Conventionalized representations such as maps, graphs, and diagrams can transmit high-dimensional data and convey complex ideas by recombining relatively few geometric primitives, such as boxes, lines, arrows, ellipses \cite[]{tversky2000lines}. Despite the ubiquity and importance of such visualizations, we lack a unified computational theory of how human perception, action, and social inference are integrated to support flexible visual communication.

Communicative uses of drawing pose a core challenge for theories of vision and communication alike: they require a detailed understanding of how sensory information is encoded, how social context guides what information is relevant to communicate, and estimating the costs of visual production. In this paper, we meet this challenge by combining high-performing models of sensory representation from deep learning with insights from Bayesian cognitive models of social reasoning in language. 

There is now abundant and converging evidence that deep convolutional neural networks (DCNNs) optimized to recognize objects in natural photographs can learn general-purpose feature representations that support a wide variety of visual tasks, including but not limited to object categorization, segmentation, and pose/size estimation \cite[]{hong2016explicit,he2017mask}.

% recent successes learning general visual features, but these inherently acontextual
In particular, despite the fact that drawings are highly abstracted away from natural visual inputs, a relatively modest amount of `fine-tuning' of models previously trained only on photos has been shown to lead to human-level performance on sketch categorization \cite[]{yu2015sketch}, as well as vast improvements in discerning image-level correspondences between sketches and photos \cite[]{sangkloy2016sketchy}. Even without fine-tuning, prior work has shown a striking isomorphism in the representations of object categories in drawings and photos at higher layers in these models \cite[]{fan2015common}. As such, models from this architecture class are a strong candidate for an encoder that can support the level visual abstraction required for representing abstract information in drawings.

However, these visual feature representations are inherently acontextual, and cannot explain why people might use one drawing to communicate an idea in one context (e.g., XX), and a different drawing in another context (e.g., XX). Rational Speech Act (RSA) models provide a probabilistic framework for deriving communicative flexibility from general principles of social cognition \cite{GoodmanFrank16_RSATiCS}.  

These models draw on ideas and insights from decision theory, probabilistic models of cognition, bounded rationality, and linguistics, taking particular inspiration from the insights of Paul Grice (\citeyear{grice1975syntax}), who provided a philosophical framework for understanding how natural language use reflects social reasoning in context. Gricean listeners assume speakers are cooperative, choosing appropriately informative utterances to convey particular meanings. Under this assumption, listeners attempt to infer the speaker's intended communicative goal, working backwards from the form of the utterance. RSA models formalizing this core idea in terms of Bayesian inference have provided unified explanations for a variety of complex linguistic patterns, achieved good quantitative fits with experimental data, and been incorporated into artificial agents that communicate with each other to solve real-world tasks.

% Cite old context-manipulation stuff like Krauss & Weinheimer (1967) or 

Previous work has shown that RSA models account for context sensitivity in human speakers \cite[]{GrafEtAl16_BasicLevel}. 

% For instance, if a speaker needs to refer to a Dalmatian that appears alongside a poodle and a bear, the model predicts that this speaker will prefer the more specific utterance (``Dalmatian'') to the shorter and lower-cost basic-level utterance (``dog''). This is because the literal meaning of the word ``dog'' fits both the Dalmatian and poodle, whereas the word ``Dalmatian'' only fits the Dalmatian. Consequently, the speaker model reasons that ``Dalmatian'' is more informative. By using the word ``Dalmatian'', the speaker encourages the listener to infer the single correct world state (i.e., the target is the Dalmatian) instead of allowing two world states to seem probable, one of which is incorrect (i.e., the target is the poodle). In a context where there is only one dog, however, the speaker will prefer to say ``dog'', which is as informative as ``Dalmatian'' but is less costly to produce. 

\section*{Methods}

\subsection*{Participants}

A total of 192 unique participants, who were recruited via Amazon Mechanical Turk (AMT) and grouped into pairs, completed the experiment. 

\subsection*{Stimuli}

Stimuli were 32 3D mesh models of objects belonging to 4 categories (i.e., birds, chairs, cars, dogs), containing eight objects each. 40 color images of each object were produced by rendering it from a 10$^{\circ}$ viewing angle (i.e., slightly above) at a fixed distance on a gray background, each rotated by an additional 9$^{\circ}$ about the vertical axis. 

\subsection*{Task}

Drawings were collected in the context of an online, sketching-based reference game (``Guess My Sketch!''). The game involved two players: a \textit{sketcher} who aims to help a \textit{viewer} pick out a target object from a set of distractor objects by representing it in a sketch. On each trial, both participants were shown an array of the same four objects; however, the positions of these objects were randomized for each participant. On each trial, one of the four objects was highlighted on the sketcher's screen to designate it as the target. 

Sketchers drew using black ink on digital canvas (pen width = 5 pixels; 300 x 300 pixels) embedded in a web browser window using Paper.js (http://paperjs.org/). Participants drew using the mouse cursor, and were not able to delete previous strokes. Each stroke of which was rendered on the viewer's screen immediately upon the completion of each stroke. There were no restrictions on how long participants could take to make their drawings. After clicking a submit button, the viewer guessed the identity of the drawn object by clicking one of the four objects in the array. Otherwise, the viewer had no other means of communicating with the sketcher. Both participants received immediate task-related feedback: the sketcher learned which object the viewer had clicked, and the viewer learned the identity of the target. Both participants earned bonus points for each correct response. 

For each pair, objects were grouped into eight quartets: Four of these quartets contained objects from the same category (``close''); the other four of these quartets contained objects from different categories (``far'' condition). Each quartet was presented four times, such that each object in the quartet served as the target exactly once. The assignment of objects to quartet and condition was randomized across pairs. 

\subsection*{Modeling}

\subsubsection*{Visual encoder module}

Features were extracted using VGG-19 \cite{simonyan2014very}, a high-performing deep convolutional neural network model architecture that had been pretrained to categorize objects on the Imagenet database, which contains millions of photographs from hundreds of object categories \cite{deng2009imagenet}. Despite the fact that drawings are highly abstracted away from natural visual inputs, prior work has shown a striking isomorphism in the similarity structure \cite{kriegeskorte2008matching} of object categories in drawings and photos at higher layers in these models \cite[]{fan2015common}, without any additional training.

While these models excel at (and were indeed optimized for) categorization of objects, two additional competencies are required to succeed at the visual communication task described above.

First, an observer must be able to represent fine-grained relationships between different images of objects from the same category; second, this observer must also be able to discern image-level correspondences between drawings and photos of the same object. 

In order to better approximate the granularity with which human observers can distinguish different images, we augment VGG19 with a smaller \textit{adaptor} network that learns a common feature representation for drawings and photos. 

This approach has been successfully used in prior work to better predict human semantic categorization judgments \cite{peterson2016adapting}, and to improve sketch-based image retrieval \cite[]{sangkloy2016sketchy}. 


\subsubsection*{Social inference module}

We compare an \emph{asocial sketcher} $S_0$ that simply aims to make a drawing with high perceptual similarity to the target against a \emph{social sketcher} $S_1$ that takes into account context. To instantiate communicative flexibility in context, we generalize the Rational Speech Act (RSA) modeling framework, originally developed for linguistic communication. At the core of the RSA framework, as applied to linguistic communication, is the Gricean maxim of quantity: to make your contribution as informative as is required by context but no more so. In particular, 

Formally, the sketcher, $\mathcal{S}$, is a decision-theoretic agent who produces drawings, $d$, by (soft) maximizing their utility function, $U$, given a particular object referent, $o$: 

{'fraction_non_alphanumeric': 0.052396638241541726, 'fraction_numerical': 0.007573192131268664, 'mean_word_length': 5.427384250098932, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '648653', 'n_tokens_mistral': 7626, 'n_tokens_neox': 7012, 'n_words': 4325}