Standard image captioning tasks such as COCO and Flickr30k are factual, neutral in tone and (to a human) state the obvious (e.g., “a man playing a guitar”). While such tasks are useful to verify that a machine understands the content of an image,  they are not engaging to humans as captions.    With this in mind we define a new task, Personality-Captions, where the goal is to be as engaging to humans as possible by incorporating controllable style and personality traits. We collect and release a large dataset of 201,858 of such captions conditioned over 215 possible traits.   We build models that combine existing work from (i) sentence representations (Mazaré et al., 2018) with Transformers trained on 1.7 billion dialogue examples; and (ii) image representations (Mahajan et al., 2018) with ResNets trained on 3.5 billion social media images.   We obtain state-of-the-art performance on Flickr30k and COCO, and strong performance on our new task. Finally, online evaluations validate that our task and models are engaging to humans, with our best model close to human performance. If we want machines to communicate with humans, they must be able to capture our interest, which means spanning both the ability to understand and the ability to be engaging, in particular to display emotion and personality as well as conversational function BID17 BID18 BID41 BID19 .Communication grounded in images is naturally engaging to humans BID15 , and yet the majority of studies in the machine learning community have so far focused on function only: standard image captioning BID36 requires the machine to generate a sentence which factually describes the elements of the scene in a neutral tone. Similarly, visual question answering BID2 and visual dialogue BID6 require the machine to answer factual questions about the contents of the image, either in single turn or dialogue form. They assess whether the machine can perform basic perception over the image which humans take for granted. Hence, they are useful for developing models that understand content, but are not useful as an end application unless the human cannot see the image, e.g. due to visual impairment BID13 .Standard image captioning tasks simply state the obvious, and are not considered engaging captions by humans. For example, in the COCO BID5 and Flickr30k BID52 tasks, some examples of captions include "a large bus sitting next to a very tall building" and "a butcher cutting an animal to sell", which describe the contents of those images in a personality-free, factual manner. However, humans consider engaging and effective captions ones that "avoid stating the obvious", as shown by advice to human captioners outside of machine learning.1 For example, "If the bride and groom are smiling at each other, don't write that they are smiling at each other. The photo already visually shows what the subject is doing. Rephrase the caption to reflect the story behind the image". Moreover, it is considered that "conversational language works best. Write the caption as though you are talking to a family member or friend".2 These instructions for human captioners to engage human readers seem to be in direct opposition to standard captioning datasets.In this work we focus on image captioning that is engaging for humans by incorporating personality. As no large dataset exists that covers the range of human personalities, we build and release a new dataset, PERSONALITY-CAPTIONS, with 201,858 captions, each conditioned on one of 215 Standard captioning output: A plate with a sandwich and salad on it. Our model with different personality traits: Sweet That is a lovely sandwich. In this work we consider models that can simultaneously understand image content and provide engaging captions for humans. To build strong models, we first leverage the latest advances in image and sentence encoding to create generative and retrieval models that perform well on standard image captioning tasks. In particular, we attain a new state-of-the-art on caption generation on COCO, and introduce a new retrieval architecture, TransResNet, that yields the highest known hits@1 score on the Flickr30k dataset.To make the models more engaging to humans, we then condition them on a set of controllable personality traits. To that end, we collect a large dataset, PERSONALITY-CAPTIONS to train such models. Using automatic metrics and human evaluations, we show that our best system is able to produce captions that are close to matching human performance in terms of engagement. Our benchmark will be made publicly available to encourage further model development, leaving the possibility of superhuman performance coming soon in this domain.A IMPACT OF PRETRAINED WORD EMBEDDINGS AND TEXT ENCODERS Table 7 : More detailed results for retrieval model performance on COCO Captions using the splits of BID20 . For our TransResNet models, we compare two types of pretraining: Full indicates a model with a pretrained text encoder, while Word indicates a model with pretrained word embeddings only.
We develop engaging image captioning models conditioned on personality that are also state of the art on regular captioning tasks.
{"src": "scitldr-AIC_train", "paper_id": "HJN6DiAcKQ"}