All-in-One Image-Grounded Conversational Agents
Da Ju,Kurt Shuster ,Y-Lan Boureau and Jason Weston
Facebook AI Research
fdaju, 
All-in-One Image-Grounded Conversational Agents
As single-task accuracy on individual language and image tasks has improved
substantially in the last few years, the long-term goal of a generally skilled
agent that can both see and talk becomes more feasible to explore. In this
work, we focus on leveraging individual language and image tasks, along with
resources that incorporate both vision and language towards that objective. We
design an architecture that combines state-of-the-art Transformer and ResNeXt
modules fed into a novel attentive multimodal module to produce a combined
model trained on many tasks. We provide a thorough analysis of the components
of the model, and transfer performance when training on one, some, or all of
the tasks. Our final models provide a single system that obtains good results
on all vision and language tasks considered, and improves the state-of-the-art
in image-grounded conversational applications.
Da Ju Kurt Shuster Y-Lan Boureau Jason Weston