Learning with Natural Language Supervision.
In parallel, there are also a series of works [1]}, [2]}, [3]}, [4]}, [5]} to learn the visual model guided by natural language features, with the target of learning transferrable models from language. Among them, VirTex [3]} pre-trains the network using dense captions (e.g., COCO Captions [7]}) to learn visual representations and transfer them to downstream visual recognition tasks.
CLIP [5]} performs the natural language supervision on a sufficiently large dataset collected from the Internet and demonstrates its powerful zero-shot transfer capability. In this paper, we leverage fine-grained object labels to enhance object encoders to improve the alignment between object features and language queries.
