Large scale datasets.
Scaling the size and scope of training datasets has widely been demonstrated to be an effective avenue of improvement for model performance. In computer vision, the adoption of early large scale datasets such as Imagenet[1]}, [2]} and MS-COCO[3]} has dramatically accelerated progress on a variety of tasks including classification, object detection, captioning, and more. Ever since, the diversity and scale of datasets have continued to grow. YFCC100M is a dataset of 99.2M images and 800K videos[4]}. OpenImages[5]} is a large scale dataset of 9M images that contains labeled subsets bounding boxes, visual relationships, segmentation masks, localized narratives, and categorical annotations. Massive web-scraped datasets containing image-text pairs such as Conceptual Captions[6]}, WIT[7]}, and LAION[8]}, [9]} have seen increased popularity recently as they have been used to train impressive models for vision-language representation learning[10]}, [11]}, [12]}, text-to-image generation[13]}, [14]}, [15]}, [12]}, and vision-language multitasking[17]}, [18]}, [19]}, [20]}.
