Massive datasets have enabled and driven rapid progress in AI. Language corpora on the web led to large language models like GPT-3 [1]}; paired image and text datasets like Conceptual Captions [2]} led to vision-and-language pretrained models like VilBERT [3]}; YouTube video datasets led to video capable models like Merlot-Reserve [4]}; and massive multimodal datasets like WebImageText [5]} and LAION [6]}, [7]} led to models like CLIP [8]} and StableDiffusion [9]}. These leaps in dataset scale and diversity were triggered by moving from manually curated datasets to harnessing the power of the web and its creative content.
