Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi &ndash; Allen Institute for Artificial Intelligence, USA; University of California, Santa Barbara, USA; Paul G. Allen School of Computer Science, University of Washington, USA; Columbia University, USA; Yonsei University, South Korea; LAION
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text
Multimodal C4 is an expansion of the text-only c4 dataset [21], which was created by taking the April 2019 snapshot from Common Crawl4 and applying several filters with the intention of retaining high-quality, natural English text. Each document in c4 consists of the text scraped from one URL. [...] e built the mmc4 dataset on top of c4 because: 1) c4 is a web-scale dataset widely adopted as a pre-training corpus [21 , 25, 9 , 29, 27 ]; 2) c4 is constructed from web pages, which frequently contain multimedia content like images: a multimodal sequence version is a natural extension; and 3) c4-en,5 the specific underlying subset from which we construct mmc4 has already been processed with several data-cleaning steps (including English- language identification by langdetect6 with at least 0.99 confidence; text deduplication removing duplicate three-sentence spans + placeholder text like “lorem ipsum{"}; and removal of any document containing any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”).7 See [ 21] for more information about the text-only c4. Importantly, by building on the popular text-only c4, prior text-only documentation efforts [ 11] can provide insight about potential biases and risks that could arise when training on our multimodal extension. We use the NLTK [4] sentence tokenizer to chunk each c4 document into a list of sentences.¶ Gathering images. We first retrieve the original webpages for each document in the c4-en dataset from the Common Crawl version 2019-18, which is the default version for c4. Next, we extract the URLs for downloadable images from the raw WAT files. We restrict the image extension to either png/jpeg/jpg, and exclude image URLs that contain the following tokens: tlogo, button, icon, plugin, widgetu. We attempt to download from these URLs, and resize images to a maximum dimension of 800px. We eliminate any c4 documents that do not contain valid, downloadable images at the time of collection (mid-to-late 2022). The starting point after this step is 115M documents and 1.37B images.