Michał Turski, Tomasz Stanisławek, Karol Kaczmarek, Paweł Dyda, Filip Graliński &ndash; Snowflake; Adam Mickiewicz University, Poznań, Poland
CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data
In recent years, the field of document understanding has progressed a lot. A significant part of this progress has been possible thanks to the use of language models pretrained on large amounts of documents. However, pretraining corpora used in the domain of document understanding are single domain, monolingual, or nonpublic. Our goal in this paper is to propose an efficient pipeline for creating a big-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl, as PDF files are the most canonical types of documents as considered in document understanding. We analysed extensively all of the steps of the pipeline and proposed a solution which is a trade-off between data quality and processing time. We also share a CCpdf corpus in a form or an index of PDF files along with a script for downloading them, which produces a collection useful for language model pretraining. The dataset and tools published with this paper offer researchers the opportunity to develop even better multilingual language models.
As our input we used web indexes created by Common Crawl. [...] They crawl webpages and save them into crawls dumps. A crawl dump contains billions of webpages (hundreds of terabytes of uncompressed data) and a new dump has been published nearly every month since March 2014. Some earlier, more irregular dumps starting from 2008 are also available.¹¹ Each dump also contains an index of the crawled pages. We decided to simply use the latest (and the largest) dump available at the time of writing this paper — the May 2022 dump.¹² [¹²https://commoncrawl.org/2022/06/may-2022-crawl-archive-now-available/] It contains 3.45 billion web pages, which amounts to 462 TB of uncompressed content. It would obviously be possible to apply the extraction procedure described in this paper to all crawls to obtain an even larger collection of PDFs, which would also allow for a diachronic analysis, but we wanted to focus on the most recent documents. Note that dumps contain only files considered as text files by the Common Crawl web robot. Mostly these are web pages in the HTML format, but, fortunately, PDFs are also treated as text files, being derivative of the PostScript page description language. This is not the case with, for instance, images, Excel files, DOCX files. Consequently, such files cannot be amassed using the methods described in the aforementioned papers.¶ 3.2 PDF links extraction¶ We experimented with two methods for extracting links to PDF files (step 1 in Figure 1):¶ 1. using CDX files, i.e., index server files provided by Common Crawl;¶ 2. looking for links to PDF files in WARC, i.e., raw crawl data files.¶ The first method is simpler, as CDX files are easy to download and take up only 225 GB in total. The second method might yield more links to PDF files, but:¶ – it is impossible for us to download all WARCs. Only a limited number of them can be processed, though still a significant number of PDF links can be added even if a small percentage of all WARC files are processed,¶ – there is lower probability that the file linked is available at all, be it in the crawl dump or simply at the original address.¶ In CDX files, the MIME type of a captured file is specified, and we limited ourselves to the application/pdf type.¶ Hence, in this paper, we focus on the first method, which allows to speed up the whole processing pipeline.