Asier Gutiérrez-Fandiño, David Pérez-Fernández, Jordi Armengol-Estapé, David Griol, Zoraida Callejas &ndash; LHF Labs; Universidad Autónoma de Madrid, Spain; University of Edinburgh, United Kingdom; Universidad de Granada, Spain
[…] In this paper, we introduce esCorpius, a Spanish crawling corpus obtained from near 1 Pb of Common Crawl data. It is the most extensive corpus in Spanish with this level of quality in the extraction, purification and deduplication of web textual content […] A total of 39,502 compressed WARC (Web Archive) from Common Crawl files were processed (see section 3.3 for more details). The compressed information occupied about 180 TB and the size of the processed decompressed information is estimated to be more than 0.8 PB. Prior to content deduplication, the downloaded corpus was composed of 106.768.594.753 words, 3.129.248.875 lines and 163.518.405 web pages. The deduplicated and cleaned corpus size is 346.262.072.705 bytes (322.5 GB), with 104.073.706 total number of lines, 50.040.055.322 tokens, 1.125.798.968 paragraphs and 2.421.598.201 sentences.