Engineering a Distributed Full-Text Index
J Fischer, F Kurpicz, P Sanders - arXiv preprint arXiv:1610.03332, 2016
... For our experiments we use the common crawl corpus as input.1 It provides world wide web crawl data and contains raw content, text only and metadata of the crawled websites from about 1.23 billion web pages. In total the corpus has a size of 541 TB (as of 27.07.2016). ...