Johannes Kiesel, Florian Kneist, Milad Alshomary, Benno Stein, Matthias Hagen, Martin Potthast &ndash; Paderborn University, Germany; Bauhaus-Universität Weimar, Germany; Martin-Luther-Universität Halle-Wittenberg, Germany; Leipzig University, Germany; Ulm University, Germany
Reproducible Web Corpora: Interactive Archiving with Automatic Quality Assessment
Paderborn University, Germany; Bauhaus-Universität Weimar, Germany; Martin-Luther-Universität Halle-Wittenberg, Germany; Leipzig University, Germany; Ulm University, Germany
web-mining, nlp/web-as-corpus
To build a solid benchmark dataset for web reproduction quality assessment, we carefully sampledweb pages with the goal of representing a wide cross-section of the different types and genres of webpages found on the web. As a population of web pages to draw a sample from, we resort to the recentbillion-page Common Crawl 2017-04 [36]. From there, we primarily sampled pages from most ofthe well-known sites—as defined by the website’s Alexa traffic rank [1]⁶—to ensure that our sampleencompasses pages using the most recent web technologies and design standards. Moreover, pagesfrom a number of less well-known sites have been included. Altogether, the Webis Web Archive 17 comprises 10,000 web pages.