Florian Hantke, Ben Stock &ndash; CISPA Helmholtz Center for Information Security, Germany
[...] we leveraged Common Crawl [22] to analyze more than 23K popular domains over the course of eight years. [...] the crawler framework first collects meta information for each of the listed domains using Common Crawl [22] as a basis for the following analyses (1). This Common Crawl approach makes it possible to take a look into the past and analyze old versions of websites as well as current snapshots. Unlike similar crawling studies before using the Internet Archive[32], with Common Crawl, we are not limited by rate limit issues as we can request the database and S3 bucket directly. This makes the process fast and enables to analyze nearly a thousand pages per minute from one IP address over multiple days. The meta information that the framework collects contains details on where an HTML document can be found in the Common Crawl’s dumps. For each domain, the framework collects meta information from up to 100 pages and hands them to the crawler.