Raffaele Sommese, Roland van Rijswijk-Deij, Mattijs Jonker &ndash; University of Twente, The Netherlands
This Is a Local Domain: On Amassing Country-Code Top-Level Domains from Public Data
Domain lists are a key ingredient for representative censuses of the Web. Unfortunately, such censuses typically lack a view on domains under country-code top-level domains (ccTLDs). This introduces unwanted bias: many countries have a rich local Web that remains hidden if their ccTLDs are not considered. The reason ccTLDs are rarely considered is that gaining access -- if possible at all -- is often laborious. To tackle this, we ask: what can we learn about ccTLDs from public sources? We extract domain names under ccTLDs from 6 years of public data from Certificate Transparency logs and Common Crawl. We compare this against ground truth for 19 ccTLDs for which we have the full DNS zone. We find that public data covers 43%-80% of these ccTLDs, and that coverage grows over time. By also comparing port scan data we then show that these public sources reveal a significant part of the Web presence under a ccTLD. We conclude that in the absence of full access to ccTLDs, domain names learned from public sources can be a good proxy when performing Web censuses.
Common Crawl – Common Crawl is a nonprofit organization that builds and maintains a sizable, open repository of Web crawl data, offering years and petabytes of Web page data. The Common Crawl data lives in Amazon S3 as part of Amazon’s Open Data Sponsorship Program and is free for anyone to access. Crawls are seeded from a set of candidate domain names and the crawler follows links leading to other pages. Crawls are performed approximately every one to two months and contain raw Web page data, metadata and text extractions, among others. Relevant to our work, crawls accumulate many tens of millions of registered domain names that one can extract from the so-called URL index. [...] For Common Crawl we consider data for crawl snapshots dated between June 2017 and June 2023 (inclusive). There are 58 such snapshots, collectively accounting for 127 million registered domain names. The combined total number of unique registered domain names in our consolidated dataset is 430 million.