<jupyter_start><jupyter_text>Comparing Common Crawl's Domain-Level Harmonic Centrality Ranks A comparison of top web sites rankings- the list of [top-1-million sites](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip) published by [Alexa](https://support.alexa.com/hc/en-us/sections/200063274-Top-Sites), based on unique visitors and page views- the [Cisco Umbrella Popularity list](http://s3-us-west-1.amazonaws.com/umbrella-static/index.html) which reflects DNS usage- the [Majestic Million](http://downloads.majestic.com/majestic_million.csv), "[ordered by the number of referring subnets](https://blog.majestic.com/development/majestic-million-csv-daily/)"- the [Tranco list](https://tranco-list.eu/)- the top million domains ranked by harmonic centrality from [Common Crawls Jun/Jul/Sep 2021 webgraph dataset](https://commoncrawl.org/2021/10/host-and-domain-level-web-graphs-jun-jul-sep-2021/)The work is largely inspired by the [Tranco research paper](https://tranco-list.eu/assets/tranco-ndss19.pdf) by <NAME>, <NAME>, <NAME>, <NAME> and <NAME>.Caveats: the lists differ in their- notion of a web site (a host, a subdomain, a registered domain)- methodology to rank sites  - Alexa: traffic / visitors  - Cisco Umbrella: DNS traffic  - Majestic: backlinks aggregated by IPv4 /24 subnets  - Tranco: weighted combination of the above (plus Quantcast)  - Common Crawl: harmonic centrality- data aggregation time: while the others provide daily updates, Common Crawl releases domain rankings 3-4 times per year  List Download and Read DataA shell script to download all lists (executed on Oct 09, 2021):```bash!/bin/bashdir="ranks-2021-10-09"mkdir -p "$dir"curl "http://s3.amazonaws.com/alexa-static/top-1m.csv.zip" | gzip -dc >"$dir"/alexa.csvcurl "http://downloads.majestic.com/majestic_million.csv" >"$dir"/majestic.csvcurl "http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip" | gzip -dc >"$dir"/umbrella.csv find latest Tranco listcurr_tranco_link=$(lynx -dump "https://tranco-list.eu/latest_list" \                    | grep -Eo 'https://tranco-list.eu/download/[^/]+/1000000')curr_tranco_id=$(echo $curr_tranco_link | cut -d/ -f5 | tr -d '\n')curl $curr_tranco_link >"$dir"/tranco.csv take top 1M from latest Common Crawl domain-level rankingscurl https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/cc-main-2021-jun-jul-sep-domain-ranks.txt.gz | gzip -dc | head -1000001 >"$dir"/cc.tsv```<jupyter_code># read the data

import os

import pandas as pd

base = 'ranks-2021-10-09'

lists = {}

lists['Alexa'] = pd.read_csv(os.path.join(base, 'alexa.csv'), header=None, names=['rank', 'domain'])
lists['Umbrella'] = pd.read_csv(os.path.join(base, 'umbrella.csv'), header=None, names=['rank', 'domain'])
lists['Tranco'] = pd.read_csv(os.path.join(base, 'tranco.csv'), header=None, names=['rank', 'domain'])

# Majestic
majestic_ = pd.read_csv(os.path.join(base, 'majestic.csv'))
majestic = pd.DataFrame()
majestic[['rank', 'domain']] = majestic_[['GlobalRank', 'Domain']]
lists['Majestic'] = majestic

# Common Crawl
def reverse_host(host):
    parts = host.split('.')
    parts.reverse()
    return '.'.join(parts)

cc = pd.read_csv(os.path.join(base, 'cc.tsv'), delimiter='\t')
cc['domain'] = cc['#host_rev'].apply(reverse_host)
cc.rename(columns={'#harmonicc_pos': 'rank'}, inplace=True)
cc.drop(columns=['#harmonicc_val', '#pr_pos', '#pr_val', '#host_rev', '#n_hosts'], inplace=True)
lists['CC'] = cc<jupyter_output><empty_output><jupyter_text>Top 20 Sites<jupyter_code>top_df = pd.DataFrame()
for l in lists:
    top_df[l] = lists[l].head(20)['domain']

top_df<jupyter_output><empty_output><jupyter_text>Count of Top-Level Domains among the Top-1M SitesSee also [TLDS in cc-crawl-statistics](https://commoncrawl.github.io/cc-crawl-statistics/plots/tlds) which reflect the TLD coverage in web page captures<jupyter_code># count TLDs

from collections import defaultdict

tlds = defaultdict(dict)

def tld(host):
    return host.split('.')[-1]

for l in lists:
    for idx, row in lists[l]['domain'].apply(tld).value_counts().iteritems():
        tlds[l][idx] = row

tld_df = pd.DataFrame(tlds, dtype=int).fillna(0).astype(int).sort_values(['CC'], ascending=0)
tld_df.head(10)
# percentage of TLD in top-million domain lists

tld_df.apply(lambda x: 100 * x / float(x.sum())).head(20)
# percentage of continents by country-code TLD

#!pip install a-world-of-countries

import awoc

from icctld import icctld2cctld

world = awoc.AWOC()
tld_continent = {}

for continent in world.get_continents_list():
    for country in world.get_countries_list_of(continent):
        tld = world.get_country_TLD(country)
        if tld:
            tld_continent[tld] = continent

for icctld in icctld2cctld:
    if icctld2cctld[icctld] in tld_continent:
        tld_continent[icctld] = tld_continent[icctld2cctld[icctld]]

continents = tld_df.reset_index()
continents['Continent'] = continents['index'].apply(
    lambda x: tld_continent[x] if x in tld_continent else '(generic etc.)')
continents.groupby(['Continent']).sum().apply(
    lambda x: 100 * x / float(x.sum())).sort_values(['CC'], ascending=0)<jupyter_output><empty_output><jupyter_text>Correlation between Ranked ListsRank-Biased Overlap (RBO), see- Webber, Moffat, Zobel, 2010: [A similarity measure for indefinite rankings](http://codalism.com/research/papers/wmz10_tois.pdf)- the [rbo](https://pypi.org/project/rbo/) Python module<jupyter_code>#!pip install rbo

import rbo

corr_matrix = defaultdict(dict)

for n in lists:
    for m in lists:
        if n == m:
            corr_matrix[n][m] = corr_matrix[m][n] = 1.0
            continue
        if n > m:
            continue
        nl = lists[n]['domain'].values
        ml = lists[m]['domain'].values
        rbo_sim = rbo.RankingSimilarity(nl, ml).rbo()
        overlap = len(set(nl) & set(ml))
        union = len(set(nl) | set(ml))
        jaccard_sim = overlap / union
        print("{:.3f}\t{:6d}\t{:7d}\t{:.3f}\t{} <> {}".format(rbo_sim, overlap, union, jaccard_sim, n, m))
        corr_matrix[n][m] = corr_matrix[m][n] = rbo_sim
corr = pd.DataFrame(corr_matrix)
corr
import matplotlib.pyplot as plt
import seaborn as sb

corr_heatmap = sb.heatmap(corr, cmap="Blues", annot=True)
plt.savefig('corr_ranks_rbo.svg')
corr_heatmap<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15451526174284358, 'fraction_numerical': 0.019136485845326585, 'mean_word_length': 4.6063829787234045, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 29, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '367894', 'license': 'Apache-2.0', 'n_tokens_mistral': 2429, 'n_tokens_neox': 2273, 'n_words': 497, 'path': 'cc-webgraph-statistics/comparison_domain_ranks.ipynb', 'repo_name': 'sebastian-nagel/cc-notebooks-var', 'size': '6378'}