<jupyter_start><jupyter_text>Test JPL webpageclassifierUsing data from THH set.<jupyter_code>!pwd
import pandas as pd
pd.set_option('display.chop_threshold', 1e-5)
pd.set_option('display.latex.repr', True)
pd.set_option('display.precision', 2)
scores = pd.read_csv('../webpageclassifier/scores.csv')
scores = scores.drop('Unnamed: 0', axis=1)
len(scores)<jupyter_output><empty_output><jupyter_text>Distribution of Confidence by Category<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
scores.plot.hist(subplots=True, layout=(4,2))
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Raw Accuracy by CategoryNote that the JPL classifier knows nothing of _search_engines_, so the 0 is uninformative. It also has a hard time telling _classified_ from _shopping_, at least in this incarnation.<jupyter_code>scores.filter(['pagetype','Correct?']).groupby('pagetype').mean()<jupyter_output><empty_output><jupyter_text>Brier ScoreAll scores are in [0,1], so squared loss on $x^2$ or $(1-x)^2$ makes sense, but it's not a true Brier score because these scores are unnormalized: they do not usually sum to 1.0 For quick benchmark, we compare them to a uniform $1/k$ forecast for $k=8$ categories.<jupyter_code>scores.ix[0]
brier_tot, hits, N = 0, 0, len(scores)
for i in range(N):
    row = scores.ix[i]
    cat = row['pagetype'][:4]
    brier_row = 0
    if row['Correct?']:
        hits += 1
    for col in ['blog','classified','forum','news','shopping','undecided','wiki']:
        val = row[col]
        if col.startswith(cat):
            brier_row += (1-val)**2
        else:
            brier_row += val**2
    brier_tot += brier_row
brier = brier_tot/N
print('Accuracy: {:5.2f} = {:d}/{:d}'.format(hits/N, hits, N))
print('Brier   : {:5.2f}'.format(brier))
print('Uniform : {:5.2f}'.format((1/7)**2 + (6/7)**2))<jupyter_output>Accuracy:  0.50 = 260/522
Brier   :  0.68
Uniform :  0.76<jupyter_text>So... hit rate so-so but clearly better than uniform guess. OTOH, Brier is only about 11% better than uniform, so not well-calibrated. Likely getting clobbered on some 1/0 guesses.  Normalize...<jupyter_code>vals = scores.filter(['blog','classified','forum','news','shopping','undecided','wiki'])
totals = vals.sum(axis=1)
probs = vals.divide(totals, axis=0)
probs['pagetype'] = scores['pagetype']
p2 = probs.groupby('pagetype')
p2.mean()
p2.sum()
import timeit
timeit.timeit("D[12]**2", setup="D = dict(((x,2*x) for x in range(1000)))")
timeit.timeit("D[12] * D[12]", setup="D = dict(((x,2*x) for x in range(1000)))")
timeit.timeit("12**2")
timeit.timeit("12*12")
X=['a','b','c','d']
X.remove('c')
X
X += 'e'
X
timeit.timeit("X += 'f'", setup="X=['a','b','c','d']")
timeit.timeit("X.append('f')", setup="X=['a','b','c','d']")
[(k,v) for k,v in {1:3, 2:3, 3:6}.items()]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18603827072997875, 'fraction_numerical': 0.03260099220411056, 'mean_word_length': 5.459954233409611, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '417726', 'license': 'Apache-2.0', 'n_tokens_mistral': 1145, 'n_tokens_neox': 1067, 'n_words': 245, 'path': 'JPL webpageclassifier Test.ipynb', 'repo_name': 'Sotera/webpageclassifier', 'size': '2822'}