<jupyter_start><jupyter_text>Naive Bayes classifiersOur goal is to "train" a NB classifier (really just count things) and apply it to some new documents.<jupyter_code>import numpy as np
from collections import Counter
import regex<jupyter_output><empty_output><jupyter_text>Simple word definition, we could do more complicated things like identify latex segments.<jupyter_code>word_pattern = regex.compile("\w+")
def read_file(filename):
    docs = []
    sum_counter = Counter()
    total_tokens = 0
    
    with open(filename) as reader:
        for line in reader:
            fields = line.rstrip().split("\t")
            if len(fields) < 3:
                continue
                
            tokens = word_pattern.findall(fields[2])
            counts = Counter(tokens)
            sum_counter.update(tokens)
            total_tokens += len(tokens)
            
            doc = { "name": fields[0], "label": fields[1], "tokens": tokens, "counts": counts }
            docs.append(doc)
    
    return (sum_counter, total_tokens, docs)
ds_counter, ds_tokens, ds_docs = read_file("datascience10k.txt")
stat_counter, stat_tokens, stat_docs = read_file("stats10k.txt")
print(ds_tokens, stat_tokens)
ds_counter.most_common(10)
stat_counter.most_common(10)
# create a union vocabulary so we have one list of all the words we know about

everything_counter = ds_counter + stat_counter
vocabulary = list(everything_counter.keys())
vocab_size = len(vocabulary)
vocab_set = set(vocabulary)
vocab_size<jupyter_output><empty_output><jupyter_text>Zipf's lawAlmost half the distinct terms occur exactly once, but about 100 words make up half of all the tokens.<jupyter_code># there are 23k word types that occur once 

Counter(everything_counter.values()).most_common(10)
# 10 words make up about 20% of the collection

cumulative_proportion = 0.0

for w, c in everything_counter.most_common(100):
    proportion = c / (ds_tokens + stat_tokens)
    cumulative_proportion += proportion
    
    print("{}\t{:.4f}\t{:.1f}%\t{}".format(c, proportion, 100 * cumulative_proportion, w))
# turn counts into probability with Dirichlet smoothing

def log_prob(word, counts, total, smoothing=0.1):
    prob = (counts[word] + smoothing) / (total + vocab_size * smoothing)
    return np.log( prob ) 
# two example docs I'm pretty sure aren't in the training set

test_stat_doc = "Suppose X1 and X2 are independent. Both have Bernoulli distribution, equal to one with probability 0.5. What is the probability distribution of the product X1X2? What is the covariance between X1 and X1X2? I am a beginner at statistics and quite confused by these conceptions. Would you please offer some guidance? Thank you!"
test_ds_doc = "Can you tell me cheap services to train ai models in the cloud, and that they are easy to configure. I would like something like Google Collab, simple but with more powerful gpu. I want the service to come with python installed so that I can program in the cloud directly and not install python on my pc."
def compare_tokens(doc, smoothing=0.1):
    total = 0
    for token in word_pattern.findall(doc):
        if token in vocab_set:
            ds_log_prob = log_prob(token, ds_counter, ds_tokens, smoothing)
            stat_log_prob = log_prob(token, stat_counter, stat_tokens, smoothing)
            diff = ds_log_prob - stat_log_prob
            total += diff
            print(f"{ds_log_prob:.02f}\t{stat_log_prob:.02f}\t{diff:.02f}\t{token}")
        else:
            print("skipping ", token)
    print(total)<jupyter_output><empty_output><jupyter_text>Intuition for log probabilitiesIt's helpful to have a rough idea what the scale of a log probability means. All log probabilities are negative or zero (why?) so if we drop the negative and exponentiate, we get the equivalent "one in ..." number.Note that in this class $\log$ means natural log.<jupyter_code>np.log(1/10)
np.exp(2.302586)
for lp in range(1,15):
    print("log prob -{} ~ one in {: 10.1f}".format(lp, np.exp(lp)))
np.exp(-1.4)
ds_counter["Python"]
stat_counter["Python"]
# Capitalization is consistently more stats-ey (negative)
# (including "the" as a baseline)

compare_tokens("R r Python python The the")
compare_tokens(test_stat_doc, 10.0)
np.exp(7.439)
compare_tokens(test_ds_doc, 10)<jupyter_output>-8.21	-8.33	0.12	Can
-4.78	-5.00	0.23	you
-8.92	-8.82	-0.10	tell
-7.11	-7.17	0.06	me
-11.20	-11.48	0.28	cheap
-11.01	-11.21	0.19	services
-3.92	-4.05	0.13	to
-7.00	-8.77	1.77	train
-10.51	-12.18	1.67	ai
-7.14	-7.18	0.04	models
-4.67	-4.62	-0.05	in
-3.37	-3.23	-0.14	the
-10.13	-11.40	1.26	cloud
-4.35	-4.33	-0.02	and
-4.83	-4.68	-0.15	that
-6.89	-6.85	-0.04	they
-5.25	-5.19	-0.06	are
-8.65	-8.83	0.18	easy
-3.92	-4.05	0.13	to
-11.49	-11.87	0.38	configure
-4.25	-4.32	0.08	I
-6.04	-5.96	-0.07	would
-6.18	-6.52	0.34	like
-7.55	-7.66	0.11	something
-6.18	-6.52	0.34	like
-9.06	-10.42	1.35	Google
-11.98	-12.27	0.29	Collab
-7.72	-7.77	0.05	simple
-5.86	-5.88	0.02	but
-5.25	-5.30	0.05	with
-6.33	-6.46	0.13	more
-10.01	-10.37	0.36	powerful
-11.09	-12.27	1.19	gpu
-4.25	-4.32	0.08	I
-6.55	-6.75	0.20	want
-3.37	-3.23	-0.14	the
-10.62	-10.86	0.24	service
-3.92	-4.05	0.13	to
-8.77	-8.78	0.01	come
-5.25	-5.30	0.05	with
-8.16	-10.12	1.96	python
-10.43	-11.48	1.05	installed
-6.40	-6.39	-0.02	so
-4.83	-4.6[...]<jupyter_text>Evaluation as a classifierNB isn't the best classifier, but it's pretty close, and the fact that it's just counting things makes leave-one-out cross validation really easy. We don't want to test on the training set, but we can easily create a test set of one for each doc by pretending we haven't seen it.Note that I'm leaving out the prior probability of data science vs. stats, since I made this 50/50 by construction.<jupyter_code>ds_tokens
smoothing = 1.0

num_correct = 0
num_docs = 0

for doc in ds_docs:
    # pretend we haven't seen this before
    for word, count in doc["counts"].items():
        ds_counter[word] -= count
        ds_tokens -= count
    
    total_diff = 0.0
    for token in doc["tokens"]:
        if token in vocab_set:
            ds_log_prob = log_prob(token, ds_counter, ds_tokens, smoothing)
            stat_log_prob = log_prob(token, stat_counter, stat_tokens, smoothing)
            diff = ds_log_prob - stat_log_prob
            total_diff += diff
    
    if total_diff > 0.0:
        num_correct += 1
    num_docs += 1
    
    # add it back in
    for word, count in doc["counts"].items():
        ds_counter[word] += count
        ds_tokens += count
    
    if ds_tokens != 1260636:
        print(doc)
        break

print(f"DS\t{num_correct}\t{num_docs}")
ds_tokens
smoothing = 1.0

num_correct = 0
num_docs = 0

for doc in stat_docs:
    # pretend we haven't seen this before
    for word, count in doc["counts"].items():
        stat_counter[word] -= count
        stat_tokens -= count
    
    total = 0.0
    for token in doc["tokens"]:
        if token in vocab_set:
            ds_log_prob = log_prob(token, ds_counter, ds_tokens, smoothing)
            stat_log_prob = log_prob(token, stat_counter, stat_tokens, smoothing)
            diff = ds_log_prob - stat_log_prob
            total += diff
    
    if total < 0.0:
        num_correct += 1
    num_docs += 1
    
    # add it back in
    for word, count in doc["counts"].items():
        stat_counter[word] += count
        stat_tokens += count

print(f"stat\t{num_correct}\t{num_docs}")<jupyter_output>stat	7421	9937
{'fraction_non_alphanumeric': 0.1105879187407507, 'fraction_numerical': 0.07184178662720302, 'mean_word_length': 2.943766578249337, 'pattern_counts': {'":': 4, '<': 21, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '138785', 'license': 'CC0-1.0', 'n_tokens_mistral': 2964, 'n_tokens_neox': 2654, 'n_words': 835, 'path': 'notebooks/template_naivebayes_6150.ipynb', 'repo_name': 'archaud/thelayoff', 'size': '7433'}