<jupyter_start><jupyter_text>Week 5: A Naive Bayes classifierSuppose President Trump gets savvy, and realizes that people can use the Android/iPhone distinction to separate his tweets from the the tweets of aides. He starts using an iPhone too. Now, how will we distinguish tweets really authored by the President?Well, one thing we can do is train a classifier to predict authorship using the text itself.<jupyter_code>import os, csv, math, random
import pandas as pd
import numpy as np

from collections import Counter

cwd = os.getcwd()
print('Current working directory: ' + cwd + '\n')
      
relativepath = os.path.join('..', 'data', 'weekfour', 'trump.csv')
trump = pd.read_csv(relativepath)
trump.head()<jupyter_output>Current working directory: /Users/tunder/Dropbox/courses/2017datasci/code<jupyter_text>Let's simplify the Trump dataframe. We only need three columns:1. Text of the tweet2. Source = Android or iphone3. A random number 0-4 that we'll use to divide the dataset into five 'folds'.<jupyter_code>def trump_test(a_data_frame, rowidx):
    if 'iphone' in a_data_frame['statusSource'][rowidx]:
        return 'iphone'
    elif 'android' in a_data_frame['statusSource'][rowidx]:
        return 'android'
    else:
        return 'other'
    
tweet_text = trump['text']

source = []
fold = []
for idx in trump.index:
    source.append(trump_test(trump, idx))
    fold.append(random.sample(list(range(5)), 1)[0])
source = pd.Series(source, index = trump.index)
fold = pd.Series(fold, index = trump.index)

tdf = pd.concat([tweet_text, source, fold], axis = 1)
tdf.columns = ['text', 'source', 'fold']

# limit the dataframe to columns with either android or iphone;
# exclude 'other'
tdf = tdf[(tdf['source'] == 'android') | (tdf['source'] == 'iphone')]
tdf.head()<jupyter_output><empty_output><jupyter_text>Now we need to divide the dataset into a training set and a test set. This is easily done using the "folds." We select one fold as our test set and use all the others as a training set.<jupyter_code>testset = tdf[tdf['fold'] == 4]
trainingset = tdf[tdf['fold'] != 4]
print('Training set includes ' + str(trainingset.shape[0]))
print('Test set includes ' + str(testset.shape[0]))<jupyter_output>Training set includes 1101
Test set includes 289<jupyter_text>Now we need some basic text-wrangling functions that we've used before.<jupyter_code>def tokenize(astring):
    ''' Breaks a string into words, and counts them.
    Designed so it strips punctuation and lowercases everything,
    but doesn't separate hashtags and at-signs.
    '''
    wordcounts = Counter()
    # create a counter to hold the counts
    
    tokens = astring.split()
    for t in tokens:
        word = t.strip(',.!?:;-—()<>[]/"\'').lower()
        wordcounts[word] += 1
        
    return wordcounts

def create_vocab(seq_of_strings, n):
    ''' Given a sequence of text snippets, this function
    returns the n most common words. We'll use this to
    create a limited 'vocabulary'.
    '''
    vocab = Counter()
    for astring in seq_of_strings:
        counts = tokenize(astring)
        vocab = vocab + counts
    topn = [x[0] for x in vocab.most_common(n)]
    return topn<jupyter_output><empty_output><jupyter_text>Now we can actually write functions that build a Naive Bayes model. ```train_nb_model``` is the central function here. It calls the other two.<jupyter_code>def categorize(df, rowidx):
    if df.loc[rowidx, 'source'] == 'android':
        return 'positive'
    elif df.loc[rowidx, 'source'] == 'iphone':
        return 'negative'
    else:
        print('error: neither iphone nor android')
        return 'other'

def get_priors(df):
    source_counts = df.groupby('source').count()['text']
    print(source_counts)
    positive_odds = source_counts['android'] / source_counts['iphone']
    negative_odds = source_counts['iphone'] / source_counts['android']
    return math.log(positive_odds), math.log(negative_odds)

def train_nb_model(df, p):
    vocab = create_vocab(df['text'], p)
    vocabset = set(vocab)
    # we make a set because membership-checking is faster
    # in sets; but we also hold onto the list, which is ordered
    
    positive_prior, negative_prior = get_priors(df)
    
    positive_counts = Counter()
    negative_counts = Counter()
    
    for i in df.index:
        tweet = df['text'][i]
        tweet_counts = tokenize(tweet)
        category = categorize(df, i)
        if category == 'negative':
            negative_counts = negative_counts + tweet_counts
        elif category == 'positive':
            positive_counts = positive_counts + tweet_counts
    
    # Now let's organize these Counters into a DataFrame
    
    negative = pd.Series(1, index = vocab)
    positive = pd.Series(1, index = vocab)
    # notice initializing to 1 -- Laplacian smoothing
    
    for word, count in positive_counts.items():
        if word in vocabset:
            positive[word] += count
    
    for word, count in negative_counts.items():
        if word in vocabset:
            negative[word] += count
    
    all_prob = (negative + positive) / (np.sum(negative) + np.sum(positive))
    
    negative_prob = negative / np.sum(negative)
    positive_prob = positive / np.sum(positive)
    
    # note that when we sum up the negative and positive
    # columns, we are also summing up all the Laplacian 1's
    # we initially added to them
    
    model = pd.concat([negative, positive, all_prob, 
                       negative_prob, positive_prob], axis = 1) 
        
    model.columns = ['neg', 'pos', 'all_prob', 'neg_prob', 'pos_prob']
    
    # The next step is unnecessary, and will not be found in
    # most published versions of naive Bayes. I'm providing it
    # because it may help you understand the logic of the
    # algorithm.
    
    model['neg_norm'] = negative_prob / all_prob
    model['pos_norm'] = positive_prob / all_prob
    
    
    model['log_neg'] = [math.log(x) for x in model['neg_norm']]
    model['log_pos'] = [math.log(x) for x in model['pos_norm']]
    return vocab, positive_prior, negative_prior, model

vocab, positive_prior, negative_prior, model = train_nb_model(trainingset, 75)
model.head()<jupyter_output>source
android    606
iphone     495
Name: text, dtype: int64<jupyter_text>Notice that we're using logarithms of the probabilities, so that we can just add them up. Our priors are logarithms, too.<jupyter_code>print(positive_prior, negative_prior)<jupyter_output>0.20232222350062407 -0.2023222235006242<jupyter_text>Now let's write a function that applies a given model to a given testset. It will have lots of arguments.<jupyter_code>pd.options.mode.chained_assignment = None

def apply_model(vocab, positive_prior, negative_prior, model, testset):
    right = 0
    wrong = 0
    vocabset = set(vocab)
    odds_pos = []
    odds_neg = []

    for i in testset.index:
        odds_positive = positive_prior
        odds_negative = negative_prior
        tweet = testset['text'][i]
        tweet_counts = tokenize(tweet)
        for word, count in tweet_counts.items():
            if word not in vocabset:
                continue
            odds_positive += model.loc[word, 'log_pos']
            odds_negative += model.loc[word, 'log_neg']
            
        if odds_positive > odds_negative:
            prediction = 'positive'
        else:
            prediction = 'negative'
        
        odds_pos.append(odds_positive)
        odds_neg.append(odds_negative)

        reality = categorize(testset, i)
        if reality != 'positive' and reality != 'negative':
            continue
        elif prediction == reality:
            right += 1
        else:
            wrong += 1

    print("Got " + str(right) + " rows right, and " + str(wrong) + " wrong.")
    accuracy = (right / (wrong + right)) * 100
    print("Accuracy was {0:.2f}%".format(accuracy))
    
    resultset = testset.copy()
    resultset['odds_positive'] = odds_pos
    resultset['odds_negative'] = odds_neg
    resultset = resultset.sort_values(by = 'odds_positive')
    
    return resultset

newtestset = apply_model(vocab, positive_prior, 
                         negative_prior, model, testset)<jupyter_output>Got 205 rows right, and 84 wrong.
Accuracy was 70.93%<jupyter_text>The ```apply_model``` function returns a version of the test set with two new columns. The dataframe is sorted by the (ascending) odds of being in the positive class, so we can find the "Trumpiest" and "least Trumpy" tweets by saying ```.tail()``` or ```.head()``` respectively.<jupyter_code>newtestset.tail(20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.106400278778023, 'fraction_numerical': 0.01126727842955047, 'mean_word_length': 3.2814520139234213, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '521138', 'license': 'CC0-1.0', 'n_tokens_mistral': 2806, 'n_tokens_neox': 2653, 'n_words': 824, 'path': 'Homework/week5bayes-copy.ipynb', 'repo_name': 'rdubnic2/lis590txl', 'size': '8609'}