<jupyter_start><jupyter_text>SMS Spam Filter We will build a basic spam filter using the UCI SMS Spam Collection Data Set: https://archive.ics.uci.edu/ml/datasets/sms+spam+collection<jupyter_code>import numpy as np
import string
import pandas as pd
import matplotlib.pyplot as plt
import re
from collections import Counter
from sklearn.model_selection import train_test_split
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>We begin by reading in the data and only keeping columns that indicate spam/ham and content of message.<jupyter_code>df = pd.read_csv('spam.csv',encoding='latin-1')

df = df[['v1','v2']]

df.columns = ['Spam','Message']
df.head()<jupyter_output><empty_output><jupyter_text>Check for any disparity between amount of spam and ham messages.<jupyter_code>df.Spam.value_counts().plot(kind='bar', title='Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Ham messages far exceed spam messages in this data set. Convert spam/ham column to 0='ham' and 1='spam'.<jupyter_code># check which rows are spam to get TRUE/FALSE then multiply Spam column by 1
df['Spam'] = df.Spam.eq('spam').mul(1)
df.head()<jupyter_output><empty_output><jupyter_text>Separate into train and test set with 25% of data going to test and utilize stratify to ensure the same proportion of spam in each set.<jupyter_code>train_set, test_set = train_test_split(df,test_size=0.25,stratify=df['Spam'],random_state=42)<jupyter_output><empty_output><jupyter_text>Check ratio of spam in both train and test set. Want the values to be close.<jupyter_code>train_SpamRatio = round(sum(train_set['Spam'])/len(train_set)*100,2)
test_SpamRatio = round(sum(test_set['Spam'])/len(test_set)*100,2)
print('train spam: %.2f%%  \ntest spam: %.2f%% ' %(train_SpamRatio,test_SpamRatio))<jupyter_output>train spam: 13.40%  
test spam: 13.42%<jupyter_text>The values are close enough so we may proceed. We will define the function messageList to transform each message into a list of strings and remove nonchar elements as well as converting to lowercase.<jupyter_code>def messageList(s):
    # split strings and convert to lowercase
    s = re.findall(r"[\w']+|[.,!?;]", s.lower())
    # remove punctuations and digits
    s = [x.translate(str.maketrans('', '', string.punctuation)) for x in s ]
    l = [x for x in s if not any(c.isdigit() for c in x)]
    final = [x for x in l if x.isalnum()]
    return(final)<jupyter_output><empty_output><jupyter_text>Separate ham and spam observations into their own dataframes to gain further insights into each group individually.<jupyter_code>ham = train_set.loc[(train_set['Spam']==0)]
spam = train_set.loc[(train_set['Spam']==1)]<jupyter_output><empty_output><jupyter_text>Apply messageList function to both dataframes.<jupyter_code>ham.loc[:,'messageList'] = pd.Series(ham['Message'].apply(messageList))
spam.loc[:,'messageList'] = pd.Series(spam['Message'].apply(messageList))<jupyter_output><empty_output><jupyter_text>Build ham and spam word libraries.<jupyter_code># isolate all words in ham messages
hamLib = ham.messageList.tolist()
hamLib = list([a for b in hamLib for a in b])

# isolate all words in spam messages
spamLib = spam.messageList.tolist()
spamLib = list([a for b in spamLib for a in b])

# build separate list of all ham and spam words
# Note one could input an integer n to the most_common function
# in order to return only the top n words in each category. 
hamLib_Common = [word for word, word_count in Counter(hamLib).most_common()]
spamLib_Common = [word for word, word_count in Counter(spamLib).most_common()]<jupyter_output><empty_output><jupyter_text>Want to remove words that show up in both ham and spam libraries.<jupyter_code># Define function to return words that are common to both ham and spam libraries.
def intersection(l1, l2): 
    temp = set(l2) 
    l3 = [val for val in l1 if val in temp] 
    return(l3)
# define library of intersecting words between ham and spam
interLib = intersection(hamLib_Common,spamLib_Common)
# remove intersecting words from the ham and spam libraries
hamLib2 = list(set(hamLib_Common)^set(interLib))
spamLib2 = list(set(spamLib_Common)^set(interLib))<jupyter_output><empty_output><jupyter_text>Form totalLib using the unique ham and unique spam words.<jupyter_code>totalLib = hamLib2 + spamLib2
# Apply messageList function to trainining set
train_set.loc[:,'messageList'] = pd.Series(train_set['Message'].apply(messageList))<jupyter_output><empty_output><jupyter_text>Find length of SMS.<jupyter_code>def messageLength(messList):
    return(len(messList))
train_set['messageLen'] = train_set['messageList'].apply(messageLength)<jupyter_output><empty_output><jupyter_text>As most Spam messages contain numbers and multiple capital letters, we will add a feature to calculate this.<jupyter_code>def howManyCapsInts(messList):
    caps = sum(1 for c in messList if c.isupper())
    ints = sum(c.isdigit() for c in messList)
    return(caps+ints)
train_set['caps&ints'] = train_set['Message'].apply(howManyCapsInts)<jupyter_output><empty_output><jupyter_text>Also we will add a feature that shows the ratio of nonalphanumeric characters to the length of the message.<jupyter_code>#function to figure out the ratio of non alphanumeric characters to length of string
def non_alphaNum(s):
    length = len(s)
    pattern = r'[^a-zA-Z0-9\_\s]'
    result = re.findall(pattern, s)
    return(len(result)/length)
train_set['ratio'] = train_set['Message'].apply(non_alphaNum)<jupyter_output><empty_output><jupyter_text>Obtain words in each message that fall into the totalLib library. We will apply bag of words to this list later.<jupyter_code># return list of words in each original message that also fall in totalLib constructed above
def topMessage(m):
    uniqueM = set(m)
    res = [x for x in totalLib if x in m]
    return(res)
train_set.loc[:,'cleanMessage'] = pd.Series(train_set['messageList'].apply(topMessage))
# define bag of words function to vectorize each SMS
def bow(m,vocab):
    bag = np.zeros(len(vocab))
    for word in m:
        for i,j in enumerate(vocab):
            if j==word:
                bag[i]+=1
                
    return(np.array(bag))<jupyter_output><empty_output><jupyter_text>Build new dataframe of vectorized messages with totalLib word library as features and merge with message length, of capital and digits, and proportion of nonalphanumeric to form full training set.<jupyter_code>vec =  pd.DataFrame()

vec['col'] = (np.array(train_set.apply(lambda row : bow(row['cleanMessage'], 
                                 totalLib), axis = 1)))

vec =  vec.col.apply(pd.Series)<jupyter_output><empty_output><jupyter_text>Now we will apply term frequency-inverse document frequency (tf-idf). The key idea is to weigh words like 'the','that','is' less as these terms likely appear frequently in the messages. More weight is added to words that appear several times in a single SMS but do not appear much in the rest of the collection of messages. Generally applying tf-idf performs better than raw word counts.<jupyter_code>def tf_idf(freq):
    # divide each entry by its row sum then multiply by log (#rows/column sum)
    size = len(freq)
    tf = freq.div(freq.sum(axis=1)+.1, axis=0)
    idf = np.log(size/freq.sum())
    tfidf = tf*idf
    return([tfidf,idf])
# save idf value to apply it to test set later on
vec,idf = tf_idf(vec)
trainX = train_set[['messageLen','caps&ints','ratio']]
trainX = trainX.reset_index().drop('index',axis=1)
trainX = pd.concat([trainX, vec], axis=1, sort=False)
trainY = train_set['Spam']<jupyter_output><empty_output><jupyter_text>We will now attempt different algorithms in search of one that offers strong performance on precision. As this is a spam filter, the biggest concern would be a user missing an SMS that should not have been classified as spam.<jupyter_code>from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import MultinomialNB,GaussianNB
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_auc_score
from sklearn.metrics import classification_report
# model =  SVC(C=10,class_weight='balanced',gamma='auto',kernel='rbf')
model = RandomForestClassifier(n_estimators=100,max_features= 'log2',
                               class_weight='balanced',random_state=42,max_depth=50) 
# model = MultinomialNB(fit_prior=True)
scores = cross_val_score(model,trainX,trainY,scoring='precision',cv=5)
print('Median precision:' ,round(np.median(scores),5))<jupyter_output>Median precision: 0.99057<jupyter_text>GridSearch or RandomizedSearch can be utilized to fine tune whichever model was chosen by user. Sample code for RandomizedSearchCV is shown below.<jupyter_code># from sklearn.model_selection import RandomizedSearchCV

# param_grid = {'n_estimators': [100,200,300], 'max_features': ['auto',None],'class_weight': 
#     ['balanced','balanced_subsample',None],'max_depth':[10,20]}


# model = RandomForestClassifier()

# random_search = RandomizedSearchCV(model,param_distributions=param_grid,cv=5,scoring='f1')<jupyter_output><empty_output><jupyter_text>Prepare test set.<jupyter_code>testY = test_set['Spam']
test_set2 = test_set.drop(['Spam'],axis=1)
test_set2.loc[:,'messageList'] = pd.Series(test_set2['Message'].apply(messageList))
test_set2['messageLen'] = test_set2['messageList'].apply(messageLength)
test_set2['caps&ints'] = test_set2['Message'].apply(howManyCapsInts)
test_set2['ratio'] = test_set2['Message'].apply(non_alphaNum)
test_set2.loc[:,'cleanMessage'] = pd.Series(test_set2['messageList'].apply(topMessage))
test_vec =  pd.DataFrame()
test_vec['col'] = (np.array(test_set2.apply(lambda row : bow(row['cleanMessage'], 
                                 totalLib), axis = 1)))
test_vec =  test_vec.col.apply(pd.Series)
test_tf = test_vec.div(test_vec.sum(axis=1)+.1, axis=0)
test_vec = test_tf * idf
testX =  test_set2[['messageLen','caps&ints','ratio']]
testX = testX.reset_index().drop('index',axis=1)
testX = pd.concat([testX,test_vec],axis=1,sort=False)<jupyter_output><empty_output><jupyter_text>Fit model and make predictions.<jupyter_code>model.fit(trainX,trainY)
feat_imp = model.feature_importances_
np.argsort(feat_imp)[-3:]<jupyter_output><empty_output><jupyter_text>2 of the top 3 important features were message length and  of captial and integers.<jupyter_code>results = model.predict(testX)
print(classification_report(testY, results))
auc = roc_auc_score(testY, results, average='macro')
print('Area under ROC:',round(auc,2))<jupyter_output>Area under ROC: 0.93<jupyter_text>Scores all around are good but most importantly the probability of an SMS actually being spam given a prediciton of spam, is at 100%. Recall dips below .9 but we can live with a few spam messages getting through. Summarize with confusion matrix.<jupyter_code>from sklearn.metrics import confusion_matrix
conf_mat = confusion_matrix(testY, results)
plt.figure(figsize=(8,8)) 
plt.imshow(conf_mat, cmap='Reds')
s = [['TN','FP'], ['FN', 'TP']]
for i in range(2):
    for j in range(2):
        plt.text(j,i, str(s[i][j])+" = "+str(conf_mat[i][j]),weight='bold')
plt.axis('off')
plt.show()<jupyter_output><empty_output><jupyter_text>Trial message<jupyter_code>def transform(message):
    m = messageList(message)
    length = messageLength(m)
    num_caps = howManyCapsInts(message)
    rat = non_alphaNum(message)
    clean = topMessage(m)
    bag = bow(clean,totalLib)
    bag = bag/(sum(bag)+.1)
    bag = bag*idf
    feat = np.array([length,num_caps,rat])
    final = np.concatenate((feat,bag))
    return(final)
    
# a spam message I get often
s = ('URGENT! Your WF account is locked.'
    'Please log in at http:// to secure your account now.')

trial = transform(s)
if model.predict(trial.reshape(1,-1))[0] ==0:
    print('Not spam')
else:
    print('Spam')<jupyter_output>Spam
{'fraction_non_alphanumeric': 0.11805205709487826, 'fraction_numerical': 0.010663308144416456, 'mean_word_length': 5.802398629354655, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 97, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '76444', 'license': 'MIT', 'n_tokens_mistral': 4057, 'n_tokens_neox': 3898, 'n_words': 1066, 'path': 'SpamFilter.ipynb', 'repo_name': 'aterzy42/SpamFilter', 'size': '11910'}