<jupyter_start><jupyter_text>0. Table of content 1. Introduction and problem statement2. Explore the data (before preprocessing)3. Data Cleaning and additional analytics (after preprocessing):4. Naïve Bayes Classifier5. Extreme Gradient Boosting (XGBoost) Classifier  1. Introduction and problem statement - Problem: SMS that we get are not always from family & friends --> Spam- Solution: develop a system that filters out spam SMS- How can we achieve a filter with a high accuracy?    - data set with 5600 actual SMS (800 Spam and 4800 actual SMS)    - analyse and visualize data to understand patterns    - build prediction model that is trained on data    - evaluate on train and test data- Challenges:    - High accuracy    - No False Positives -> better to have some spam SMS than to filter out actual SMS    - Unclean data set: SMS contain a lot of mistakes<jupyter_code>#load libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import re
import xgboost as xgb
import numpy as np

import nltk
from nltk import word_tokenize
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn import metrics
from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import log_loss
from sklearn.metrics import roc_auc_score

from collections import Counter<jupyter_output><empty_output><jupyter_text>2. Explore the data (before preprocessing)  2.1. Load in the data:<jupyter_code>#load in the text file:
with open("SMSSpamCollection.txt") as f:
    content = f.readlines()
#initialize emty list that later will be transformed to final data frame:
l = []
#loop trhough lines, get rid of new line signs and save in list:
for line in content:
    line = line.strip("\n")
    c = line.split("\t")
    l.append(c)
#transform list to df:
df = pd.DataFrame.from_records(l)
#rename columns:
df.columns = ["class","sms"]
#change ham to 0 and spam to 1
ind = df.loc[:,"class"]=="ham"
df.loc[ind,"class"] = 0
df.loc[-ind,"class"] = 1
#print the head of the df:
df.head(5)<jupyter_output><empty_output><jupyter_text>2.2) Data Analysis before cleaning - Before cleaning the data, analyse following characteristics:    - Uniqueness of SMS    - Length of SMS    - Presence of stopwors    - Presence of special characters    - Presence of hyperlink    - Presence of phone numbers    - Count of upper-case characters    - Presence of smileys<jupyter_code>print(df.shape)
#check for Nas:
print(df.isnull().values.any())
#generate a count plot:
sns.countplot(x="class",data=df)<jupyter_output>(5574, 2)
False<jupyter_text>The df has 5574 rows and 2 columns (i.e. is it a spam or not and the content of the sms). The data set contains no NA's, so the reading of the file was successful. We can see that the data set is unbalanced with +/- 4800 sms being not spam, and around 800 sms being spam. Lets check for doubles in the data:<jupyter_code>print('Number of SMS:', df.sms.count())
print('Number of unique SMS: ', df.sms.nunique())

#Are the non-unique sms rather spam?

#set index to group on indexes:
df.set_index(['class','sms'],inplace=True)
#group by and get a count column:
df2 = pd.DataFrame(df.groupby(["sms","class"]).size())
df2.reset_index(inplace=True)
#get index of count larger than 1:
i = df2.iloc[:,2] > 1
df2.columns=["sms","class","count"]
df2 = df2[i]
#generate count plot:
sns.countplot(x="count",hue="class",data=df2)
plt.title('count commets that appear more than once per class')
plt.show()


#reset the index for the main data frame:
df.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>2.2.1. SMS Length Lets check the distribution of the length of SMS, once in general, then for spams and not spams and see if we can get a conclusion out of this:<jupyter_code>#distribution of comment lengths:

#whole data set:
sns.distplot(df['sms'].apply(len))
plt.title('Characters distributions for whole data set')
plt.show()

#spam:
ind = df.loc[:,"class"]==1
df_spam=df[ind]
sns.distplot(df_spam['sms'].apply(len),label="Spam")
#not spam:
df_not_spam=df[-ind]
ax = sns.distplot(df_not_spam['sms'].apply(len),label="no spam")
plt.legend()
plt.title('Characters distributions for spam and not spam')
ax.set(xlabel='Number of characters', ylabel='Frequency')
plt.show()<jupyter_output>C:\Users\rick\Anaconda3\lib\site-packages\scipy\stats\stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>We can observe that the spam sms are longer on average, so it might be interesting to include the length of the sms as a feature.<jupyter_code>df3 = df.copy()
df3['length'] = df3['sms'].apply(len)
df3.head(10)<jupyter_output><empty_output><jupyter_text>2.2.2. Stopword presence Another idea is the presence of stopwords, there could be a repetition of 'you' in spam messages as it would be an oriented commercial sms for example...  It's also to check if we can clean the sms later on with stopwords.<jupyter_code>#SMS special characters count 
# creating a list with all reviews
sms = df.sms.tolist()

# creating special characters list
stopW = stopwords.words('english')

#empty list to save counts:
count_list = []
for s in sms:
    count = 0
    for c in s:
        if c in stopW :
            count = count + 1
    count_list.append(count)
    
#transform to series and add to df2 (which is a copy of df):
count_list = pd.Series(count_list)

df2 = df.copy()

df2['sw_count'] = count_list
df2.head()
#spam:
ind = df2.loc[:,"class"]==1
df_spam=df2[ind]
sns.distplot(df_spam.loc[:,'sw_count'],label="Spam")

#not spam:
df_not_spam=df2[-ind]
ax = sns.distplot(df_not_spam.loc[:,'sw_count'],label="no spam")
plt.legend()
plt.title('Stopword count in spam vs ham')
ax.set(xlabel='Average of stopwords', ylabel='Frequency')
plt.show()
df3['sw_count'] = count_list
df3.head(10)<jupyter_output><empty_output><jupyter_text>2.2.3. Special Characters count Same idea as numerics in text, we could expect a lot of exclamation points, question marks etc. This special characters list can be modified to find the pattern that distinguishes between spam and no spam, like using $ or a lot of ! and ?<jupyter_code>#SMS special characters count 
# creating a list with all reviews
sms = df.sms.tolist()

# creating special characters list
special_characters = ["@","?","$",".","£"] # "?" distinguishes a lot!! "?" & "." works nicely as well!

#empty list to save counts:
count_list = []
for s in sms:
    count = 0
    for c in s:
        if c in special_characters :
            count = count + 1
    count_list.append(count)
    
#transform to series and add to df2 (which is a copy of df):
count_list = pd.Series(count_list)

df2 = df.copy()

df2['sc_count'] = count_list
df2.head(10)
#spam:
ind = df2.loc[:,"class"]==1
df_spam=df2[ind]
sns.distplot(df_spam.loc[:,'sc_count'],label="Spam")

#not spam:
df_not_spam=df2[-ind]
ax = sns.distplot(df_not_spam.loc[:,'sc_count'],label="no spam")
plt.legend()
plt.title('Special characters count in spam vs ham')
ax.set(xlabel='Number of special characters', ylabel='Frequency')
plt.show()<jupyter_output><empty_output><jupyter_text>Not the most clear-cut result... But could serve! Maybe we can improve by modifying the special characters list.<jupyter_code>df3['sc_count'] = count_list
df3.head()<jupyter_output><empty_output><jupyter_text>2.2.4. Hyperlink presence<jupyter_code>sms = df.sms.tolist()

#empty list to save counts:
count_list = []
for s in sms:
    count = 0
    count += s.count('www') + s.count('http') 
    count_list.append(count)
    
count_list = pd.Series(count_list)
count_list = (count_list >= 1)

df2 = df.copy()
df2['hyperlink'] = count_list
#hyperlinks = (df2.loc[:,'hyp'] >=1)
#df2['hyperlinks'] = hyperlinks

df2.head(20)
#count phone numbers per class:
sns.countplot(x="hyperlink",hue="class",data=df2)
plt.title('Presence of hyperlink per class')
plt.show()
df3['hyperlink'] = count_list
df3.head(10)<jupyter_output><empty_output><jupyter_text>All messages that have an url are spam.  2.2.5. Digit count + Phone nb presence We might be interested to see if there is a phone number in a sms. A simple idea is to count hte number of single digits in a sms, and if the number is larger then 8, the probability is high that it is a phone number (an adress for example should have less than 8 digits). As a result, we need to come up with a function counting the digits in each sms:<jupyter_code>#SMS digit count 
# creating a list with all reviews
sms = df.sms.tolist()
#empty list to save counts:
count_list = []
for s in sms:
    count = 0
    for c in s:
        if c.isdigit():
            count = count + 1
    count_list.append(count)
    
#transform to series and add to df2 (which is a copy of df):
count_list = pd.Series(count_list)

df2 = df.copy()

df2['digit_count'] = count_list
phone = (df2.loc[:,'digit_count'] > 8)
df2['phone'] = phone
df2.head()
#make a count plot with phone numbers:
sns.countplot(x="phone",data=df2)
plt.title('Number of phone number (>8 digits) in the sms')
plt.show()

#count phone numbers per class:
sns.countplot(x="phone",hue="class",data=df2)
plt.title('Number of phone number (>8 digits) in the sms per class')
plt.show()

#look precisely at phone numbers for different classes:
ind = (df2.loc[:,"phone"] == True)
df4 = df2[ind]
sns.countplot(x="phone",hue="class",data=df4)
plt.title('Distribution (count) of phone numbers between classes')
plt.show()

df3['digit_count'] = count_list
phone = (df3.loc[:,'digit_count'] > 8)
df3['phone'] = phone
df3.head(10)<jupyter_output><empty_output><jupyter_text>We get a very interesting insight from analysing phone numbers.1) about 4900 sms's have not more than 8 digits (i.e. no phone numbers)  2) around 600 sms's have a phone number in the text  3) out of those 600, about 570 are marked as spam, only about 30 are not spamAs such, having more than 8 digits in the content might be a strong indicator for a spam sms.  2.2.6. Smiley presence<jupyter_code>sms = df.sms.tolist()

#empty list to save counts:
smileys = [':-)',':)','(-:','(:',':-(',':(',')-:','):',':o',':O',':0',':‑)'
           ,':)',':-]',':]',':-3',':3',':->',':>','8-)','8)',':-}',':}',':o)',
           ':c)',':^)','=]','=)']


count_list = []
for s in sms:
    count = 0
    for i in range(len(smileys)):
        count += s.count(smileys[i])
    count_list.append(count)
    
count_list = pd.Series(count_list)
weeb = (count_list > 1)

df2 = df.copy()
df2['smileys'] = count_list
df2['weeb'] = weeb

df2.head(10)
#count phone numbers per class:
sns.countplot(x="smileys",hue="class",data=df2)
plt.title('Presence of smileys per class')
plt.show()<jupyter_output><empty_output><jupyter_text>Some spam can have on (very rarely 2) but only non spam have multiple emojis!<jupyter_code>df3['smileys'] = count_list
df3['weeb'] = weeb
df3.head()<jupyter_output><empty_output><jupyter_text>2.2.7. Upper-case presence Next, we might be interested to count upper case letters. The underlying assumption is that spam sms's might use upper case to attract attention, while normal sms's might use a lot less upp-case letters.<jupyter_code>#count upper-case:
count_list = []
for s in sms:
    count = 0
    for c in s:
        if c.isupper():
            count = count + 1
    count_list.append(count)
count_list

#transform to series and add to df2 (which is a copy of df):
count_list = pd.Series(count_list)

df2['upper_case'] = count_list
#spam:
ind = df2.loc[:,"class"]==1
df_spam=df2[ind]
sns.distplot(df_spam.loc[:,'upper_case'],label="Spam")

#not spam:
df_not_spam=df2[-ind]
ax = sns.distplot(df_not_spam.loc[:,'upper_case'],label="no spam")
plt.legend()
plt.title('Count the number of upper-case characters in spam sms and not spam sms')
ax.set(xlabel='Number of upper-case characters', ylabel='Frequency')
plt.show()
df3['upper_case'] = count_list
df3.head()<jupyter_output><empty_output><jupyter_text>While a lot of normal sms seem to have none upper-case characters, the spam emails tend to have more upper-case characters, which seems to confirm our. It might be a good idea to include the number of upper-case characters as a feature in our model.  2.3. Conclusion of data analysis before cleaning: The main conclusions of the data analysis are as follows:    1)Data set contains around 5600 sms  2)Unbalanced data set, i.e. 800 spam, 4800 not spam  3)Spam sms tend to be longer than not spam sms  4)Around 600 sms contain phone number, almost all are spam  5)Spam sms tend to have more upper-case letters than non-spam sms    3. Data Cleaning and additional analytics (after preprocessing):  3.1. Stopword removal: We get rid of stopwords and certain punctuations (not dollar signs for example)<jupyter_code>stopW = stopwords.words('english')
special_characters = ["@", "/", "#", ".", ",", "!", "?", "(", ")",
                      "-", "_","’","'", "\"", ":","=","+","&"
                      ,"`","*","0","1","2","3","4","5",
                      "6","7","8","9","'",'.','‘',';']
transformation_sc_dict = {initial:" " for initial in special_characters}

def preprocessing(sms):
  
    # Tokenization
    tokens = word_tokenize(sms)
    
    # Deleting words with  only one caracter
    tokens = [token for token in tokens if len(token)>2]
    
    # stopwords + lowercase
    tokens = [token.lower() for token in tokens if token.lower() not in stopW]   
    
    # Deleting specific characters
    tokens = [token.translate(str.maketrans(transformation_sc_dict)) for token in tokens]
        
    return tokens
  

# Creating a new column swith tokenized reviews
df3['tokens'] = df3['sms'].apply(preprocessing)
df3.head()
# creating a list with all reviews and more pre-processing
textos = df3.tokens.tolist()
texts = [[text.strip() for text in sms] for sms in textos]
clean_tokens = [[word for word in sms if word != ''] for sms in texts]
df3['clean_tokens'] = clean_tokens<jupyter_output><empty_output><jupyter_text>- Now we define a function to consistently plot the most frequent words:<jupyter_code>def plot_frequent_words(list_of_words):    
    dist = nltk.FreqDist(list_of_words)
    X = [nb[1] for nb in dist.most_common(20)]
    y = [nb[0] for nb in dist.most_common(20)]
    ax = sns.barplot(X,y)
    ax.set(xlabel='Word frequencies',ylabel='Words',title='Most common words in the corpus')
    plt.show()

all_words = []
for sms in clean_tokens:
    for text in sms:
        all_words.append(text)
    
plot_frequent_words(all_words)
print(len(all_words))<jupyter_output><empty_output><jupyter_text>3.2. Additional analytics after cleaning We can go more in detail and see the specificities of most frequent words in ham and spam seperately. * Identifying spam and ham<jupyter_code>ind = df3.loc[:,"class"]==1
df_spam = df3[ind]
df_ham = df3[-ind]<jupyter_output><empty_output><jupyter_text>* Applying the previous pre-processing as before<jupyter_code># Separating ham and spam and pre-processing them:
df_spam['tokens'] = df_spam['sms'].apply(preprocessing)
df_ham['tokens'] = df_ham['sms'].apply(preprocessing)

# creating a list with spam and ham respectively
textos_spam = df_spam.tokens.tolist()
textos_ham = df_ham.tokens.tolist()

# Additionnal pre-processing
texts_spam = [[text.strip() for text in sms] for sms in textos_spam]
texts_ham = [[text.strip() for text in sms] for sms in textos_ham]

clean_tokens_spam = [[word for word in sms if word != ''] for sms in texts_spam]
clean_tokens_ham = [[word for word in sms if word != ''] for sms in texts_ham]<jupyter_output>C:\Users\rick\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  
C:\Users\rick\Anaconda3\lib\site-packages\ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until<jupyter_text>* Plotting the results<jupyter_code>all_words_ham = []
for sms in clean_tokens_ham:
    for text in sms:
        all_words_ham.append(text)
plot_frequent_words(all_words_ham)
print(len(all_words))

all_words_spam = []
for sms in clean_tokens_spam:
    for text in sms:
        all_words_spam.append(text)
plot_frequent_words(all_words_spam)
print(len(all_words))<jupyter_output><empty_output><jupyter_text>We can already see that a few key-words will certainly help distinguish spam from ham : £, call, free, txt, mobile, prize etc... are buzzwords that you could expect from any commercial spam. Although we must be carefull as to not overfit our model and be able to detect only sms spams that resemble our data!  3.3. Lemmatiztion We could consider using lemmatization to keep the root of the word only, although I don't think this might change much since we have a lot more grammar mistakes (done on purpose or not) given that sms are written more loosely. It might actually be an indicator for spam detection... * Applying lemmatization We just take the clean tokens that were preporcessed before for both ham and spam separately<jupyter_code>df_ham['clean_tokens'] = clean_tokens_ham
df_spam['clean_tokens'] = clean_tokens_spam

lemmatizer = WordNetLemmatizer()
def lemmatize(tokens):
    tokens = [lemmatizer.lemmatize(lemmatizer.lemmatize(lemmatizer.lemmatize(token,pos='a'),pos='v'),pos='n') for token in tokens]
    return tokens  

df_ham['Lemmatisation'] = df_ham['clean_tokens'].apply(lemmatize)
df_spam['Lemmatisation'] = df_spam['clean_tokens'].apply(lemmatize)
all_words_ham = []
for sms in df_ham['Lemmatisation']:
    for text in sms:
        all_words_ham.append(text)
plot_frequent_words(all_words_ham)
print(len(all_words_ham))

all_words_spam = []
for sms in df_spam['Lemmatisation']:
    for text in sms:
        all_words_spam.append(text)
plot_frequent_words(all_words_spam)
print(len(all_words_spam))<jupyter_output><empty_output><jupyter_text>We can't see any immediate effect but this might help for TF-IDF computations. Let's apply lemmatisation to our original data including both ham and spam:<jupyter_code>df3['Lemmatisation'] = df3['clean_tokens'].apply(lemmatize)<jupyter_output><empty_output><jupyter_text>3.4. Conclusion of EDA and preprocessing<jupyter_code>### CONCLUSION<jupyter_output><empty_output><jupyter_text>- Now that we have a good idea of what distinguishes spam sms from non-spam we can move on to predicting wether they appartein to one class or the other- We'll be considering two models, the first one being a Naïve Bayes classifier which will use NLP info translated numerically to distinguish sms- The second one will be an XGBoost classifier, which will use the features we've found throughout our EDA to classify spam and non-spam  4. Naïve Bayes Classifier - Naïve bayes classification    - Probalistic classification method relying on Bayes formula and strong independence (hence the term naïve!)    - Use conditional law of X|y and law of y to retrieve an estimation of y|X    - Given the law of y|X classify as +1 if P(y|X) >= 1/2, 0 otherwise - Naïve bayes classification for spam detection    - Naïve Bayes classifier very famous for spam detection (emails, sms...)    - Uses NLP features such as wordcount, TF-IDF etc...    - We compare Naïve bayes with XGBoost (implemented in next section) to benchmark our models and check scalability  4.1 Load in the data - Loading the data    - Naïve bayes classifier relies on NLP information retrieved from sms    - Thus it will have its own version of pre-processed sms as these are its features in essence    - We use a 80/20 train test split<jupyter_code>data = df3.copy()
X = data['sms']
y = data['class']

#make train test split:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=111)<jupyter_output><empty_output><jupyter_text>4.2 Preprocessing the data - Pre-processing data    - As mentionned before, the processing is key for this classifier    - We reverse engineer the preprocessing we apply (stopwords, punctuation etc)    - Some characters are a major distinguisher between ham and spam ("£" for example!)<jupyter_code>def clean_text(text):
    
    # convert text to lowercase
    text = text.strip().lower()
    
    # replace punctuation characters with spaces
    filters='"\'%&()*,-./:;<=>?[\\]^_`{|}~\t\n'
    translate_dict = dict((c, " ") for c in filters)
    translate_map = str.maketrans(translate_dict)
    text = text.translate(translate_map)

    return text<jupyter_output><empty_output><jupyter_text>4.3 Vectorization    - We need to transform strings into numerical data for the classifier to understand them   - Various methods exist, Wordcount and TF-IDF will be our focus here   - Drawbacks of wordcount will be explained in next slide<jupyter_code>#spam:
ind = data.loc[:,"class"]==1
data_spam=data[ind]
sns.distplot(data_spam.loc[:,'length'],label="Spam")

#not spam:
data_ham=data[-ind]
ax = sns.distplot(data_ham.loc[:,'length'],label="no spam")
plt.legend()
plt.title('length of sms')
ax.set(xlabel='Number of characters in sms', ylabel='Frequency')
plt.show()<jupyter_output><empty_output><jupyter_text>4.3.1 Wordcount vectorization    - As its title states it, we basically count the number of times a word appears in an sms   - Main drawback is that it is subject to the length of the sms (i.e higher count just because there are more words in an sms)   - The classifier computes conditional laws for classification on these transformed numerical features<jupyter_code>vectorizer = CountVectorizer(
    stop_words="english",
    preprocessor=clean_text)

data_train_count = vectorizer.fit_transform(X_train)
data_test_count = vectorizer.transform(X_test)<jupyter_output><empty_output><jupyter_text>4.3.2. TF-IDF vectorization    - Solves length of sms problem mentionned in wordcount vectorization by weighting word counts (loosely speeaking)   - TF score(word,text) :  of word in text / total  of words in text | IDF score(word): log( of texts/  of texts where word appears)   - TF-IDF(word,text):<jupyter_code>vectorizer = TfidfVectorizer(
    stop_words="english",
    preprocessor=clean_text)

data_train_tfidf = vectorizer.fit_transform(X_train)
data_test_tfidf = vectorizer.transform(X_test)<jupyter_output><empty_output><jupyter_text>4.4 Building the model  4.4.1 Model using wordcount vectorization<jupyter_code>clf_count = MultinomialNB()
clf_count.fit(data_train_count,y_train)

#predict on test data:
pred_count = clf_count.predict(data_test_count)
pred_count_prob = clf_count.predict_proba(data_test_count)[:,1]<jupyter_output><empty_output><jupyter_text>4.4.2 Model using TF-IDF vectorization<jupyter_code>clf_tfidf = MultinomialNB()
clf_tfidf.fit(data_train_tfidf,y_train)

#predict on test data:
pred_tfidf = clf_tfidf.predict(data_test_tfidf)
pred_count_tfidf = clf_tfidf.predict_proba(data_test_tfidf)[:,1]<jupyter_output><empty_output><jupyter_text>4.5 Model evaluation  4.5.1 Results for wordcount vectoriation<jupyter_code>print("\n")
print("Using count vectorization")
print("\n")
acc_count = accuracy_score(y_test,pred_count)
prec_count = precision_score(y_test, pred_count)
sens_count = recall_score(y_test,pred_count)

print("Accuracy :", acc_count)
print("Precision :", prec_count)
print("Sensitivity :", sens_count)
confusion = metrics.confusion_matrix(y_test, pred_count)
sns.heatmap(confusion, annot = True, cbar = False, square = True, cmap = 'Blues', fmt = 'g', linewidths = 0.5)

TP = confusion[1, 1]
TN = confusion[0, 0]
FP = confusion[0, 1]
FN = confusion[1, 0]

print("We correctly classified",TP,"sms as spam")
print("\n")
print("We correctly classified",TN,"sms as not spam")
print("\n")
print("We inccorrectly classified",FP,"sms as spam")
print("\n")
print("We inccorrectly classified",FN,"sms as not spam")
print("\n")
fpr, tpr, thresholds = metrics.roc_curve(y_test,pred_count_prob)

plt.plot(fpr, tpr)
plt.xlim([-0.1, 1.1])
plt.ylim([-0.1, 1.1])
plt.rcParams['font.size'] = 12
plt.title('ROC curve for spam classifier using count vectorization')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.grid(True)
print("AUC score :",metrics.roc_auc_score(y_test, pred_count_prob))
print("\n")<jupyter_output>AUC score : 0.9973752717702254<jupyter_text>4.5.2 Results for TF-IDF vectorization<jupyter_code>print("\n")
print("Using TF-IDF vectorization")
print("\n")

acc_tfidf = accuracy_score(y_test,pred_tfidf)
prec_tfidf = precision_score(y_test, pred_tfidf)
sens_tfidf = recall_score(y_test,pred_tfidf)

print("Accuracy :", acc_tfidf)
print("Precision :", prec_tfidf)
print("Sensitivity :", sens_tfidf)
confusion = metrics.confusion_matrix(y_test, pred_tfidf)
sns.heatmap(confusion, annot = True, cbar = False, square = True, cmap = 'Blues', fmt = 'g', linewidths = 0.5)

TP = confusion[1, 1]
TN = confusion[0, 0]
FP = confusion[0, 1]
FN = confusion[1, 0]

print("We correctly classified",TP,"sms as spam")
print("\n")
print("We correctly classified ",TN," sms as not spam")
print("\n")
print("We inccorrectly classified ",FP," sms as spam")
print("\n")
print("We inccorrectly classified ",FN," sms as not spam")
print("\n")
fpr, tpr, thresholds = metrics.roc_curve(y_test,pred_count_tfidf)

plt.plot(fpr, tpr)
plt.xlim([-0.1, 1.1])
plt.ylim([-0.1, 1.1])
plt.rcParams['font.size'] = 12
plt.title('ROC curve for spam classifier using TF-IDF vectorization')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.grid(True)
print("AUC score :",metrics.roc_auc_score(y_test, pred_count_tfidf))
print("\n")<jupyter_output>AUC score : 0.9955658542167297<jupyter_text>4.6. Conclusion    - What matters the most for us is to not classify non-spam sms as spam (False positives must be 0)   - We can see that TF-IDF vectorization does exactly that but at the cost of more spam not being correctly classified (more Fasle Negatives)   - Wordcount vectorization has some False positives, we thus prefer TF-IDF vectorization over Wordcount    - Disregarding these details both models are very precise and accurate   - the auc score is very similar for both    - We'll see how XGBoost fares in comparison in the next section  5. Extreme Gradient Boosting (XGBoost) Classifier - Extreme Gradient Boosting method- Boosting: ensemble learning technique of building many models sequentially    - new model attempting to correct for the deficiencies in the previous model    - this is an ensemble method that seeks to create a strong classifier (model) based on “weak” classifiers-	Gradient boosting:  updating model using gradient descent-	XGBoost implements this algorithm for decision tree boosting![Drag Racing -fullwidth](mystery-box.jpg)  5.1. Load and prepare data for XGBoost<jupyter_code>#import file with features:
xgbdata = df3.copy()
xgbdata.head(3)
#change False to 0 and True to 1:
#define function:
def bool_to_bin(colname):
    index_false = xgbdata.loc[:,colname]==False
    xgbdata.loc[index_false,colname]=0
    xgbdata.loc[-index_false,colname]=1

bool_to_bin("hyperlink")
bool_to_bin("phone")
bool_to_bin("weeb")

xgbdata = xgbdata.loc[:,["class","length","sc_count","hyperlink","digit_count","smileys","upper_case"]]

xgbdata.head(5)
#separate data in target and predictor variables:
X = xgbdata.iloc[:,1:]
y = xgbdata.iloc[:,0]

#transform into Dmatrix (faster and supported by xgboost):
dmatrix = xgb.DMatrix(data=X,label=y)
#make train test split (80-20)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=111)
#set seed in order to be able to reproduce results<jupyter_output><empty_output><jupyter_text>5.2 Fine-tuning of the XGBoost model with Cross-Validation<jupyter_code>%%capture
#CROSS VALIDATION 


#generate classifier:
params={"max_depth":5, 
        "min_child_weight":2, 
        "eta": 0.1, 
        "subsamples":0.9, 
        "objective" : "binary:logistic", 
        "eval_metric": ["error","logloss","auc"]}
rounds = 200
result = xgb.cv(params=params, nfold=5, dtrain=dmatrix, num_boost_round=rounds, as_pandas=True, seed=10,early_stopping_rounds=50)

result.iloc[105,:]<jupyter_output><empty_output><jupyter_text>- early stopping: 106 --> num_boosting_rounds = 106- max_depth: maximum depth of a tree -> increasing this value will make the model more complex and more likely to overfit    - max_depth = 5- min_child_weight: corresponds to minimum number of instances needed to be in each node. The larger min_child_weight is, the more conservative the algorithm will be.    - min_child_weight = 2- eta: Step size shrinkage used in update to prevents overfitting    - eta = 0.1- subsample: Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees    - subsample = 0.9<jupyter_code>test_error = result.loc[:,"test-error-mean"]
train_error = result.loc[:,"train-error-mean"]
test_logloss = result.loc[:,"test-logloss-mean"]
train_logloss = result.loc[:,"train-logloss-mean"]
test_auc = result.loc[:,"test-auc-mean"]
train_auc = result.loc[:,"train-auc-mean"]
n = range(len(test_auc))

fig = plt.figure(figsize=(20, 10))
ax1 = fig.add_subplot(3,1,1)
ax2 = fig.add_subplot(3,1,2)
ax3 = fig.add_subplot(3,1,3)

sns.lineplot(n,test_error,ax=ax1,label="test error")
sns.lineplot(n,train_error,ax=ax1, label="train error")
ax1.set_title("Test error in function of number of boosting rounds")
ax1.legend()

sns.lineplot(n,test_logloss,ax=ax2,label="test logloss")
sns.lineplot(n,train_logloss,ax=ax2,label="train logloss")
ax2.set_title("Logloss in function of number of boosting rounds")
ax2.legend()

sns.lineplot(n,test_auc,ax=ax3,label="test AUC-score")
sns.lineplot(n,train_auc,ax=ax3,label="train AUC-score")
ax3.set_title("AUC score in function of number of boosting rounds")
ax3.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>5.3Train the model and evaluate it on test set<jupyter_code>xg_reg = xgb.XGBRegressor(max_depth=5,
                          eta = 0.1,
                          min_child_weight = 2,
                          subsamples = 0.9,
                          objective = "binary:logistic",
                          num_boost_round=106
                         )
#make prediction:
xg_reg.fit(X_train,y_train)

xg_reg.save_model("xg_reg")
import joblib
joblib.dump(xg_reg,"xg_reg2.bin")<jupyter_output><empty_output><jupyter_text>We see that all the metrics converge with the number of boosting rounds. There seems to be no issue with overfitting either. As such, lets chooser 50 boosting rounds for the following tests.<jupyter_code>#TEST ERROR:
y_pred = xg_reg.predict(X_test)
predictions = [round(value) for value in y_pred]
# evaluate predictions
accuracy2 = accuracy_score(y_test, predictions)
print("Accuracy for test set: %.2f%%" % (accuracy2 * 100.0))

#plot area under the curve:
roc = roc_curve(y_test,y_pred)
x = roc[0]
y = roc[1]
sns.lineplot(x,y)
print(roc_auc_score(y_test,y_pred))

#generate a confusion matrix
cm = confusion_matrix(y_test,np.round(y_pred))
sns.heatmap(cm,annot=True,cbar=False,square=True,cmap='Blues',fmt='g',linewidths=0.5)<jupyter_output><empty_output><jupyter_text>5.4 Model Interpretation:<jupyter_code># plot feature importance
xgb.plot_importance(xg_reg,importance_type="weight")
xgb.plot_importance(xg_reg,importance_type="cover")
xgb.plot_importance(xg_reg,importance_type="gain")<jupyter_output><empty_output><jupyter_text>- result completely different, why?    - weight: The number of times a feature is used to split the data across all trees    - cover: The number of times a feature is used to split the data across all trees weighted by the number of training data points that go through those splits    - gain: The average training loss reduction gained when using a feature for splitting- Which one to choose? Problem: Neither one of the 3 is consitent    - consitecy:  Whenever we change a model such that it relies more on a feature, then the attributed importance for that feature should not decrease    Tree SHAP method is mathematically equivalent to averaging differences in predictions over all possible orderings of the features, rather than just the ordering specified by their position in the tree -> consistent<jupyter_code>#import shap
#shap.TreeExplainer(xg_reg).shap_values(X_train) 
#call shap.summary_plot(shap_values, X_train)<jupyter_output><empty_output><jupyter_text>6. Conclusion - Summary of results:    - Naive Bayes:        - Wordcount vectorization            - Accuracy = 0.9892            - AUC score = 0.9966            - Number of FP = 5            - Number of FN = 7        - TF-IDF vectorization            - Accuracy = 0.9713            - AUC score = 0.9948            - Number of FP = 0            - Number of FN = 31    - XGBoost:        - Specific features extracted from SMS            - Accuracy = 0.9881            - AUC score = 0.9937            - Number of FP = 3            - Number of FN = 7- As we want no FP, we chose TF-IDF vectoization with Naive Bayes- A lot of potential for the XGBoost model- Future work:    - Try XGBoost on TF-IDF vectorization    - Develop new features    - Fine-tune parameters (NOT looking at test-set)    - Find more data (more divers), and test models on this data as well<jupyter_code>print("compiled successfully!")<jupyter_output>compiled successfully!
{'fraction_non_alphanumeric': 0.11341320864991233, 'fraction_numerical': 0.01580946814728229, 'mean_word_length': 4.814953271028037, 'pattern_counts': {'":': 6, '<': 152, '<?xml version=': 0, '>': 169, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '363724', 'license': 'MIT', 'n_tokens_mistral': 11872, 'n_tokens_neox': 11275, 'n_words': 3568, 'path': 'Final/.ipynb_checkpoints/SMS_SPAM_FINAL_NOTEBOOK-checkpoint.ipynb', 'repo_name': 'ALaks96/SMS_spam_detection', 'size': '34220'}