<jupyter_start><jupyter_text>**Description:** This notebook is used to implement a classifier to devide "SPAM" and "no SPAM" messages. The dataset `SMS Spam Collection v. 1` can be found ([here](http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/)).**Project Name:** Natural Language SPAM Classifier**Author:** <NAME>**Date:** 2020-12-09  Setup<jupyter_code>import pandas as pd
import numpy as np

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import LinearSVC
from sklearn.metrics import roc_auc_score, confusion_matrix, classification_report

import nltk

# warnings handler
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Load dataset<jupyter_code>df = pd.read_csv("data/SMSSpamCollection.txt", encoding="utf-8", header=None, delimiter="\t", names=["target", "text"])
df.head()
# Encoding target variable
df["target"] = np.where(df["target"] == "spam", 1, 0)
df.head(11)<jupyter_output><empty_output><jupyter_text>Short EDA<jupyter_code>print("Dataset contains {} instances of {} variables.".format(df.shape[0], df.shape[1]))

print(
    "It contains {} spam messages ({:.1%} of all)".format(
        df[df.target == 1].shape[0],
        df[df.target == 1].shape[0] / df.shape[0],
    )
)
print("\nExample 'no SPAM'")
print(df.text[0])
print(df.text[1])
print("\nExample 'SPAM'")
print(df.text[2])
print(df.text[5])
print(df.text[8])<jupyter_output>Example 'no SPAM'
Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...
Ok lar... Joking wif u oni...

Example 'SPAM'
Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
FreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv
WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.<jupyter_text>It seems like SPAM messages start with wordslike "winner" or "free". Let´s keep this in mind while evaluation.  Spam classificationThe model will be a "vanilla" classifier, without pouring too many thoughts about what the actual messages, spam or not, look like. To improve your model you can of course have a closer look and investigate the data more in detail.<jupyter_code># Split dataset between train and test sets
X_train, X_test, y_train, y_test = train_test_split(df["text"], df["target"], random_state=101)<jupyter_output><empty_output><jupyter_text>We'll start by using a count vectorizer on our data. It will convert our text and return a sparse matrix.  CountVectorizer<jupyter_code>vect = CountVectorizer().fit(X_train)

# transform the documents in the training data to a document-term matrix
X_train_vectorized = vect.transform(X_train)
print("X_train_vectorized: ")
X_train_vectorized
print("X_train shape     = {}".format(X_train.shape[0]))
print("Vocabulary length = {}".format(len(vect.vocabulary_)))
# vect.vocabulary_
# Train Logistic Regression
log_reg = LogisticRegression(max_iter=1500)
log_reg.fit(X_train_vectorized, y_train)
log_reg_predictions = log_reg.predict(vect.transform(X_test))
print("\nLogisticRegression") 
print(f"AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(confusion_matrix(y_test, log_reg_predictions))
print(classification_report(y_test, log_reg_predictions))


# Train KNN
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train_vectorized, y_train)
knn_predictions = knn.predict(vect.transform(X_test))
print("\nNearest Neighbors Classifier")
print(f"AUC = {round(roc_auc_score(y_test, knn_predictions),3)}")
print(confusion_matrix(y_test, knn_predictions))
print(classification_report(y_test, knn_predictions))

# Train SVC
svc = LinearSVC()
svc.fit(X_train_vectorized, y_train)
svc_predictions = svc.predict(vect.transform(X_test))
print("\nSupport Vector Classifier")
print(f"AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")
print(confusion_matrix(y_test, svc_predictions))
print(classification_report(y_test, svc_predictions))<jupyter_output>LogisticRegression
AUC = 0.876
[[1225    2]
 [  41  125]]
              precision    recall  f1-score   support

           0       0.97      1.00      0.98      1227
           1       0.98      0.75      0.85       166

    accuracy                           0.97      1393
   macro avg       0.98      0.88      0.92      1393
weighted avg       0.97      0.97      0.97      1393


Nearest Neighbors Classifier
AUC = 0.822
[[1227    0]
 [  59  107]]
              precision    recall  f1-score   support

           0       0.95      1.00      0.98      1227
           1       1.00      0.64      0.78       166

    accuracy                           0.96      1393
   macro avg       0.98      0.82      0.88      1393
weighted avg       0.96      0.96      0.95      1393


Support Vector Classifier
AUC = 0.944
[[1222    5]
 [  18  148]]
              precision    recall  f1-score   support

           0       0.99      1.00      0.99      1227
           1       0.97      0.89      0.93[...]<jupyter_text>Evaluation<jupyter_code>print(f"LogisticRegression AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(f"KNN                AUC = {round(roc_auc_score(y_test, knn_predictions),3)}")
print(f"SVC                AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")<jupyter_output>LogisticRegression AUC = 0.917
KNN                AUC = 0.822
SVC                AUC = 0.935<jupyter_text>The best model is a linear support vector classifier with an AUC of 0.935, second best is a logistic regression with AUC 0.917.<jupyter_code># get the feature names as numpy array
feature_names = np.array(vect.get_feature_names())

# Sort the coefficients from the model
log_reg_sorted_coef_index = log_reg.coef_[0].argsort()
print("Log Reg features:\n")
print("Smallest Coefs:\n{}\n".format(feature_names[log_reg_sorted_coef_index[:10]]))
print("Largest Coefs: \n{}".format(feature_names[log_reg_sorted_coef_index[:-11:-1]]))

svc_sorted_coef_index = svc.coef_[0].argsort()
print("\n" + "SVC features:\n")
print("Smallest Coefs:\n{}\n".format(feature_names[svc_sorted_coef_index[:10]]))
print("Largest Coefs: \n{}".format(feature_names[svc_sorted_coef_index[:-11:-1]]))<jupyter_output>Log Reg features:

Smallest Coefs:
['my' 'but' 'gt' 'lt' 'sir' 'fullonsms' 'him' 'me' 'can' 'place']

Largest Coefs: 
['txt' 'text' 'call' 'ringtone' 'reply' 'chat' 'won' 'uk' '150p' 'stop']

SVC features:

Smallest Coefs:
['liked' 'place' 'sir' 'once' 'fullonsms' 'gt' 'can' 'lunch' 'lt' 'missed']

Largest Coefs: 
['146tf150p' 'ringtoneking' '84484' 'ringtone' 'stories' 'filthy' 'txt'
 '88066' 'won' 'chat']<jupyter_text>TfidfVectorizer<jupyter_code>vect = TfidfVectorizer(min_df=3).fit(X_train)

# transform the documents in the training data to a document-term matrix
X_train_vectorized = vect.transform(X_train)
print("X_train_vectorized: ")
X_train_vectorized
# vect.vocabulary_
# Train Logistic Regression
log_reg = LogisticRegression(max_iter=1500)
log_reg.fit(X_train_vectorized, y_train)
log_reg_predictions = log_reg.predict(vect.transform(X_test))
print("\nLogisticRegression") 
print(f"AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(confusion_matrix(y_test, log_reg_predictions))
print(classification_report(y_test, log_reg_predictions))


# Train KNN
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train_vectorized, y_train)
knn_predictions = knn.predict(vect.transform(X_test))
print("\nNearest Neighbors Classifier")
print(f"AUC = {round(roc_auc_score(y_test, knn_predictions),3)}")
print(confusion_matrix(y_test, knn_predictions))
print(classification_report(y_test, knn_predictions))

# Train SVC
svc = LinearSVC()
svc.fit(X_train_vectorized, y_train)
svc_predictions = svc.predict(vect.transform(X_test))
print("\nSupport Vector Classifier")
print(f"AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")
print(confusion_matrix(y_test, svc_predictions))
print(classification_report(y_test, svc_predictions))<jupyter_output>LogisticRegression
AUC = 0.876
[[1225    2]
 [  41  125]]
              precision    recall  f1-score   support

           0       0.97      1.00      0.98      1227
           1       0.98      0.75      0.85       166

    accuracy                           0.97      1393
   macro avg       0.98      0.88      0.92      1393
weighted avg       0.97      0.97      0.97      1393


KNN
AUC = 0.822
[[1227    0]
 [  59  107]]
              precision    recall  f1-score   support

           0       0.95      1.00      0.98      1227
           1       1.00      0.64      0.78       166

    accuracy                           0.96      1393
   macro avg       0.98      0.82      0.88      1393
weighted avg       0.96      0.96      0.95      1393


SVC
AUC = 0.944
[[1222    5]
 [  18  148]]
              precision    recall  f1-score   support

           0       0.99      1.00      0.99      1227
           1       0.97      0.89      0.93       166

    accuracy                       [...]<jupyter_text>Evaluation<jupyter_code>print(f"LogisticRegression AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(f"KNN                AUC = {round(roc_auc_score(y_test, knn_predictions),3)}")
print(f"SVC                AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")<jupyter_output>LogisticRegression AUC = 0.876
KNN                AUC = 0.822
SVC                AUC = 0.944<jupyter_text>The SVC is the best and increased to an AUC score of 0.944. Both other models performances decreased.<jupyter_code># Sort the coefficients from the model
log_reg_sorted_coef_index = log_reg.coef_[0].argsort()
print("Log Reg features:\n")
print("Smallest Coefs:\n{}\n".format(feature_names[log_reg_sorted_coef_index[:10]]))
print("Largest Coefs: \n{}".format(feature_names[log_reg_sorted_coef_index[:-11:-1]]))

svc_sorted_coef_index = svc.coef_[0].argsort()
print("\n" + "SVC features:\n")
print("Smallest Coefs:\n{}\n".format(feature_names[svc_sorted_coef_index[:10]]))
print("Largest Coefs: \n{}".format(feature_names[svc_sorted_coef_index[:-11:-1]]))<jupyter_output>Log Reg features:

Smallest Coefs:
['beautiful' 'blacko' '2geva' 'aroundn' 'costs' 'barolla' 'alive' 'bought'
 '2stoptx' 'anything']

Largest Coefs: 
['cutest' '2morrow' 'corect' 'admit' 'bergkamp' 'competition' 'director'
 'cricketer' 'casting' 'advisors']

SVC features:

Smallest Coefs:
['affection' 'beautiful' '2stoptx' 'alive' 'desparate' 'b4280703'
 'barolla' '2geva' 'blacko' 'bsnl']

Largest Coefs: 
['cutest' 'corect' 'da' '08000930705' '382' 'bergkamp' 'director' 'child'
 '402' 'celebration']<jupyter_text>StemmingIt is also possible to use stemming (remove morphological affixes from words, leaving only the word stem) as a preprocessing step.<jupyter_code># Initializing stemmer and countvectorizer 
stemmer = nltk.PorterStemmer()
analyzer = CountVectorizer().build_analyzer()

def stemmed_words(doc):
    return (stemmer.stem(w) for w in analyzer(doc))

stemm_vectorizer = CountVectorizer(analyzer=stemmed_words)

# Transform X_train
X_train_stemm_vectorized = stemm_vectorizer.fit_transform(X_train)

# Train Logistic Regression
log_reg_stemm = LogisticRegression(max_iter=1500)
log_reg_stemm.fit(X_train_stemm_vectorized, y_train)
log_reg_predictions = log_reg_stemm.predict(stemm_vectorizer.transform(X_test))
print("\nLogisticRegression") 
print(f"AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(confusion_matrix(y_test, log_reg_predictions))
print(classification_report(y_test, log_reg_predictions))

# Train SVC
svc_stemm = LinearSVC()
svc_stemm.fit(X_train_stemm_vectorized, y_train)
svc_predictions = svc_stemm.predict(stemm_vectorizer.transform(X_test))
print("\nSupport Vector Classifier") 
print(f"AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")
print(confusion_matrix(y_test, svc_predictions))
print(classification_report(y_test, svc_predictions))<jupyter_output>LogisticRegression
AUC = 0.93
[[1225    2]
 [  23  143]]
              precision    recall  f1-score   support

           0       0.98      1.00      0.99      1227
           1       0.99      0.86      0.92       166

    accuracy                           0.98      1393
   macro avg       0.98      0.93      0.95      1393
weighted avg       0.98      0.98      0.98      1393


Support Vector Classifier
AUC = 0.951
[[1224    3]
 [  16  150]]
              precision    recall  f1-score   support

           0       0.99      1.00      0.99      1227
           1       0.98      0.90      0.94       166

    accuracy                           0.99      1393
   macro avg       0.98      0.95      0.97      1393
weighted avg       0.99      0.99      0.99      1393<jupyter_text>Evaluation<jupyter_code>print(f"LogisticRegression AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(f"SVC                AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")<jupyter_output>LogisticRegression AUC = 0.93
SVC                AUC = 0.951<jupyter_text>Stemming improved the model.  LemmatizationThe same way we used stemming we can also apply lemmatization (grouping together the inflected forms of a word) to the data.<jupyter_code>#nltk.download('wordnet')
# Initialization
WNlemma = nltk.WordNetLemmatizer()
analyzer = CountVectorizer().build_analyzer()

def lemmatize_word(doc):
    return (WNlemma.lemmatize(t) for t in analyzer(doc))

lemm_vectorizer = CountVectorizer(analyzer=lemmatize_word)

# Transform X_train
X_train_lemm_vectorized = lemm_vectorizer.fit_transform(X_train)
# Train Logistic Regression
log_reg_lemm = LogisticRegression(max_iter=1500)
log_reg_lemm.fit(X_train_lemm_vectorized, y_train)
log_reg_predictions = log_reg_lemm.predict(lemm_vectorizer.transform(X_test))
print("\nLogisticRegression") 
print(f"AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(confusion_matrix(y_test, log_reg_predictions))
print(classification_report(y_test, log_reg_predictions))

# Train SVC
svc_lemm = LinearSVC()
svc_lemm.fit(X_train_lemm_vectorized, y_train)
svc_predictions = svc_lemm.predict(lemm_vectorizer.transform(X_test))
print("\nSupport Vector Classifier") 
print(f"AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")
print(confusion_matrix(y_test, svc_predictions))
print(classification_report(y_test, svc_predictions))<jupyter_output>LogisticRegression
AUC = 0.92
[[1223    4]
 [  26  140]]
              precision    recall  f1-score   support

           0       0.98      1.00      0.99      1227
           1       0.97      0.84      0.90       166

    accuracy                           0.98      1393
   macro avg       0.98      0.92      0.95      1393
weighted avg       0.98      0.98      0.98      1393


Support Vector Classifier
AUC = 0.938
[[1223    4]
 [  20  146]]
              precision    recall  f1-score   support

           0       0.98      1.00      0.99      1227
           1       0.97      0.88      0.92       166

    accuracy                           0.98      1393
   macro avg       0.98      0.94      0.96      1393
weighted avg       0.98      0.98      0.98      1393<jupyter_text>Evaluation<jupyter_code>print(f"LogisticRegression AUC = {round(roc_auc_score(y_test, log_reg_predictions),3)}")
print(f"SVC                AUC = {round(roc_auc_score(y_test, svc_predictions),3)}")<jupyter_output>LogisticRegression AUC = 0.92
SVC                AUC = 0.938
{'fraction_non_alphanumeric': 0.11019318759532283, 'fraction_numerical': 0.05840111845449924, 'mean_word_length': 3.4068888266591992, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '132096', 'license': 'MIT', 'n_tokens_mistral': 6157, 'n_tokens_neox': 5575, 'n_words': 1234, 'path': 'SPAM-classifier.ipynb', 'repo_name': 'sls-mdr/rdc-nlp-SPAM-classifier', 'size': '15743'}