<jupyter_start><jupyter_text>IntroductionThe [SMS Spam Collection v.1](http://dcomp.sor.ufscar.br/talmeida/smspamcollection/) is a public set of SMS labeled messages that have been collected for mobile phone spam research. It has one collection composed by 5,574 English, real and non-enconded messages, tagged according being legitimate (ham) or spam. Some questions arise when we take a look at the data set are:- Is there any pattern that could help us classify each message as ham or spam at a first view?- Which are the most common and most significant words for ham and spam messages?- How well our model predicts a new message as ham or spam?  Import Libraries<jupyter_code>import pandas as pd
import numpy as np
import re

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.set_style('whitegrid')<jupyter_output>[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\MyPc\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\MyPc\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\MyPc\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>----  Exploratory Data Analysis<jupyter_code># Load the data set
sms = pd.read_csv('SMSSpamCollection.txt', sep='\t', names=['label', 'text'])
sms.head()
sms.info()
sms.describe()
print('The number of ham messages in the dataset is {}'.format(sms['label'].value_counts()[0]))
print('The number of spam messages in the dataset is {}'.format(sms['label'].value_counts()[1]))<jupyter_output>The number of ham messages in the dataset is 4825
The number of spam messages in the dataset is 747<jupyter_text>As we can see, the dataset is unbalanced. This could affect our model prediction and the accuracy. Lets move on... ---  Data Visualizations<jupyter_code>sms['text_length'] = sms['text'].apply(lambda col: len(col))
sms.head()
sms.groupby('label')['text_length'].describe()
# Distribution Plot of lenghts
fig, ax = plt.subplots(figsize=(12,6))
sns.distplot(a=sms[sms['label']=='ham']['text_length'], kde=False, hist=True, bins=50, label='Ham Sms')
sns.distplot(a=sms[sms['label']=='spam']['text_length'], kde=False, hist=True, bins=50, color='red', label='Spam Sms')
plt.legend()
plt.ylabel('Number of Messages')
plt.title('Distribution of lengths for Ham and Spam SMS')
plt.show()<jupyter_output><empty_output><jupyter_text>As we see in previous table, *ham* messages maximum length is 910. It could be outliers so let's take a look on it, remove it and plot the distributions again.<jupyter_code>sms.loc[sms['text_length'] == 910]['text'].values[0]
# Distribution Plot of lenghts
fig, ax = plt.subplots(figsize=(12,6))

sns.distplot(a=sms.drop(axis=0, index=1085)[sms.drop(axis=0, index=1085)['label']=='ham']['text_length'], \
             kde=False, hist=True, color='blue', label='Ham Sms')

sns.distplot(a=sms.drop(axis=0, index=1085)[sms.drop(axis=0, index=1085)['label']=='spam']['text_length'], \
             kde=False, hist=True, color='red', label='Spam Sms')

plt.legend()
plt.ylabel('Number of Messages')
plt.title('Distribution of lengths for Ham and Spam SMS')
plt.show()
# Portion of spam messages that have 133<=length<=157
sms.loc[(sms['label'] == 'spam') & (sms['text_length'] >= 133) & \
        (sms['text_length'] <=157)]['label'].count() \
        /sms.loc[(sms['label'] == 'spam')]['label'].count()
# Portion of ham messages that have 133<=length<=157
sms.loc[(sms['label'] == 'ham') & (sms['text_length'] >= 133) & \
        (sms['text_length'] <=157)]['label'].count() \
        /sms.loc[(sms['label'] == 'ham')]['label'].count()<jupyter_output><empty_output><jupyter_text>>Insights of the distributions and the statistical analysis:- The distribution of the spam messages has a smaller standard deviation and the mean value is much higher than the mean value of ham messages,- Almost 50% of the spam messages has length from 133 to 157, on the other hand 75% of the ham messages have a length lower or equal to 93>These insights could help us answer the first question. We could say that there is a hidden pattern to classify a message as spam or ham from its length. We could not be sure 100% but there is a big chance to classified it as a spam and be right. ----  Bag of words modelMachine learning models needs to ingest data in a structured form, a matrix where the rows represents observations and the columns are features/attributes. When working with text data, we need a method to convert this unstructured data into a form that the machine learning model can work with. One technique to transform text data into a matrix is to count the number of appearances of each word in each document. This technique is called the **bag of words model**. The model gets its name because each document is viewed as a bag holding all the words, disregarding word order, context, and grammar. After applying the bag of words model to a corpus, the resulting matrix will exhibit patterns that a machine learning model can exploit.   The CountVectorizer transformerThe bag of words model is found in scikit-learn with the *CountVectorizer* transformer. Note, scikit-learn uses the word **Vectorizer** to refer to transformers that convert a data structure (like a dictionary) into a NumPy array. Since it is a transformer, we need to first fit the object and then call transform.     Stemming and lemmatization**Stemming** is the process of reducing a word to its stem. Note, the stemming process is not 100% effective and sometimes the resulting stem is not an actual word. For example, the popular Porter stemming algorithm applied to "argues" and "arguing" returns `"argu"`.**Lemmatization** is the process of reducing a word to its lemma, or the dictionary form of the word. It is a more sophisticated process than stemming as it considers context and part of speech. Further, the resulting lemma is an actual word.  > We are going to choose `lemmatization` for our process<jupyter_code>def tokenize(text):
    """
    Helper function to tokenize the messages
    INPUT:
    text: Messages
    OUTPUT:
    Tokens of word
    """

    stop_words = stopwords.words("english")
    lemmatizer = WordNetLemmatizer()

    text = re.sub(r"[^a-zA-Z0-9]", " ", text.lower())

    tokens = word_tokenize(text)

    tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words]

    return tokens<jupyter_output><empty_output><jupyter_text>Time to answer the second question> Which are the most common and most significant words for ham and spam messages?<jupyter_code>bag_of_words = CountVectorizer(tokenizer=tokenize)
# The most common words in spam messages

spam_word_counts = bag_of_words.fit_transform(sms.loc[sms['label']=='spam']['text'].values);

spam_words = bag_of_words.vocabulary_

spam_words_df = pd.DataFrame.from_dict(spam_words, orient='index')
spam_words_df.reset_index(inplace=True)
spam_words_df.columns = ['word', 'counts']

spam_words_df.sort_values(by='counts', ascending=False, inplace=True)
spam_words_df.head(15)
# The most common words in ham messages

ham_word_counts = bag_of_words.fit_transform(sms.loc[sms['label']=='ham']['text'].values)

ham_words = bag_of_words.vocabulary_

ham_words_df = pd.DataFrame.from_dict(ham_words, orient='index')
ham_words_df.reset_index(inplace=True)
ham_words_df.columns = ['word', 'counts']

ham_words_df.sort_values(by='counts', ascending=False, inplace=True)
ham_words_df.head(15)<jupyter_output><empty_output><jupyter_text>Term frequency-inverse document frequencyThe `CountVectorizer` creates a feature matrix of raw counts. Using raw counts has two problems, documents vary widely in length and the counts will be large for common words such as "the" and "is". We need to use a weighting scheme that considers the aforementioned attributes. The term frequency-inverse document frequency, **tf-idf** for short, is a popular weighting scheme to improve the simple count based data from the bag of words model. It is the product of two values, the term frequency and the inverse document frequency. There are several variants but the most popular is defined below.* **Term Frequency:**$$ \mathrm{tf}(t, d) = \frac{\mathrm{counts}(t, d)}{\sqrt{\sum_{t \in d} \mathrm{counts}(t, d)^2}}, $$    where $\mathrm{counts}(t, d)$ is the raw count of term $t$ in document $d$ and $t \in d$ are the terms in document $d$. The normalization results in a vector of unit length.* **Inverse Document Frequency:**$$ \mathrm{idf}(t, D) = \ln\left(\frac{\text{number of documents in corpus } D}{1 + \text{number of documents with term } t}\right). $$    Every counted term $t$ in the corpus will have its own idf weight. The $1+$ in the denominator is to ensure no division by zero if a term does not appear in the corpus. The idf weight is simply the log of the inverse of a term's document frequency.    With both $\mathrm{tf}(t, d)$ and $\mathrm{idf}(t, D)$ calculated, the tf-idf weight is$$ \mathrm{tfidf}(t, d, D) = \mathrm{tf}(t, d) \mathrm{idf}(t, D).$$With the idf weighting, words that are very common throughout the documents get weighted down. The reverse is true; the count of rare words get weighted up. With the tf-idf weighting scheme, a machine learning model will have an easier time to learn patterns to properly predict labels.<jupyter_code>tfidf = TfidfTransformer(smooth_idf=False)
# Most significant words in spam messages

spam_word_weights = tfidf.fit_transform(spam_word_counts)
top_idf_indices = tfidf.idf_.argsort()[:-20:-1]
ind_to_word = bag_of_words.get_feature_names()

print('The most significant words for spam messages are:')
for ind in top_idf_indices:
    print(tfidf.idf_[ind], ind_to_word[ind])
# Most significant words in ham messages

ham_word_weights = tfidf.fit_transform(ham_word_counts)
top_idf_indices = tfidf.idf_.argsort()[:-20:-1]
ind_to_word = bag_of_words.get_feature_names()

print('The most significant words for ham messages are:')
for ind in top_idf_indices:
    print(tfidf.idf_[ind], ind_to_word[ind])<jupyter_output>The most significant words for ham messages are:
8.788626065625031 zyada
8.788626065625031 iknow
8.788626065625031 ifink
8.788626065625031 ignorant
8.788626065625031 ignoring
8.788626065625031 ihave
8.788626065625031 ijust
8.788626065625031 ikno
8.788626065625031 ileave
8.788626065625031 immunisation
8.788626065625031 illspeak
8.788626065625031 ilol
8.788626065625031 imagination
8.788626065625031 imat
8.788626065625031 imf
8.788626065625031 imin
8.788626065625031 iff
8.788626065625031 ie
8.788626065625031 idu<jupyter_text>---  Building our Model<jupyter_code>X = sms.text.values
y = sms.label.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)<jupyter_output><empty_output><jupyter_text>Random Forest Classifier<jupyter_code>clf = RandomForestClassifier()

pipeline = Pipeline([
    ('bagOfWords', bag_of_words),
    ('tfidf', tfidf),
    ('clf', clf)
])

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
def display_results(cv, y_test, y_pred):
    labels = np.unique(y_pred)
    confusion_mat = confusion_matrix(y_test, y_pred, labels=labels)
    accuracy = (y_pred == y_test).mean()

    print("Labels:", labels)
    print("Confusion Matrix:\n", confusion_mat)
    print("Accuracy:", accuracy)
display_results(pipeline, y_test, y_pred)<jupyter_output>Labels: ['ham' 'spam']
Confusion Matrix:
 [[1207    1]
 [  30  155]]
Accuracy: 0.9777458722182341<jupyter_text>> Nice one with the first model. We will try to improve it using GridSearchCv and tuning its parameters.  Tune Parameters with GridSearchCv<jupyter_code>pipeline.get_params()
parameters = {
    'bagOfWords__ngram_range': ((1, 1), (1, 2)),
    'tfidf__use_idf': (True, False),
    'clf__min_samples_leaf': [1,3,5],
    'clf__n_estimators': [10, 30, 50],
    'clf__min_samples_split': [2, 3, 4]
}

clf_pipeline = GridSearchCV(pipeline, param_grid=parameters, n_jobs=-1, verbose=2)
clf_pipeline.fit(X_train, y_train)
y_pred = clf_pipeline.predict(X_test)
display_results(clf_pipeline, y_test, y_pred)<jupyter_output>Labels: ['ham' 'spam']
Confusion Matrix:
 [[1207    1]
 [  24  161]]
Accuracy: 0.9820531227566404<jupyter_text>Excellent!!! We have an improvement from the first model. Now we are going to work with a second model, a `Stochastic Gradient Descent` classifier.  Stochastic Gradient Descent<jupyter_code>sgd = SGDClassifier()
pipeline = Pipeline([
    ('bagOfWords', bag_of_words),
    ('tfidf', tfidf),
    ('sgd', sgd)
])

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
display_results(pipeline, y_test, y_pred)<jupyter_output>Labels: ['ham' 'spam']
Confusion Matrix:
 [[1205    3]
 [  13  172]]
Accuracy: 0.9885139985642498<jupyter_text>Tune the parameters of SGD classifier<jupyter_code>pipeline.get_params()
parameters = {
    'bagOfWords__ngram_range': ((1, 1), (1, 2)),
    'tfidf__use_idf': (True, False),
    'sgd__alpha': [0.0001, 0.0003, 0.0009, 0.001, 0.003, 0.009],
    'sgd__max_iter': [1000, 3000, 9000],
}
sgd_pipeline = GridSearchCV(pipeline, param_grid=parameters, n_jobs=-1, verbose=2)

sgd_pipeline.fit(X_train, y_train)
y_pred = sgd_pipeline.predict(X_test)
display_results(sgd_pipeline, y_test, y_pred)<jupyter_output>Labels: ['ham' 'spam']
Confusion Matrix:
 [[1205    3]
 [   9  176]]
Accuracy: 0.9913854989231874
{'fraction_non_alphanumeric': 0.10931317128321086, 'fraction_numerical': 0.04140757490107405, 'mean_word_length': 5.453716370269038, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '188362', 'license': 'MIT', 'n_tokens_mistral': 5048, 'n_tokens_neox': 4609, 'n_words': 1503, 'path': 'SMS Spam Collection.ipynb', 'repo_name': 'EvanMath/SMS-Spam-Collection', 'size': '14152'}