<jupyter_start><jupyter_text>SMS SPAM DETECTION FILTERI used this Data set from UCI datasets.It contains text data of 5000 SMS along with whether the sms is spam or ham.<jupyter_code># As this is text file we created list from this text file
messages = [line.rstrip() for line in open('smsspamcollection/SMSSpamCollection')]
print(len(messages))
# To check what messages are there in list we create a loop to read messages
for message_no, message in enumerate(messages[:5]):
    print(message_no, message)
    print('\n')
# To properly manipulate the data we created data frame called messages 
# Also it has separater space so tab(\t) is used

import pandas as pd

messages = pd.read_csv('smsspamcollection/SMSSpamCollection', sep='\t',
                           names=["label", "message"])
messages.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>messages.describe()
messages.groupby('label').describe()
# We are creating a new column to calcuate the length of messages

messages['length'] = messages['message'].apply(len)
messages.head()<jupyter_output><empty_output><jupyter_text>Data Visualizations<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
# Histogram showing general length of text messages
messages['length'].plot(bins=50, kind='hist') 
messages.length.describe()
# We are now creating a histogram to check lengths of spam and ham messages

messages.hist(column='length', by='label', bins=100,figsize=(12,4))<jupyter_output><empty_output><jupyter_text>We can see from the graph that spam messages have more clusters around 150 length while ham messages have more clusters around 100 length.So we can predict that spam messages are more in length that ham messages  Text Processing<jupyter_code>#Importing Natural Processing library
import nltk
#nltk.download_shell()

#Creating a function to remove punctuation
import string

mess = 'Sample message! Notice: it has punctuation.'

# Check characters to see if they are in punctuation
nopunc = [char for char in mess if char not in string.punctuation]

# Join the characters again to form the string.
nopunc = ''.join(nopunc)
#cleanig the text with removing the stop words
clean_mess = [word for word in nopunc.split() if word.lower() not in stopwords.words('english')]
# We are using stop words from NLTK library
# Stop words are words which are generally not useful in predicting whether sms will be spam or ham
from nltk.corpus import stopwords
stopwords.words('english')[0:10] # Show some stop words
def text_process(mess):
    """
    Takes in a string of text, then performs the following:
    1. Remove all punctuation
    2. Remove all stopwords
    3. Returns a list of the cleaned text
    """
    # Check characters to see if they are in punctuation
    nopunc = [char for char in mess if char not in string.punctuation]

    # Join the characters again to form the string.
    nopunc = ''.join(nopunc)
    
    # Now just remove any stopwords
    return [word for word in nopunc.split() if word.lower() not in stopwords.words('english')]<jupyter_output><empty_output><jupyter_text>Now we are going to "tokenize" these messages. Tokenization is just the term used to describe the process of converting the normal text strings in to a list of tokens (words that we actually want).<jupyter_code>messages['message'].head(5).apply(text_process)<jupyter_output><empty_output><jupyter_text>Vectorization I am using a bag of words model here. For implementing any algorithm we need to convert this texts in vectors. This process is known as vectorization. We'll do that in three steps using the bag-of-words model:Count how many times does a word occur in each message (Known as term frequency)Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)Normalize the vectors to unit length, to abstract from the original text length (L2 norm)I will use Count Vectorizer.This model will convert a collection of text documents to a matrix of token counts.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
# creating transformer with analyzer set at text process
bow_transformer = CountVectorizer(analyzer=text_process).fit(messages['message'])

# Print total number of vocab words
print(len(bow_transformer.vocabulary_))
message4 = messages['message'][3]
print(message4)<jupyter_output>U dun say so early hor... U c already then say...<jupyter_text>We can see the vector representation<jupyter_code>bow4 = bow_transformer.transform([message4])
print(bow4)
print(bow4.shape)
#Transforming Bag-of-Words (bow) transformed object and transform the entire DataFrame of messages
messages_bow = bow_transformer.transform(messages['message'])<jupyter_output><empty_output><jupyter_text>Now we will use TF-IDF Term frequency and Inverse Document Frequencyhis weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer

tfidf_transformer = TfidfTransformer().fit(messages_bow)
tfidf4 = tfidf_transformer.transform(bow4)
print(tfidf4)
# This is used to transform whole bag of words into tf-idf corpus
messages_tfidf = tfidf_transformer.transform(messages_bow)
print(messages_tfidf.shape)
#Training the model
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = \
train_test_split(messages['message'], messages['label'], test_size=0.2)


# This creates pipeline to remeber all functions for future refreneces
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('bow', CountVectorizer(analyzer=text_process)),  # strings to token integer counts
    ('tfidf', TfidfTransformer()),  # integer counts to weighted TF-IDF scores
    ('classifier', MultinomialNB()),  # train on TF-IDF vectors w/ Naive Bayes classifier
])
# fitting the model
pipeline.fit(X_train,y_train)
# Now trying to predict on basis of model
predictions = pipeline.predict(X_test)
# Measuring accuracy
from sklearn.metrics import classification_report
print(classification_report(predictions,label_test))<jupyter_output>precision    recall  f1-score   support

        ham       0.89      0.86      0.88       993
       spam       0.12      0.16      0.14       122

avg / total       0.81      0.78      0.79      1115
{'fraction_non_alphanumeric': 0.07243024432557467, 'fraction_numerical': 0.011131993638860778, 'mean_word_length': 4.974093264248705, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '414010', 'license': 'MIT', 'n_tokens_mistral': 2005, 'n_tokens_neox': 1917, 'n_words': 800, 'path': 'SMS Spam Filter.ipynb', 'repo_name': 'so19/Airbnb-Project', 'size': '6917'}