<jupyter_start><jupyter_text>Spam Ham filter using sklearn<jupyter_code>import warnings
warnings.filterwarnings("ignore")
%matplotlib inline
import matplotlib.pyplot as plt
import csv
import pandas as pd
import sklearn
import cPickle
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import SVC, LinearSVC
from sklearn.metrics import classification_report, f1_score, accuracy_score, confusion_matrix
from sklearn.pipeline import Pipeline
from sklearn.grid_search import GridSearchCV
from sklearn.cross_validation import StratifiedKFold, cross_val_score, train_test_split 
from textblob import TextBlob
from wordcloud import WordCloud<jupyter_output>/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
  DeprecationWarning)<jupyter_text>Data prepration download dataset from https://archive.ics.uci.edu/ml/machine-learning-databases/00228<jupyter_code>df = pd.read_csv('./data/SMSSpamCollection' , sep='\t' , names=['status', 'message'])
df.head()
len(df)
len(df[df.status=='spam'])<jupyter_output><empty_output><jupyter_text>Aggregate statistics analysis of dataUsing pandas and matplotlib we can easily anlaysis the dataset<jupyter_code>df.groupby('status').describe()<jupyter_output><empty_output><jupyter_text>how long the message is ?getting length of each message=><jupyter_code>df['length'] = df['message'].map(lambda msg:len(msg))
df.head()
df['length'].plot(bins=40,kind='hist',color='orange')
#describing message length
print(df['length'].describe())
#describing HAM message by length
print('HAM',df['length'].loc[df['status']=='ham'].describe())
#describing SPAM message by length
print('SPAM',df['length'].loc[df['status']=='spam'].describe())
df['length'].plot(kind='hist',bins=20,color='red',facecolor='green',alpha=0.5,normed=1)
#length difference betwwn SPAM(1) and HAM(0)
df.hist(column='length', by='status', bins=20)<jupyter_output><empty_output><jupyter_text>getting largest and smallest message<jupyter_code>print("Largest messge is:",list(df.message[df.length>900]))  #we can get this 900 from above df.describe
print("Smallest message is ",list(df.message[df.length<3]))  #we can get this 3 from above df.describe<jupyter_output>('Largest messge is:', ["For me the love should start with attraction.i should feel that I need her every time around me.she should be the first thing which comes in my thoughts.I would start the day and end it with her.she should be there every time I dream.love will be then when my every breath has her name.my life should happen around her.my life will be named to her.I would cry for her.will give all my happiness and take all her sorrows.I will be ready to fight with anyone for her.I will be in love when I will be doing the craziest things for her.love will be when I don't have to proove anyone that my girl is the most beautiful lady on the whole planet.I will always be singing praises for her.love will be when I start up making chicken curry and end up makiing sambar.life will be the most beautiful then.will get every morning and thank god for the day because she is with me.I would like to say a lot..will tell later.."])
('Smallest message is ', ['Ok', 'Ok', 'Ok', 'Ok'])<jupyter_text>so we can see that most of the SPAM messages are of length 110 to 160 Data preprocessingIt is necessary because we can't apply any Machine learning model on text type data. We we need to tokenize the data.So we going to conver raw message into vectors. Steps:1. Split all data into invidual words2. normalizing data 3. data to vectors dependency:first you need to downlaod **punkt** otherwise it will show you an error.start python terminal and type  >>import nltk  >>nltk.download('punkt)  >>nltk.download('wordnet')  1. split into individual words 2.normalize words into their base form which is Lemmatisation Lemmatisation of dataThe verb ' to go' may appear as 'go', 'goes', 'gone', 'going'. The base form, 'go', that one might look up in a dictionary, is called the lemma for the word.<jupyter_code>class Split:
    
    """This class is for spliting data into individual words and lemmatistion of words."""
    
    def into_tokens(self,msg):
        msg = unicode(msg, 'utf8')  # convert bytes into proper unicode
        return TextBlob(msg).words

    def into_lemmas(self,message):
        message = unicode(message, 'utf8').lower()
        words = TextBlob(message).words
        # for each word, take its "base form" = lemma 
        return [word.lemma for word in words]
split = Split()
df.message.head().apply(split.into_tokens)  
df.message.head().apply(split.into_lemmas)
# print(split_into_lemmas("go goes"))<jupyter_output><empty_output><jupyter_text>3. Getting vectors from data What is text feature extraction?Convert a collection of text documents to a matrix of token countsText Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length.<jupyter_code>vectorizer = CountVectorizer(analyzer=split.into_lemmas)
#example
X = vectorizer.fit_transform(['hellow','hellow we are back','we we we are back'])
print(X)
# .vocabulary_ gives counts of all words in corpus
print(vectorizer.vocabulary_) 
X = vectorizer.fit_transform(df['message'])<jupyter_output><empty_output><jupyter_text>**getting feature names using vectorizer.get_feature_names**<jupyter_code>vectorizer.get_feature_names()[:10]   #printing 10 feature 
bag_of_words = vectorizer.fit_transform(df['message'])
X = vectorizer.transform(['win win win this'])
print(bag_of_words.shape,X.shape)<jupyter_output>((5572, 8859), (1, 8859))<jupyter_text>**Now ***bag_of_word*** contains full SMS coupus in the from of vector**  Creating weigthage list using tf-idf using scikit-learn's TfidfTransformer**tf–idf or TFIDF**, is short form of **term frequency–inverse document frequency**, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. The tf-idf value increases proportionally to the number of times a word appears in the document, but is often offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general. Nowadays, tf-idf is one of the most popular term-weighting schemes.[more](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)<jupyter_code>tfidf_transformer = TfidfTransformer()
tfidf = tfidf_transformer.fit_transform(bag_of_words)
print(tfidf.shape)
df.loc[df["status"]=='ham',"status"]=1
df.loc[df["status"]=='spam',"status"]=0
df['status'] = df['status'].astype('int')<jupyter_output><empty_output><jupyter_text>Training modelNow we will train our classifier using numerical message vector.This is very easy using sklearn.We will use **```MultinomialNB```** classifier of sklearn<jupyter_code>class Train:
    """Traning and testing our model."""
    
    def __init__(self):
        self.clf = MultinomialNB()
        
    def train(self,x,y):
        self.clf.fit(x,y)
        
    def score(self,x,y):
        return self.clf.score(x,y)
    
    def test(self,y):
        return self.clf.predict(y)
    
    def accuracy(self,x,y):
        acc = accuracy_score(x,y,normalize=False)
        print('accuracy', acc)
        return acc
X_train, X_test, Y_train, Y_test = train_test_split(tfidf,df['status'],test_size=.1)
clf = Train()
clf.train(X_train,Y_train)
print(clf.score(X_test,Y_test))
all_predictions = clf.test(tfidf)
print all_predictions
print 'accuracy', accuracy_score(df['status'], all_predictions)
print 'confusion matrix\n', confusion_matrix(df['status'], all_predictions)
print '(row=expected, col=predicted)'<jupyter_output>[1 1 0 ..., 1 1 1]
accuracy 0.965003589375
confusion matrix
[[ 552  195]
 [   0 4825]]
(row=expected, col=predicted)<jupyter_text>Testing on some random message string<jupyter_code>message = ['attending workshop']
X = vectorizer.transform(message)
print(X.shape)
clf.test(X)<jupyter_output>(1, 8859)<jupyter_text>Creating Word Cloud of Spam and Ham messages<jupyter_code>#spam  = 0 , Ham = 1 generating spam hand ham wordlist
spam_wl = ' '.join(list(df[df['status']==0]['message']))
ham_wl = ' '.join(list(df[df['status']==1]['message']))
spam_wc = WordCloud(background_color='white',width=512,height=512).generate(spam_wl)
ham_wc = WordCloud(background_color='white',width=512,height=512).generate(ham_wl)<jupyter_output><empty_output><jupyter_text>Spam word cloud<jupyter_code>plt.figure(figsize=(10,8))
plt.imshow(spam_wc)<jupyter_output><empty_output><jupyter_text>Ham word cloud<jupyter_code>plt.figure(figsize=(10,8))
plt.imshow(ham_wc)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10510385137250809, 'fraction_numerical': 0.015029746373029956, 'mean_word_length': 5.766949152542373, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '607361', 'license': 'Unlicense', 'n_tokens_mistral': 3088, 'n_tokens_neox': 2920, 'n_words': 978, 'path': 'Spam_filter.ipynb', 'repo_name': 'lalityadav1509/NLP', 'size': '9581'}