<jupyter_start><jupyter_text>Naive Bayes Classifier: SMS - spam/ham 1. Use sklearn.naive_bayes modules as the classifier to determine if a text message is SPAM or not (HAM).2. We will compare different approaches to feature selection CountVectorizer / tf-idf $c=\sqrt{a^2}$3. We will compare different flavors of Naive Bayes: Gaussian, Multinomial, Bernoulli.$$c = \sqrt{a^2 + b^2}$$  Import ModulesLet's get the usual suspects for data science.Also we will use matplotlib and seaborn for plotting<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.naive_bayes import GaussianNB,MultinomialNB,BernoulliNB
import csv
import seaborn as sns
from textblob import TextBlob
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.svm import SVC, LinearSVC
from sklearn.metrics import classification_report, f1_score, accuracy_score, confusion_matrix
from sklearn.pipeline import Pipeline
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split, GridSearchCV, learning_curve 
from sklearn.tree import DecisionTreeClassifier<jupyter_output><empty_output><jupyter_text>Import SMS dataUse rstrip() to remove any additional whitespace after each message<jupyter_code>messages = [ line.rstrip() for line in open('SMSSpamCollection')]<jupyter_output><empty_output><jupyter_text>Take a look at the first few messages<jupyter_code>for number,entry in enumerate(messages[:10]):
    print number, entry<jupyter_output>0 ham	Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...
1 ham	Ok lar... Joking wif u oni...
2 spam	Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
3 ham	U dun say so early hor... U c already then say...
4 ham	Nah I don't think he goes to usf, he lives around here though
5 spam	FreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv
6 ham	Even my brother is not like to speak with me. They treat me like aids patent.
7 ham	As per your request '<NAME> (<NAME>)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune
8 spam	WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 [...]<jupyter_text>Now we'll create a dataframe to help digest the data. The columns are tab-separated. Instruct pandas not to use quotes when reading in fields: csv.QUOTE_NONE.<jupyter_code>data = pd.read_csv('SMSSpamCollection.txt',sep='\t',names=["label","message"],quoting=csv.QUOTE_NONE)<jupyter_output><empty_output><jupyter_text>Let's get some info about our data set.<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>So we have a corpus of 5574 messages, of which 5171 are unique. Most of them (4827) are actually messages (HAM), of which 'Sorry, I'll call later' is the most frequent.<jupyter_code>data.groupby('label').describe()<jupyter_output><empty_output><jupyter_text>Let's introduce another field into the data frame: 'length.' Which is the number of words in the message<jupyter_code>data['length']=data['message'].map(lambda msg: len(msg))
data.head()<jupyter_output><empty_output><jupyter_text>Now let's look at the distribution of lengths for each label<jupyter_code>sns.distplot(data['length'])<jupyter_output><empty_output><jupyter_text>Now see some statistics on the lengths of messages<jupyter_code>data['length'].describe()<jupyter_output><empty_output><jupyter_text>What is the very long message?<jupyter_code>print list(data.message[data.length>900])<jupyter_output>["For me the love should start with attraction.i should feel that I need her every time around me.she should be the first thing which comes in my thoughts.I would start the day and end it with her.she should be there every time I dream.love will be then when my every breath has her name.my life should happen around her.my life will be named to her.I would cry for her.will give all my happiness and take all her sorrows.I will be ready to fight with anyone for her.I will be in love when I will be doing the craziest things for her.love will be when I don't have to proove anyone that my girl is the most beautiful lady on the whole planet.I will always be singing praises for her.love will be when I start up making chicken curry and end up makiing sambar.life will be the most beautiful then.will get every morning and thank god for the day because she is with me.I would like to say a lot..will tell later.."]<jupyter_text>View the length distributions by label<jupyter_code>data.hist(column='length', by='label', bins=50)<jupyter_output><empty_output><jupyter_text>With seaborn this can be done using facet grid<jupyter_code>facet = sns.FacetGrid(data,hue='label',col='label')
facet.map(sns.distplot,'length')<jupyter_output><empty_output><jupyter_text>PreprocessingNow we need to take out symbols and punctuation to get a better representation of the messageFirst, we convert everything to unicode, then use the TextBlob module to extract words...<jupyter_code>def split_into_tokens(message):
    message=unicode(message,'utf8')
    return TextBlob(message).words
data.message.head()
data.message.head().apply(split_into_tokens)
TextBlob('Learning Naive Bayes is Fun!')<jupyter_output><empty_output><jupyter_text>Now reduce words to their base form 'lemmas'<jupyter_code>def split_into_lemmas(message):
    message=unicode(message,'utf8').lower()
    words = TextBlob(message).words
    return [word.lemma for word in words]
data.message.head().apply(split_into_lemmas)
bow_transformer = CountVectorizer(analyzer=split_into_lemmas).fit(data['message'])

print len(bow_transformer.vocabulary_)
print bow_transformer.get_stop_words()
test_msg = data['message'][4]
print test_msg
test_bow = bow_transformer.transform([test_msg])
print test_bow
print test_bow.shape<jupyter_output>(0, 1320)	1
  (0, 2770)	1
  (0, 3628)	1
  (0, 3858)	2
  (0, 3915)	1
  (0, 4098)	1
  (0, 4708)	1
  (0, 5316)	1
  (0, 5328)	1
  (0, 7709)	1
  (0, 7728)	1
  (0, 7801)	1
  (0, 8139)	1
(1, 8859)<jupyter_text>which entry has 2 words?<jupyter_code>print bow_transformer.get_feature_names()[3858]
data_bow = bow_transformer.transform(data['message'])
print 'now we have a sparse matrix of shape', data_bow.shape
print 'with {} number of nonzer entries'.format(data_bow.nnz)
print 'Sparsity: {:.2f}%'.format(100. * data_bow.nnz \
                                 / (data_bow.shape[0] * data_bow.shape[1]))
% time spam_detector_MNB = MultinomialNB().fit(data_bow,data['label'])
% time spam_detector_GNB = GaussianNB().fit(data_bow.toarray(),data['label'])
% time spam_detector_BNB = BernoulliNB().fit(data_bow,data['label'])
print spam_detector_MNB.predict(test_bow)
print test_msg
print spam_detector_MNB.predict_proba(test_bow)
print test_bow<jupyter_output>(0, 1320)	1
  (0, 2770)	1
  (0, 3628)	1
  (0, 3858)	2
  (0, 3915)	1
  (0, 4098)	1
  (0, 4708)	1
  (0, 5316)	1
  (0, 5328)	1
  (0, 7709)	1
  (0, 7728)	1
  (0, 7801)	1
  (0, 8139)	1<jupyter_text>ok let's test predictive power between MNB,GNB,BBB (multinomial,gaussian,bernollui)We want to set up a cross validation pipeline using Bernoulli and Multinmoial Classifiers.First we need a training set and test set...Use 80% of the data to train, 20% to test using train_test_split()<jupyter_code>msg_train, msg_test, label_train, label_test = \
    train_test_split(data['message'],data['label'],test_size=0.2)
print len(msg_train), len(msg_test)<jupyter_output>4459 1115<jupyter_text>Create a Pipeline which allows multiple steps to be performed on the input data1. Vectorize the words2. Weight using the TF-IDF scheme3. Specify the classifier<jupyter_code>pipeline=Pipeline([
    ('bow', CountVectorizer(analyzer=split_into_lemmas)), #strings to lemmas
    ('tfidf', TfidfTransformer()), #weighted TF-IDF
    ('classifier', MultinomialNB()) # Naive Bayes Classifier
])<jupyter_output><empty_output><jupyter_text>Further subdivide the training data using Stratified Cross Validation, to make sure that our model is stable to different training sets.<jupyter_code>scores = cross_val_score(pipeline, # convert raw messages to a model
                        msg_train, # training data
                        label_train, # training labels
                        cv = 10, # split into 10 groups: 9 for training, 1 for testing
                        scoring = 'accuracy', # scoring metric
                        n_jobs = -1
                        )
print scores<jupyter_output>[ 0.93064877  0.95973154  0.95964126  0.9573991   0.96188341  0.95067265
  0.9573991   0.94157303  0.94382022  0.95280899]<jupyter_text>All scores are above 90% indicating decent accuracy and a stable model<jupyter_code>print scores.mean(), scores.std()<jupyter_output>0.951557807609 0.00951900372923<jupyter_text>Now let's see if we can find the best combination of parameters for a model.i.e.1. how much feature engineering to do: lemmas or tokens?2. what weighting scheme: just tf or iff as well?3. BernoulliNB or MultinomialNB<jupyter_code>pipeline = Pipeline([
    ('bow', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('classifier',MultinomialNB())
])
params = {
    'tfidf__use_idf': (True,False),
    'bow__analyzer' : (split_into_lemmas,split_into_tokens),
}
grid = GridSearchCV(
    pipeline,
    params,
    refit = True,
    n_jobs = -1,
    scoring = 'accuracy',
    cv = 10
)
%time MNB_detector = grid.fit(msg_train, label_train)
print MNB_detector.cv_results_['mean_test_score']
print MNB_detector.cv_results_['params']
pipeline_BNB = Pipeline([
    ('bow' , CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('classifier',BernoulliNB())
])
params = {
    'tfidf__use_idf' : (True, False),
    'bow__analyzer' : (split_into_lemmas,split_into_tokens)
}
grid_BNB = GridSearchCV(
    pipeline,
    params,
    refit = True,
    n_jobs = -1,
    scoring = 'accuracy',
    cv = 10
)
%time BNB_detector = grid_BNB.fit(msg_train,label_train)
print BNB_detector.cv_results_['mean_test_score']
print BNB_detector.cv_results_['params']

print MNB_detector.predict(['hi mom how are you ?'])
print MNB_detector.predict_proba(['hi mom how are you ?'])
print MNB_detector.predict(['WINNER! Credit for free!'])
print MNB_detector.predict_proba(['WINNER! Credit for free!'])
predictions = MNB_detector.predict(msg_test)
print confusion_matrix(label_test, predictions)
print classification_report(label_test, predictions)
pipeline_SVM = Pipeline([
    ('bow', CountVectorizer(analyzer=split_into_lemmas)),
    ('tfidf', TfidfTransformer()),
    ('classifier', SVC())
])

param_svm = [
    {'classifier__C' : [1,10,100,1000], 'classifier__kernel' : ['linear']},
    {'classifier__C' : [1,10,100,1000], 'classifier__gamma': [0.001,0.0001], 'classifier__kernel': ['rbf']}
]
grid_SVM = GridSearchCV(
    pipeline_SVM,
    param_svm,
    refit = True,
    n_jobs = -1,
    scoring = 'accuracy',
    cv = 10
)
%time SVM_detector = grid_SVM.fit(msg_train,label_train)
print SVM_detector.cv_results_['mean_test_score']
print SVM_detector.cv_results_['params']
predictions = SVM_detector.predict(msg_test)
print confusion_matrix(label_test,predictions)
print classification_report(label_test,predictions)
print SVM_detector.predict(['hi mom how are you ?'])
print SVM_detector.predict(['WINNER! Credit for free!'])
correlation_matrix = data.corr()
plt.figure(figsize=(10,8))
ax = sns.heatmap(correlation_matrix, vmax=1, square=True,annot=True,cmap='RdBu_r')
plt.title('Correlation matrix')
plt.show()
scores = cross_val_score(pipeline, # specifies classifier
                        data_train, # training data
                        label_train, # training labels
                        cv = 10, # split into 10 groups: 9 for training, 1 for testing
                        scoring = 'accuracy', # scoring metric
                        n_jobs = -1
                        )
print(scores)
% time wine_detector = grid.fit(data_train,label_train)
% time wine_detector = GaussianNB().fit(data_train,label_train)
predictions = wine_detector.predict(data_test)
print('accuracy', accuracy_score(label_test, predictions))
print('confusion matrix\n', 
      confusion_matrix(label_test, predictions))
print('(row=expected, col=predicted)')
print(svm_detector.best_estimator_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10221016516396712, 'fraction_numerical': 0.03614457831325301, 'mean_word_length': 4.943100995732575, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '121386', 'license': 'MIT', 'n_tokens_mistral': 4497, 'n_tokens_neox': 4109, 'n_words': 1237, 'path': 'content/downloads/notebooks/pipeline_NB.ipynb', 'repo_name': 'jbechtel/jbechtel.github.io-source', 'size': '12566'}