<jupyter_start><jupyter_text>**SMS Spam classification**The given dataset is a set of SMS tagged messages that have been collected for SMS Spam research. It contains one set of SMS messages in English of 5,574 messages, tagged acording being ham (legitimate) or spam.**Task is to build a prediction model that will accurately classify which texts are spam**  **Steps:**- Importing Libraries- Exploring the Dataset & removing unwanted columns- Exploratory Data Analysis - Univariate Analysis - Text Analysis - Feature Engineering - CountVectorisation- preprocessing data before loading to data - Predictive analysis - Conclusion  **1. Importing Libraries**<jupyter_code>#importing the necessary packages to process or plot the data
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from collections import Counter
from sklearn import feature_extraction, model_selection, naive_bayes, metrics, svm
from IPython.display import Image
import warnings
warnings.filterwarnings("ignore")
%matplotlib inline<jupyter_output><empty_output><jupyter_text>**Read the text file with the help of pandas read_table function**<jupyter_code>data = pd.read_table('/content/SMSSpamCollection',sep='\t',names=['v1','v2'])

# using the head function see what is originlly present in the dataset
data.head()<jupyter_output><empty_output><jupyter_text>**Shape of the dataset**<jupyter_code># shape method gives the shape of dataframe
data.shape<jupyter_output><empty_output><jupyter_text>Again take a look at the data,run below cells<jupyter_code>data.describe(include='all')
#describe data by group - use pandas groupby('v1').describe() method
data.groupby('v1').describe()<jupyter_output><empty_output><jupyter_text>whats your inference??<jupyter_code>#Calculating the length of all the messages
data['length']=data['v2'].apply(len)
data.head()



data.head()
data.describe()<jupyter_output><empty_output><jupyter_text>Distribution spam/non-spam plots<jupyter_code>count_Class=pd.value_counts(data["v1"], sort= True)
count_Class.plot(kind= 'bar', color= ["blue", "orange"])
plt.title('Bar chart')
plt.show()
count_Class.plot(kind = 'pie',  autopct='%1.0f%%')
plt.title('Pie chart')
plt.show()<jupyter_output><empty_output><jupyter_text>**Text Analysis**It is an important step when dealing with text data **We want to find the frequencies of words in the spam and non-spam messages. We use the function Counter.**<jupyter_code>#import counter function
from collections import Counter

#using counter
count1 = Counter(" ".join(data[data['v1']=='spam']["v2"]).split()).most_common(20)
df1 = pd.DataFrame.from_dict(count1)
df1 = df1.rename(columns={0: "words in spam", 1 : "count_"})
df1.plot.bar(legend = False, color = 'orange')
y_pos = np.arange(len(df1["words in spam"]))
plt.xticks(y_pos, df1["words in spam"])
plt.title('More frequent words in spam messages')
plt.xlabel('words')
plt.ylabel('number')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the majority of frequent words in msgs are **'the', 'to', 'a', 'or' and so on. These are called stop words and are a part of english grammer,but they don't really convey your message, we need to find the actual vocabulary words which contribute to the meaning**With stop words we refer to the most common words in a lenguage, there is no simgle, universal list of stop words. Using skearn , we will import the stopwords dataset listed,and remove them from our messages and again analyse the messages Now we will remove the stopwords and again do the analysis<jupyter_code>from sklearn.feature_extraction import text
stop = text.ENGLISH_STOP_WORDS
#removing the stop-words from the messages(complete dataset)
data['v2']=data['v2'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
## todo: rewrite the counter code here, and check the most frequent words 
#using counter
count1 = Counter(" ".join(data[data['v1']=='spam']["v2"]).split()).most_common(20)
df1 = pd.DataFrame.from_dict(count1)
df1 = df1.rename(columns={0: "words in spam", 1 : "count_"})
df1.plot.bar(legend = False, color = 'orange')
y_pos = np.arange(len(df1["words in spam"]))
plt.xticks(y_pos, df1["words in spam"])
plt.title('More frequent words in spam messages')
plt.xlabel('words')
plt.ylabel('number')
plt.show()<jupyter_output><empty_output><jupyter_text>Above after removing the stop words , we can actually find some meaningful words, like "Free","Contact" which can be actially found in spam mails<jupyter_code>##todo:Do the same for finding most frequent words in Non-Spam messages:

#note: stop words are already removed from all ham/spam sentences 

#using counter
count2 = Counter(" ".join(data[data['v1']=='ham']["v2"]).split()).most_common(20)
df2 = pd.DataFrame.from_dict(count2)
df2 = df2.rename(columns={0: "words in non-spam", 1 : "count_"})
df2.plot.bar(legend = False, color = 'blue')
y_pos = np.arange(len(df2["words in non-spam"]))
plt.xticks(y_pos, df2["words in non-spam"])
plt.title('More frequent words in non-spam messages')
plt.xlabel('words')
plt.ylabel('number')
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the majority of frequent words in both classes are **stop words** such as 'to', 'a', 'or' and so on.With stop words we refer to the most common words in a lenguage, there is no simgle, universal list of stop words. **In the next step,  feature engineering we will remove these stop words which don't majorly contribute in prediction**  **Feature engineering**  **First we need to understand how to Prepare Text Data for Machine Learning** **1. Intution/logic:**- Text data requires special preparation before you can start using it for predictive modeling.- The text must be parsed to remove words, called tokenization. - Then the words need to be encoded as integers or floating point values for use as input to a machine learning algorithm, called feature extraction (or vectorization).-scikit-learn library offers easy-to-use tools to perform both tokenization and feature extraction of your text data- **Since we cannot work with words directly we convert text to fixed-length vectors of numbers - also called Bag-of-words model**  **So Intution is to input Vectors --> model -->output class(spam/ham)**-The basic idea of BoW is to take a piece of text and count the frequency of the words in that text. It is important to note that the BoW concept treats each word individually and the order in which the words occur does not matter.- Using a process which we will go through now, we can convert a collection of documents to a matrix, with each document being a row and each word(token) being the column, and the corresponding (row,column) values being the frequency of occurrence of each word or token in that document. **2. Vectorisation:**- This can be done by assigning each word a unique number. - Then any document we see can be encoded as a fixed-length vector with the length of the vocabulary of known words. - The value in each position in the vector could be filled with a count or frequency of each word in the encoded document.- An encoded vector is returned with a length of the entire vocabulary and an integer count for the number of times each word appeared in the document.-  eg. he is a boy             attribute name assigned to words : he ->1 , is ->2 , a->3 , boy->4            considering the above sentence -count the occurance of each word             so vector is [1 1 1 1]-  The above logic is applied for complete corpus(text dataset)(Note: This is just a basic intution for understanding in detail pls refer this link)              **3. Scikit learn Count Vectorizers:** - Count Vectorizers provides a simple way to both tokenize a collection of text documents and build a vocabulary of known words, but also to encode new documents using that vocabulary.     Below is an example for reference:<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
# list of text documents
text = ["The quick brown fox jumped over the lazy dog."]
# create the transform
vectorizer = CountVectorizer()
# tokenize and build vocab
vectorizer.fit(text)
# summarize
print(vectorizer.vocabulary_)
# encode document
vector = vectorizer.transform(text)
# summarize encoded vector
print(vector.shape)
print(type(vector))
print(vector.toarray())<jupyter_output>{'the': 7, 'quick': 6, 'brown': 0, 'fox': 2, 'jumped': 3, 'over': 5, 'lazy': 4, 'dog': 1}
(1, 8)
<class 'scipy.sparse.csr.csr_matrix'>
[[1 1 1 1 1 1 1 2]]<jupyter_text>**Text preprocessing, tokenizing and filtering of stopwords are included in a high level component that is able to build a dictionary of features and transform documents to feature vectors.****We remove the stop words in order to improve the analytics**  **TODO: Vectorize spam message data ie. data["v2"]** You will be amazed by looking at the vocabulary<jupyter_code>#todo : vectorise the messages

from sklearn.feature_extraction.text import CountVectorizer
# list of text documents
text = data['v2']
# create the transform
vectorizer = CountVectorizer()
# tokenize and build vocab
vectorizer.fit(text)
# summarize
#print the vocalbulary - very large
# print(vectorizer.vocabulary_)
# encode document
vector = vectorizer.transform(text)
# summarize encoded vector
print(vector.shape)
print(type(vector))
print(vector.toarray())<jupyter_output>(5572, 8675)
<class 'scipy.sparse.csr.csr_matrix'>
[[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]<jupyter_text>We have created more than 8400 new features. The new feature $j$ in the row $i$ is equal to 1 if the word $w_{j}$ appears in the text example $i$. It is zero if not.  **Predictive Analysis** First we transform the variable spam/non-spam into binary variable(Binary Classification), then we split our data set in training set and test set.<jupyter_code>data.head()
data['label_num']=data.v1.map({'ham':0,'spam':1})
data.head()<jupyter_output><empty_output><jupyter_text>**train-test split**<jupyter_code>X_train, X_test, y_train, y_test = model_selection.train_test_split(X, data['label_num'], test_size=0.33, random_state=42)
print([np.shape(X_train), np.shape(X_test)])
print(y_train[:5])
from sklearn.naive_bayes import MultinomialNB,GaussianNB
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score

#Loading all classifier
svc = SVC(kernel = 'linear')
mnb = MultinomialNB(alpha =0.2)
gnb  = GaussianNB()
lr = LogisticRegression(solver='liblinear', penalty='l1')
rfc = RandomForestClassifier(n_estimators=100,random_state=11)
abc = AdaBoostClassifier(n_estimators =100,random_state=11)
#defining functions for training and testing data

def training(clf,X_train,y_train):
    clf.fit(X_train,y_train)
    
#function for predicting labels

def predict(clf,X_test):
    return clf.predict(X_test)
#defining a dictionary of classifier
classifier={'SVM': svc , 'MultinomialNB': mnb,'GaussianNB': gnb,'logistic': lr,'RandomForest': rfc,'Adaboost': abc}
#predict and storing score of each classifier
score = []
for n,c in classifier.items():
    training(c,X_train.toarray(),y_train)
    pred = predict(c,X_test.toarray())
    score.append((n,accuracy_score(y_test,pred),precision_score(y_test,pred),recall_score(y_test,pred),f1_score(y_test,pred)))

print(score)
score_df=pd.DataFrame(score,columns=['classifier','acc','precision','recall','f1_score'])
#Adding accuracy column
score_df['Accuracy (%)']=score_df['acc']*100
score_df
# Extra : You can also use GridSearch for fine tuning the models.<jupyter_output><empty_output><jupyter_text>Between these models with the highest possible precision, we are going to select the one which has more test accuracy. Clearly SVM is a WINNER  Confusion matrix with support vector machine classifier.<jupyter_code>m_confusion_test = metrics.confusion_matrix(y_test, svc.predict(X_test.toarray()))
pd.DataFrame(data = m_confusion_test, columns = ['Predicted 0(ham)', 'Predicted 1(spam)'],
            index = ['Actual 0(ham)', 'Actual 1(spam)'])<jupyter_output><empty_output><jupyter_text>We misclassify 19 spam as non-spam messages whereas only 1 ham(non-spam) message is misclassified.  Conclusion **The best model found is support vector machine with 98.9% validation accuracy.** **It classifies 99.5% non-spam message correctly (Model precision)**  **It classifies the 92.2% of spam messages correctly (Model recall)**<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1054287962234461, 'fraction_numerical': 0.014240755310778914, 'mean_word_length': 5.648012552301255, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '557877', 'license': 'MIT', 'n_tokens_mistral': 4003, 'n_tokens_neox': 3774, 'n_words': 1454, 'path': 'SMS-SPAM-FILTERING/spamfiltersms(Solution).ipynb', 'repo_name': 'abhisngh/Data-Science', 'size': '12710'}