<jupyter_start><jupyter_text>1. Introduction- Email spam, also referred to as junk email, is unsolicited messages sent in bulk by email (spamming).- The name comes from a Monty Python sketch in which Spam is ubiquitous, unavoidable, and repetitive.[1] Email spam has steadily grown since the early 1990s, and by 2014 was estimated to account for around 90% of total email traffic.[2]-Most email spam messages are commercial in nature. Whether commercial or not, many are not only annoying, but also dangerous because they may contain links that lead to phishing web sites or sites that are hosting malware - or include malware as file attachments.  1.1Defining the Questions1. Predict if an email is spam or not   1.2 Metrics for success* Accuracy and FI score will be used todeterrmine efficiency,the highest the score ,the better the model*different split sizes 80-20,70-30 and 60-40  will be used to determine the best of the three   1.3Appropriateness of the data**Dataset 2 link:*** [link text](https://archive.ics.uci.edu/ml/datasets/Spambase)* The collection of spam e-mails came from the postmaster and individuals who had filed spam.* Drawback :This dataset has no column names.  Data Understanding  2.1 Importing libraries<jupyter_code># Importing Libraries we will use for this project analysis

import pandas as pd  # for data processing
import numpy as np  # for numerical calculations


# Data Visualization Libraries
import seaborn as sns
import seaborn as sb

import matplotlib.pyplot as plt

%matplotlib inline



# Sklearn libraries for data preparartion and performance measures
from sklearn.model_selection import train_test_split, RandomizedSearchCV, GridSearchCV, KFold, cross_val_predict
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, normalize, Normalizer
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.decomposition import PCA


# Algorithms
from sklearn.naive_bayes import MultinomialNB, GaussianNB, BernoulliNB
from sklearn.neighbors import KNeighborsClassifier

from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from xgboost import XGBClassifier<jupyter_output><empty_output><jupyter_text>1.2 Loading the datasets<jupyter_code># Loading the dataset

sm= pd.read_csv('spambase.csv')<jupyter_output><empty_output><jupyter_text>2.3 Checking data<jupyter_code># Viewing the first  five observations of the  dataset
sm.head(5)
# Viewing the first five observations of the  dataset
sm.head(5)
# Checking the number of rows and columns in the dataset

# Get the number of rows and columns 
rows = len(sm.axes[0]) 
cols = len(sm.axes[1]) 
  
# Print the number of rows and columns 
print("Number of Rows in this  dataset is: " + str(rows)) 
print("Number of Columns in this  dataset is : " + str(cols))<jupyter_output>Number of Rows in this  dataset is: 4600
Number of Columns in this  dataset is : 58<jupyter_text>3 Data Preparation<jupyter_code># viewing summary information for our dataset set

sm.info()
# Summary statistics of the dataset
# 
sm.describe()<jupyter_output><empty_output><jupyter_text>3.1 Missing values<jupyter_code># Checking for null values

sm.isnull().sum()<jupyter_output><empty_output><jupyter_text>there are no missing values  3.2 Duplicates<jupyter_code># checking for duplicates
sm.duplicated().sum()<jupyter_output><empty_output><jupyter_text>There are 391  duplicates and we will drop them<jupyter_code># Dropping for duplicates 

sm.drop_duplicates(inplace = True)

# Cconfriming that we no longer have duplicates

sm.duplicated().sum()<jupyter_output><empty_output><jupyter_text>Duplicates have been successfully dropped 4 Building  Machine Learning Models  4.1Naive Bayes Classifier* The Naive Bayes Classifier is a statistical classification technique based on the Bayes Theorem. * It has high accuracy and speed on large datasets.* This type of classifier takes into account the assumption that the effect of a particular feature in a class is independent of other features. Multinomial NB considers the frequency count (occurrences) of the features  while Bernoulli NB cares only about the presence or absence of a particular feature (word) in the document. The latter is adequate for features that are binary-valued (Bernoulli, boolean). Whereas, with Gaussian NB, features are real-valued or continuous and their distribution is Gaussian,  splitting data in 80-20 set<jupyter_code>X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)




lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

gnb = GaussianNB()

model = gnb.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')

print(classification_report(y_test, y_pred))<jupyter_output>0.8978622327790974


[[475  20]
 [ 66 281]]


              precision    recall  f1-score   support

           0       0.88      0.96      0.92       495
           1       0.93      0.81      0.87       347

    accuracy                           0.90       842
   macro avg       0.91      0.88      0.89       842
weighted avg       0.90      0.90      0.90       842<jupyter_text>* The 80,20 split model yielded 89.78% accuracy.* Interpreting the confusion matrix;* The first row is about the non-spam-predictions:      *475 emails were correctly classified as Ham (called true negatives)       * 66 were wrongly classified as ham (false positives).* The second row is about the spam-predictions:      * 20 emails where wrongly classified as spam (false negatives) and     * 281 were correctly classified as Spam (true positives).      Splitting  into 70, 30 sets<jupyter_code>X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)




lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

gnb = GaussianNB()

model = gnb.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')
print(classification_report(y_test, y_pred))<jupyter_output>0.9036231884057971


[[789  33]
 [100 458]]


              precision    recall  f1-score   support

           0       0.89      0.96      0.92       822
           1       0.93      0.82      0.87       558

    accuracy                           0.90      1380
   macro avg       0.91      0.89      0.90      1380
weighted avg       0.91      0.90      0.90      1380<jupyter_text>* The 70,30 split model also yielded 90.36% accuracy.(higher than previous model)* Interpreting the confusion matrix;* The first row is about the non-spam-predictions:     * 789 emails were correctly classified as Ham (called true negatives)      *100 were wrongly classified as ham (false positives).* The second row is about the spam-predictions:      * 33 emails where wrongly classified as spam (false negatives) and     * 458 were correctly classified as Spam (true positives).      Splitting  into 60, 40 sets<jupyter_code>X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)




lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

gnb = GaussianNB()

model = gnb.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')
print(classification_report(y_test, y_pred))<jupyter_output>0.9081521739130435


[[1060   37]
 [ 132  611]]


              precision    recall  f1-score   support

           0       0.89      0.97      0.93      1097
           1       0.94      0.82      0.88       743

    accuracy                           0.91      1840
   macro avg       0.92      0.89      0.90      1840
weighted avg       0.91      0.91      0.91      1840<jupyter_text>* The 60,40 split model yielded 90.81% accuracy(Higest of all the models)* Interpreting the confusion matrix;* The first row is about the non-spam-predictions:     * 1060 emails were correctly classified as Ham (called true negatives)      * 132 were wrongly classified as ham (false positives).* The second row is about the spam-predictions:      * 37 emails where wrongly classified as spam (false negatives) and     * 611 were correctly classified as Spam (true positives)..      5 Challenging the solution SVM builds a classifier by searching for a separating hyperplane (optimal hyperplane) which is optimal and maximises the margin that separates the categories (in our case spam and ham/not spam). Thus, SVM has the advantage of robustness in general and effectiveness when the number of dimensions is greater than the number of samples.The 60-40 split set will be used to challenge the nayes bayes as it was the best split set<jupyter_code># Fitting the Support Vector Classifier
# Splitting the data

X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)




lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

svc = SVC(C=0.1, gamma=0.01, kernel = 'linear')

model = svc.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')

print(classification_report(y_test, y_pred))
# Fitting the Support Vector Classifier
# Splitting the data

X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)



lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

svc = SVC(C=0.1, gamma=0.01, kernel = 'rbf')

model = svc.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')

print(classification_report(y_test, y_pred))
# Fitting the Support Vector Classifier
# Splitting the data

X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)


lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

svc = SVC(C=0.1, gamma=0.01, kernel = 'poly')

model = svc.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')

print(classification_report(y_test, y_pred))
# Fitting the Support Vector Classifier
# Splitting the data

X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)



lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

svc = SVC(C=0.1, gamma=0.01, kernel = 'sigmoid')

model = svc.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')

print(classification_report(y_test, y_pred))
#e linear kernel perform best for this partical classification problem.

# Here we will use Grid search to find the best parameters for the model

# Creating a dictionary
# Specifying the parameters we want to tune

params = {'C': [0.00001,0.001,0.1],
          'gamma': [0.01, 0.001,0.0001]}

svc = SVC()

search = GridSearchCV(estimator = svc, 
                     param_grid= params, 
                      scoring = 'accuracy',
                      cv = 10 )
                
                
search.fit(X,y)
search.best_params_
# Fitting the Support Vector Classifier
# Splitting the data

X = sm.iloc[:, 0:-1]
y = sm.iloc[:,-1]



X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)




lda = LDA(n_components=10)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

svc = SVC(C=0.1, gamma=0.001, kernel = 'linear')

model = svc.fit(X_train, y_train)

y_pred = model.predict(X_test)


# Evaluating our model using accuracy score, confusion matrix and classification report.
print(accuracy_score(y_test, y_pred))

print('\n')

print(confusion_matrix(y_test, y_pred))

print('\n')

print(classification_report(y_test, y_pred))<jupyter_output>0.9217391304347826


[[1041   56]
 [  88  655]]


              precision    recall  f1-score   support

           0       0.92      0.95      0.94      1097
           1       0.92      0.88      0.90       743

    accuracy                           0.92      1840
   macro avg       0.92      0.92      0.92      1840
weighted avg       0.92      0.92      0.92      1840
{'fraction_non_alphanumeric': 0.09027584285316244, 'fraction_numerical': 0.041864028977431036, 'mean_word_length': 3.755548194766479, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '292770', 'license': 'MIT', 'n_tokens_mistral': 5192, 'n_tokens_neox': 4722, 'n_words': 1479, 'path': 'aurelia_wambui_week_9_assignment2.ipynb', 'repo_name': 'AureliaWambui19/EMAIL-SPAM-DETECTION-NAYES-BAYES', 'size': '14356'}