<jupyter_start><jupyter_text>**Week 9 Ip on k-Nearest Neighbours(KNN)** Defining the question **a) Specifying the analytic question** Build a model using Naive Bayes Classifier to predict whether a message is a spam or not. b) Defining the Metric of Success If the model has an accuracy higher than 80% it will be considered successful c)**Understanding the context** The "spam" concept is diverse: advertisements for products/web sites, make money fast schemes, chain letters, pornography...The collection of spam e-mails came from our postmaster and individuals who had filed spam. Our collection of non-spam e-mails came from filed work and personal e-mails, and hence the word 'george' and the area code '650' are indicators of non-spam. These are useful when constructing a personalized spam filter. One would either have to blind such non-spam indicators or get a very wide collection of non-spam to generate a general purpose spam filter. d)**Recording the Experimental Design** importing libraries-loading data-data cleaning-exploratory data analysis-modeling-implementing model-conclusion-recommendations 2.**Reading the Data**<jupyter_code>#importing libraries
import pandas as pd 
import numpy as np
import seaborn as sns
from matplotlib import pyplot as plt 
# Loading the Data from the source
data=pd.read_csv('spambase.csv')
data<jupyter_output><empty_output><jupyter_text>3. Checking the Data<jupyter_code># Determining the no. of columns in our dataset
data.columns
# Previewing the top of our dataset
data.head()
# Previewing the bottom of our dataset
data.tail()
#check the data types for all the variables 
data.dtypes

data.head(5)
data.describe()<jupyter_output><empty_output><jupyter_text>5. Tidying the Dataset<jupyter_code>#The first step will be check whether to drop any column in our dataset
#All our data seems to be valuable thus we will not drop any column
data.head(5)
#Let us preview our data now
data.head(5)
#Checking the existence of null values in our dataset
data.isnull().sum()

# There are no null values in our dataset 
#Checking for duplicates in the dataset
data.duplicated().sum()
#There are 391 duplicates in our dataset
# dropping duplicate values
data.drop_duplicates(keep=False,inplace=True)
#Let us check for any duplicates after dropping them
data.duplicated().sum()
#We have no duplicates now
#Let us check for unique values in our dataset
data.nunique().sum
#Our data seems to have many unique values
#checcking the numerical and categorical variables
numerical = data._get_numeric_data().columns
categorical = set(data.columns) - set(numerical)
numerical
categorical<jupyter_output><empty_output><jupyter_text>BIVARIATE and UNIVARIATE ANALYSIS<jupyter_code>data['Spam'].value_counts()
f, ax = plt.subplots(figsize=(6, 8))
ax = sns.countplot(x="Spam", data=data, palette="Set1")
plt.show()
#PLOTTING HORIZONTALLY
f, ax = plt.subplots(figsize=(8, 4))
ax = sns.countplot(y="Spam", data=data, palette="Set1")
plt.show()<jupyter_output><empty_output><jupyter_text>The number of unique values in Spam variable is 2.The two unique values are No and Yes.The No variable have 2457 entries, andThe Yes variable have 1573 entries.<jupyter_code>num_of_rows = 4
num_of_cols = 4
fig, ax = plt.subplots(num_of_rows, num_of_cols, figsize=(15,15))
print(numerical)
i=0;j=0;k=0;
while i<num_of_rows:
    while j<num_of_cols:
        sns.boxplot(data[numerical[k]], ax=ax[i, j])
        k+=1;j+=1
    j=0;i+=1
plt.savefig('before_removing_outliers_from_numerical_columns.png')
plt.show()<jupyter_output>Index(['word_freq_make', 'word_freq_address', 'word_freq_all', 'word_freq_3d',
       'word_freq_our', 'word_freq_over', 'word_freq_remove',
       'word_freq_internet', 'word_freq_order', 'word_freq_mail',
       'word_freq_receive', 'word_freq_will', 'word_freq_people',
       'word_freq_report', 'word_freq_addresses', 'word_freq_free',
       'word_freq_business', 'word_freq_email', 'word_freq_you',
       'word_freq_credit', 'word_freq_your', 'word_freq_font', 'word_freq_000',
       'word_freq_money', 'word_freq_hp', 'word_freq_hpl', 'word_freq_george',
       'word_freq_650', 'word_freq_lab', 'word_freq_labs', 'word_freq_telnet',
       'word_freq_857', 'word_freq_data', 'word_freq_415', 'word_freq_85',
       'word_freq_technology', 'word_freq_1999', 'word_freq_parts',
       'word_freq_pm', 'word_freq_direct', 'word_freq_cs', 'word_freq_meeting',
       'word_freq_original', 'word_freq_project', 'word_freq_re',
       'word_freq_edu', 'word_freq_table', 'word_freq_conference',
[...]<jupyter_text>Removing outliers<jupyter_code>lsUpper = []
lsLower = []
def removeOutliers(numerical):
    for i in range(len(numerical)):
        q1 = data[numerical[i]].quantile(0.25)
        q3 = data[numerical[i]].quantile(0.75)
        IQR = q3-q1
        minimum = q1 - 1.5 * IQR
        maximum = q3 + 1.5 * IQR
        data.loc[(data[numerical[i]] <= minimum), numerical[i]] = minimum
        data.loc[(data[numerical[i]] >= maximum), numerical[i]] = maximum  
removeOutliers(numerical)
num_of_rows = 4
num_of_cols = 4
fig, ax = plt.subplots(num_of_rows, num_of_cols, figsize=(15,15))
print(numerical)
i=0;j=0;k=0;
while i<num_of_rows:
    while j<num_of_cols:
        sns.boxplot(data[numerical[k]], ax=ax[i, j])
        k+=1;j+=1
    j=0;i+=1
plt.savefig('after_removing_outliers_from_numerical_columns.png')
plt.show()<jupyter_output>Index(['word_freq_make', 'word_freq_address', 'word_freq_all', 'word_freq_3d',
       'word_freq_our', 'word_freq_over', 'word_freq_remove',
       'word_freq_internet', 'word_freq_order', 'word_freq_mail',
       'word_freq_receive', 'word_freq_will', 'word_freq_people',
       'word_freq_report', 'word_freq_addresses', 'word_freq_free',
       'word_freq_business', 'word_freq_email', 'word_freq_you',
       'word_freq_credit', 'word_freq_your', 'word_freq_font', 'word_freq_000',
       'word_freq_money', 'word_freq_hp', 'word_freq_hpl', 'word_freq_george',
       'word_freq_650', 'word_freq_lab', 'word_freq_labs', 'word_freq_telnet',
       'word_freq_857', 'word_freq_data', 'word_freq_415', 'word_freq_85',
       'word_freq_technology', 'word_freq_1999', 'word_freq_parts',
       'word_freq_pm', 'word_freq_direct', 'word_freq_cs', 'word_freq_meeting',
       'word_freq_original', 'word_freq_project', 'word_freq_re',
       'word_freq_edu', 'word_freq_table', 'word_freq_conference',
[...]<jupyter_text>Distribution of the numerical columns<jupyter_code>def plotHist():
    fig, ax =plt.subplots(5,3, figsize=(15,15))
    i=0;j=0;k=0
    while i<=4:
        while j<=2:
            sns.distplot(data[numerical[k]], ax=ax[i, j])
            j+=1;k+=1
        j=0;i+=1;
    plt.savefig('distribution_after_removing_outliers_from_numerical_columns.png')
    plt.show()
plotHist()
#Pie chart on spam rates
#

data['Spam'].value_counts().head(5).plot.pie(figsize=(8,6),autopct='%1.1f%%')

plt.title('Spam',size=18)
plt.show()
# Heat map showing correlation

plt.figure(figsize=(120, 80))

dfcorr = data.corr()
sns.heatmap(dfcorr, 
            xticklabels = dfcorr.columns.values,
            yticklabels = dfcorr.columns.values,
            annot = True);<jupyter_output><empty_output><jupyter_text>MODELLING Naive Bayes Classifier<jupyter_code>X=data[['word_freq_make', 'word_freq_address', 'word_freq_all', 'word_freq_3d',
       'word_freq_our', 'word_freq_over', 'word_freq_remove',
       'word_freq_internet', 'word_freq_order', 'word_freq_mail',
       'word_freq_receive', 'word_freq_will', 'word_freq_people',
       'word_freq_report', 'word_freq_addresses', 'word_freq_free',
       'word_freq_business', 'word_freq_email', 'word_freq_you',
       'word_freq_credit', 'word_freq_your', 'word_freq_font', 'word_freq_000',
       'word_freq_money', 'word_freq_hp', 'word_freq_hpl', 'word_freq_george',
       'word_freq_650', 'word_freq_lab', 'word_freq_labs', 'word_freq_telnet',
       'word_freq_857', 'word_freq_data', 'word_freq_415', 'word_freq_85',
       'word_freq_technology', 'word_freq_1999', 'word_freq_parts',
       'word_freq_pm', 'word_freq_direct', 'word_freq_cs', 'word_freq_meeting',
       'word_freq_original', 'word_freq_project', 'word_freq_re',
       'word_freq_edu', 'word_freq_table', 'word_freq_conference',
       'char_freq_;', 'char_freq_(:', 'char_freq_[', 'char_freq_!: ',
       'char_freq_$:', 'char_freq_#', 'capital_run_length_average',
       'capital_run_length_longest', 'capital_run_length_total']].values
y=data['Spam'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=6) 
from sklearn.naive_bayes import BernoulliNB
classifier=BernoulliNB()
model=classifier.fit(X_train,y_train)
predicted = model.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix,accuracy_score
print(confusion_matrix(y_test, predicted))
print(classification_report(y_test, predicted))
print(np.mean(predicted == y_test))<jupyter_output>[[665  81]
 [ 91 372]]
              precision    recall  f1-score   support

         0.0       0.88      0.89      0.89       746
         1.0       0.82      0.80      0.81       463

    accuracy                           0.86      1209
   macro avg       0.85      0.85      0.85      1209
weighted avg       0.86      0.86      0.86      1209

0.8577336641852771<jupyter_text>Our model did very well as it had an accuracy of 85.77 using the BernoulliNB method.  GaussianNB Method<jupyter_code>import numpy as np
from sklearn.naive_bayes import GaussianNB
classifier=GaussianNB()
mode=classifier.fit(X_train,y_train)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=6)
predict = mode.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix,accuracy_score
print(confusion_matrix(y_test, predict))
print(classification_report(y_test, predict))
print(np.mean(predict == y_test))<jupyter_output>[[673  73]
 [ 53 410]]
              precision    recall  f1-score   support

         0.0       0.93      0.90      0.91       746
         1.0       0.85      0.89      0.87       463

    accuracy                           0.90      1209
   macro avg       0.89      0.89      0.89      1209
weighted avg       0.90      0.90      0.90      1209

0.8957816377171216
{'fraction_non_alphanumeric': 0.11858131150735653, 'fraction_numerical': 0.03351846438663159, 'mean_word_length': 4.512352309344791, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '153592', 'license': 'MIT', 'n_tokens_mistral': 3840, 'n_tokens_neox': 3584, 'n_words': 806, 'path': 'Week_9_Ip_on_Naive_Bayes_Classifier.ipynb', 'repo_name': 'Jobby-John/KNN-AND-NAIVE-BAYES', 'size': '10263'}