<jupyter_start><jupyter_text>**Research Objective** To predict whether a passenger survived or not.to predict whether an email is a spam or not  **Metric of Success** create a model of above 80%ensure data is clean having passed the validity,complete,accurate and uniformity tests  **Context** titanic and spam base datasets  **Experimental design**<jupyter_code>load the data

clean the data

perform EDA

create model and evaluate

tune the model

get the insights<jupyter_output><empty_output><jupyter_text>**Appropriateness of Data** All the datasets were relevant for the research objectives  **Import important Libraries**<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import missingno as msno
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn import metrics
from fancyimpute import KNN
from sklearn.preprocessing import Imputer 
from sklearn.metrics import precision_recall_fscore_support
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import confusion_matrix
from sklearn.naive_bayes import GaussianNB
from sklearn.naive_bayes import BernoulliNB
from sklearn.naive_bayes import MultinomialNB<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/externals/six.py:31: DeprecationWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).
  "(https://pypi.org/project/six/).", DeprecationWarning)
Using TensorFlow backend.<jupyter_text>**Loading the data**<jupyter_code>df = pd.read_csv('train_titanic.csv')
#we check our data
#lets check the size
df.shape
#we check the top rows
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>From the descriptive statistics we can see that a majority of the passengers were between the ages 14 and 43.In a number of the features there was a presence of variances  **Exploratory Data Analysis**<jupyter_code>#lets see the distribution of survivors
sns.countplot(x='Survived', data = df)
plt.title('distribution of survivors')
plt.ylabel('count')
plt.xlabel('survived')<jupyter_output><empty_output><jupyter_text>From the above plot we can see that the dataset is not balanced and indicates more deaths than survivors<jupyter_code>#lets look at the distribution of the sex
sns.countplot(x='Sex', data = df)
plt.title('distribution of Sex')
plt.ylabel('count')
plt.xlabel('Sex')<jupyter_output><empty_output><jupyter_text>We can see that from the datasets there were more males than females. over 40% of men more than the females<jupyter_code>#lets view the distribution
numerical = ['PassengerId','Survived','Pclass','Age','SibSp','Parch','Fare']
df[numerical].hist(bins=15, figsize=(15, 6), layout=(2, 4));<jupyter_output><empty_output><jupyter_text>From the above plots we see a high variance in the economic statuses of the passengers represented by the pclass.we also see a high variance in the fares paid by the different passengers. A majority of the passengers preffered cheaper fares of not more than 100.most of the passengers were aged between 14 and 43there were more deaths than survivors represented by survived 1 representing those who survived and 0 representing non survivors. most of those who survived were of upper class with regard to socio economic status<jupyter_code>#we want to see how the ages related to the survivor so we put the ages into categories
sns.stripplot(x='Survived', y='Age', data=df)
plt.title('age vs survived')
plt.ylabel('Age')
plt.xlabel('survived')<jupyter_output><empty_output><jupyter_text>From the plot above more children below age ten survived compared to those who did not survive. as for those aged between 14 to 43 more died compsred to those who survived.<jupyter_code>sns.barplot(x='Pclass',y='Age',data = df)
plt.title('pclass vs Age')
plt.ylabel('Age')
plt.xlabel('Pclass')<jupyter_output><empty_output><jupyter_text>most of those in upper class are also elderly compared to those in lower class.<jupyter_code>sns.barplot(x='Pclass',y='Fare',data = df)
plt.title('pclass vs fare')
plt.ylabel('fare')
plt.xlabel('Pclass')<jupyter_output><empty_output><jupyter_text>Those in upper class also paid more fare.<jupyter_code>colormap = plt.cm.RdBu
plt.figure(figsize=(32,10))
plt.title('Pearson Correlation of Features', y=1.05, size=15)
sns.heatmap(df.corr(),linewidths=0.1,vmax=1.0, 
            square=True, cmap=colormap, linecolor='white', annot=True)<jupyter_output><empty_output><jupyter_text>There exists some correlation between the independent variables though not very strong ones.we can see a strong correlation between fare and pclassthere is also a moderate correlation between pclass and survived, parch and sibsp  **DATA CLEANING** We now want to clean our data and ensure its valid,complete,uniform, consistent and accurate. **Valid**<jupyter_code>#here we remove outliers,drop irrelevant attributes and syntax errors if any
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3-Q1
df2 = df[~((df < (Q1-1.5*IQR))|(df > (Q3+1.5*IQR))).any(axis=1)]
df2.shape
#there are columns that we dont need so we shall drop them. they include name,passengerid,ticket, Fare 
df2 = df2.drop(['PassengerId', 'Name', 'Ticket', 'Fare','Cabin'],axis=1)
df2.head()<jupyter_output><empty_output><jupyter_text>**Complete**<jupyter_code>#check for completeness of the data by checking the missing values
df2.isnull().sum()
#we fill in the missing values in age
df2["Age"] = df2["Age"].fillna(df2["Age"].median())
df2.head()<jupyter_output><empty_output><jupyter_text>our dataset is uniform as all the columns and their values have the same names that refer to the same columns and values.<jupyter_code>#we now want to convert sex and embarked columns to dummy variables to enable modelling
categorical = ['Sex','Embarked']
dummy = pd.get_dummies(df2[categorical])
dummy.head()
#we combine the dummy values with the df2 and drop the original sex and embarked columns
df2 = pd.concat([df2,dummy], axis=1)
df2.head(3)
#we drop the sex and embarked columns
df2.drop(['Sex','Embarked'], axis=1,inplace=True)
df2.head()
#looking at the above dataframe age has large values we can categorize people in these ages as either adults and children by 
#creating another column to represent the same information.we can call it adults
#adults =1
#children=0
df2['adult'] = 0
df2['adult'][df2["Age"] >= 18] = 1
#we check if this is done successfully
df2.head(2)
#lets now drop the age column as we have the same information in the column adults
df2.drop(['Age'], axis=1, inplace=True)
df2.tail(3)
#we can now model but first we want to separete our target and predictor variables
y = df2['Survived']
X = df2.drop(['Survived'], axis=1)
#we then split the dataset into train test sets first 80-20,then 70-30 and finally 60-40 and observe our moel performances
X_train,X_test,y_train,y_test = train_test_split(X,y, test_size=0.2, random_state=100 )
#implementation
for K in range(25):
 K_value = K+1
 neigh = KNeighborsClassifier(n_neighbors = K_value, weights='uniform', algorithm='auto')
 neigh.fit(X_train, y_train) 
 y_pred = neigh.predict(X_test)
 print("Accuracy is:" , accuracy_score(y_test,y_pred)*100,"% for K-Value:",K_value)


confusion_matrix(y_test, y_pred)

#lets split our data in a ratio of 70-30
X1_train,X1_test,y1_train,y1_test = train_test_split(X,y, test_size=0.3, random_state=100 )
for K in range(25):
 K_value = K+1
 neigh1 = KNeighborsClassifier(n_neighbors = K_value, weights='uniform', algorithm='auto')
 neigh1.fit(X1_train, y1_train) 
 y1_pred = neigh.predict(X1_test)
 print("Accuracy is:" , accuracy_score(y1_test,y1_pred)*100,"% for K-Value:",K_value)


confusion_matrix(y1_test, y1_pred)

print(metrics.classification_report(y1_test, y1_pred))
#lastly lets split our data set into 60-40
X2_train,X2_test,y2_train,y2_test = train_test_split(X,y, test_size=0.4, random_state=100 )
for K in range(25):
 K_value = K+1
 neigh2 = KNeighborsClassifier(n_neighbors = K_value, weights='uniform', algorithm='auto')
 neigh2.fit(X2_train, y2_train) 
 y2_pred = neigh.predict(X2_test)
 print("Accuracy is:" , accuracy_score(y2_test,y2_pred)*100,"% for K-Value:",K_value)


confusion_matrix(y2_test, y2_pred)

print(metrics.classification_report(y2_test, y2_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.80      0.96      0.87       166
           1       0.83      0.46      0.59        74

    accuracy                           0.80       240
   macro avg       0.81      0.71      0.73       240
weighted avg       0.81      0.80      0.78       240<jupyter_text>from the above three models the accuracy increases with an increased testing _size or reducing training size.we also notice that for a smaller training size the value of k does not change the accuracy as the accuracy remains constant throught!  **Tuning our KNN best model**<jupyter_code>knn = KNeighborsClassifier( )
k_range = list(range(1,25))
weights_options = ['uniform','distance']
k_grid = dict(n_neighbors=k_range, weights = weights_options)
grid = GridSearchCV(knn, k_grid, cv=10, scoring = 'precision')
grid.fit(X, y)
print ("Best Score: ",str(grid.best_score_))


print ("Best Parameters: ",str(grid.best_params_))

print ("Best Estimators: ",str(grid.best_estimator_))

# predicting scores

y2_pred = grid.predict(X2_test)
# Calculating Accuracy

acc_clf = metrics.accuracy_score(y2_test,y2_pred)
print ("classifier's accuracy: ",str(acc_clf) )
confusion_matrix(y2_test, y2_pred)<jupyter_output><empty_output><jupyter_text>When we tune our best model it yields an accuracy of 82%The confusion matrix of the tuned model repors better classification as the number of true positives increase and true negatives reduce. **Predicting survival rate for test_csv values**<jupyter_code>#we load the dataset
df_test = pd.read_csv('test_titanic.csv')
df_test.columns
#I will need only the values i used in prediction so i drop the ones i dropped earlier
df_test = df_test.drop(['PassengerId', 'Name','Ticket', 'Cabin','Fare' ], axis=1)
df_test.shape
df_test.isnull().sum()
df_test["Age"] = df_test["Age"].fillna(df_test["Age"].median())

Q1 = df_test.quantile(0.25)
Q3 = df_test.quantile(0.75)
IQR = Q3-Q1
df_clean = df_test[~((df_test < (Q1-1.5*IQR))|(df > (Q3+1.5*IQR))).any(axis=1)]
df_clean.shape
#I created a new column so i create the same here
df_clean['adult']=0
df_clean['adult'][df_clean['Age'] >= 18] = 1

df_clean.head()
#we need to drop the  age column and convert the sex and Embarked coolumns to dummies
df_clean.drop(['Age'],axis=1, inplace=True)
cat = ['Sex','Embarked']
dummies = pd.get_dummies(df_clean[cat])
dummies.head(2)
#we then combine the dummies
df_clean = pd.concat([df_clean,dummies], axis=1)
df_clean.head()
df_clean.drop(['Sex','Embarked'], axis=1, inplace=True)
# predicting survivals on test set

ytest_pred = grid.predict(df_clean)
ytest_pred
df_clean['Survived'] = pd.Series(ytest_pred, index=df_clean.index)
df_clean.head()
# Exporting predictions as csv
df_clean.to_csv("titanic_pred.csv", encoding='utf-8')<jupyter_output><empty_output><jupyter_text>**Naive Bayes Classifier<jupyter_code>#we load the dataset
data = pd.read_csv('spambase.data')
#lets check the data
data.head()<jupyter_output><empty_output><jupyter_text>looking at the dataset above the target is binary in nature ie spam=1 and not spam=0.<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>From the above descriptive statistics we notice an existence of high variances betwen the variables. Because we shall be using naive bayes classifier methods this will not worry us because the naive bayes assumes that each variable is independent .<jupyter_code>#we check the size of the dataset
data.shape
#lets check for missing values
data.isnull().sum()<jupyter_output><empty_output><jupyter_text>there are no missing values  **EXPLORATORY DATA ANALYSIS**<jupyter_code>#We take a look at the distribution of the dataset per column
sns.countplot(x='1', data=data)
plt.title('distribution of spamemail')
plt.ylabel('count')
plt.xlabel('spam email')<jupyter_output><empty_output><jupyter_text>We see that more non spam emails were present compared to spam emails<jupyter_code>sns.scatterplot(x='61',y='1', data=data)
plt.title('61 vs 1 ')
plt.ylabel('1')
plt.xlabel('61 ')<jupyter_output><empty_output><jupyter_text>attribute 61 pointed more to an email being spam as shown in the above plot<jupyter_code>sns.scatterplot(x='278',y='1', data=data)
plt.title('278 vs 1 ')
plt.ylabel('1')
plt.xlabel('278 ')<jupyter_output><empty_output><jupyter_text>attribute 278  showed more of spam email.<jupyter_code>data.columns
predictors=['0', '0.64', '0.64.1', '0.1', '0.32', '0.2', '0.3', '0.4', '0.5', '0.6',
       '0.7', '0.64.2', '0.8', '0.9', '0.10', '0.32.1', '0.11', '1.29', '1.93',
       '0.12', '0.96', '0.13', '0.14', '0.15', '0.16', '0.17', '0.18', '0.19',
       '0.20', '0.21', '0.22', '0.23', '0.24', '0.25', '0.26', '0.27', '0.28',
       '0.29', '0.30', '0.31', '0.32.2', '0.33', '0.34', '0.35', '0.36',
       '0.37', '0.38', '0.39', '0.40', '0.41', '0.42', '0.778', '0.43', '0.44',
       '3.756', '61', '278']
#we separate the data into target and predictor variables
y = data['1']
X = data[predictors]
#we split the data into train test sets
X_train,X_test,y_train,y_test = train_test_split(X,y, test_size=0.30,random_state=100)<jupyter_output><empty_output><jupyter_text>**TRAINING THE THREE CLASSIFIERS USING A TRAIN SIZE OF 70**<jupyter_code>#we shall build all the naive_bayes classifier. lets begin with bernoullis
BernNB = BernoulliNB(binarize=True)
BernNB.fit(X_train,y_train)
print(BernNB)
y_expect = y_test
y_predict = BernNB.predict(X_test)
accuracy_score(y_expect,y_predict)
confusion_matrix(y_expect,y_predict)
print(metrics.classification_report(y_expect, y_predict))
#Lets build a multinomial classifier
MultiNB = MultinomialNB()
MultiNB.fit(X_train,y_train)
print(MultiNB)
y_expect = y_test
y_predict = MultiNB.predict(X_test)
accuracy_score(y_expect,y_predict)
confusion_matrix(y_expect,y_predict)
print(metrics.classification_report(y_expect, y_predict))
#lets now build a gaussian classifier
GaussNB = GaussianNB()
GaussNB.fit(X_train,y_train)
print(GaussNB)
y_expect = y_test
y_predict = GaussNB.predict(X_test)
accuracy_score(y_expect,y_predict)
confusion_matrix(y_expect,y_predict)
print(metrics.classification_report(y_expect, y_predict))<jupyter_output>precision    recall  f1-score   support

           0       0.95      0.72      0.82       808
           1       0.71      0.95      0.81       572

    accuracy                           0.82      1380
   macro avg       0.83      0.84      0.82      1380
weighted avg       0.85      0.82      0.82      1380<jupyter_text>From the above models bernoullis yields the best model because ontop of a high accuracy it also does a few misclassification compared to the other two models according to the confusion matrix<jupyter_code>#lets now change the train set and test set sizes
X3_train,X3_test, y3_train, y3_test = train_test_split(X,y, test_size=0.4, random_state=100)<jupyter_output><empty_output><jupyter_text>TRAINING THE THREE CLASSIFIERS USING A TRAIN SIZE OF 60<jupyter_code>#we shall build all the naive_bayes classifier. lets begin with bernoullis
BernNB = BernoulliNB(binarize=True)
BernNB.fit(X3_train,y3_train)
print(BernNB)
y3_expect = y3_test
y3_predict = BernNB.predict(X3_test)
accuracy_score(y3_expect,y3_predict)
confusion_matrix(y3_expect,y3_predict)
print(metrics.classification_report(y3_test, y3_predict))
#Lets build a multinomial classifier
MultiNB = MultinomialNB()
MultiNB.fit(X3_train,y3_train)
print(MultiNB)
y3_expect = y3_test
y3_predict = MultiNB.predict(X3_test)
accuracy_score(y3_expect,y3_predict)
confusion_matrix(y3_expect,y3_predict)
print(metrics.classification_report(y3_test, y3_predict))
#lets now build a gaussian classifier
GaussNB = GaussianNB()
GaussNB.fit(X3_train,y3_train)
print(GaussNB)
y3_expect = y3_test
y3_predict = GaussNB.predict(X3_test)
accuracy_score(y3_expect,y3_predict)
confusion_matrix(y3_expect,y3_predict)
print(metrics.classification_report(y3_test, y3_predict))<jupyter_output>precision    recall  f1-score   support

           0       0.96      0.72      0.82      1081
           1       0.71      0.96      0.81       759

    accuracy                           0.82      1840
   macro avg       0.83      0.84      0.82      1840
weighted avg       0.86      0.82      0.82      1840<jupyter_text>As the training size reduces bernoullis still out preforms the multinomial and gaussian. this is a typical case for bayes classification as it performs better with smaller training sets. the accuracy also goes up with a reduced training set and the classification according to the classification matrix bernoullis still yields the best results.from the classification report we also see bernoullis recording a higher precision, recall and f1_score   **Tuning the best model**<jupyter_code>#according to our models Bernoullis model is the best. lets try to improve its performance 
#we shall do this by setting the binarize to 0.25 and see the effect on our accuracy and results yielded in the confusion matrix
#we shall build all the naive_bayes classifier. lets begin with bernoullis
BernNB = BernoulliNB(binarize=0.25)
BernNB.fit(X3_train,y3_train)
print(BernNB)
y3_expect = y3_test
y3_predict = BernNB.predict(X3_test)
accuracy_score(y3_expect,y3_predict)
confusion_matrix(y3_expect,y3_predict)
print(metrics.classification_report(y3_test, y3_predict))<jupyter_output>precision    recall  f1-score   support

           0       0.91      0.93      0.92      1081
           1       0.90      0.86      0.88       759

    accuracy                           0.90      1840
   macro avg       0.90      0.90      0.90      1840
weighted avg       0.90      0.90      0.90      1840
{'fraction_non_alphanumeric': 0.10555954286976205, 'fraction_numerical': 0.0409098437586264, 'mean_word_length': 5.082605775688381, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 104, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '202844', 'license': 'MIT', 'n_tokens_mistral': 6622, 'n_tokens_neox': 6108, 'n_words': 1791, 'path': 'Moringa_Data_Science_Core_W9_Independent_Project_2019_07_Elizabeth_Okello_Python_Notebook.ipynb', 'repo_name': 'EOKELLO/IP-WEEK1-CORE', 'size': '18113'}