<jupyter_start><jupyter_text>1.0 Defining the question  a.Specific Question Implementing a K-nearest neighbor (kNN) classifier on the titanic train set data using the 80:20,70:30 and 60:40 partitioning to predict passenger survival.   b.Defining Metrics for success Create an accurate model that best predicts passenger survival.  c.Understanding the context Doing a background search to understand the titanic disaster and review the dataset description in order to be able to draw a concrete data analysis plan.  d.Recording the experimental design Data CleaningExploratory Data AnalysiskNN model implementationImproving Model Performance  e.Data Relevance Cross checking the data to ensure it is relevant to the topic of study and adequate enough to answer the research question.  Importing the Libraries<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats<jupyter_output><empty_output><jupyter_text>Reading and checking the data<jupyter_code># train dataset url = /content/train (5).csv
# test dataset url = /content/test (1).csv
df = pd.read_csv('/content/train (5).csv')
df.head()
# Checking the shape of the dataset
df.shape
# Checking the datatypes
df.dtypes
# Checking the information on the data
df.info()
# Getting statistical summary of the data
df.describe()<jupyter_output><empty_output><jupyter_text>Data Cleaning<jupyter_code># checking for duplicates
df.duplicated().sum()
# checking for unique values in each column using a for loop

for column in df.columns:           
  print(column)                                
  print(df[column].unique()) 
  print('\n')         
  print(df[column].nunique())
          
df.head(0)
# There are null values in embarked, age and cabin columns.
# We replace the null values.
# Replacing the null value in age with the mean
df['Age'].fillna(value=df['Age'].mean(), inplace=True)
# Dropping null values in the embarked column and cabin columns
df.dropna(inplace=True)
df.isnull().sum()
# Dropping irrelevant columns
# Some columns are irrelevant to our prediction so we drop them
df.drop(columns=['PassengerId', 'Cabin','Name','Fare'], axis=1, inplace=True)

df.drop(['Ticket'], axis=1)
# Checking for outliers for numeric variables
cols =cols = ['Pclass', 'Age', 'SibSp', 'Parch']

fig, axs = plt.subplots(len(cols), figsize=(6,44))

for i, col_val in enumerate(cols):
        
    sns.boxplot(y=df[col_val], ax=axs[i])
    axs[i].set_title('Boxplot for '+col_val, fontsize=10)
    axs[i].set_xlabel(col_val, fontsize=10)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code># Histogram showing age distribution of the passengers
sns.displot(df.Age, kde=False);<jupyter_output><empty_output><jupyter_text>Most of the passengers on board were aged between 20 to 50 years.<jupyter_code># count plot showing how many pasengers survived
sns.countplot(x ='Survived', data = df)
 
# Show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>Survival (0 = No; 1 = Yes)The survival rate was much higher than the death rate.<jupyter_code># count plot on sex of the passengers
sns.countplot(x ='Sex', data = df)
 
# Show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>Most of the passengers on board were male.<jupyter_code># count plot on survival rate by sex
sns.countplot(x ='Sex', hue = "Survived", data = df)
 
# Show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>In comparison, there was a higher survival rate among the female passengers as opposed to male passengers where the death rate was higher.<jupyter_code># count plot showing the number of passengers that survived per class
sns.countplot(x ='Pclass', hue = "Survived", data = df)
 
# Show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>More people survived in the first class.There was a higher rate of survival than death in second classThe survival and death rate was equal in third class.<jupyter_code># Heatmap showing correlation of variables
plt.figure(figsize=(12, 8))

corr = df.corr()
sns.heatmap(corr, 
            xticklabels = corr.columns.values,
            yticklabels = corr.columns.values,
            annot = True);<jupyter_output><empty_output><jupyter_text>There's a weak correlation between the variables.  Implementing the solution.  a. Using 80:20 partitioning ratio<jupyter_code>df.head(2)
# Dropping more irrelevant columns
df.drop(columns=['Ticket', 'SibSp','Parch'], axis=1, inplace=True)
df.head(2)
# Using the 80-20 partitioning
# Import the relevant libraries
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
# Defining the target and the predictor variables
X = df.drop('Survived', axis = 1)
y = df.Survived.values
# Encoding the categorical variables
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()

df[['Sex']] = df[['Sex']].apply(LabelEncoder().fit_transform)
df[['Embarked']] = df[['Embarked']].apply(LabelEncoder().fit_transform)
df.head()
df['Sex'].unique()
 X = pd.get_dummies(X)
df.head(2)
# Splitting the dataset into training and testing (ratio of 80:20)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=101)
# Scaling the data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
sc.fit(X_train)

X_train = sc.transform(X_train)
X_test = sc.transform(X_test)
# Train the model with the help of KNeighbours classifier
classifier = KNeighborsClassifier(n_neighbors=3,metric='euclidean')
classifier.fit(X_train,y_train)
# make predictions
y_pred = classifier.predict(X_test)
y_pred
# print the results
from sklearn.metrics import classification_report,confusion_matrix,accuracy_score

result =confusion_matrix(y_test,y_pred)
print("Confusion Matrix")
print(result)
print('\n')
result1 = classification_report(y_test,y_pred)
print("Classification Report")
print(result1)
print('\n')
result2 = accuracy_score(y_test,y_pred)
print("Accuracy:",result2)<jupyter_output>Confusion Matrix
[[ 7  7]
 [ 4 23]]


Classification Report
              precision    recall  f1-score   support

           0       0.64      0.50      0.56        14
           1       0.77      0.85      0.81        27

    accuracy                           0.73        41
   macro avg       0.70      0.68      0.68        41
weighted avg       0.72      0.73      0.72        41



Accuracy: 0.7317073170731707<jupyter_text>Using the 80:20 ratio, the model achieved an accuracy of 73%.  b. Using 70-30 partitioning ratio<jupyter_code># Defining the target and the predictor variables
X = df.drop('Survived', axis = 1)
y = df.Survived.values
# Splitting the dataset into training and testing (ratio of 70:30)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)
 X = pd.get_dummies(X)
# Scaling the data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
sc.fit(X_train)

X_train = sc.transform(X_train)
X_test = sc.transform(X_test)
# Train the model with the help of KNeighbours classifier
classifier = KNeighborsClassifier(n_neighbors=3,metric='euclidean')
classifier.fit(X_train,y_train)
# make predictions
y_pred = classifier.predict(X_test)
y_pred
# print the results
from sklearn.metrics import classification_report,confusion_matrix,accuracy_score

result =confusion_matrix(y_test,y_pred)
print("Confusion Matrix")
print(result)
print('\n')
result1 = classification_report(y_test,y_pred)
print("Classification Report")
print(result1)
print('\n')
result2 = accuracy_score(y_test,y_pred)
print("Accuracy:",result2)<jupyter_output>Confusion Matrix
[[11  9]
 [ 7 34]]


Classification Report
              precision    recall  f1-score   support

           0       0.61      0.55      0.58        20
           1       0.79      0.83      0.81        41

    accuracy                           0.74        61
   macro avg       0.70      0.69      0.69        61
weighted avg       0.73      0.74      0.73        61



Accuracy: 0.7377049180327869<jupyter_text>Using 70-30 partitioning ratio, the model achieved an accuracy of 73.7%.  c. Using 60-40 partitioning ratio<jupyter_code># Defining the target and the predictor variables
X = df.drop('Survived', axis = 1)
y = df.Survived.values
# Splitting the dataset into training and testing (ratio of 60:40)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=101)
# Scaling the data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
sc.fit(X_train)

X_train = sc.transform(X_train)
X_test = sc.transform(X_test)
# Train the model with the help of KNeighbours classifier
classifier = KNeighborsClassifier(n_neighbors=3,metric='euclidean')
classifier.fit(X_train,y_train)
# make predictions
y_pred = classifier.predict(X_test)
y_pred
# print the results
from sklearn.metrics import classification_report,confusion_matrix,accuracy_score

result =confusion_matrix(y_test,y_pred)
print("Confusion Matrix")
print(result)
print('\n')
result1 = classification_report(y_test,y_pred)
print("Classification Report")
print(result1)
print('\n')
result2 = accuracy_score(y_test,y_pred)
print("Accuracy:",result2)<jupyter_output>Confusion Matrix
[[15 12]
 [ 8 46]]


Classification Report
              precision    recall  f1-score   support

           0       0.65      0.56      0.60        27
           1       0.79      0.85      0.82        54

    accuracy                           0.75        81
   macro avg       0.72      0.70      0.71        81
weighted avg       0.75      0.75      0.75        81



Accuracy: 0.7530864197530864<jupyter_text>Using 60-40 partitioning ratio, we achieved an accuracy of 75%.  d. Hyperparameter Tuning to Improve Model Performance.<jupyter_code># Import relevant libraries
from sklearn.model_selection import GridSearchCV
# List the hyperparameters to tune
grid_param = {
    'n_neighbors': [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31],
    'leaf_size': [10,20,30,40,50],
    'metric':['minkowski','euclidean'],
    'p': [1,2]
}
# Instantiating GridSearchCV object
gd_sr= GridSearchCV(KNeighborsClassifier(), grid_param, refit = True, cv = 10 ,n_jobs= 1, verbose  = 0)
# Calling the fit method
gd_sr.fit(X_train, y_train)
#  Checking the parameters that return the highest accuracy
best_parameters = gd_sr.best_params_
print(best_parameters)<jupyter_output>{'leaf_size': 10, 'metric': 'minkowski', 'n_neighbors': 5, 'p': 2}<jupyter_text>From GridSearch, the best number of leaf_size is 10, while the best metric method is minkowski and the optimal number of K is 5.<jupyter_code># Finding the obtained accuracy
best_result = gd_sr.best_score_
print(best_result)<jupyter_output>0.7365384615384616<jupyter_text>Using Hyperparameter Tuning has not improved the model performance. We achieved an accuracy of 73.6% which is almost similar to that achieved using the 80:20 and 70:30 partitioning ratios, and n_neighbours:5, with metric : euclidean, leaf size=30 and p=2.The 60:40 partitioning had performed slightly better with an accuracy of 75%.  Challenging the solution We compare the kNN model performance with Random Forest to see which one performs better.<jupyter_code>#Import Random Forest Model
from sklearn.ensemble import RandomForestClassifier
# Defining the target and the predictor variables
X = df.drop('Survived', axis = 1)
y = df.Survived.values
# Splitting into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.4, random_state = 101)
#Create a Gaussian Classifier
clf=RandomForestClassifier(n_estimators=100,random_state=20, min_samples_split = 20, max_depth=5)
#Train the model using the training sets y_pred=clf.predict(X_test)
clf.fit(X_train,y_train)
# making predictions
y_pred=clf.predict(X_test)
y_pred
# Evaluating the accuracy of the model
result =confusion_matrix(y_test,y_pred)
print("Confusion Matrix")
print(result)
print('\n')
result1 = classification_report(y_test,y_pred)
print("Classification Report")
print(result1)
print('\n')
result2 = accuracy_score(y_test,y_pred)
print("Accuracy:",result2)<jupyter_output>Confusion Matrix
[[15 12]
 [10 44]]


Classification Report
              precision    recall  f1-score   support

           0       0.60      0.56      0.58        27
           1       0.79      0.81      0.80        54

    accuracy                           0.73        81
   macro avg       0.69      0.69      0.69        81
weighted avg       0.72      0.73      0.73        81



Accuracy: 0.7283950617283951
{'fraction_non_alphanumeric': 0.09372003835091083, 'fraction_numerical': 0.0421061041866411, 'mean_word_length': 4.098574338085539, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '655767', 'license': 'MIT', 'n_tokens_mistral': 4391, 'n_tokens_neox': 3997, 'n_words': 1199, 'path': 'DSC9_Independent_Project_kNN_Week_9_Njoki_Mbugua.ipynb', 'repo_name': 'NjokiMbugua/spamdata-predictions-kNN-naive-bayes', 'size': '12516'}