<jupyter_start><jupyter_text>*To start working on this notebook, or any other notebook that we will use in the Moringa Data Science Course, we will need to save our own copy of it. We can do this by clicking File > Save a Copy in Drive. We will then be able to make edits to our own copy of this notebook.*  K-Nearest Neighbours (KNN): Titanic Dataset Analysis **1**. Defining the Question  a) Specifying the Data Analytic Question Determine whether a passenger survived the accident or not.  b) Defining the Metric for SuccessThe project will be a success when we are able to create a model to accurately predicts whether na passenger survived the titanic accident or not.  c) Understanding the contextTitanic was a British passenger liner operated by the White Star Line that sank in the North Atlantic Ocean in the early morning hours of 15 April 1912, after striking an iceberg during her maiden voyage from Southampton to New York City. Of the estimated 2,224 passengers and crew aboard, more than 1,500 died, making the sinking one of modern history's deadliest unforgatable commercial marine disasters.• Survived: This feature has value 0 and 1. 0 for not survived and 1 for survived.• Pclass: There are 3 classes: Class 1, Class 2 and Class 3.• Name: Name of passenger.• Sex: Gender of passenger.• Age: Age of passenger.• SibSp: Indication that passengers have siblings and spouses.• Parch: Whether a passenger is alone or has a family.• Ticket: Ticket number of passenger.• Fare: Indicating the fare.• Cabin: The cabin of passenger.• Embarked: The port embarked category; C = Cherbourg, Q = Queenstown, S = Southampton  d) Recording the Experimental Design The process will entail: * Define the question, the metric for success, the context, experimental design taken.* Read and explore the given dataset.* Define the appropriateness of the available data to answer the given question.* Find and deal with outliers, anomalies, and missing data within the dataset.* Perform univariate and bivariate analysis recording my observations.* Build our classification models using KNN.* Do different dataset splits(80-20, 70-30, 60-40) for the train and test set and make observations of the model performance.* Optimize model performance and challenge the solution* Give a conclusion and recommendations from the analysis.  e) Data RelevanceThe data provided is very relevant and has a wide range of variables to consider in achieving our main objective of this project. [Train dataset](https://archive.org/download/train5_202002/train%20%285%29.csv)[Test dataset](https://archive.org/download/test1_202002/test%20%281%29.csv) **2**. Reading the Data<jupyter_code>#importing our necessary libraries
# 
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import  accuracy_score
from sklearn.metrics import classification_report, confusion_matrix
%matplotlib inline

# Loading the Data from the source i.e. csv
# ---
#
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')<jupyter_output><empty_output><jupyter_text>**3**. Checking the Data<jupyter_code># Determining the no. of records in our dataset
#
print(train.shape)
print(test.shape)

# Previewing the top of our dataset
#
train.head()

test.head()
# Previewing the bottom of our dataset
# 

train.tail()

test.tail()

# Checking whether each column has an appropriate datatype
#
train.dtypes

train.info()

# get a statistical summary of the continuous variables
# 
train.describe()<jupyter_output><empty_output><jupyter_text>**4**. External Data Source Validation  a.Validation **5**. Tidying the Dataset<jupyter_code># converting the column names to lowercase
# 
train.columns = train.columns.str.lower()
train.columns

# Checking for Anomalies
# checking for duplicates
# 
train.duplicated().sum()

# Identifying the Missing Data
#
train.isnull().sum()

# Dealing with the Missing Data
# imputing the missing age values with the mean

train.groupby(['pclass', 'sex'])['age'].mean()

[train['age'].fillna(train.groupby(['pclass','sex'])['age'].transform('mean'), inplace=True)]

# to confirm that there are no missing values in the age column
# 
train.isnull().sum()

# drop the cabin column since it has about 77% missing values
# 
train.drop(['cabin'], axis=1, inplace=True)
train.head()
# dropping the missing values in the embarked column
# 
train.dropna(inplace=True)

# confirming we dont have any missing values
# 
train.isnull().sum()

# determing the entries present after dealing with missing values
# 
train.shape

# dropping the passengerid, name and ticket columns as they are not pertinent in predicting the survival of passengers
# 
train.drop(['passengerid', 'name', 'ticket'], axis=1, inplace=True)
train.head()

# getting our dataframes' datatypes
# 
train.dtypes

# checking for outliers
# 

import matplotlib.pyplot as plt
import seaborn as sns

num_col = ['age','survived', 'pclass', 'sibsp', 'parch', 'fare']

fig, ax = plt.subplots(len(num_col), figsize=(8,40))

for i, col_val in enumerate(num_col):

    sns.boxplot(y=train[col_val], ax=ax[i])
    ax[i].set_title('Box plot - {}'.format(col_val), fontsize=10)
    ax[i].set_xlabel(col_val, fontsize=8)

plt.show()<jupyter_output><empty_output><jupyter_text>**6**. Exploratory Analysis  6.1 Univariate analysis Categorical variables<jupyter_code>train.dtypes
# plotting a bar chart to show the frequency distribution of sex
# 
train['sex'].value_counts().plot.bar(title='Frequency distribution of sex')

# plotting a barchart to show frequency distribution of embarked category
# 
train['embarked'].value_counts().plot.bar(title='Frequency distribution of embarked categories')<jupyter_output><empty_output><jupyter_text>Continuous variables<jupyter_code># survived variable piechart
#
plt.figure(figsize = (6, 6))
train.survived.value_counts().plot(kind = 'pie', autopct = '%0.1f%%')
plt.show()

# plotting passenger class bar graph
#
train.pclass.value_counts().plot(kind = 'bar', rot = 0)
plt.show()

# plotting a histogram for age
# 

train['age'].hist()

# Ploting a summary of the distribution of numerical columns which contains histograms and estimated PDF
#
num_col = ['age','survived', 'pclass', 'sibsp', 'parch', 'fare']

fig, ax = plt.subplots(len(num_col), figsize=(16,40))

for i, col_val in enumerate(num_col):

    sns.distplot(train[col_val], hist=True, ax=ax[i])
    ax[i].set_title('Freq dist '+col_val, fontsize=10)
    ax[i].set_xlabel(col_val, fontsize=8)
    ax[i].set_ylabel('Count', fontsize=8)

plt.show()
# statistical summary of the continuous variables
# 
train.describe()
# getting the median of the continuous variables

num_col = ['age','survived', 'pclass', 'sibsp', 'parch', 'fare']

median1 = train[num_col].median()

median1.sort_values(ascending=False)
# getting the skewness of the continuous variables
# 

num_col = ['age','survived', 'pclass', 'sibsp', 'parch', 'fare']

skewed = train[num_col].skew()

skewed.sort_values(ascending=False)
# getting the kurtosis of the continuous variables
# 
num_col = ['age','survived', 'pclass', 'sibsp', 'parch', 'fare']

kurt1 = train[num_col].kurt()

kurt1.sort_values(ascending=False)
# getting the variance of the continuous variables
# 
num_col = ['age','survived', 'pclass', 'sibsp', 'parch', 'fare']

var1 = train[num_col].var()

var1.sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Observations:*   There were more males1 than females in the titanic.*   Most passengers were in the third class.*   Southampton(S) was the port where most people embarked the ship.*   Those who perished were more than those who survived.*   The mean age is 29years.  6.2 Bivariate **Analysis**<jupyter_code># Ploting the bivariate summaries and recording our observations
#
sns.pairplot(train)

# creating a correlation matrix
# 
correlations = train.corr()
correlations

# creating a heatmap from our correlations
# 
sns.heatmap(correlations)<jupyter_output><empty_output><jupyter_text>The variables are not correlated. **7**. Implementing the Solution  Modelling: K-Nearest Neighbors Checking for Multicollinearity<jupyter_code># compute the VIF
# 
pd.DataFrame(np.linalg.inv(correlations.values), index = correlations.index, columns=correlations.columns)<jupyter_output><empty_output><jupyter_text>The VIF score for each variable is found alongside the downwards sloping diagonal. In this case they are fairly okay since they are below 4 a) Splitting data 80-20<jupyter_code>train.head()
# Implementing the Solution
# define our x and y
# 
test_features = train.columns.to_list()
test_features.remove('survived')

X = train[test_features]
X = pd.get_dummies(X)
y = train['survived'].values

# splitting data into train and test sets
# 
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2, random_state=0)

# scaling our features
# 

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# training our model KNN

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier(n_neighbors=11, metric='euclidean')
classifier.fit(X_train, y_train)

# make predictions with our trained model
# 
y_pred = classifier.predict(X_test)

comparison_frame = pd.DataFrame({'actual':y_test.flatten(), 'predicted':y_pred.flatten()})
comparison_frame

# evaluate performance of our model
# 
from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
print('model accuracy', accuracy_score(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.76      0.83      0.79       105
           1       0.71      0.62      0.66        73

    accuracy                           0.74       178
   macro avg       0.74      0.72      0.73       178
weighted avg       0.74      0.74      0.74       178

[[87 18]
 [28 45]]
model accuracy 0.7415730337078652<jupyter_text>b) Splitting the data 70-30<jupyter_code># Implementing the Solution
# define our x and y
# 
test_features = train.columns.to_list()
test_features.remove('survived')

X = train[test_features]
X = pd.get_dummies(X)
y = train['survived'].values
# splitting data into train and test sets
# 

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3, random_state=0)

# scaling our features
# 

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
# train our model KNN

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier(n_neighbors=11, metric='euclidean')
classifier.fit(X_train, y_train)

# making predictions with our trained model
# 
y_pred = classifier.predict(X_test)

comparison_frame = pd.DataFrame({'actual':y_test.flatten(), 'predicted':y_pred.flatten()})
comparison_frame
# evaluate performance of our model
# 
from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
print('model accuracy', accuracy_score(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.78      0.89      0.83       157
           1       0.81      0.65      0.72       110

    accuracy                           0.79       267
   macro avg       0.79      0.77      0.78       267
weighted avg       0.79      0.79      0.79       267

[[140  17]
 [ 39  71]]
model accuracy 0.7902621722846442<jupyter_text>There is a great improvement in the accuracy score, from 74%(80-20 split) to 79% (70-30 split). c) Splitting our data 60-40<jupyter_code># defining our x and y
# 
test_features = train.columns.to_list()
test_features.remove('survived')

X = train[test_features]
X = pd.get_dummies(X)
y = train['survived'].values

# splitting data into train and test sets
# 

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.4, random_state=0)

# scaling our features
# 

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# train our model KNN

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier(n_neighbors=11, metric='euclidean')
classifier.fit(X_train, y_train)

# making predictions with our trained model
# 
y_pred = classifier.predict(X_test)

comparison_frame = pd.DataFrame({'actual':y_test.flatten(), 'predicted':y_pred.flatten()})
comparison_frame

# evaluate performance of our model
# 
from sklearn.metrics import classification_report, confusion_matrix

print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))
print('model accuracy', accuracy_score(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.79      0.89      0.84       216
           1       0.80      0.64      0.71       140

    accuracy                           0.79       356
   macro avg       0.80      0.77      0.78       356
weighted avg       0.80      0.79      0.79       356

[[193  23]
 [ 50  90]]
model accuracy 0.7949438202247191<jupyter_text>There is only a slight improvement in the accuracy score, compared to the 70-30 split. This would mean a bigger sample doesn't really improve the model's performance as much. **8**. Challenging the solution Improving model performance using LDA<jupyter_code># To challenge the solution, we'll perform LDA and see if our model works better
#
# define our x and y
# 
test_features = train.columns.to_list()
test_features.remove('survived')

X = train[test_features]
X = pd.get_dummies(X)
y = train['survived'].values

# splitting data into train and test sets
# 

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3, random_state=0)

# feature scaling
# 
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# perform LDA
# 
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA(n_components=3)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

# train our model KNN

from sklearn.neighbors import KNeighborsClassifier

classifier = KNeighborsClassifier(n_neighbors=11, metric='euclidean')
classifier.fit(X_train, y_train)

# Make predictions using our trained model
# 
y_pred_lda = classifier.predict(X_test)

comparison_frame = pd.DataFrame({'actual':y_test.flatten(), 'predicted':y_pred_lda.flatten()})
comparison_frame
# evaluating performance of our model
# 
print(classification_report(y_test, y_pred_lda))
print(confusion_matrix(y_test, y_pred_lda))
print('accuracy score', accuracy_score(y_test, y_pred_lda))<jupyter_output>precision    recall  f1-score   support

           0       0.77      0.85      0.81       157
           1       0.75      0.64      0.69       110

    accuracy                           0.76       267
   macro avg       0.76      0.74      0.75       267
weighted avg       0.76      0.76      0.76       267

[[134  23]
 [ 40  70]]
accuracy score 0.7640449438202247
{'fraction_non_alphanumeric': 0.09491525423728814, 'fraction_numerical': 0.03108410617204989, 'mean_word_length': 4.4329395413481585, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '609528', 'license': 'MIT', 'n_tokens_mistral': 5328, 'n_tokens_neox': 4916, 'n_words': 1547, 'path': 'WEEK_9_DS_CORE_TITANIC_DATASET_ANALYSIS_K_Nearest_Neighbors_(KNN).ipynb', 'repo_name': 'joykiriba/MORINGA-DS-CORE-WEEK9-IP--KNN-NAIVE-BAYES', 'size': '15635'}