<jupyter_start><jupyter_text>**DEFINING THE QUESTION**  **a) Specifying the Question**Implementing a K-nearest neighbor (kNN) classifier on the provided dataset.from the dataset given, we can make predictions on the genders that were in the train whether it was male or female  **b) Defining the metrics for success**Implementing a K-nearest neighbor (kNN) classifier on the provided dataset.  **c) Understanding the context**Implementing a K-nearest neighbor (kNN) classifier on the provided dataset.  **d) Recording the Experimental Design**1.   Define the question, the metric for success, the context, experimental design taken.2. Read and explore the given dataset.3. Define the appropriateness of the available data to answer the given question.4. Find and deal with outliers, anomalies, and missing data within the dataset.5. Perform univariate, and bivariate analysis recording your observations.6. Randomly partition each dataset into two parts i.e 80 - 20  sets7. Performing  perform K-nearest neighbor classification. Compute the accuracy (percentage of correct classification).> Report the confusion matrix of each classifier.8. Repeat step 6 to step 7 twice, each time splitting the datasets differently i.e. 70-30, 60-40, then note the outcomes of your modeling.9. Suggest and apply at least one of the optimization techniques that you learned earlier this week.Provide further recommendations to improve both classifiers.10. Provide a recommendation based on your analysis.   **e) Relevance of the data**The data used for this project is necessary for building a model that implements the KNN classifier[https://archive.org/download/train5_202002/train%20%285%29.csv].  **DATA ANALYSIS**  **Data Checking**<jupyter_code># importing libraries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib
from matplotlib import pyplot as plt
%matplotlib inline
# reading datasets
df = pd.read_csv('https://archive.org/download/train5_202002/train%20%285%29.csv')
df
# previewing the dataset
df.shape
# previewing the top of the dataset
df.head()
# previewing the tail of the dataset
df.tail()
# printing the the number of unique values of the columns
print(df.nunique())
# previewing the columns of the datasets
df.columns
# previwin
print(df.dtypes) 
# previewing a sample of the dataset
df.sample(8)
# view summary of dataset
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB<jupyter_text>**Data Cleaning**<jupyter_code># dropping redundant columns from dataset
df = df.drop(['PassengerId'], axis=1)

# this is because it is not essential for the modelling process
# checking count of null values
df.isnull().sum()

# there are null values in the age, cabin and embarked column
# replacing the null values in the dataset by 0
df1 = df.fillna(0)
df1
# checking count of null values
df1.isnull().sum()
# checking for duplicates
df1.duplicated().any()

# there are no duplicates in the dataset
# checking count of duplicate entries
df1.duplicated().sum()
# previewing the dataset
df1.shape

# the dataset has 891 rows and 11 columns
# previewing the columns
df1.columns
# converting variables from float to int 
df1['Age'] = df1['Age'].astype(int)
df1['Fare'] = df1['Fare'].astype(int)
# describing the numerical features
df1.describe()
# previewing the datatypes
df1.dtypes

# five of the columns contain string datatypes while the rest have integers
# checking for anomalies

q11 = df1['Age'].quantile(.25)
q31 = df1['Age'].quantile(.75)

iqr11 = q31 - q11
iqr11
##
q11, q31 = np.percentile(df1['Age'], [25, 75]) 

iqr = q31 - q11

l_bound = q11 - (1.5*iqr)
u_bound = q31 + (1.5 * iqr)

print(iqr11, iqr)

#
# there are no anomalies in the data
# previewing the columns
df1.dtypes
# checking for outliers
# saving the column names onto a dictionary
columns_dict = {'Fare': 1, 'Survived': 2, 'Pclass': 3, 'Age': 4, 'SibSp': 5, 'Parch': 6}

plt.figure(figsize=(20,30))

# make a boxplot for each numerical column
for variable,i in columns_dict.items():
  plt.subplot(5,4,i)
  plt.boxplot(df1[variable])
  plt.title(variable)

plt.show()

# the dataset clearly has outliers as shown below
# view summary statistics in numerical variables
# Outliers in numerical variables
print(round(df.describe(),2))
# dealing with the outliers using quantiles
Q1 = df1.quantile(0.25)
Q3 = df1.quantile(0.75)
IQR = Q3 - Q1

# Removing outliers based on the IQR range 
df2 = df1[~((df1 < (Q1 - 1.5 * IQR)) | (df1 > (Q3 + 1.5 * IQR))).any(axis=1)]

# Printing the shape of our new dataset
print(df2.shape)

# Printing the shape of our old dataset
print(df1.shape)


#
# after the removal of the outliers, our new dataset had 608 rows and 11 columns<jupyter_output>(608, 11)
(891, 11)<jupyter_text>**EXPLORATORY DATA ANALYSIS**  **a) Univariate Analysis**<jupyter_code># Frequency distribution of values in variables
for var in df2.columns:
    
    print(df[var].value_counts())

# displays the summary frequencies of the dataset (df2)
# plot histograms of the variables
plt.rcParams['figure.figsize']=(30,25)
df2.plot(kind='hist', bins=10, subplots=True, layout=(5,2), sharex=False, sharey=False)
plt.show()
# Ploting the univariate summaries and recording our observations
# plot histograms for all numeric columns
# we use the dictionary we had earlier on saved
plt.figure(figsize = (20,30))
for variable,i in columns_dict.items():
  
  plt.subplot(5,4,i)
  sns.distplot(df2[variable])
  plt.title(variable)

plt.show()

# this shows the relationship between the variables
# describing the statistical features of the data
df2.describe()

# this way it is easier to understand the dataset and identify whether or 
# not there are outliers in the dataset<jupyter_output><empty_output><jupyter_text>**b) Bivariate Analysis**<jupyter_code># Ploting the bivariate summaries and recording our observations
# scatterplot
sns.pairplot(df2)
plt.show()
### Categoric Feature Analysis
# Boxplot gives a good idea about how the age is distributed across the categories.
# Distribution charts show how well each category is populated.
c = 0
len_c = 3 # (len(df2.columns)-2)
fig, axes = plt.subplots(len_c, 2, figsize=(10, 13))     # fig height = 70 -> in figsize(width,height)
for i, ax in enumerate(fig.axes):
    if (c < len_c) & (i % 2 == 0):
        ax.set_xticklabels(ax.xaxis.get_majorticklabels(), rotation=45)
        sns.countplot(x=df2.columns[c], alpha=0.7, data=df2, ax=ax)

    if (c < len_c) & (i % 2 != 0):
        ax.set_xticklabels(ax.xaxis.get_majorticklabels(), rotation=45)
        sns.boxplot(data = df2, x=df2.columns[c], y='Age', ax=ax)
        c = c + 1
fig.tight_layout()
# plotting the correlation using a heatmap
sns.heatmap(df2.corr(),annot=True)
plt.show()
# tabulating the correlation coefficients
corr = df2.corr()
corr<jupyter_output><empty_output><jupyter_text>InterpretationThe correlation coefficient ranges from -0.6 to +1.if the coefficients are close to +1, then that shows the strong positive correlation between variables. however in this case, there are no cases of strong positive correlation between the variables. instead, they are positive and close to zero, and indication that the correlation is zero or something close to zero.with the correlation coefficients going beyond the zero line to -0.689, it is said that the variables, in this case, fare and pclass are strongly negatively correlated.  **IMPLEMENTING THE SOLUTION**<jupyter_code># the exercise expects us to implement a K-nearest neighbor (kNN) classifier.
# it is an experiment that demands the metrics be calcuated carefully and
# all observatios noted.
# therefore, after splitting the dataset into two parts i.e 80 - 20  sets,
# we have to further make conclusions based on second ad third experiments with 
# different partitionin schemes 70-30, 60-40.
# this experiment expects a computation of the accuracy matrix which is the 
# percentage of correct classification
# it is then required that the confusion matric=x be calculated and 
# optimization done on the models.
# the whole process is as below<jupyter_output><empty_output><jupyter_text>**PART 1: 80:20 partition**<jupyter_code># importing the required libraries
from sklearn.model_selection import train_test_split
# previewing the dataset before modelling
df2.dtypes
# dropping redundant columns
df3 = df2.drop(['Name', 'Ticket', 'Cabin', 'Embarked'], axis=1)
df3.info()
# previewing the head of the dataset df3
df3.head()
# implementing the solution with the firt partitioning
# preprocessing
# declaring feature vector and target variables
X = df3.drop(['Sex'], axis=1)
y = df3['Sex']
# splitting the dataset into training and test sets
# Train using 80% of the data.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)
# checking the shapes of the train and test sets
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
# feature scaling 
# normalization
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
#
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# X_train = pd.DataFrame(X_train, columns=[cols])
# X_test = pd.DataFrame(X_test, columns=[cols])
# X_train.describe()
X_train
# Fitting K Neighbours Classifier to the training eet
# Training and Making Predictions
from sklearn.neighbors import KNeighborsClassifier
# instantiating the model
classifier = KNeighborsClassifier(n_neighbors=5)
# fitting the model to the training set
classifier.fit(X_train, y_train)
# comparing the accuracy of both the train and test sets
# Predicting the Test set results
y_pred = classifier.predict(X_test)
y_pred

# displays the predicted values of y
# evaluating the algorithm
from sklearn.metrics import classification_report, confusion_matrix
# Print the Confusion Matrix with k =5 and slice it into four pieces
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)
print('\nTrue Positives(TP) = ', cm[0,0])
print('\nTrue Negatives(TN) = ', cm[1,1])
print('\nFalse Positives(FP) = ', cm[0,1])
print('\nFalse Negatives(FN) = ', cm[1,0])<jupyter_output>Confusion matrix

 [[18 13]
 [ 4 87]]

True Positives(TP) =  18

True Negatives(TN) =  87

False Positives(FP) =  13

False Negatives(FN) =  4<jupyter_text>The confusion matrix is used to measure the performance of an machine learning algorithm, usually a supervised learning one. Each row of the confusion matrix represents the instances of an actual class and each column represents the instances of a predicted class.that is evident with the matrix shown above True Positives(TP) =  18True Negatives(TN) =  87False Positives(FP) =  13False Negatives(FN) =  4The confusion matrix shows 18 + 87 = 105 correct predictions and 13 + 4 = 17 incorrect predictions.<jupyter_code># visualizing the confusion matrix with seaborn heatmap
cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')
# Classification metrices
print(classification_report(y_test, y_pred))
# classification accuracies
TP = cm[0,0]
TN = cm[1,1]
FP = cm[0,1]
FN = cm[1,0]
# printing the classification accuracy

classification_accuracy = (TP + TN) / float(TP + TN + FP + FN)
print('Classification accuracy : {0:0.4f}'.format(classification_accuracy))
# printing the classification error

classification_error = (FP + FN) / float(TP + TN + FP + FN)
print('Classification error : {0:0.4f}'.format(classification_error))
# printing the precision score

precision = TP / float(TP + FP)
print('Precision : {0:0.4f}'.format(precision))
# printing recall

recall = TP / float(TP + FN)
print('Recall or Sensitivity : {0:0.4f}'.format(recall))
# true positive rate

true_positive_rate = TP / float(TP + FN)
print('True Positive Rate : {0:0.4f}'.format(true_positive_rate))
# specificity

specificity = TN / (TN + FP)
print('Specificity : {0:0.4f}'.format(specificity))
# false positive rate
false_positive_rate = FP / float(FP + TN)
print('False Positive Rate : {0:0.4f}'.format(false_positive_rate))
# Checking the accuracy accuracy_score
# printing the accuracy of the training set
from sklearn.metrics import accuracy_score
print('Accuracy' + str(accuracy_score(y_test, y_pred)))
print('Model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))
## the accuracy of this evaluation is less than 1<jupyter_output>Accuracy0.860655737704918
Model accuracy score: 0.8607<jupyter_text>**PART 2: 70:30 partition**<jupyter_code># importing the required libraries
from sklearn.model_selection import train_test_split
# implementing the solution with the firt partitioning
# preprocessing
# declaring feature vector and target variables
X = df3.drop(['Sex'], axis=1)
y = df3['Sex']
# splitting the dataset into training and test sets
# Train using 70% of the data.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30)
# checking the shapes of the train and test sets
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
# feature scaling 
# normalization
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
#
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# X_train = pd.DataFrame(X_train, columns=[cols])
# X_test = pd.DataFrame(X_test, columns=[cols])
# X_train.describe()
X_train
# Fitting K Neighbours Classifier to the training eet
# Training and Making Predictions
from sklearn.neighbors import KNeighborsClassifier
# instantiating the model
classifier = KNeighborsClassifier(n_neighbors=5)
# fitting the model to the training set
classifier.fit(X_train, y_train)
# comparing the accuracy of both the train and test sets
# Predicting the Test set results
y_pred = classifier.predict(X_test)
y_pred

# displays the predicted values of y
# evaluating the algorithm
from sklearn.metrics import classification_report, confusion_matrix
# Print the Confusion Matrix with k =5 and slice it into four pieces
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)
print('\nTrue Positives(TP) = ', cm[0,0])
print('\nTrue Negatives(TN) = ', cm[1,1])
print('\nFalse Positives(FP) = ', cm[0,1])
print('\nFalse Negatives(FN) = ', cm[1,0])<jupyter_output>Confusion matrix

 [[ 27  19]
 [ 12 125]]

True Positives(TP) =  27

True Negatives(TN) =  125

False Positives(FP) =  19

False Negatives(FN) =  12<jupyter_text>The confusion matrix is used to measure the performance of an machine learning algorithm, usually a supervised learning one. Each row of the confusion matrix represents the instances of an actual class and each column represents the instances of a predicted class.that is evident with the matrix shown above True Positives(TP) =  27True Negatives(TN) =  125False Positives(FP) =  19False Negatives(FN) =  12The confusion matrix shows 27 + 125 = 152 correct predictions and 19 + 12 = 31 incorrect predictions.<jupyter_code># visualizing the confusion matrix with seaborn heatmap
cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')
# Classification metrices
print(classification_report(y_test, y_pred))
# classification accuracies
TP = cm[0,0]
TN = cm[1,1]
FP = cm[0,1]
FN = cm[1,0]
# printing the classification accuracy

classification_accuracy = (TP + TN) / float(TP + TN + FP + FN)
print('Classification accuracy : {0:0.4f}'.format(classification_accuracy))
# printing the classification error

classification_error = (FP + FN) / float(TP + TN + FP + FN)
print('Classification error : {0:0.4f}'.format(classification_error))
# printing the precision score

precision = TP / float(TP + FP)
print('Precision : {0:0.4f}'.format(precision))
# printing recall

recall = TP / float(TP + FN)
print('Recall or Sensitivity : {0:0.4f}'.format(recall))
# true positive rate

true_positive_rate = TP / float(TP + FN)
print('True Positive Rate : {0:0.4f}'.format(true_positive_rate))
# specificity

specificity = TN / (TN + FP)
print('Specificity : {0:0.4f}'.format(specificity))
# false positive rate
false_positive_rate = FP / float(FP + TN)
print('False Positive Rate : {0:0.4f}'.format(false_positive_rate))
# Checking the accuracy accuracy_score
# printing the accuracy of the training set
from sklearn.metrics import accuracy_score
print('Accuracy' + str(accuracy_score(y_test, y_pred)))
print('Model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))
## the accuracy of this evaluation is less than 1<jupyter_output>Accuracy0.8306010928961749
Model accuracy score: 0.8306<jupyter_text>**PART 3: 60:40 partition**<jupyter_code># importing the required libraries
from sklearn.model_selection import train_test_split
# implementing the solution with the firt partitioning
# preprocessing
# declaring feature vector and target variables
X = df3.drop(['Sex'], axis=1)
y = df3['Sex']
# splitting the dataset into training and test sets
# Train using 60% of the data.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.40)
# checking the shapes of the train and test sets
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
# feature scaling 
# normalization
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
#
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# X_train = pd.DataFrame(X_train, columns=[cols])
# X_test = pd.DataFrame(X_test, columns=[cols])
# X_train.describe()
X_train
# Fitting K Neighbours Classifier to the training eet
# Training and Making Predictions
from sklearn.neighbors import KNeighborsClassifier
# instantiating the model
classifier = KNeighborsClassifier(n_neighbors=5)
# fitting the model to the training set
classifier.fit(X_train, y_train)
# comparing the accuracy of both the train and test sets
# Predicting the Test set results
y_pred = classifier.predict(X_test)
y_pred

# displays the predicted values of y
# evaluating the algorithm
from sklearn.metrics import classification_report, confusion_matrix
# Print the Confusion Matrix with k =5 and slice it into four pieces
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)
print('\nTrue Positives(TP) = ', cm[0,0])
print('\nTrue Negatives(TN) = ', cm[1,1])
print('\nFalse Positives(FP) = ', cm[0,1])
print('\nFalse Negatives(FN) = ', cm[1,0])<jupyter_output>Confusion matrix

 [[ 40  25]
 [  8 171]]

True Positives(TP) =  40

True Negatives(TN) =  171

False Positives(FP) =  25

False Negatives(FN) =  8<jupyter_text>The confusion matrix is used to measure the performance of an machine learning algorithm, usually a supervised learning one. Each row of the confusion matrix represents the instances of an actual class and each column represents the instances of a predicted class.that is evident with the matrix shown above True Positives(TP) =  40True Negatives(TN) =  171False Positives(FP) =  25False Negatives(FN) =  8The confusion matrix shows 40 + 171 = 211 correct predictions and 25 + 8 = 33 incorrect predictions.<jupyter_code># visualizing the confusion matrix with seaborn heatmap
cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')
# Classification metrices
print(classification_report(y_test, y_pred))
# classification accuracies
TP = cm[0,0]
TN = cm[1,1]
FP = cm[0,1]
FN = cm[1,0]
# printing the classification accuracy

classification_accuracy = (TP + TN) / float(TP + TN + FP + FN)
print('Classification accuracy : {0:0.4f}'.format(classification_accuracy))
# printing the classification error

classification_error = (FP + FN) / float(TP + TN + FP + FN)
print('Classification error : {0:0.4f}'.format(classification_error))
# printing the precision score

precision = TP / float(TP + FP)
print('Precision : {0:0.4f}'.format(precision))
# printing recall

recall = TP / float(TP + FN)
print('Recall or Sensitivity : {0:0.4f}'.format(recall))
# true positive rate

true_positive_rate = TP / float(TP + FN)
print('True Positive Rate : {0:0.4f}'.format(true_positive_rate))
# specificity

specificity = TN / (TN + FP)
print('Specificity : {0:0.4f}'.format(specificity))
# false positive rate
false_positive_rate = FP / float(FP + TN)
print('False Positive Rate : {0:0.4f}'.format(false_positive_rate))
# Checking the accuracy accuracy_score
# printing the accuracy of the training set
from sklearn.metrics import accuracy_score
print('Accuracy' + str(accuracy_score(y_test, y_pred)))
print('Model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))
## the accuracy of this evaluation is less than 1<jupyter_output>Accuracy0.8647540983606558
Model accuracy score: 0.8648<jupyter_text>**`Hyperparameter Optimization using GridSearch CV`**<jupyter_code># import GridSearchCV
from sklearn.model_selection import GridSearchCV
k_range=range(1,31)
print(k_range)
param_grid=dict(n_neighbors=k_range)
print (param_grid)
#
grid_search = GridSearchCV(classifier,param_grid,cv=10,scoring='accuracy')
#
grid_search.fit(X_train, y_train)
# examining the best model
# best score achieved during the GridSearchCV
print('GridSearch CV best score : {:.4f}\n\n'.format(grid_search.best_score_))
# print parameters that give the best results
print('Parameters that give the best results :','\n\n', (grid_search.best_params_))
# print estimator that was chosen by the GridSearch
print('\n\nEstimator that was chosen by the search :','\n\n', (grid_search.best_estimator_))
# calculating GridSearch CV score on test set

print('GridSearch CV score on test set: {0:0.4f}'.format(grid_search.score(X_test, y_test)))<jupyter_output>GridSearch CV score on test set: 0.8607
{'fraction_non_alphanumeric': 0.10192693347283983, 'fraction_numerical': 0.030473450170023543, 'mean_word_length': 5.062103594080338, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '656477', 'license': 'MIT', 'n_tokens_mistral': 7779, 'n_tokens_neox': 7123, 'n_words': 2441, 'path': 'KNN_IPWK9_CORE_The_KNN_Classifier_ELIZABETH_JOSEPHINE.ipynb', 'repo_name': 'alfafimel/IPWK9-CORE', 'size': '22938'}