<jupyter_start><jupyter_text>ML Group Project - Data Cleaning & KNN Algorithm testing of data 1. Import of the Python libraries<jupyter_code>import pandas as pd  #build dataframe
import numpy as np
import matplotlib.pyplot as plt #to graph data
import seaborn as sns # for heatmap of features

from sklearn.model_selection import train_test_split #training algorithm libraries
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score<jupyter_output><empty_output><jupyter_text>2. Read in CSV files<jupyter_code>opal_corsa_01 = pd.read_csv("opel_corsa_01.csv", sep = ';')
opal_corsa_01["Make"] = "Opal Corsa"
opal_corsa_01["Source File"] = 1

opal_corsa_02 = pd.read_csv("opel_corsa_02.csv", sep = ';')
opal_corsa_02["Make"] = "Opal Corsa"
opal_corsa_02["Source File"] = 2

peugeot_207_01 = pd.read_csv("peugeot_207_01.csv", sep = ';')
peugeot_207_01["Make"] = "Peugeot 207"
peugeot_207_01["Source File"] = 1

peugeot_207_02 = pd.read_csv("peugeot_207_02.csv", sep = ';')
peugeot_207_02["Make"] = "Peugeot 207"
peugeot_207_02["Source File"] = 2<jupyter_output><empty_output><jupyter_text>3. Review the record counts for each data set using a column with a string value and to observe if there is any imbalance in the dataset<jupyter_code>print(opal_corsa_01['Make'].value_counts()[:])
print(opal_corsa_02['Make'].value_counts()[:])

print(peugeot_207_01['Make'].value_counts()[:])
print(peugeot_207_02['Make'].value_counts()[:])<jupyter_output>Opal Corsa    7038
Name: Make, dtype: int64
Opal Corsa    4092
Name: Make, dtype: int64
Peugeot 207    8199
Name: Make, dtype: int64
Peugeot 207    4446
Name: Make, dtype: int64<jupyter_text>4. The count values for each data set do not suggest any imbalance in the data set, 47% records Opel Corsa, 53% records Peugeot 207 5. Combine the four datasets into one dataset<jupyter_code>dataset = [opal_corsa_01, opal_corsa_02, peugeot_207_01, peugeot_207_02]

combined = pd.concat(dataset)

data_pre_cleaning = combined.drop('Unnamed: 0', axis = 1)  #remove the first column

data_pre_cleaning.to_csv("pre_cleaning_dataset.csv",index = False)

data_pre_cleaning.shape<jupyter_output><empty_output><jupyter_text>6. Check the structure of the combined dataset, check for null values, remove any rows with NaN values and remove columns that are not relevant to the analysis<jupyter_code>#display the first five rows of the dataset
data_pre_cleaning.head()
#find all columns that contain null values using 'isna' function but present the data in column format

data_pre_cleaning.isna().any()
#drop any rows with NaN values

clean_dataset = data_pre_cleaning.dropna()
#confirm that there are now no rows with NaN in the dataset

clean_dataset.isna().any()
#drop the source file column

clean_dataset1 = clean_dataset.drop(['Source File'], axis=1)
#check the dataset structure

clean_dataset1.head()<jupyter_output><empty_output><jupyter_text>7. Check the data statistics for the dataset<jupyter_code># summuary of the statistically breakdown for the integer based features

clean_dataset1.describe()<jupyter_output><empty_output><jupyter_text>8. The project team reviewed the output types variables for the dataset are "Driving Style", "Road Surface Conditions", & "Traffic", and decided to analysis the "Driving Style" variable. There are two classes Even Pace and Aggressive for the Driving output. These two classes will be converted into integer values for the remainder of the analysis.<jupyter_code># replace the driving style string values to binary values as there are only two class possibilities

clean_dataset1 = clean_dataset1.replace(to_replace="EvenPaceStyle",value="0")

clean_dataset1 = clean_dataset1.replace(to_replace="AggressiveStyle", value="1")<jupyter_output><empty_output><jupyter_text>9. Carry out a correlation analysis on the features<jupyter_code>#run a correlation check on the dataframe to identify what features we want to use

clean_dataset1.corr()<jupyter_output><empty_output><jupyter_text>Visualise the correlation data into a Heatmap to make it easier to identify features that have correlation.<jupyter_code>#Create a heatmap is add to the visualisaton of the correlation (Pearson Correlation) 
plt.figure(figsize=(12,10))
cor = clean_dataset1.corr()
#sns.heatmap(cor, annot=True, cmap=plt.cm.Reds)

sns.heatmap(cor, annot=True, cmap="coolwarm")
plt.show()<jupyter_output><empty_output><jupyter_text>The following features on the heatmap have a correlation higher that 0.5 and -0.5 VehicleSpeedInstantaneous - VehicleSpeedAverage - 0.87VehicleSpeedInstantaneous - ManifoldAbsolutePressure - 0.72VehicleSpeedInstantaneous - EngineRPM - 0.84VehicleSpeedInstantaneous - MassAirFlow - 0.82VehicleSpeedAverage - ManifoldAbsolutePressure - 0.6VehicleSpeedAverage - EngineRPM - 0.65VehicleSpeedAverage - MassAirFlow - 0.64 LongitudinalAcceleration - VerticalAcceleration - 0.74ManifoldAbsolutePressure - EngineRPM - 0.71ManifoldAbsolutePressure - MassAirFlow - 0.87EngineRPM - MassAirFlow - 0.88MassAirFlow - ManifoldAbsolutePressure - 0.87VehicleSpeedAverage - FuelConsumptionAverage - -0.62We will remove the following features as they are highly correlated to a remaining featureEngineRPMVehicleSpeedAverageManifoldAbsolutePressureMassAirflowVerticalAccelerationFuelConsumptionAverageThis leaves the following features for the algorithmAltitudeVariationVehicleSpeedInstantaneousVehicleSpeedVarianceVehicleSpeedVariationLongitudinalAccelerationEngineLoadEngineCoolantTemperatureIntakeAirTemperatureVehicleSpeedAverage 8. Create a new dataset for the remaining features that will be analysed and remove the other output variables, Road Surface Conditions, & Traffic<jupyter_code>#create a new dataset with an appropriate name and drop the feature and output variable columns no longer required.

driving_style_features = clean_dataset1.drop(['ManifoldAbsolutePressure','EngineRPM', 'MassAirFlow', 'VerticalAcceleration', 'FuelConsumptionAverage', 'roadSurface', 'traffic', 'Make'], axis=1)
#check new dataset structure

driving_style_features.head()
#check if there area any columns that contain null values using 'isna' function 

driving_style_features.isna().any()<jupyter_output><empty_output><jupyter_text>No future null values found. 9. Start the KNN analysis of the driving_style_features dataset<jupyter_code>#split data set into test and training. The test set was set at 20% and training set was at 80%
#and 9 features are to be analysed

X = driving_style_features.iloc[:, 0:9]
y = driving_style_features.iloc[:, 9]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, test_size=0.2)
# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)
#calculation used to identify the k value for the analysis

import math
math.sqrt(len(y_test))<jupyter_output><empty_output><jupyter_text>Define the model parameters. The calculated K value from the test data was so 68.942, so it is rounded up 69 and therefore reduces the risk of a tie. P = 2 as we have two parameters, and metric = euclidean which is the common metric used for KNN algorithms<jupyter_code>#set the KNN classification algorithm settings

classifier = KNeighborsClassifier(n_neighbors=69, p=2, metric='euclidean')
#run the algorithm

classifier.fit(X_train, y_train)
#predict the test results

y_pred = classifier.predict(X_test)
y_pred
#Evaluate the model using confusion matrix

cm = confusion_matrix(y_test, y_pred)

print(cm)<jupyter_output>[[4180   13]
 [ 518   42]]<jupyter_text>The confusion matrix results show 4180 postive results for the "Even Pace" driving style class and 42 positive results for the "Aggressive" driving style<jupyter_code>#Run the KNN algorithn statistically report

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.89      1.00      0.94      4193
           1       0.76      0.07      0.14       560

    accuracy                           0.89      4753
   macro avg       0.83      0.54      0.54      4753
weighted avg       0.87      0.89      0.85      4753
{'fraction_non_alphanumeric': 0.07773894178618777, 'fraction_numerical': 0.0367602747981198, 'mean_word_length': 5.824013157894737, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '282061', 'license': 'MIT', 'n_tokens_mistral': 2817, 'n_tokens_neox': 2562, 'n_words': 801, 'path': 'ML Group Project KNN Algorithm Analysis.ipynb', 'repo_name': 'darraghs/SligoIT_ML2021_GroupB', 'size': '8297'}