<jupyter_start><jupyter_text>PREPROCESSING & MODELINGTHis is where after the data has been preprocessed,  machine learning models will be built, tested and the best performing model to be implemented during deployment<jupyter_code># Load the clean dataset

import pandas as pd

data = pd.read_csv('/Users/RyanMburu/Desktop/DS Projects/Thyroid Disease Detector/Datasets/clean_hypothyroid2.csv')
data.head()
# Drop the redundant index column
data.drop(columns=('Unnamed: 0'), axis=1, inplace=True)

data.head()<jupyter_output><empty_output><jupyter_text>The project is a classification problem, as the output is YES/NO or 1/2We need to perform preprocessing  Data Preprocessing  1. Feature EncodingThis is where categorical data that is in string form (t/f, etc) will be transformed to numerical data<jupyter_code># will perform label encoding on the categorical data all over the dataset

# Will make use of codes

codes = {'f' : 0, 't' : 1}
codes

# Apply on every column necessary

data['on thyroxine'] = data['on thyroxine'].replace(codes)
data
# Function that inputs codes per indicated column

def label_encoder(column):
    data[column] = data[column].replace(codes)
label_encoder('query on thyroxine')
# Copy paste the rest of the columns and run at once
label_encoder('on antithyroid medication')
label_encoder('sick')
label_encoder('pregnant')
label_encoder('thyroid surgery')
label_encoder('I131 treatment')
label_encoder('query hypothyroid')
label_encoder('query hyperthyroid')
label_encoder('lithium')
label_encoder('goitre')
label_encoder('tumor')
label_encoder('hypopituitary')
label_encoder('psych')

# Will do the same process on gender column

# This is important so as to avoid using alot of libraries,and practice my craft as a programmer

codes = {'F' : 0, 'M' : 1, 'NS' : 2}

data['sex'] = data['sex'].replace(codes)
# Drop referal source column, as the hospital you were refered from cannot influence whether you have an ailment or not

data.drop(columns = ('referral source'), axis = 1, inplace = True)
# Output final dataset
data<jupyter_output><empty_output><jupyter_text>2. Feature Reduction / Feature Selection - Disclaimer :  Decided to do reduction before normalization as when conducting research, I saw that decision tree models are normally not affected by the scale of the inputs, unlike distance-based models such as K-Nearest Neighbours models and SVM's Feature selection is where we will choose the most important features for our modelling phaseWill use Random Forest and check for feature importance as a trick to see which features contributed the most to prediction<jupyter_code># Will split the data into arrays of data i.e X and y using the traintest split module
from sklearn.model_selection import train_test_split
# X are the features
# y is the label
X = data.loc[:, 'age' : 'FTI'].values
y = data['output'].values
# Will split X and y into training data and testing data of 80% and 20% splits
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)

X_test

# Will quickly build a random forest model and check for feature importance

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 500, max_depth = 8, min_samples_split = 30)
forest.fit(X_train, y_train)
# Will check for feature importance, as thats our main priority
X_columns = data.loc[:, 'age' : 'FTI']
df_FI = pd.DataFrame(forest.feature_importances_, X_columns.columns, columns = ['feature importance'])

# Will first import plotting module
import matplotlib.pyplot as plt
# Plot the dataframe as a horizontal bar graph of its values
df_FI.plot(kind = 'barh', figsize = (20, 20))
plt.title('A graph showing the most important features in the dataset ')<jupyter_output><empty_output><jupyter_text>The most important features as per the Feature Importance plot are : 1. FTI2. TFU3. TT44. T35. TSH (most important)6. Thyroid Surgery7. On Thyroxine8. Age Will go ahead and extract a new dataset for modelling only using the said features<jupyter_code># New dataset with reduced features

data_modeling = data[['age', 'on thyroxine', 'thyroid surgery', 'TSH', 'T3', 'TT4', 'T4U', 'FTI', 'output']]
data_modeling
# Export modeling dataset
data_modeling.to_csv('modeling_hypothyroid.csv')<jupyter_output><empty_output><jupyter_text>3. Feature normalization / ScalingThis is when the data is in different scales eg age and sex. Age ranges until 100 whereas sex is 1 and 0This often leads to machine learning models performing poorly.<jupyter_code># Will import MinMaxScaler to have all data range from 0 - 1
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
# Will split the data to X and y then scale data in X
X = data_modeling.loc[:, 'age' : 'FTI'].values
y = data_modeling['output'].values

scaled_X = scaler.fit_transform(X)
scaled_X<jupyter_output><empty_output><jupyter_text>The X values have been normalized. They are now all in a range of 0 - 1  4. Data SplittingThis is where the data is split into two or sometimes 3 sets1. Training set (80% of the data) -  this is the data that will train the model2. Testing set (20%) - This will be used during testing the model's accuracy<jupyter_code># Will use the train test module that was used when building the Random Forest Model
X_train, X_test, y_train, y_test = train_test_split(scaled_X, y, test_size = 0.2, random_state = 0)

# Length of training data
print(X_train.shape)

# Length of testing data
print(X_test.shape)
import numpy as np
np.unique(y_test, return_counts = True)<jupyter_output><empty_output><jupyter_text>The splitting is a success, where the training split has 2199 records of data and the testing split has 550  Modeling Here is where machine learning models will be built and the best performing one is chosen. ML models to be built include : 1. Naive Bayes2. Random Forest3. Boosted trees (XGBoosted trees and ADA trees)4. K-Nearest Neighbours (KNN) / Standard Vector Machines (SVM)5. Neural Networks   1. Naive Bayes - This is a predominantly classification algorithm that aims to calculate the probability of on escenario (in this case if a patient has hypothyroid disease) based on its relationship with anotherother scenario(their ages, TSH score etc)- As our output is boolean (0/1), we will use bernoulli's NaiveBayes  i. Bernoulli NB  Training the naive Bayes Model<jupyter_code># Will import the bernoullis naive bayes module hosted by sklearn
from sklearn.naive_bayes import BernoulliNB
#Initializing the model
NB = BernoulliNB()
# Fitting the model with training data
NB.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Prediction using the trained model<jupyter_code># Prediction
y_pred = NB.predict(X_test)

# Testing the accuracy
from sklearn.metrics import accuracy_score

print('accuracy of the NB model is : ', accuracy_score(y_test, y_pred))
np.unique(y_pred)<jupyter_output><empty_output><jupyter_text>Besides having a 90% accuracy, the model was explicitly predicting 0 as the output, which is highly inaccurate.  ii. Multinomial NB Will do the same process, but in this case will use Multinomial Model<jupyter_code># Will import the Multinomial naive bayes module hosted by sklearn
from sklearn.naive_bayes import MultinomialNB
#Initializing the model
NB2 = MultinomialNB()
# Fitting the model with training data
NB2.fit(X_train, y_train)
# Prediction
y_pred = NB2.predict(X_test)

# Testing the accuracy
print('accuracy of the NB model is : ', accuracy_score(y_test, y_pred))

# Print number of outputs
print(np.unique(y_pred))<jupyter_output>accuracy of the NB model is :  0.9072727272727272
[0 1]<jupyter_text>In the case of Multinomial NB, it has a 90% accuracy, but logically performs better than the Bernoulli's one as it predicts both positive cases and negative  iii. Naive Bayes with hot encoded labels<jupyter_code>data_dummy = data_modeling
data_dummy = pd.get_dummies(data_dummy, columns = ['output'])
data_dummy
# Normalize data in X and have y as two columns
X2 = data_dummy[['age', 'on thyroxine', 'TSH', 'T3', 'TT4', 'T4U', 'FTI']].values
y2 = data_dummy[['output_0', 'output_1']]
# Scaling
scaled_X2 = scaler.fit_transform(X2)
scaled_X2
#Splitting
# Will use the train test module that was used when building the Random Forest Model
X_train2, X_test2, y_train2, y_test2 = train_test_split(scaled_X2, y2, test_size = 0.2, random_state = 0)

# Length of training data
print(X_train2.shape)

# Length of testing data
print(X_test2.shape)
# Modeling
# Will import the Multinomial naive bayes module hosted by sklearn
#Initializing the model
NB3 = MultinomialNB()
# Fitting the model with training data
NB3.fit(X_train2, y_train2)<jupyter_output><empty_output><jupyter_text>Hot encoded values cant work as y. This is only applicable in Neural Networks, as it can have more than one output.  2. Random Forests This is a Machine Learning technique where multiple decision trees are created and combined to derive an output. As known by most ML engineers, random forests usually do not require the data to be normalized/standadized in any form<jupyter_code># Load dataset

data_RF = pd.read_csv('Datasets/modeling_hypothyroid.csv')
data_RF

# Splitting
# Will split the data into arrays of data i.e X and y using the traintest split module
from sklearn.model_selection import train_test_split
# X are the features
# y is the label
X = data_RF.loc[:, 'age' : 'FTI'].values
y = data_RF['output'].values
# Will split X and y into training data and testing data of 80% and 20% splits
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 42)

X_test.shape
#Building model
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 300, max_depth = 8, min_samples_split = 30)
forest.fit(X_train, y_train)
# Checking the model's accuracy

y_pred = forest.predict(X_test)

from sklearn.metrics import accuracy_score
print('accuracy of the RF model is : ', accuracy_score(y_test, y_pred))<jupyter_output>accuracy of the RF model is :  0.9915151515151515<jupyter_text>UUm ok the RF model has an accuracy of 98%.<jupyter_code># Feature importance
X_columns = data_RF.loc[:, 'age' : 'FTI']
df_FI = pd.DataFrame(forest.feature_importances_, X_columns.columns, columns = ['feature importance'])

# Will first import plotting module
import matplotlib.pyplot as plt
# Plot the dataframe as a horizontal bar graph of its values
df_FI.plot(kind = 'barh', figsize = (20, 20))
plt.title('A graph showing the most important features in the dataset ')<jupyter_output><empty_output><jupyter_text>As seen, the importances of the inputs have been displayed, TSH being the most important feature and thyroid surgery is the least important  Draft Deployment Will perform a mock deployment using foreign data, so as to test the performance of our model.<jupyter_code># Function that asks for user input and displays the inputs

import numpy as np

def thyroid_predictor():
    # Patients inputs
    age = int(input('Whats the patients age ? :'))
    on_thyroxine = int(input('Is the patient on thyroxine medication ? :'))
    thyroid_surgery = int(input('Has the patient had previous surgery on thyroid gland? : '))
    TSH = float(input('Whats the patients TSH value? : '))
    T3 = float(input('Whats the patients T3 value? : '))
    TT4 = float(input('Whats the patients TT4 value? : '))
    T4U = float(input('Whats the patients T4U value? : '))
    FTI = float(input('Whats the patients FTI value? : '))

    patient_details = np.array([age, on_thyroxine, thyroid_surgery, TSH, T3, TT4, T4U, FTI]).reshape(1, -1)

    # Inputs are fed into the Random Forest model created and tested
    diagnosis = forest.predict(patient_details)

    return diagnosis
# Situation where patient has the disease
thyroid_predictor()
# Situation where the patient does not have the disease
thyroid_predictor()<jupyter_output><empty_output><jupyter_text>Exporting model<jupyter_code>import pickle
pickle_model = 'thyroid_model.pkl'

with open(pickle_model, 'wb') as file:
    pickle.dump(forest, file)
# Loading the model
with open(pickle_model, 'rb') as file:
    forest_model = pickle.load(file)
# Called model
forest_model<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08035348989444399, 'fraction_numerical': 0.01505605105965142, 'mean_word_length': 5.453009503695882, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '639898', 'license': 'Unlicense', 'n_tokens_mistral': 3921, 'n_tokens_neox': 3699, 'n_words': 1466, 'path': 'modeling.ipynb', 'repo_name': 'mbururyan/Hypothyroid-Disease-Detection-System', 'size': '12221'}