<jupyter_start><jupyter_text>Thyroid_Disease_PredicitionProblem Statement:Thyroid disease is a common cause of medical diagnosis and prediction, with an onsetthat is difficult to forecast in medical research. The thyroid gland is one of our body'smost vital organs. Thyroid hormone releases are responsible for metabolic regulation.Hyperthyroidism and hypothyroidism are one of the two common diseases of the thyroidthat releases thyroid hormones in regulating the rate of body's metabolism.The main goal is to predict the estimated risk on a patient's chance of obtaining thyroiddisease or not.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import dtale
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler,MinMaxScaler,LabelEncoder
import imblearn.over_sampling
import sklearn.impute

from sklearn.ensemble import RandomForestClassifier

import warnings
warnings.filterwarnings('ignore')
data=pd.read_csv('hypothyroid.csv')
data.columns
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3772 entries, 0 to 3771
Data columns (total 30 columns):
 #   Column                     Non-Null Count  Dtype 
---  ------                     --------------  ----- 
 0   age                        3772 non-null   object
 1   sex                        3772 non-null   object
 2   on_thyroxine               3772 non-null   object
 3   query_on_thyroxine         3772 non-null   object
 4   on_antithyroid_medication  3772 non-null   object
 5   sick                       3772 non-null   object
 6   pregnant                   3772 non-null   object
 7   thyroid_surgery            3772 non-null   object
 8   I131_treatment             3772 non-null   object
 9   query_hypothyroid          3772 non-null   object
 10  query_hyperthyroid         3772 non-null   object
 11  lithium                    3772 non-null   object
 12  goitre                     3772 non-null   object
 13  tumor                      3772 non-null   object
 14  hypopitu[...]<jupyter_text>All feature data types are object<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>Data PreprocessingSeems some of the columns has '?' as empty values<jupyter_code># dtale.show(data)<jupyter_output><empty_output><jupyter_text>Have to replace '?' to 'nan' to remove empty values,because dtypes of features are objects<jupyter_code>for column in data.columns:
    count = data[column][data[column]=='?'].count()
    if count != 0:
        print(column, count)<jupyter_output>age 1
sex 150
TSH 369
T3 769
TT4 231
T4U 387
FTI 385
TBG 3772<jupyter_text>"TBG" has 37772 as null value it is good to drop and some columns too and unwanted columns 'TSH_measured','T3_measured','TT4_measured','T4U_measured','FTI_measured','TBG_measured'<jupyter_code>data= data.drop(['TBG','TSH_measured','T3_measured','TT4_measured','T4U_measured','FTI_measured','TBG_measured'],axis =1)
for column in data.columns:
    count = data[column][data[column]=='?'].count()
    if count!=0:
        data[column] = data[column].replace('?',np.nan)
for column in data.columns:
    count = data[column][data[column]=='?'].count()
    if count==0:
        print(column, count)<jupyter_output>age 0
sex 0
on_thyroxine 0
query_on_thyroxine 0
on_antithyroid_medication 0
sick 0
pregnant 0
thyroid_surgery 0
I131_treatment 0
query_hypothyroid 0
query_hyperthyroid 0
lithium 0
goitre 0
tumor 0
hypopituitary 0
psych 0
TSH 0
T3 0
TT4 0
T4U 0
FTI 0
referral_source 0
Class 0<jupyter_text>'?' is replaced with 'Nan' Sucessfully<jupyter_code>data.isna().sum()<jupyter_output><empty_output><jupyter_text>Replacing Categorical to numbers<jupyter_code>data['sex'] = data['sex'].replace({'F' : 0, 'M' : 1})
for column in data.columns:
    if  len(data[column].unique())==2:
        data[column] = data[column].replace({'f' : 0, 't' : 1})
data['referral_source'].unique()
data['referral_source'] = data['referral_source'].replace({'SVHC' : 1, 'SVI' : 2, 'STMW' : 3 , 'SVHD' :4 , 'other' :5})
data['Class']. value_counts()
data['Class'] = data['Class'].replace({'negative' : 0, 'compensated_hypothyroid' : 1,'primary_hypothyroid' :2,'secondary_hypothyroid':3})
data['Class'].value_counts()
data["Class"] = data["Class"].apply(lambda value : 1 if value >= 1 else 0)
data['Class'].value_counts()<jupyter_output><empty_output><jupyter_text>As 'Class' has some imbalance.... we can add all Hypothyroid as (1) one and negative as (0) seperate  output<jupyter_code>data.tail()
imputer=sklearn.impute.KNNImputer(n_neighbors=3, weights='uniform',missing_values=np.nan)
new_array=imputer.fit_transform(data)
data=pd.DataFrame(data=np.round(new_array), columns=data.columns)
data.isnull().sum()<jupyter_output><empty_output><jupyter_text>No null value  Feature SelectionSeems some of the columns are not impacts on predicting output.As it the update data verson,previous versio n of the data has clear lables and records.As model concern feature selection has to be done......<jupyter_code>y=data["Class"].copy()
X=data.drop('Class',axis=1).copy()
X_train,X_test,y_train,y_test=train_test_split(X,y,train_size=0.8,shuffle=True,random_state=123)
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
bestfeatures = SelectKBest(score_func=chi2, k=10)
fit = bestfeatures.fit(X,y)
dfscores = pd.DataFrame(fit.scores_)
dfcolumns = pd.DataFrame(X.columns)
#concat two dataframes for better visualization 
featureScores = pd.concat([dfcolumns,dfscores],axis=1)
featureScores.columns = ['Specs','Score']  #naming the dataframe columns
print(featureScores.nlargest(10,'Score'))  #print 10 best features<jupyter_output>Specs         Score
16                TSH  77121.541469
20                FTI   3757.920430
18                TT4   3569.940747
17                 T3     37.389572
9   query_hypothyroid     26.338754
2        on_thyroxine     21.736074
1                 sex      7.381573
6            pregnant      4.430623
15              psych      2.929734
12             goitre      2.842287<jupyter_text>We are taking 1st 10 feature that make impact on output  New Data by Feature Selection<jupyter_code>new_data = data.filter(['TSH','FTI','TT4','T3','query_hypothyroid','on_thyroxine','sex','pregnant','psych','Class'])
new_data.to_csv('new_Thyroid_data.csv', index=False)
new_data['pregnant'].value_counts()
#dtale.show(new_data)
y=new_data["Class"].copy()
X=new_data.drop('Class',axis=1).copy()
X_train,X_test,y_train,y_test=train_test_split(X,y,train_size=0.8,shuffle=True,random_state=123)
y.value_counts()<jupyter_output><empty_output><jupyter_text>Seems imbalanced class,RandomOverSample is done to over come this<jupyter_code>from imblearn.over_sampling import RandomOverSampler
os=RandomOverSampler(sampling_strategy='all')
X_train_res,y_train_res=os.fit_resample(X_train,y_train)
from collections import Counter
print('Resampled dataset shape %s' % Counter(y_train_res))
import joblib as jb
rfc=RandomForestClassifier( )

forest_params = [{'max_depth': list(range(10, 15)), 'max_features': list(range(0,14))}]

from sklearn.model_selection import GridSearchCV

clf = GridSearchCV (rfc, forest_params, cv = 10, scoring='accuracy')
clf.fit(X_train, y_train)
print(clf.best_params_)

print(clf.best_score_)
X_train.shape,y_train.shape<jupyter_output><empty_output><jupyter_text>Nearly 98% of accuracy<jupyter_code>model=RandomForestClassifier( n_estimators=150,
                               criterion='gini',
                               max_depth=10)
model.fit(X_train,y_train)

from sklearn.metrics import confusion_matrix,accuracy_score,classification_report
y_pred=model.predict(X_test)
cf_matrix=np.array(confusion_matrix(y_test,y_pred))
print(cf_matrix)
print("Accuracy Score {}".format(accuracy_score(y_test,y_pred)))
print(format(classification_report(y_test,y_pred)))
y_pred=.predict([[12.0,87.0,99.0,2.0,1.0,0.0,0.0,0.0,0.0]])
y_pred
import joblib as jb<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11001375171896487, 'fraction_numerical': 0.043005375671958994, 'mean_word_length': 4.46448087431694, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '113039', 'license': 'MIT', 'n_tokens_mistral': 2984, 'n_tokens_neox': 2672, 'n_words': 625, 'path': 'notebooks/Thyroid_Disease_Prediction.ipynb', 'repo_name': 'yuvraj123-verma/Thyroid_Disease-Detection', 'size': '7999'}