<jupyter_start><jupyter_text>Disease Prediction using Machine Learning - Disease:       Malaria- Data:          Haematological data- Classifier:    Random Forest- Python Libraries    - numpy    - scipy    - pandas    - matplotlib    - seaborn    - scikit-learn This tutorial is based on this paper: <NAME>., <NAME>., <NAME>. et al. Machine learning approaches classify clinical malaria outcomes based on haematological parameters. BMC Med 18, 375 (2020). https://doi.org/10.1186/s12916-020-01823-3 Explanation of the code can be found here  Download Dataset  Import libraries for the analysis<jupyter_code>import pandas as pd        #read,explore & clean data
import numpy as np         # data manipulation
import matplotlib.pyplot as plt   # data visualization
import seaborn as sns             # data visualization<jupyter_output><empty_output><jupyter_text>Read the data<jupyter_code>#read the data with pandas
dataframe=pd.read_excel("https://static-content.springer.com/esm/art%3A10.1186%2Fs12916-020-01823-3/MediaObjects/12916_2020_1823_MOESM2_ESM.xlsx",sheet_name="Table S2 Hematological Raw Data")<jupyter_output><empty_output><jupyter_text>Explore, clean and preprocess the data<jupyter_code>#find the number of rows and columns in the dataframe
dataframe.shape
#get the first n rows in the dataframe
dataframe.head(n=5)
# list the column names
dataframe.columns

#obtain some information about the data 
#i.e. columns,datatypes,missing values,etc
dataframe.info()
#we are interested in the columns : 'Clinical_diagnosis' up to 'RBC_dist_width_Percent'
#meaning we will subset the data from column 16 - the last column
subset=dataframe.iloc[:,16:]
subset.shape
subset.info()
#Check the mising data. We are interested in how many missing data are present in each column
subset.isnull().sum()
# handling missing values
# drop / remove all rows with missing values
subset.dropna(inplace=True)
subset.shape
subset.columns
#Let us get the different malaria outcomes. 
#The outcomes will be our labels/classes in the data
subset['Clinical_Diagnosis'].unique()
labels=pd.Categorical(subset['Clinical_Diagnosis'])
labels
subset.head()
#class distribution
subset['Clinical_Diagnosis'].value_counts()
# plot a bar chat to display the class distribution
subset['Clinical_Diagnosis'].value_counts().plot.bar()
#descriptive statistics on the data
subset.iloc[:,1:].describe().transpose()
#check the correlation for the features
subset.corr()
#lets visualize the correlation matrix using seaborn
sns.heatmap(subset.corr(),cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>Data Preprocessing<jupyter_code># separate the labels/classes from the features/measurement
X=subset.iloc[:,1:]
y=subset.iloc[:,0]
X.shape
y.shape
#encode the labels. 
#This is required by scikit learn when performing supervised learning
from sklearn.preprocessing import LabelEncoder
label_encoder=LabelEncoder()
label_encoder.fit(y)
y_encoded=label_encoder.transform(y)
y_encoded[0:5]
y[0:5]
classes=label_encoder.classes_
classes<jupyter_output><empty_output><jupyter_text>Split data into train and test sets<jupyter_code># train test ratio 80:20
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y_encoded,test_size=0.2)
X_train.shape
y_train.shape
X_test.shape
y_test.shape<jupyter_output><empty_output><jupyter_text>Standardization of the data<jupyter_code># scale data between 0 and 1
from sklearn.preprocessing import MinMaxScaler
min_max_scaler=MinMaxScaler()
X_train_scaled=min_max_scaler.fit_transform(X_train)
X_test_scaled=min_max_scaler.fit_transform(X_test)
X_train_scaled[0,0]
X_train.iloc[0,0]<jupyter_output><empty_output><jupyter_text>Training Phase<jupyter_code>#create random forest classifier
from sklearn.ensemble import RandomForestClassifier
clf=RandomForestClassifier()
clf.fit(X_train_scaled,y_train)<jupyter_output><empty_output><jupyter_text>Testing phase<jupyter_code># model prediction on the test set
y_pred=clf.predict(X_test_scaled)
y_pred[0:3]
y_test[0:3]
classes<jupyter_output><empty_output><jupyter_text>Evaluating the model The following metrics will be used- accuracy- f1 score- recall- precision- confusion matrix<jupyter_code># import the metrics
from sklearn.metrics import balanced_accuracy_score,f1_score,precision_score, recall_score
from sklearn.metrics import plot_confusion_matrix
#balanced accuracy
balanced_accuracy=balanced_accuracy_score(y_test,y_pred)
balanced_accuracy=round(balanced_accuracy,2)
print('balanced accuracy:',balanced_accuracy)
#f1score
f1score=f1_score(y_test,y_pred,average='weighted')
f1score=round(f1score,2)
print('f1score:',f1score)
#precision 
precision=precision_score(y_test,y_pred,average='weighted')
precision=round(precision,2)
print('precision:',precision)
#recall
recall=recall_score(y_test,y_pred,average='weighted')
recall=round(recall,2)
print('recall:',recall)
#confusion matrix
disp=plot_confusion_matrix(clf,X_test_scaled,y_test,xticks_rotation='vertical',
                     cmap='Blues',display_labels=classes)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09344132753852233, 'fraction_numerical': 0.02113789016199131, 'mean_word_length': 7.341021416803954, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '442737', 'license': 'MIT', 'n_tokens_mistral': 1756, 'n_tokens_neox': 1642, 'n_words': 391, 'path': 'Machine learning for predicting Malaria using Haematological data.ipynb', 'repo_name': 'vappiah/Machine-Learning-Tutorials', 'size': '5080'}