<jupyter_start><jupyter_text>Importing Libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
df=pd.read_excel('dataset.xlsx')
df.head()<jupyter_output><empty_output><jupyter_text>Data Wrangling<jupyter_code>pd.Series(df.columns)
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5644 entries, 0 to 5643
Columns: 111 entries, Patient ID to ctO2 (arterial blood gas analysis)
dtypes: float64(70), int64(4), object(37)
memory usage: 4.8+ MB<jupyter_text>Analysing the missing data<jupyter_code>a= df.isnull().sum()/5643
b=pd.DataFrame(a)
b
d=b.loc[b[0]<0.8]
d<jupyter_output><empty_output><jupyter_text>All the columns having missing values more than 80% of total data points are dropped since they do not carry relavnt information and can sometimes leads to unexpected error.  New data is created which contains only the columns which have less than 80% missing values<jupyter_code>data= df[['SARS-Cov-2 exam result','Patient ID','Patient age quantile','Patient addmited to regular ward (1=yes, 0=no)',
         'Patient addmited to semi-intensive unit (1=yes, 0=no)','Patient addmited to intensive care unit (1=yes, 0=no)',
         'Respiratory Syncytial Virus','Influenza A','Influenza B','Parainfluenza 1',
        'CoronavirusNL63','Rhinovirus/Enterovirus','Coronavirus HKU1','Parainfluenza 3','Chlamydophila pneumoniae','Adenovirus',
        'Parainfluenza 4','Coronavirus229E','CoronavirusOC43','Inf A H1N1 2009','Bordetella pertussis','Metapneumovirus','Parainfluenza 2']]
data.head()
data=data.drop(['Patient ID'], axis=1)
data.head()
data['Influenza A'].unique()
data['CoronavirusOC43'].unique()<jupyter_output><empty_output><jupyter_text>Missing Value Treatment<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>Since we have included the columns which have even more than 50% of missing values, so we fill "UNAVAILABLE" in place of missing values of the columns<jupyter_code>data['Respiratory Syncytial Virus']= data['Respiratory Syncytial Virus'].fillna('unavailable')
data['Influenza A']= data['Influenza A'].fillna('unavailable')
data['Influenza B']= data['Influenza B'].fillna('unavailable')
data['Parainfluenza 1']= data['Parainfluenza 1'].fillna('unavailable')
data['CoronavirusNL63']= data['CoronavirusNL63'].fillna('unavailable')
data['Rhinovirus/Enterovirus']= data['Rhinovirus/Enterovirus'].fillna('unavailable')
data['Coronavirus HKU1']= data['Coronavirus HKU1'].fillna('unavailable')
data['Parainfluenza 3']= data['Parainfluenza 3'].fillna('unavailable')
data['Chlamydophila pneumoniae']= data['Chlamydophila pneumoniae'].fillna('unavailable')
data['Adenovirus']= data['Adenovirus'].fillna('unavailable')
data['Parainfluenza 4']= data['Parainfluenza 4'].fillna('unavailable')
data['Coronavirus229E']= data['Coronavirus229E'].fillna('Coronavirus229E')
data['CoronavirusOC43']= data['CoronavirusOC43'].fillna('unavailable')
data['Inf A H1N1 2009']= data['Inf A H1N1 2009'].fillna('unavailable')
data['Bordetella pertussis']= data['Bordetella pertussis'].fillna('unavailable')
data['Metapneumovirus']= data['Metapneumovirus'].fillna('unavailable')
data['Parainfluenza 2']= data['Parainfluenza 2'].fillna('unavailable')
data.head()<jupyter_output><empty_output><jupyter_text>-------------------------------------------------------------------------------------------------- Convert the values : negative ---> 0 and positive ----> 1 for the target column<jupyter_code>data['SARS-Cov-2 exam result']= data['SARS-Cov-2 exam result'].replace({'negative': 0 , 'positive': 1}).astype(int)
data.head()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5644 entries, 0 to 5643
Data columns (total 22 columns):
 #   Column                                                 Non-Null Count  Dtype 
---  ------                                                 --------------  ----- 
 0   SARS-Cov-2 exam result                                 5644 non-null   int32 
 1   Patient age quantile                                   5644 non-null   int64 
 2   Patient addmited to regular ward (1=yes, 0=no)         5644 non-null   int64 
 3   Patient addmited to semi-intensive unit (1=yes, 0=no)  5644 non-null   int64 
 4   Patient addmited to intensive care unit (1=yes, 0=no)  5644 non-null   int64 
 5   Respiratory Syncytial Virus                            5644 non-null   object
 6   Influenza A                                            5644 non-null   object
 7   Influenza B                                            5644 non-null   object
 8   Parainfluenza 1                                        564[...]<jupyter_text>------------------------------------------------------------------------------------------- Statistically describe the Numerical data<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>Bi-variate Analysis Analysis of variables or features with respect to the target label, to gain the insights and relationship between the input features and their impact on the spread of the Covid  1. Age group vs Covid<jupyter_code>sns.barplot(x='SARS-Cov-2 exam result',y='Patient age quantile',data=data)<jupyter_output><empty_output><jupyter_text>Note : The top of the bar is the mean; the width of the bar is the standard deviation; the vertical line is the range of values or standard error<jupyter_code>sns.distplot(data['Patient age quantile'])<jupyter_output><empty_output><jupyter_text>It is kind of Bimodular Histogram which depicts that the children/person in the age group of 5-10 and 10-15 years have the maximum rate of positive test<jupyter_code>sns.boxplot(data['Patient age quantile'])<jupyter_output><empty_output><jupyter_text>Box Plot -----------------------------------------------------------------------------------------<jupyter_code>data['Patient age quantile'].unique()<jupyter_output><empty_output><jupyter_text>Creating age groups for the analysis<jupyter_code>bins = [0,4,8,12,16,20]
labels = ['below 4', 'bet 4 and 8', 'bet 8 and 12', 'bet 12 and 16', 'above 16']

data['AgeGroup'] = pd.cut(data['Patient age quantile'], bins=bins, labels=labels, right=False)

data.head()
plt.figure(figsize=(10,8))
sns.barplot(x= 'AgeGroup', y='SARS-Cov-2 exam result', data=data)<jupyter_output><empty_output><jupyter_text>From this Bar graph it can be easily depicted which age group have maximum positive rate of Covid testing<jupyter_code>data.groupby('AgeGroup')['SARS-Cov-2 exam result'].agg(np.mean)<jupyter_output><empty_output><jupyter_text>Here we can clearly see that the positive rate of Covid spread is high in the age group of 8 to 12 and minimum for the age group below 4 years and above 15 years -------------------------------------------------------------------------  Patient Admitted Unit vs Covid Result<jupyter_code>sns.barplot(x= 'Patient addmited to regular ward (1=yes, 0=no)', y='SARS-Cov-2 exam result', data=data)
sns.barplot(x= 'Patient addmited to semi-intensive unit (1=yes, 0=no)', y='SARS-Cov-2 exam result', data=data)
sns.barplot(x= 'Patient addmited to intensive care unit (1=yes, 0=no)', y='SARS-Cov-2 exam result', data=data)<jupyter_output><empty_output><jupyter_text>From these above visualizations it can be easily inferred that the person/children who were addmitted to any of the three facilities are having high rate of Positive Covid testing ---------------------------------------------------------------------  Effect of different other diseases detected vs Covid Result<jupyter_code>data.columns
plt.figure(figsize=(20,30))

plt.subplot(6,3,1)
sns.barplot(x='Respiratory Syncytial Virus', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,2)
sns.barplot(x='Influenza A', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,3)
sns.barplot(x='Influenza B', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,4)
sns.barplot(x='Parainfluenza 1', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,5)
sns.barplot(x='CoronavirusNL63', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,6)
sns.barplot(x='Rhinovirus/Enterovirus', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,7)
sns.barplot(x='Coronavirus HKU1', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,8)
sns.barplot(x='Parainfluenza 3', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,9)
sns.barplot(x='Chlamydophila pneumoniae', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,10)
sns.barplot(x='Adenovirus', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,11)
sns.barplot(x='Parainfluenza 4', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,12)
sns.barplot(x='Coronavirus229E', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,13)
sns.barplot(x='CoronavirusOC43', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,14)
sns.barplot(x='Inf A H1N1 2009', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,15)
sns.barplot(x='<NAME>', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,16)
sns.barplot(x='Metapneumovirus', y='SARS-Cov-2 exam result' , data=data )

plt.subplot(6,3,17)
sns.barplot(x='Parainfluenza 2', y='SARS-Cov-2 exam result' , data=data )<jupyter_output><empty_output><jupyter_text>From these illustration we can clearly see that only ---> Influenza B , CoronavirusNL63 , Coronavirus229E diseases have a slight impact on the positive spread of COVID-19 ----------------------------------------------------------------------------------------------------------------  Data Prepration for Model So from the above Exploartory Data Analysis, the features which contribute significantly as well as the features that have atleast a slight impact on the results of Covid are used in the model Building<jupyter_code>final_data= data[['SARS-Cov-2 exam result','AgeGroup','Patient addmited to regular ward (1=yes, 0=no)',
                 'Patient addmited to semi-intensive unit (1=yes, 0=no)','Patient addmited to intensive care unit (1=yes, 0=no)',
                 'Influenza B','CoronavirusNL63','Coronavirus229E']]
final_data.head()<jupyter_output><empty_output><jupyter_text>------------------------------------------------------------------------------------ Label Encoding is done here below<jupyter_code>final_data['AgeGroup']=final_data['AgeGroup'].replace({'below 4': 1, 'bet 4 and 8':2, 'bet 8 and 12':3, 'bet 12 and 16':4, 'above 16':5}).astype(int)
final_data['Influenza B']= final_data['Influenza B'].replace({'not_detected':0, 'unavailable':1 , 'detected':2})
final_data['CoronavirusNL63']= final_data['CoronavirusNL63'].replace({'not_detected':0, 'unavailable':1 , 'detected':2})
final_data['Coronavirus229E']= final_data['Coronavirus229E'].replace({'not_detected':0, 'Coronavirus229E':1 , 'detected':1})
final_data.head()
final_data.head()
final_data.info()
final_data['SARS-Cov-2 exam result'].mean()<jupyter_output><empty_output><jupyter_text>From this statistics it can be easily seen that this dataset is a Skewed dataset which means that positive class contribute to 90% of the dataset while the neagtive class contribute to around 10% only  Machine Leraning Model<jupyter_code>y = final_data['SARS-Cov-2 exam result']
X= final_data.drop(['SARS-Cov-2 exam result'], axis = 1)
X.head()
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
X_train, X_test , y_train , y_test = train_test_split(X , y, random_state = 0 , test_size= 0.2)
rm= RandomForestClassifier(n_estimators=50).fit(X_train, y_train)
from sklearn.metrics import accuracy_score , confusion_matrix , precision_score , recall_score , f1_score
predict = rm.predict(X_test)
accuracy_score(y_test , predict)<jupyter_output><empty_output><jupyter_text>Since it is skewed data so accuracy is not the major metric to evaluate our learning algorithm,  So we use f1 score as our Evaluating metric of algorithm<jupyter_code>confusion_matrix(y_test , predict)
precision_score(y_test , predict)
recall_score(y_test , predict)
f1_score(y_test , predict, average='weighted')
f1_score(y_test , predict, average='micro')
f1_score(y_test , predict, average='macro')
f1_score(y_test , predict, average='binary')
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import plot_precision_recall_curve


disp = plot_precision_recall_curve(rm, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Hyper-parameter Tuning<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.metrics import f1_score, make_scorer
parameters = [{'n_estimators':[10, 50 ,100, 150]}]

grid_search = GridSearchCV(estimator = rm ,
                          param_grid = parameters,
                          cv = 5,
                          n_jobs = -1)

grid_search = grid_search.fit(X_train , y_train)
grid_search.best_score_
grid_search.best_params_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1691380908248378, 'fraction_numerical': 0.032514674080939145, 'mean_word_length': 4.074059561128527, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 93, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '380823', 'license': 'Apache-2.0', 'n_tokens_mistral': 4617, 'n_tokens_neox': 4175, 'n_words': 1040, 'path': 'Fiverr-Project/COVID-19.ipynb', 'repo_name': 'mahaveer-rulaniya/competitions-and-hackathons', 'size': '12962'}