<jupyter_start><jupyter_text>KEN3450, Data Analysis 2020 **Kaggle Competition 2020**<jupyter_code>import numpy as np
import pandas as pd
import scipy as sp
import seaborn as sns
from sklearn import preprocessing
from sklearn.model_selection import KFold
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
sns.set(context='paper')

#import your classifiers here

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Diagnosing the Maastricht Flu You are given the early data for an outbreak of a dangerous virus originating from a group of primates being kept in a Maastricht biomedical research lab in the basement of Henri-Paul Spaaklaan building, this virus is dubbed the "Maastricht Flu".You have the medical records of $n$ number of patients in `flu_train.csv`. There are two general types of patients in the data, flu patients and healthy (this is recorded in the column labeled `flu`, a 0 indicates the absences of the virus and a 1 indicates presence). Notice that the dataset is unbalanced and you can expect a similar imbalance in the testing set.**Your task:** build a model to predict if a given patient has the flu. Your goal is to catch as many flu patients as possible without misdiagnosing too many healthy patients.**The deliverable:** submit your final solution via Kaggle competition using the `flu_test.csv` data.Maastricht Gemeente will use your model to diagnose sets of future patients (held by us). You can expect that there will be an increase in the number of flu patients in any groups of patients in the future.Here are some benchmarks for comparison and for expectation management. Notice that because the dataset is unbalanced, we expect that there is going to be a large difference in the accuracy for each class, thus `accuracy` is a metric that might be misleading in this case (see also below). That's why the baselines below are based on the expected accuracy **per class** and also they give you an estimate for the AUROC on all patients in the testing data. This is the score you see in the Kaggle submission as well.**Baseline Model:** - ~50% expected accuracy on healthy patients in training data- ~50% expected accuracy on flu patients in training data- ~50% expected accuracy on healthy patients in testing data (future data, no info on the labels)- ~50% expected accuracy on flu patients in testing data (future data, no info on the labels)- ~50% expected AUROC on all patients in testing data (future data, no info on the labels)**Reasonable Model:** - ~70% expected accuracy on healthy patients in training data- ~55% expected accuracy on flu patients, in training data- ~70% expected accuracy on healthy patients in testing data (future data, no info on the labels, to be checked upon your submission)- ~57% expected accuracy on flu patients, in testing data (future data, no info on the labels, to be checked upon your submission)- ~65% expected AUROC on all patients, in testing data (future data, no info on the labels, to be checked from Kaggle)**Grading:**Your grade will be based on:1. your model's ability to out-perform the benchmarks (they are kind of low, so we won't care much about this)2. your ability to carefully and thoroughly follow the data analysis pipeline3. the extend to which all choices are reasonable and defensible by methods you have learned in this class  Step 1: Read the data, clean and explore the dataThere are a large number of missing values in the data. Nearly all predictors have some degree of missingness. Not all missingness are alike: NaN in the `'pregnancy'` column is meaningful and informative, as patients with NaN's in the pregnancy column are males, where as NaN's in other predictors may appear randomly. **What do you do?:** We make no attempt to interpret the predictors and we make no attempt to model the missing values in the data in any meaningful way. We replace all missing values with 0.However, it would be more complete to look at the data and allow the data to inform your decision on how to address missingness. For columns where NaN values are informative, you might want to treat NaN as a distinct value; You might want to drop predictors with too many missing values and impute the ones with few missing values using a model. There are many acceptable strategies here, as long as the appropriateness of the method in the context of the task and the data is discussed.<jupyter_code>#Train
df = pd.read_csv('data/flu_train.csv')
df = df[~np.isnan(df['flu'])]
df.head()
#Test
df_test = pd.read_csv('data/flu_test.csv')
df_test.head()
#What's up in each set

x = df.values[:, :-1]
y = df.values[:, -1]

x_test = df_test.values[:, :-1]

print('x train shape:', x.shape)
print('x test shape:', x_test.shape)
print('train class 0: {}, train class 1: {}'.format(len(y[y==0]), len(y[y==1])))
def fill_bin_mean(dataframe, category, bin_category, bin_size):  # Only works for numerical categories
    df_meancat = pd.DataFrame(columns=['interval', 'mean'])
    for age_bin, subset in dataframe.groupby(pd.cut(dataframe[bin_category], np.arange(0, dataframe[bin_category].max()+1, bin_size), include_lowest=True)):
        row = [age_bin, subset[category].mean()]
        df_meancat.loc[len(df_meancat)] = row
    for index, row_df in df[df[category].isna()].iterrows():
        for _, row_meancat in df_meancat.iterrows():
            if row_df[bin_category] in row_meancat['interval']:
                dataframe.at[index, category] = row_meancat['mean']
    return dataframe
def fill_bin_num(dataframe, feature, bin_feature, bin_size, stat_measure, min_bin=None, max_bin=None, default_val='No'):
    if min_bin is None:
        min_bin = dataframe[bin_feature].min()
    if max_bin is None:
        max_bin = dataframe[bin_feature].max()
    new_dataframe = dataframe.copy()
    df_meancat = pd.DataFrame(columns=['interval', 'stat_measure'])
    for num_bin, subset in dataframe.groupby(pd.cut(dataframe[bin_feature], np.arange(min_bin, max_bin+bin_size, bin_size), include_lowest=True)):
        if stat_measure is 'mean':
            row = [num_bin, subset[feature].mean()]
        elif stat_measure is 'mode':
            mode_ar = subset[feature].mode().values
            if len(mode_ar) > 0:
                row = [num_bin, mode_ar[0]]
            else:
                row = [num_bin, default_val]
        else:
            raise Exception('Unknown statistical measure: ' + stat_measure)
        df_meancat.loc[len(df_meancat)] = row
    for index, row_df in dataframe[dataframe[feature].isna()].iterrows():
        for _, row_meancat in df_meancat.iterrows():
            if row_df[bin_feature] in row_meancat['interval']:
                new_dataframe.at[index, feature] = row_meancat['stat_measure']
    return new_dataframe


def make_dummy_cols(dataframe, column, prefix, drop_dummy):
    dummy = pd.get_dummies(dataframe[column], prefix=prefix)
    dummy = dummy.drop(columns=prefix+'_'+drop_dummy)
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns=column)
    return dataframe

def cleaning(dataframe_raw):
    dataframe = dataframe_raw.copy()

    dataframe.loc[(dataframe['Age']<=20) & (dataframe['MaritalStatus'].isna()), 'MaritalStatus'] = 'NeverMarried'
    dataframe.at[dataframe['MaritalStatus'].isna(), 'MaritalStatus'] = 'Married'  # For now this is hardcoded

    dataframe = dataframe.drop(columns=['HHIncome'])

    dataframe.loc[dataframe['HHIncomeMid'].isna(), 'HHIncomeMid'] = dataframe['HHIncomeMid'].mean()

    dataframe.loc[dataframe['Poverty'].isna(), 'Poverty'] = dataframe['Poverty'].mean()

    dataframe.loc[dataframe['HomeRooms'].isna(), 'HomeRooms'] = dataframe['HomeRooms'].mean()

    dataframe.loc[dataframe['HomeOwn'].isna(), 'HomeOwn'] = dataframe['HomeOwn'].mode().values[0]

    dataframe.loc[(dataframe['Work'].isna()) & (dataframe['Education'].isna()) & (dataframe['Age']<=20), 'Work'] = 'NotWorking'

    dataframe.loc[dataframe['Work'].isna(), 'Work'] = dataframe['Work'].mode().values[0]

    fill_bin_mean(dataframe, 'Weight', 'Age', 2)

    dataframe = dataframe.drop(columns=['HeadCirc'])

    for index, row in dataframe.iterrows():
        if np.isnan(row['Height']) and not np.isnan(row['Length']):
            dataframe.at[index, 'Height'] = row['Length']
    fill_bin_mean(dataframe, 'Height', 'Age', 2)

    dataframe = dataframe.drop(columns=['Length'])

    for index, row in dataframe[dataframe['BMI'].isna()].iterrows():
        dataframe.at[index, 'BMI'] = row['Weight'] / ((row['Height']/100)**2)

    dataframe = dataframe.drop(columns='BMICatUnder20yrs')

    dataframe = dataframe.drop(columns='BMI_WHO')

    fill_bin_mean(dataframe, 'Pulse', 'Age', 10)





    dataframe = dataframe.drop(columns='BPSys1')

    dataframe = dataframe.drop(columns='BPDia1')

    dataframe = dataframe.drop(columns='BPSys2')

    dataframe = dataframe.drop(columns='BPDia2')

    dataframe = dataframe.drop(columns='BPSys3')

    dataframe = dataframe.drop(columns='BPDia3')

    dataframe = dataframe.drop(columns=['Testosterone'])




    
    dataframe = dataframe.drop(columns=['UrineVol1'])

    dataframe = dataframe.drop(columns=['UrineFlow1'])

    dataframe = dataframe.drop(columns=['UrineVol2'])

    dataframe = dataframe.drop(columns=['UrineFlow2'])

    dataframe.loc[dataframe['Diabetes'].isna(), 'Diabetes'] = 'No'

    dataframe['DiabetesAge'] = dataframe['DiabetesAge'].fillna(0)






    dataframe['nPregnancies'] = dataframe['nPregnancies'].fillna(0)

    dataframe['nBabies'] = dataframe['nBabies'].fillna(0)

    dataframe['Age1stBaby'] = dataframe['Age1stBaby'].fillna(0)


    # Making dummy variables
    dummy = pd.get_dummies(dataframe['Gender'])
    dummy = dummy.drop(columns='female')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='Gender')

    dataframe['white'] = np.where(dataframe['Race1'] == 'white',1,0)
    dataframe = dataframe.drop(columns=['Race1'])

    dummy = pd.get_dummies(dataframe['Education'], prefix='education')
    dummy = dummy.drop(columns='education_8th Grade')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='Education')
    
    dummy = pd.get_dummies(dataframe['MaritalStatus'], prefix='maritalstatus')
    dummy = dummy.drop(columns='maritalstatus_Separated')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='MaritalStatus')

    dummy = pd.get_dummies(dataframe['HomeOwn'], prefix='homeown')
    dummy = dummy.drop(columns='homeown_Other')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='HomeOwn')

    dummy = pd.get_dummies(dataframe['Work'], prefix='work')
    dummy = dummy.drop(columns='work_Looking')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='Work')

    dummy = pd.get_dummies(dataframe['Diabetes'], prefix='diabetes')
    dummy = dummy.drop(columns='diabetes_No')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='Diabetes')

    dummy = pd.get_dummies(dataframe['Marijuana'], prefix='marijuana')
    dummy = dummy.drop(columns='marijuana_No')
    dataframe = pd.concat([dataframe, dummy], axis=1)
    dataframe = dataframe.drop(columns='Marijuana')

    dataframe = dataframe.fillna(0)

    return dataframe<jupyter_output><empty_output><jupyter_text>Step 2: Model ChoiceThe first task is to decide which classifier to use (from the ones that we learned this block), i.e. which one would best suit our task and our data. Note that our data are heavily unbalanced, thus you need to do some exploration on how different classifiers handle inbalances in the data (we will discuss some of these techniques during week 3 lecture).It would be possible to do brute force model comparison here - i.e. tune all models and compare which does best with respect to various benchmarks. However, it is also reasonable to do a first round of model comparison by running models (with out of the box parameter settings) on the training data and eliminating some models which performed very poorly.Let the best model win!<jupyter_code>def expected_score(model, x_test, y_test):
    overall = 0
    class_0 = 0
    class_1 = 0
    for i in range(100):
        sample = np.random.choice(len(x_test), len(x_test))
        x_sub_test = x_test[sample]
        y_sub_test = y_test[sample]
        
        overall += model.score(x_sub_test, y_sub_test)
        class_0 += model.score(x_sub_test[y_sub_test==0], y_sub_test[y_sub_test==0])
        class_1 += model.score(x_sub_test[y_sub_test==1], y_sub_test[y_sub_test==1])

    return pd.Series([overall / 100., 
                      class_0 / 100.,
                      class_1 / 100.],
                      index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])

score = lambda model, x_test, y_test: pd.Series([model.score(x_test, y_test), 
                                                 model.score(x_test[y_test==0], y_test[y_test==0]),
                                                 model.score(x_test[y_test==1], y_test[y_test==1])], 
                                                index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])
### fancy models that solve the problem
num = df.select_dtypes(include = 'number')
num = num.set_index('ID')
num.fillna(0, inplace=True)
from sklearn import tree
from sklearn.model_selection import train_test_split


train, test = train_test_split(num, test_size=0.2, stratify=num['flu'])

X_train = train.drop('flu', axis=1)
X_test = test.drop('flu', axis=1)
y_train = train['flu']
y_test = test['flu']<jupyter_output><empty_output><jupyter_text>Random Forests<jupyter_code>from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier()
rf.fit(X_train, y_train)
from sklearn.model_selection import cross_val_score

rf_s = cross_val_score(rf, num.drop('flu', axis=1), num['flu'], scoring='roc_auc', cv=10)
sns.violinplot(rf_s)
rf_s
def random_forest(it, X_train, y_train, score='roc_auc',cv=5):
    df = pd.DataFrame()
    models = {}
    for i in it:
        rf = RandomForestClassifier(max_depth=4,n_estimators=i, class_weight='balanced')
        rf.fit(X_train, y_train)
        models[i] = rf
        df[i] = cross_val_score(rf, num.drop('flu', axis=1), num['flu'], scoring=score, cv=cv)
    return df,models
rf, models = random_forest(range(22,202,10), X_train, y_train)
ax = sns.violinplot(data = rf)
ax.set(xlabel='n_estimators', ylabel='AUROC')
rf.describe()
score(models[112], X_test, y_test)
data = cleaning(df).select_dtypes(include = 'number')

train, test = train_test_split(data, stratify=data['flu'], test_size=0.2)

X_train = train.drop('flu', axis=1)
X_test = test.drop('flu', axis=1)
y_train = train['flu']
y_test = test['flu']
rf, models = random_forest(range(2,202,10), X_train, y_train, score='roc_auc', cv=10)
ax = sns.boxplot(data = rf)
ax.set(xlabel='n_estimators', ylabel='AUROC')
rf.describe()
def flu_accuracy(models):
    accuracy = {}
    for k, mod in models.items():
        accuracy[k] = score(mod[1],X_test,y_test)[2]
    plt.scatter(accuracy.keys(), accuracy.values())
flu_accuracy(models)
score(models[72], X_test, y_test)<jupyter_output><empty_output><jupyter_text>Now I'm gonna try to improve the class weight<jupyter_code>rf = RandomForestClassifier(max_depth=4,n_estimators=62, class_weight={0:0.05, 1:0.95})
rf.fit(X_train, y_train)
ma = cross_val_score(rf, num.drop('flu', axis=1), num['flu'], scoring='roc_auc', cv=10)
sns.violinplot(data = ma)
ma
score(rf, X_test, y_test)<jupyter_output><empty_output><jupyter_text>Final results and printing<jupyter_code>num_test = cleaning(df_test.set_index('ID')).select_dtypes(include='number')
#num_test = num_test.set_index('ID')
#num_test.fillna(0, inplace=True) 
#num_test = cleaning(df_test)
num_test
a = np.array(rf.predict(num_test))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_c_rf_b.csv', index=False)
result.to_csv('result2.csv', index=False)<jupyter_output><empty_output><jupyter_text>Random Forests with Grid Search<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
w0 = 0.0599

cw = []
for i in np.linspace(start = 0.001, stop = 0.15, num = 10):
    cw.append({0:i, 1:1-i})
cw.append('balanced')

param_grid = {
    'n_estimators' : [20,50,70,110, 130, 150, 200],
    'max_features' : ['auto', 'sqrt'], 
    'max_depth':[3, 5, 7, 10, 15, None],
    'criterion' : ['gini', 'entropy'],
    'min_samples_split' : [2, 3, 5, 7],
    'min_samples_leaf' : [2, 3, 5, 7],
    'class_weight': cw
}
print(param_grid)
rfs = RandomForestClassifier()
rfs_random = RandomizedSearchCV(rfs, param_grid, scoring='balanced_accuracy', cv=5, return_train_score=True, random_state=42, n_jobs=-1, n_iter=1500)
best = pd.DataFrame(rfs_random.cv_results_).sort_values(by='mean_test_score', ascending=False)
best
rfs_random.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>rfs_random.fit(X_train, y_train)<jupyter_code>best.iloc[2]
params = rfs_random.best_params_
notify_me('The best parameters are {} giving an average ROC AUC score of {:.4f}'.format(params, rfs_random.best_score_))
rf = RandomForestClassifier(**best.iloc[2]['params'])
rf.fit(X_train, y_train)
rf = RandomForestClassifier(class_weight= {0:0.05,1:0.95}, criterion='gini', max_depth = 3, max_features = 'auto', min_samples_leaf = 5, min_samples_split=3, n_estimators=70)

rf.fit(data.drop('flu', axis=1), data['flu'])
rf.fit(X_train, y_train)
a = np.array(rf.predict(ntest))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_c_rf_gs_ba_n_2.csv', index=False)<jupyter_output><empty_output><jupyter_text>PCA<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

pca = PCA(n_components = 10)
pca_ready = pd.DataFrame(StandardScaler().fit_transform(num.drop('flu', axis=1)))
pca.fit_transform(pca_ready)
pca_ready
print(sum(pca.explained_variance_ratio_))<jupyter_output><empty_output><jupyter_text>Support Vector Machine<jupyter_code>from sklearn.svm import SVC

clf = SVC(class_weight='balanced', C=10)
clf.fit(X_train, y_train)
score(clf, X_test, y_test)
ma = cross_val_score(clf, X_train, y_train, scoring='roc_auc', cv=10)
sns.boxplot(data = ma)
ma.mean()
a = np.array(clf.predict(num_test))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_svm_b.csv', index=False)<jupyter_output><empty_output><jupyter_text>SVM grid search<jupyter_code>cw = []
from sklearn.model_selection import cross_val_score

for i in np.linspace(start = 0.006, stop = 0.08, num = 5):
    cw.append({0:i, 1:1-i})
cw.append('balanced')
C = [x for x in np.linspace(start = 0.2, stop = 1.5, num = 5)]
C.append(1)

param_grid = {
    'C':C,
    'kernel':['linear', 'rbf'],
    'degree':[2,3,4,5,6,7,8],
    'gamma':['auto'],
    'shrinking':[True, False],
    'class_weight': cw
}
from sklearn.model_selection import RandomizedSearchCV
sv = SVC()
sv_r = RandomizedSearchCV(sv, param_grid, scoring=scorel, cv=3, return_train_score=True, verbose=2, random_state=42, n_jobs=-2, n_iter=600)
sv_r.fit(X_train, y_train)
params = sv_r.best_params_
notify_me('The best parameters are {} giving an average Balanced Accuract of {:.4f}'.format(params, sv_r.best_score_))
res = pd.DataFrame(sv_r.cv_results_).sort_values(by=['mean_test_score'], ascending=False)
res[res['rank_test_score']<10]
d = res.iloc[1]['params']
#d['kernel'] = 'linear'
j = 0.06
d['class_weight'] = {0:j, 1:1-j}
lol = SVC(**d)
lol.fit(ndata.drop('flu', axis=1), ndata['flu'])
a = np.array(lol.predict(ntest))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_svm_gs_ba_7.csv', index=False)<jupyter_output><empty_output><jupyter_text>XGBoost<jupyter_code>import xgboost as xgb
data_dmatrix = xgb.DMatrix(data=X_train,label=y_train)
xg_reg = xgb.XGBClassifier(objective ='reg:logistic', learning_rate = 0.1, max_depth = 5, alpha = 100, n_estimators = 50, eval_metric='aucpr', scale_pos_weight=25)
xg_reg.fit(X_train, y_train)
score(xg_reg, X_test, y_test)
ma = cross_val_score(xg_reg, X_train, y_train, scoring='roc_auc', cv=10)
sns.boxplot(data = ma)
ma.mean()
a = np.array(xg_reg.predict(num_test))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_xg.csv', index=False)<jupyter_output><empty_output><jupyter_text>XGBoost with Grid Search<jupyter_code>xg_c = xgb.XGBClassifier(max_depth=3)
param_grid = {
    'objective':['reg:squarederror', 'reg:logistic', 'binary:logistic'],
    'scale_pos_weight':[20,21,22],
    'colsample_bytree':[0.3],
    'eval_metric':['aucpr', 'auc', 'mae', 'map'],
    'alpha':[5, 10, 20],
    'n_estimators': [5, 10, 25, 40, 50, 100, 125],
    'learning_rate': [0.05, 0.1, 0.15]
}
xg_s = GridSearchCV(xg_c, param_grid, scoring='balanced_accuracy', cv=3, return_train_score=True)
xg_s.fit(ndata.drop('flu', axis=1), ndata['flu'])
params = xg_s.best_params_
print('The best parameters are {} giving an average ROC AUC score of {:.4f}'.format(params, xg_s.best_score_))
xg = xg_s.best_estimator_
#score(xg_s.best_estimator_, X_test, y_test)
#data_dmatrix = xgb.DMatrix(data=ndata.drop('flu', axis=1),label=ndata['flu'])
#xg= xgb.XGBClassifier(objective ='reg:squarederror', colsample_bytree = 0.3,alpha=10, max_depth=3, n_estimators=100, scale_pos_weight=22, learning_rate=0.1, eval_metric='aucpr')
#xg.fit(ndata.drop('flu', axis=1).values, ndata['flu'])
#xg= xgb.XGBClassifier(objective ='reg:squarederror', colsample_bytree = 0.3,alpha=10, max_depth=3, n_estimators=50, scale_pos_weight=20, learning_rate=0.1, eval_metric='aucpr')
#xg.fit(ndata.drop('flu', axis=1).values, ndata['flu'])
#xg= xgb.XGBClassifier(objective ='reg:squarederror', colsample_bytree = 0.3,alpha=10, max_depth=3, n_estimators=10, scale_pos_weight=22, learning_rate=0.2, eval_metric='aucpr')
#xg.fit(ndata.drop('flu', axis=1).values, ndata['flu'])
#xg= xgb.XGBClassifier(objective ='binary:logistic', colsample_bytree = 0.3,alpha=20, max_depth=3, n_estimators=10, scale_pos_weight=20, learning_rate=0.1, eval_metric='mae')
#xg.fit(ndata.drop('flu', axis=1).values, ndata['flu'])
xg= xgb.XGBClassifier(objective ='reg:logistic', colsample_bytree = 0.3,alpha=5, max_depth=3, n_estimators=50, scale_pos_weight=20, learning_rate=0.05, eval_metric='aucpr')
xg.fit(ndata.drop('flu', axis=1).values, ndata['flu'])
params = xg_s.best_params_
print('The best parameters are {} giving an average ROC AUC score of {:.4f}'.format(params, xg_s.best_score_))
xg = xg_s.best_estimator_
a = np.array(xg.predict(ntest.values))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_xg_7.csv', index=False)<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(class_weight={0:0.06,1:0.94}, C=10)
clf.fit(X_train, y_train)
score(clf, X_test, y_test)
ma = cross_val_score(clf, X_train, y_train, scoring='roc_auc', cv=10)
sns.boxplot(data = ma)
ma.mean()
a = np.array(clf.predict(num_test))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_lr.csv', index=False)<jupyter_output><empty_output><jupyter_text>LR with Grid Search<jupyter_code>import telegram 
import json
import os
def notify_me(message='Done'):
    filename = os.environ['HOME']+'/.telegram'
    with open(filename) as f:
        json_blob = f.read()
        credentials = json.loads(json_blob)
    bot = telegram.Bot(token=credentials['api_key'])
    bot.send_message(chat_id=credentials['chat_id'], text=message)
notify_me()
cw = []
for i in np.linspace(start = 0.001, stop = 0.4, num = 40):
    cw.append({0:i, 1:1-i})
cw.append('balanced')
w0 = 0.0599
param_grid = {
    'C':[x for x in np.linspace(start = 0.001, stop = 100, num = 40)],
    'penalty':['l1', 'l2', 'elasticnet'],
    'max_iter':[10, 100, 1000, 10000],
    'class_weight': cw
}
lr = LogisticRegression()
lr_r = GridSearchCV(lr, param_grid, scoring='balanced_accuracy', cv=5, return_train_score=True, verbose=0, n_jobs=-1)
lr_r.fit(X_train, y_train)
lr_r.best_params_
params = lr_r.best_params_

notify_me('The best parameters are {} giving an average ROC AUC score of {:.4f}'.format(params, lr_r.best_score_))
ndata.drop('flu', axis=1)
lr_f = LogisticRegression(**lr_r.best_params_)
lr_f.fit(ndata.drop('flu', axis=1), ndata['flu'])
a = np.array(lr_r.best_estimator_.predict(num_test))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_lr_gs_n_ba_9.csv', index=False)
np.mean([1,3,4])
def scorel(model, X_test, y_test):
    return 0.6*np.mean(cross_val_score(model,X_train,y_train,scoring='balanced_accuracy', cv=5))+0.4*score(model,X_test, y_test)[1]
scorel(lr_r.best_estimator_, X_test, y_test)
ma = cross_val_score(lr_r.best_estimator_, X_train, y_train, scoring='roc_auc', cv=10)
sns.boxplot(data = ma)
ma.mean()<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>clf = tree.DecisionTreeClassifier()
param_grid = {
    'max_features' : ['auto', 'sqrt'], 
    'max_depth':[3, 4, 5,6, 7, 10, None],
    'criterion' : ['gini', 'entropy'],
    'min_samples_split' : [2, 3,4, 5, 7],
    'min_samples_leaf' : [2, 3,4, 5, 7],
    'class_weight': cw
}
clf_r = RandomizedSearchCV(clf, param_grid, scoring='balanced_accuracy', cv=3, return_train_score=True, verbose=0, n_iter=2000)
clf_r.fit(X_train, y_train)
params = clf_r.best_params_
print('The best parameters are {} giving an average ROC AUC score of {:.4f}'.format(params, clf_r.best_score_))
score(models[3], X_test, y_test)
a = np.array(clf_r.best_estimator_.predict(ntest))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_dt_n.csv', index=False)<jupyter_output><empty_output><jupyter_text>With normalization<jupyter_code>from sklearn import preprocessing
norm = preprocessing.MinMaxScaler()
data_n = norm.fit_transform(data.drop('flu', axis=1))
ndata = pd.DataFrame(data_n, index = data.index)
ndata['flu'] = data['flu']
ntest = pd.DataFrame(norm.fit_transform(num_test), index=num_test.index)

train, test = train_test_split(ndata, stratify=ndata['flu'])

X_train = train.drop('flu', axis=1)
X_test = test.drop('flu', axis=1)
y_train = train['flu']
y_test = test['flu']
rfs_n = RandomForestClassifier()
rfs_n_random = RandomizedSearchCV(rfs_n, param_grid, scoring='balanced_accuracy', cv=3, return_train_score=True, random_state=42, n_jobs=-1, n_iter=1000)
rfs_n_random.fit(X_train, y_train)
params = rfs_n_random.best_params_
print('The best parameters are {} giving an average ROC AUC score of {:.4f}'.format(params, rfs_n_random.best_score_))
rf_n = RandomForestClassifier(n_estimators=70, min_samples_split=2, min_samples_leaf=7, max_features='auto', max_depth=3, criterion='gini', class_weight={0:0.1,1:0.9})
rf_n.fit(train.drop('flu', axis=1), train['flu'])
a = np.array(rfs_n_random.best_estimator_.predict(ntest))
result = pd.DataFrame(np.array([num_test.index, a], dtype=np.int32).T, columns=['ID', 'Prediction'])
result.to_csv('result_rf_n.csv', index=False)<jupyter_output><empty_output><jupyter_text>On evaluation AUROCAs mentioned abbove, we will use the accuracy scores for each class and for the whole dataset, as well as the AUROC score from Kaggle platform. You can coimpute AUROC locally (e.g. on your train/validation set) by calling the relevant scikit learn function:<jupyter_code>###AUROC locally

#score = roc_auc_score(real_labels, predicted_labels)

#real_labels: the ground truth (0 or 1)
#predicted_labels: labels predicted by your algorithm (0 or 1)<jupyter_output><empty_output><jupyter_text>Accuracy (per class)Below there is a function that will be handy for your models. It computes the accuracy per-class, based on a model you pass as parameter and a dataset (split to x/y)<jupyter_code>def extended_score(model, x_test, y_test):
    overall = 0
    class_0 = 0
    class_1 = 0
    for i in range(100):
        sample = np.random.choice(len(x_test), len(x_test))
        x_sub_test = x_test[sample]
        y_sub_test = y_test[sample]
        
        overall += model.score(x_sub_test, y_sub_test)
        class_0 += model.score(x_sub_test[y_sub_test==0], y_sub_test[y_sub_test==0])
        class_1 += model.score(x_sub_test[y_sub_test==1], y_sub_test[y_sub_test==1])

    return pd.Series([overall / 100., 
                      class_0 / 100.,
                      class_1 / 100.],
                      index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])
#same job as before, but faster?

score = lambda model, x_val, y_val: pd.Series([model.score(x_val, y_val), 
                                                 model.score(x_val[y_val==0], y_val[y_val==0]),
                                                 model.score(x_val[y_val==1], y_val[y_val==1])], 
                                                index=['overall accuracy', 'accuracy on class 0', 'accuracy on class 1'])
#same job as before, but faster?

score1 = lambda model, x_val, y_val: model.score(x_val[y_val==1], y_val[y_val==1])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13924435290542594, 'fraction_numerical': 0.02056251477192153, 'mean_word_length': 5.137173642768338, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '398159', 'license': 'MIT', 'n_tokens_mistral': 10787, 'n_tokens_neox': 10262, 'n_words': 2443, 'path': 'models.ipynb', 'repo_name': 'mavrokefalidis/MaastrichtFlu', 'size': '29617'}