<jupyter_start><jupyter_text>Stratified KFold+XGBoost+EDA Tutorial **<NAME>, Sungkyunkwan University** 2017/10/8 Outline* **1. Introduction*** **2. Data preparation**    * 2.1 Load data    * 2.2 Check for missing values    * 2.3 Split features and targets from the data    * 2.4 Exploratory Visualization* **3. Training/Predicting Pipeline**    * 3.1  Define Gini metric    * 3.2 Drop Unnecessary Features    * 3.3 Stratified KFold    * 3.4 XGBoost* **4. Prediction and submission**    * 4.1 Predict and Submit results  **1. Introduction**This is a full walkthrough for building the machine learning model for Porto Seguro’s Safe Driver Prediction dataset provided by Porto Seguro. Stratified KFold is used due to inbalance of the output variable. XGBoost is used because it is like the winning ticket for classification problem with formatted data. You can check its success on this link. ([XGBoost winning solutions](https://github.com/dmlc/xgboost/tree/master/demomachine-learning-challenge-winning-solutions)) First, I will prepare the data (driver's information and whether the driver initiated auto insurance or not) then I will focus on prediction.For more information on XGBoost, click this link. [XGBoost](https://xgboost.readthedocs.io/en/latest/)<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.model_selection import StratifiedKFold
import xgboost as xgb<jupyter_output><empty_output><jupyter_text>**2. Data Preparation** **2.1 Load Data**<jupyter_code>train=pd.read_csv('../input/train.csv', na_values=-1)
test=pd.read_csv('../input/test.csv', na_values=-1)<jupyter_output><empty_output><jupyter_text>2.2 Check for missing values(NaN)<jupyter_code>train.isnull().values.any()<jupyter_output><empty_output><jupyter_text>Fill it with median value of the columnthis does not harm the distribution of the model  2.3 Split features and targets from the data<jupyter_code>features = train.drop(['id','target'], axis=1).values
targets = train.target.values<jupyter_output><empty_output><jupyter_text>2.3 Exploratory Visualization Distribution of targets<jupyter_code>ax = sns.countplot(x = targets ,palette="Set2")
sns.set(font_scale=1.5)
ax.set_xlabel(' ')
ax.set_ylabel(' ')
fig = plt.gcf()
fig.set_size_inches(10,5)
ax.set_ylim(top=700000)
for p in ax.patches:
    ax.annotate('{:.2f}%'.format(100*p.get_height()/len(targets)), (p.get_x()+ 0.3, p.get_height()+10000))

plt.title('Distribution of 595212 Targets')
plt.xlabel('Initiation of Auto Insurance Claim Next Year')
plt.ylabel('Frequency [%]')
plt.show()<jupyter_output><empty_output><jupyter_text>The plot shows that:- the target is imbalanced- high bias is expected to 0- class weight has to be balanced on training  Correlation matrix<jupyter_code>sns.set(style="white")


# Compute the correlation matrix
corr = train.corr()


# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(220, 10, as_cmap=True)

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(corr, cmap=cmap, vmax=.3, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})

plt.show()<jupyter_output><empty_output><jupyter_text>It can be seen that: - ps\_calc\_\*  features are not related to target at all. - Removing them would prevent the curse of dimensionality.          3. Training/Predicting Pipeline  3.1 Define Gini Metric<jupyter_code># Define the gini metric - from https://www.kaggle.com/c/ClaimPredictionChallenge/discussion/703#5897
def gini(actual, pred, cmpcol = 0, sortcol = 1):
    assert( len(actual) == len(pred) )
    all = np.asarray(np.c_[ actual, pred, np.arange(len(actual)) ], dtype=np.float)
    all = all[ np.lexsort((all[:,2], -1*all[:,1])) ]
    totalLosses = all[:,0].sum()
    giniSum = all[:,0].cumsum().sum() / totalLosses
    
    giniSum -= (len(actual) + 1) / 2.
    return giniSum / len(actual)
 
def gini_normalized(a, p):
    return gini(a, p) / gini(a, a)

def gini_xgb(preds, dtrain):
    labels = dtrain.get_label()
    gini_score = gini_normalized(labels, preds)
    return 'gini', gini_score<jupyter_output><empty_output><jupyter_text>3.2 Drop Unnecessary Columns<jupyter_code>unwanted = train.columns[train.columns.str.startswith('ps_calc_')]

train = train.drop(unwanted, axis=1)  
test = test.drop(unwanted, axis=1)<jupyter_output><empty_output><jupyter_text>3.3 Stratified KFoldStratified KFold is used to keep the distribution of each label consistent for each training batch.<jupyter_code>kfold = 5
skf = StratifiedKFold(n_splits=kfold, random_state=42)<jupyter_output><empty_output><jupyter_text>3.4. XGBoost  Set parameters<jupyter_code># More parameters has to be tuned. Good luck :)
params = {
    'min_child_weight': 10.0,
    'objective': 'binary:logistic',
    'max_depth': 7,
    'max_delta_step': 1.8,
    'colsample_bytree': 0.4,
    'subsample': 0.8,
    'eta': 0.025,
    'gamma': 0.65,
    'num_boost_round' : 700
    }<jupyter_output><empty_output><jupyter_text>4.  Prediction and submission  4.1. Predict and Submit results  Define X and y<jupyter_code>X = train.drop(['id', 'target'], axis=1).values
y = train.target.values
test_id = test.id.values
test = test.drop('id', axis=1)<jupyter_output><empty_output><jupyter_text>Create a submission fileThis is a pipeline originated from [StratifiedShuffleSplit + XGBoost example (0.28)](https://www.kaggle.com/kueipo/stratifiedshufflesplit-xgboost-example-0-28) Original [Simple XGBoost BTB (0.27+)](https://www.kaggle.com/anokas/simple-xgboost-btb-0-27?scriptVersionId=1551232)<jupyter_code>sub = pd.DataFrame()
sub['id'] = test_id
sub['target'] = np.zeros_like(test_id)
for i, (train_index, test_index) in enumerate(skf.split(X, y)):
    print('[Fold %d/%d]' % (i + 1, kfold))
    X_train, X_valid = X[train_index], X[test_index]
    y_train, y_valid = y[train_index], y[test_index]
    # Convert our data into XGBoost format
    d_train = xgb.DMatrix(X_train, y_train)
    d_valid = xgb.DMatrix(X_valid, y_valid)
    d_test = xgb.DMatrix(test.values)
    watchlist = [(d_train, 'train'), (d_valid, 'valid')]

    # Train the model! We pass in a max of 1,600 rounds (with early stopping after 70)
    # and the custom metric (maximize=True tells xgb that higher metric is better)
    mdl = xgb.train(params, d_train, 1600, watchlist, early_stopping_rounds=70, feval=gini_xgb, maximize=True, verbose_eval=100)

    print('[Fold %d/%d Prediciton:]' % (i + 1, kfold))
    # Predict on our test data
    p_test = mdl.predict(d_test, ntree_limit=mdl.best_ntree_limit)
    sub['target'] += p_test/kfold<jupyter_output>[Fold 1/5]
[0]	train-error:0.036447	valid-error:0.036449	train-gini:0.023568	valid-gini:0.030105
Multiple eval metrics have been passed: 'valid-gini' will be used for early stopping.

Will train until valid-gini hasn't improved in 70 rounds.
[100]	train-error:0.036447	valid-error:0.036449	train-gini:0.321717	valid-gini:0.263208
[200]	train-error:0.036443	valid-error:0.036449	train-gini:0.363434	valid-gini:0.272687
[300]	train-error:0.03643	valid-error:0.036449	train-gini:0.39765	valid-gini:0.278288
[400]	train-error:0.036424	valid-error:0.036457	train-gini:0.426342	valid-gini:0.280264
[500]	train-error:0.036418	valid-error:0.036449	train-gini:0.448977	valid-gini:0.279886
Stopping. Best iteration:
[438]	train-error:0.036424	valid-error:0.036457	train-gini:0.435328	valid-gini:0.280578

[Fold 1/5 Prediciton:]
[Fold 2/5]
[0]	train-error:0.036447	valid-error:0.036449	train-gini:0.023179	valid-gini:0.036347
Multiple eval metrics have been passed: 'valid-gini' will be used for early stopping.[...]<jupyter_text>Put submission to csv file<jupyter_code>sub.to_csv('StratifiedKFold.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12924298706289228, 'fraction_numerical': 0.054438324580504675, 'mean_word_length': 5.123921568627451, 'pattern_counts': {'":': 1, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '638595', 'license': 'MIT', 'n_tokens_mistral': 3139, 'n_tokens_neox': 2861, 'n_words': 687, 'path': '10 poer sugero safe driver prediction/stratified-kfold-xgboost-eda-tutorial-0-281.ipynb', 'repo_name': 'MLVPRASAD/KaggleProjects', 'size': '7814'}