<jupyter_start><jupyter_text>From now on, everything is generic and directly applies to most datasets. We assume that all feature engineering is done at this stage and focus on model tuning. For multi-class problems, you can use `h2o.logloss()` or `h2o.confusion_matrix()` instead of `h2o.auc()` and for regression problems, you can use `h2o.mean_residual_deviance()` or `h2o.mse()`.  Split the data for Machine LearningWe split the data into three pieces: 70% for training, 20% for validation, 10% for final testing. Here, we use random splitting, but this assumes i.i.d. data. If this is not the case (e.g., when events span across multiple rows or data has a time structure), you'll have to sample your data non-randomly.<jupyter_code>train, valid, test = df.split_frame(
    ratios=[0.8,0.1], 
    seed=1234, 
    destination_frames=['train.hex','valid.hex','test.hex']
)<jupyter_output><empty_output><jupyter_text>Establish baseline performanceAs the first step, we'll build some default models to see what accuracy we can expect. Let's use the [AUC metric](http://mlwiki.org/index.php/ROC_Analysis) for this demo, but you can use `h2o.logloss()` and `stopping_metric="logloss"` as well. It ranges from 0.5 for random models to 1 for perfect models.The first model is a default GBM, trained on the 70% training split<jupyter_code>#We only provide the required parameters, everything else is default
#gbm = H2OGradientBoostingEstimator()
#gbm.train(x=predictors, y=response, training_frame=train)

## Show a detailed model summary
#print (gbm)
## Get the AUC on the validation set
#perf = gbm.model_performance(valid)
#print (perf.auc())<jupyter_output><empty_output><jupyter_text>The second model is another default GBM, but trained on 80% of the data (here, we combine the training and validation splits to get more training data), and cross-validated using 4 folds.Note that cross-validation takes longer and is not usually done for really large datasets.<jupyter_code>## rbind() makes a copy here, so it's better to use split_frame with `ratios = c(0.9)` instead above
#cv_gbm = H2OGradientBoostingEstimator(nfolds = 4, seed = 0xDECAF)
#cv_gbm.train(x = predictors, y = response, training_frame = train.rbind(valid))
## Show a detailed summary of the cross validation metrics
## This gives you an idea of the variance between the folds
#cv_summary = cv_gbm.cross_validation_metrics_summary().as_data_frame()
#print(cv_summary) ## Full summary of all metrics
#print(cv_summary.iloc[4]) ## get the row with just the AUCs

## Get the cross-validated AUC by scoring the combined holdout predictions.
## (Instead of taking the average of the metrics across the folds)
#perf_cv = cv_gbm.model_performance(xval=True)
#print (perf_cv.auc())<jupyter_output><empty_output><jupyter_text>Next, we train a GBM with "I feel lucky" parameters.We'll use early stopping to automatically tune the number of trees using the validation AUC. We'll use a lower learning rate (lower is always better, just takes more trees to converge).We'll also use stochastic sampling of rows and columns to (hopefully) improve generalization.<jupyter_code>#gbm_lucky = H2OGradientBoostingEstimator(
#  ## more trees is better if the learning rate is small enough 
#  ## here, use "more than enough" trees - we have early stopping
#  ntrees = 10000,                                                            

#  ## smaller learning rate is better (this is a good value for most datasets, but see below for annealing)
#  learn_rate = 0.01,                                                         

#  ## early stopping once the validation AUC doesn't improve by at least 0.01% for 5 consecutive scoring events
#  stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = "AUC", 

#  ## sample 80% of rows per tree
#  sample_rate = 0.8,                                                       

#  ## sample 80% of columns per split
#  col_sample_rate = 0.8,                                                   

#  ## fix a random number generator seed for reproducibility
#  seed = 1234,                                                             

#  ## score every 10 trees to make early stopping reproducible (it depends on the scoring interval)
#  score_tree_interval = 10)

#gbm_lucky.train(x=predictors, y=response, training_frame=train, validation_frame=valid)<jupyter_output><empty_output><jupyter_text>This model doesn't seem to be better than the previous models:<jupyter_code>#perf_lucky = gbm_lucky.model_performance(valid)
#print (perf_lucky.auc())<jupyter_output><empty_output><jupyter_text>For this small dataset, dropping 20% of observations per tree seems too aggressive in terms of adding regularization. For larger datasets, this is usually not a bad idea. But we'll let this parameter tune freshly below, so no worries.  Hyper-Parameter SearchNext, we'll do real hyper-parameter optimization to see if we can beat the best AUC so far (around 94%).The key here is to start tuning some key parameters first (i.e., those that we expect to have the biggest impact on the results). From experience with gradient boosted trees across many datasets, we can state the following "rules":1. Build as many trees (`ntrees`) as it takes until the validation set error starts increasing.2. A lower learning rate (`learn_rate`) is generally better, but will require more trees. Using `learn_rate=0.02 `and `learn_rate_annealing=0.995` (reduction of learning rate with each additional tree) can help speed up convergence without sacrificing accuracy too much, and is great to hyper-parameter searches. For faster scans, use values of 0.05 and 0.99 instead.3. The optimum maximum allowed depth for the trees (`max_depth`) is data dependent, deeper trees take longer to train, especially at depths greater than 10.4. Row and column sampling (`sample_rate` and `col_sample_rate`) can improve generalization and lead to lower validation and test set errors. Good general values for large datasets are around 0.7 to 0.8 (sampling 70-80 percent of the data) for both parameters. Column sampling per tree (`col_sample_rate_per_tree`) can also be tuned. Note that it is multiplicative with `col_sample_rate`, so setting both parameters to 0.8 results in 64% of columns being considered at any given node to split.5. For highly imbalanced classification datasets (e.g., fewer buyers than non-buyers), stratified row sampling based on response class membership can help improve predictive accuracy.  It is configured with `sample_rate_per_class` (array of ratios, one per response class in lexicographic order).6. Most other options only have a small impact on the model performance, but are worth tuning with a Random hyper-parameter search nonetheless, if highest performance is critical.First we want to know what value of `max_depth` to use because it has a big impact on the model training time and optimal values depend strongly on the dataset.We'll do a quick Cartesian grid search to get a rough idea of good candidate `max_depth` values. Each model in the grid search will use early stopping to tune the number of trees using the validation set AUC, as before.We'll use learning rate annealing to speed up convergence without sacrificing too much accuracy.<jupyter_code>## Depth 10 is usually plenty of depth for most datasets, but you never know
#hyper_params = {'max_depth' : range(1,30,2)}
hyper_params = {'max_depth' : [4,6,8,12,16,20,24,28]} ##faster for larger datasets

#Build initial GBM Model
gbm_grid = H2OGradientBoostingEstimator(
        balance_classes = True,
        ## more trees is better if the learning rate is small enough 
        ## here, use "more than enough" trees - we have early stopping
        ntrees=10000,
        ## smaller learning rate is better
        ## since we have learning_rate_annealing, we can afford to start with a 
        #bigger learning rate
        learn_rate=0.5,
        ## learning rate annealing: learning_rate shrinks by 1% after every tree 
        ## (use 1.00 to disable, but then lower the learning_rate)
        learn_rate_annealing = 0.99,
        ## sample 80% of rows per tree
        sample_rate = 0.8,
        ## sample 80% of columns per split
        col_sample_rate = 0.8,
        ## fix a random number generator seed for reproducibility
        seed = 1234,
        ## score every 10 trees to make early stopping reproducible 
        #(it depends on the scoring interval)
        score_tree_interval = 10, 
        ## early stopping once the validation AUC doesn't improve by at least 0.01% for 
        #5 consecutive scoring events
        stopping_rounds = 5,
        stopping_metric = "AUC",
        stopping_tolerance = 1e-4)

#Build grid search with previously made GBM and hyper parameters
grid = H2OGridSearch(gbm_grid,hyper_params,
                         grid_id = 'depth_grid',
                         search_criteria = {'strategy': "Cartesian"})


#Train grid search
grid.train(x=predictors, 
           y=response,
           training_frame = train,
           validation_frame = valid)
## by default, display the grid search results sorted by increasing logloss (since this is a classification task)
print (grid)
## sort the grid models by decreasing AUC
sorted_grid = grid.get_grid(sort_by='auc',decreasing=True)
print(sorted_grid)
max_depths = sorted_grid.sorted_metric_table()['max_depth'][0:2]
new_max = int(max(max_depths, key=int))
new_min = int(min(max_depths, key=int))

print ("MaxDepth", new_max)
print ("MinDepth", new_min)<jupyter_output>MaxDepth 6
MinDepth 4<jupyter_text>Now that we know a good range for max_depth, we can tune all other parameters in more detail. Since we don't know what combinations of hyper-parameters will result in the best model, we'll use random hyper-parameter search to "let the machine get luckier than a best guess of any human".<jupyter_code># create hyperameter and search criteria lists (ranges are inclusive..exclusive))
hyper_params_tune = {'max_depth' : list(range(new_min,new_max+1,1)),
                'sample_rate': [x/100. for x in range(20,101)],
                'col_sample_rate' : [x/100. for x in range(60,101)],
                'col_sample_rate_per_tree': [x/100. for x in range(20,101)],
                'col_sample_rate_change_per_level': [x/100. for x in range(90,211)],
                'min_rows': [2**x for x in range(0,int(math.log(train.nrow,2)-1)+1)],
                'nbins': [2**x for x in range(4,11)],
                'nbins_cats': [2**x for x in range(4,13)],
                'min_split_improvement': [0,1e-8,1e-6,1e-4,1e-2],
                'histogram_type': ["UniformAdaptive","QuantilesGlobal","RoundRobin"]}
search_criteria_tune = {'strategy': "RandomDiscrete",
                   'max_runtime_secs': 3600,  ## limit the runtime to 60 minutes
                   'max_models': 100,  ## build no more than 100 models
                   'seed' : 1234,
                   'stopping_rounds' : 5,
                   'stopping_metric' : "AUC",
                   'stopping_tolerance': 1e-3
                   }
gbm_final_grid = H2OGradientBoostingEstimator(distribution='bernoulli',
                    #keep_cross_validations = True,
                    balance_classes = True,
                    ## more trees is better if the learning rate is small enough 
                    ## here, use "more than enough" trees - we have early stopping
                    ntrees=20000,
#                    nfolds=5,
                    ## smaller learning rate is better
                    ## since we have learning_rate_annealing, we can afford to start with a 
                    #bigger learning rate
                    learn_rate=0.1,
                    ## learning rate annealing: learning_rate shrinks by 1% after every tree 
                    ## (use 1.00 to disable, but then lower the learning_rate)
                    learn_rate_annealing = 0.99,
                    ## score every 10 trees to make early stopping reproducible 
                    #(it depends on the scoring interval)
                    score_tree_interval = 10,
                    ## fix a random number generator seed for reproducibility
                    seed = 1234,
                    ## early stopping once the validation AUC doesn't improve by at least 0.01% for 
                    #5 consecutive scoring events
                    stopping_rounds = 5,
                    stopping_metric = "AUC",
                    stopping_tolerance = 1e-4)
            
#Build grid search with previously made GBM and hyper parameters
final_grid = H2OGridSearch(gbm_final_grid, hyper_params = hyper_params_tune,
                                    grid_id = 'final_grid',
                                    search_criteria = search_criteria_tune)
#Train grid search
final_grid.train(x=predictors, 
           y=response,
           ## early stopping based on timeout (no model should take more than 1 hour - modify as needed)
           max_runtime_secs = 3600, 
           training_frame = train,
           validation_frame = valid)

print (final_grid)<jupyter_output>gbm Grid Build progress: |████████████████████████████████████████████████| 100%
    col_sample_rate col_sample_rate_change_per_level col_sample_rate_per_tree  \
0              0.87                             1.11                     0.68   
1              0.97                             0.97                     0.39   
2              0.92                              1.7                     0.65   

    histogram_type max_depth min_rows min_split_improvement nbins nbins_cats  \
0  UniformAdaptive         6   4096.0                1.0E-6   128        128   
1       RoundRobin         5    128.0                1.0E-6    16         64   
2       RoundRobin         5   2048.0                   0.0   512        512   

  sample_rate           model_ids             logloss  
0        0.87  final_grid_model_3  0.2573324970528222  
1        0.57  final_grid_model_2  0.2596684503675237  
2        0.64  final_grid_model_1  0.2604051826705137<jupyter_text>We can see that the best models have even better validation AUCs than our previous best models, so the random grid search was successful!<jupyter_code>## Sort the grid models by AUC
sorted_final_grid = final_grid.get_grid(sort_by='auc',decreasing=True)

print (sorted_final_grid)<jupyter_output>col_sample_rate col_sample_rate_change_per_level col_sample_rate_per_tree  \
0              0.97                             0.97                     0.39   
1              0.87                             1.11                     0.68   
2              0.92                              1.7                     0.65   

    histogram_type max_depth min_rows min_split_improvement nbins nbins_cats  \
0       RoundRobin         5    128.0                1.0E-6    16         64   
1  UniformAdaptive         6   4096.0                1.0E-6   128        128   
2       RoundRobin         5   2048.0                   0.0   512        512   

  sample_rate           model_ids                 auc  
0        0.57  final_grid_model_2  0.8594015641514952  
1        0.87  final_grid_model_3  0.8583873614135801  
2        0.64  final_grid_model_1  0.8556348907443877<jupyter_text>Model Inspection and Final Test Set ScoringLet's see how well the best model of the grid search (as judged by validation set AUC) does on the held out test set:<jupyter_code>#Get the best model from the list (the model name listed at the top of the table)
best_model = h2o.get_model(sorted_final_grid.sorted_metric_table()['model_ids'][0])
performance_best_model = best_model.model_performance(test)
print (performance_best_model.auc())
performance_best_model = best_model.model_performance(test)
print (performance_best_model.auc())
params_list = []
for key, value in best_model.params.items():
    params_list.append(str(key)+" = "+str(value['actual']))
params_list

gbm = h2o.get_model(sorted_final_grid.sorted_metric_table()['model_ids'][0])
##get the parameters from the Random grid search model and modify them slightly
#params = gbm.params
#new_params = {"nfolds":5, "model_id":None}
#for key in new_params.keys():
#    params[key]['actual'] = new_params[key] 
gbm_best = gbm #H2OGradientBoostingEstimator()
#for key in params.keys():
#    if key in dir(gbm_best) and getattr(gbm_best,key) != params[key]['actual']:
#        setattr(gbm_best,key,params[key]['actual']) 
print(gbm.params)
gbm_best.train(x=predictors, y=response, training_frame=train, validation_frame=valid)
print (gbm_best.cross_validation_metrics_summary())<jupyter_output>No cross-validation metrics summary for this model
None<jupyter_text>It looks like the winning model performs slightly better on the validation and test sets than during cross-validation on the training set as the mean AUC on the 5 folds is estimated to be only 97.04%, but with a fairly large standard deviation of 0.7%. For small datasets, such a large variance is not unusual. To get a better estimate of model performance, the Random hyper-parameter search could have used `nfolds = 5` (or 10, or similar) in combination with 80% of the data for training (i.e., not holding out a validation set, but only the final test set). However, this would take more time, as `nfolds+1` models will be built for every set of parameters.Instead, to save time, let's just scan through the top 5 models and cross-validate their parameters with `nfolds=5` on the entire dataset:<jupyter_code>#for i in range(5): 
#    gbm = h2o.get_model(sorted_final_grid.sorted_metric_table()['model_ids'][i])
#    #get the parameters from the Random grid search model and modify them slightly
#    params = gbm.params
#    new_params = {"nfolds":5, "model_id":None}
#    for key in new_params.keys():
#        params[key]['actual'] = new_params[key]
#    new_model = H2OGradientBoostingEstimator()
#    for key in params.keys():
#        if key in dir(new_model) and getattr(new_model,key) != params[key]['actual']:
#            setattr(new_model,key,params[key]['actual'])
#    new_model.train(x = predictors, y = response, training_frame = train)  
#    cv_summary = new_model.cross_validation_metrics_summary().as_data_frame()
#    print(gbm.model_id)
#    print(cv_summary.iloc[1]) ## AUC<jupyter_output><empty_output><jupyter_text>The avid reader might have noticed that we just implicitly did further parameter tuning using the "final" test set (which is part of the entire dataset `df`), which is not good practice - one is not supposed to use the "final" test set more than once. Hence, we're not going to pick a different "best" model, but we're just learning about the variance in AUCs. It turns out, for this tiny dataset, that the variance is rather large, which is not surprising.Keeping the same "best" model, we can make test set predictions as follows:<jupyter_code>preds = best_model.predict(test)
preds.head()
preds = gbm_best.predict(test)
preds.head()<jupyter_output>gbm prediction progress: |████████████████████████████████████████████████| 100%<jupyter_text>Note that the label (survived or not) is predicted as well (in the first predict column), and it uses the threshold with the highest F1 score (here: 0.528098) to make labels from the probabilities for survival (`p1`). The probability for death (`p0`) is given for convenience, as it is just `1-p1`.<jupyter_code>print(best_model.model_performance(valid))
print(gbm_best.model_performance(valid))<jupyter_output>ModelMetricsBinomial: gbm
** Reported on test data. **

MSE: 0.07603561763700735
RMSE: 0.27574556684923757
LogLoss: 0.2596684503675237
Mean Per-Class Error: 0.21355126432998306
AUC: 0.8594015641514952
AUCPR: 0.5135066632042186
Gini: 0.7188031283029903

Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.13965869291900493:<jupyter_text>The model and the predictions can be saved to file as follows:<jupyter_code># uncomment if you want to export the best model
# h2o.save_model(best_model, "/tmp/bestModel.csv", force=True)
# h2o.export_file(preds, "/tmp/bestPreds.csv", force=True)
import pandas as pd
dtest = h2o.import_file("../input/test.csv")

predict = gbm_best.predict(dtest)
resultdf = pd.DataFrame()
resultdf = dtest["ID_code"]
resultdf["target"] = predict['predict']
h2o.export_file(resultdf, 'submission.csv', force= True)

print(resultdf.shape)

predict.shape

# print pojo to screen, or provide path to download location
# h2o.download_pojo(best_model)<jupyter_output><empty_output><jupyter_text>Ensembling TechniquesAfter learning above that the variance of the test set AUC of the top few models was rather large, we might be able to turn this into our advantage by using ensembling techniques. The simplest one is taking the average of the predictions (survival probabilities) of the top `k` grid search model predictions (here, we use `k=10`):<jupyter_code>prob = None
k=len(sorted_final_grid.sorted_metric_table()['model_ids'])
for i in range(0,k): 
    gbm = h2o.get_model(sorted_final_grid.sorted_metric_table()['model_ids'][i])
    if (prob is None):
        prob = gbm.predict(dtest)["p1"]
    else:
        prob = prob + gbm.predict(dtest)["p1"]
prob = prob/k
prob.head()<jupyter_output><empty_output><jupyter_text>We can bring those ensemble predictions to our Python session's memory space and use other Python packages.<jupyter_code>from sklearn.metrics import roc_auc_score
# convert prob and test[response] h2oframes to pandas' frames and then convert them each to numpy array
np_array_prob = prob.as_data_frame().as_matrix()
np_array_test = dtest[response].as_data_frame().as_matrix()
probInPy = np_array_prob
labeInPy = np_array_test
# compare true scores (test[response]) to probability scores (prob)
roc_auc_score(labeInPy, probInPy)<jupyter_output>/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:3: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  This is separate from the ipykernel package so we can avoid doing imports until
{'fraction_non_alphanumeric': 0.08320051004144087, 'fraction_numerical': 0.03442779725852726, 'mean_word_length': 3.191639625882802, 'pattern_counts': {'":': 5, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '555825', 'license': 'MIT', 'n_tokens_mistral': 6923, 'n_tokens_neox': 6208, 'n_words': 2310, 'path': 'santander-tryouts.ipynb', 'repo_name': 'cp-sadag/deehive', 'size': '21959'}