<jupyter_start><jupyter_text>Machine Learning with H2O - Tutorial 4b: Classification Models (Ensembles)**Objective**:- This tutorial explains how to create stacked ensembles of classification models for better out-of-bag performance.**Titanic Dataset:**- Source: https://www.kaggle.com/c/titanic/data    **Steps**:1. Build GBM models using random grid search and extract the best one.2. Build DRF models using random grid search and extract the best one. 3. Use model stacking to combining different models.**Full Technical Reference:**- http://docs.h2o.ai/h2o/latest-stable/h2o-py/docs/modeling.html- http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/stacked-ensembles.html<jupyter_code># Import all required modules
import h2o
from h2o.estimators.gbm import H2OGradientBoostingEstimator
from h2o.estimators.random_forest import H2ORandomForestEstimator
from h2o.estimators.deeplearning import H2ODeepLearningEstimator
from h2o.estimators.stackedensemble import H2OStackedEnsembleEstimator
from h2o.grid.grid_search import H2OGridSearch

# Start and connect to a local H2O cluster
h2o.init(nthreads = -1)<jupyter_output>Checking whether there is an H2O instance running at http://localhost:54321..... not found.
Attempting to start a local H2O server...
  Java Version: openjdk version "1.8.0_131"; OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11); OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
  Starting server from /home/joe/anaconda3/lib/python3.6/site-packages/h2o/backend/bin/h2o.jar
  Ice root: /tmp/tmpbf3ymcq4
  JVM stdout: /tmp/tmpbf3ymcq4/h2o_joe_started_from_python.out
  JVM stderr: /tmp/tmpbf3ymcq4/h2o_joe_started_from_python.err
  Server is running at http://127.0.0.1:54321
Connecting to H2O server at http://127.0.0.1:54321... successful.<jupyter_text><jupyter_code># Import Titanic data (local CSV)
titanic = h2o.import_file("kaggle_titanic.csv")
titanic.head(5)
# Convert 'Survived' and 'Pclass' to categorical values
titanic['Survived'] = titanic['Survived'].asfactor()
titanic['Pclass'] = titanic['Pclass'].asfactor()
# Define features (or predictors) manually
features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
# Split the H2O data frame into training/test sets
# so we can evaluate out-of-bag performance
titanic_split = titanic.split_frame(ratios = [0.8], seed = 1234)

titanic_train = titanic_split[0] # using 80% for training
titanic_test = titanic_split[1]  # using the rest 20% for out-of-bag evaluation
titanic_train.shape
titanic_test.shape<jupyter_output><empty_output><jupyter_text>Define Search Criteria for Random Grid Search<jupyter_code># define the criteria for random grid search
search_criteria = {'strategy': "RandomDiscrete", 
                   'max_models': 9,
                   'seed': 1234}<jupyter_output><empty_output><jupyter_text>Step 1: Build GBM Models using Random Grid Search and Extract the Best Model<jupyter_code># define the range of hyper-parameters for GBM grid search
# 27 combinations in total
hyper_params = {'sample_rate': [0.7, 0.8, 0.9],
                'col_sample_rate': [0.7, 0.8, 0.9],
                'max_depth': [3, 5, 7]}
# Set up GBM grid search
# Add a seed for reproducibility
gbm_rand_grid = H2OGridSearch(
                    H2OGradientBoostingEstimator(
                        model_id = 'gbm_rand_grid', 
                        seed = 1234,
                        ntrees = 10000,   
                        nfolds = 5,
                        fold_assignment = "Modulo",               # needed for stacked ensembles
                        keep_cross_validation_predictions = True, # needed for stacked ensembles
                        stopping_metric = 'mse', 
                        stopping_rounds = 15,     
                        score_tree_interval = 1),
                    search_criteria = search_criteria, # full grid search
                    hyper_params = hyper_params)
# Use .train() to start the grid search
gbm_rand_grid.train(x = features, 
                    y = 'Survived', 
                    training_frame = titanic_train)
# Sort and show the grid search results
gbm_rand_grid_sorted = gbm_rand_grid.get_grid(sort_by='auc', decreasing=True)
print(gbm_rand_grid_sorted)
# Extract the best model from random grid search
best_gbm_model_id = gbm_rand_grid_sorted.model_ids[0]
best_gbm_from_rand_grid = h2o.get_model(best_gbm_model_id)
best_gbm_from_rand_grid.summary()<jupyter_output>Model Summary:<jupyter_text>Step 2: Build DRF Models using Random Grid Search and Extract the Best Model<jupyter_code># define the range of hyper-parameters for DRF grid search
# 27 combinations in total
hyper_params = {'sample_rate': [0.5, 0.6, 0.7],
                'col_sample_rate_per_tree': [0.7, 0.8, 0.9],
                'max_depth': [3, 5, 7]}
# Set up DRF grid search
# Add a seed for reproducibility
drf_rand_grid = H2OGridSearch(
                    H2ORandomForestEstimator(
                        model_id = 'drf_rand_grid', 
                        seed = 1234,
                        ntrees = 200,   
                        nfolds = 5,
                        fold_assignment = "Modulo",                # needed for stacked ensembles
                        keep_cross_validation_predictions = True), # needed for stacked ensembles
                    search_criteria = search_criteria, # full grid search
                    hyper_params = hyper_params)
# Use .train() to start the grid search
drf_rand_grid.train(x = features, 
                    y = 'Survived', 
                    training_frame = titanic_train)
# Sort and show the grid search results
drf_rand_grid_sorted = drf_rand_grid.get_grid(sort_by='auc', decreasing=True)
print(drf_rand_grid_sorted)
# Extract the best model from random grid search
best_drf_model_id = drf_rand_grid_sorted.model_ids[0]
best_drf_from_rand_grid = h2o.get_model(best_drf_model_id)
best_drf_from_rand_grid.summary()<jupyter_output>Model Summary:<jupyter_text>Model Stacking<jupyter_code># Define a list of models to be stacked
# i.e. best model from each grid
all_ids = [best_gbm_model_id, best_drf_model_id]
# Set up Stacked Ensemble
ensemble = H2OStackedEnsembleEstimator(model_id = "my_ensemble",
                                       base_models = all_ids)
# use .train to start model stacking
# GLM as the default metalearner
ensemble.train(x = features, 
               y = 'Survived', 
               training_frame = titanic_train)<jupyter_output>stackedensemble Model Build progress: |███████████████████████████████████| 100%<jupyter_text>Comparison of Model Performance on Test Data<jupyter_code>print('Best GBM model from Grid (AUC) : ', best_gbm_from_rand_grid.model_performance(titanic_test).auc())
print('Best DRF model from Grid (AUC) : ', best_drf_from_rand_grid.model_performance(titanic_test).auc())
print('Stacked Ensembles        (AUC) : ', ensemble.model_performance(titanic_test).auc())<jupyter_output>Best GBM model from Grid (AUC) :  0.8892284186401833
Best DRF model from Grid (AUC) :  0.8903106697224344
Stacked Ensembles        (AUC) :  0.8918385536032595
{'fraction_non_alphanumeric': 0.0946154927597357, 'fraction_numerical': 0.03430338816251933, 'mean_word_length': 3.2855421686746986, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '26158', 'license': 'MIT', 'n_tokens_mistral': 2531, 'n_tokens_neox': 2269, 'n_words': 578, 'path': 'introduction_to_machine_learning/py_04b_classification_ensembles.ipynb', 'repo_name': 'woobe/h2o_tutorials', 'size': '7113'}