<commit_before>Grid (Hyperparameter) Search
============================


H2O supports two types of grid search -- traditional (or "cartesian") grid search and random grid search.  In a cartesian grid search, users specify a set of values for each hyperparameter that they want to search over, and H2O will train a model for every combination of the hyperparameter values.  This means that if you have three hyperparameters and you specify 5, 10 and 2 values for each, your grid will contain a total of 5*10*2 = 100 models.

In random grid search, the user specifies the hyperparameter space in the exact same way, except H2O will sample uniformly from the set of all possible hyperparameter value combinations.  In random grid search, the user also specifies a stopping criterion, which controls when the random grid search is completed.  The user can tell the random grid search to stop by specifying a maximum number of models or the maximum number of seconds allowed for the search.  The user may also specify a performance-metric-based stopping criterion, which will stop the random grid search when the performance stops improving by a specified amount. 

Once the grid search is complete, the user can query the grid object and sort the models by a particular performance metric (for example, "AUC").  All models are stored in the H2O cluster and are accessible by model id.

Examples of how to perform cartesian and random grid search in all of H2O's APIs follow below.  There are also longer grid search tutorials available for `R <https://github.com/h2oai/h2o-tutorials/blob/master/h2o-open-tour-2016/chicago/grid-search-model-selection.R>`__ and `Python <https://github.com/h2oai/h2o-tutorials/blob/master/h2o-open-tour-2016/chicago/grid-search-model-selection.ipynb>`__.

Quick Links
-----------

- `Grid Search in R <#grid-search-in-r>`__
- `Grid Search in Python <#grid-search-in-python>`__
- `Java API <#grid-search-java-api>`__
- `REST API <#rest-api>`__
- `Supported Grid Search Hyperparameters <#supported-grid-search-hyperparameters>`__
- `Grid Testing <#grid-testing>`__
- `Caveats/In Progress <#caveats-in-progress>`__
- `Additional Documentation <#additional-documentation>`__


Grid Search in R and Python
---------------------------

.. tabs::
   .. group-tab:: R

    Grid search in R provides the following capabilities:

    -  ``H2OGrid class``: Represents the results of the grid search
    -  ``h2o.getGrid(<grid_id>, sort_by, decreasing)``: Displays the
       specified grid
    -  ``h2o.grid()``: Starts a new grid search parameterized by

       -  model builder name (e.g., ``gbm``)
       -  model parameters (e.g., ``ntrees = 100``)
       -  ``hyper_parameters`` attribute for passing a list of hyper
          parameters (e.g., ``list(ntrees = c(1,100), learn_rate = c(0.1, 0.001))``)
       -  ``search_criteria`` optional attribute for specifying a more
          advanced search strategy  

    More about ``search_criteria``:  

    This is a named list of control parameters for smarter hyperparameter search.  The list can include values for: ``strategy``, ``max_models``, ``max_runtime_secs``, ``stopping_metric``, ``stopping_tolerance``, ``stopping_rounds`` and ``seed``. The default value for ``strategy``, "Cartesian", covers the entire space of hyperparameter combinations.  If you want to use cartesian grid search, you can leave the ``search_criteria`` argument unspecified.  Specify the "RandomDiscrete" strategy to perform a random search of all the combinations of your hyperparameters. RandomDiscrete should be usually combined with at least one early stopping criterion, ``max_models`` and/or ``max_runtime_secs``.  Some examples below:

    .. code-block:: r 

        list(strategy = "RandomDiscrete", max_models = 10, seed = 1)
        list(strategy = "RandomDiscrete", max_runtime_secs = 3600)
        list(strategy = "RandomDiscrete", max_models = 42, max_runtime_secs = 28800)
        list(strategy = "RandomDiscrete", stopping_tolerance = 0.001, stopping_rounds = 10)
        list(strategy = "RandomDiscrete", stopping_metric = "misclassification", stopping_tolerance = 0.0005, stopping_rounds = 5)

   .. group-tab:: Python

    -  Class is ``H2OGridSearch``
    -  ``<grid_name>.show()``: Display a list of models (including model
       IDs, hyperparameters, and MSE) explored by grid search (where
       ``<grid_name>`` is an instance of an ``H2OGridSearch`` class)
    -  ``grid_search = H2OGridSearch(<model_type), hyper_params=hyper_parameters)``:
       Start a new grid search parameterized by:

       -  ``model_type`` is the type of H2O estimator model with its
          unchanged parameters
       -  ``hyper_params`` in Python is a dictionary of string parameters
          (keys) and a list of values to be explored by grid search (values)
          (e.g., ``{'ntrees':[1,100], 'learn_rate':[0.1, 0.001]}``
       -  ``search_criteria`` is the optional dictionary for specifying more a
          advanced search strategy


    More about ``search_criteria``:  

    This is a dictionary of control parameters for smarter hyperparameter search.  The dictionary can include values for: ``strategy``, ``max_models``, ``max_runtime_secs``, ``stopping_metric``, ``stopping_tolerance``, ``stopping_rounds`` and ``seed``. The default value for ``strategy``, "Cartesian", covers the entire space of hyperparameter combinations.  If you want to use cartesian grid search, you can leave the ``search_criteria`` argument unspecified.  Specify the "RandomDiscrete" strategy to perform a random search of all the combinations of your hyperparameters. RandomDiscrete should be usually combined with at least one early stopping criterion, ``max_models`` and/or ``max_runtime_secs``.  Some examples below:

    .. code-block:: python

        {'strategy': "RandomDiscrete", 'max_models': 10, 'seed': 1}
        {'strategy': "RandomDiscrete", 'max_runtime_secs': 3600}
        {'strategy': "RandomDiscrete", 'max_models': 42, 'max_runtime_secs': 28800}
        {'strategy': "RandomDiscrete", 'stopping_tolerance': 0.001, 'stopping_rounds': 10}
        {'strategy': "RandomDiscrete", 'stopping_metric': "misclassification", 'stopping_tolerance': 0.0005, 'stopping_rounds': 5}

Saving and Loading a Grid Search
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

H2O supports saving and loading grids even after a cluster wipe or complete cluster restart. The ``save_grid`` function will export a grid and its models into a given folder while the ``load_grid`` function loads a previously saved grid and all its models from the given folder.

There are two modes to save a grid (in both R and Python):

- Use auto-checkpointing and supply the ``export_checkpoitns_dir`` parameter<commit_msg>PUBDEV-7411: added 'parallelism' to grid parameters<commit_after>Grid (Hyperparameter) Search
============================


H2O supports two types of grid search -- traditional (or "cartesian") grid search and random grid search.  In a cartesian grid search, users specify a set of values for each hyperparameter that they want to search over, and H2O will train a model for every combination of the hyperparameter values.  This means that if you have three hyperparameters and you specify 5, 10 and 2 values for each, your grid will contain a total of 5*10*2 = 100 models.

In random grid search, the user specifies the hyperparameter space in the exact same way, except H2O will sample uniformly from the set of all possible hyperparameter value combinations.  In random grid search, the user also specifies a stopping criterion, which controls when the random grid search is completed.  The user can tell the random grid search to stop by specifying a maximum number of models or the maximum number of seconds allowed for the search.  The user may also specify a performance-metric-based stopping criterion, which will stop the random grid search when the performance stops improving by a specified amount. 

Once the grid search is complete, the user can query the grid object and sort the models by a particular performance metric (for example, "AUC").  All models are stored in the H2O cluster and are accessible by model id.

Examples of how to perform cartesian and random grid search in all of H2O's APIs follow below.  There are also longer grid search tutorials available for `R <https://github.com/h2oai/h2o-tutorials/blob/master/h2o-open-tour-2016/chicago/grid-search-model-selection.R>`__ and `Python <https://github.com/h2oai/h2o-tutorials/blob/master/h2o-open-tour-2016/chicago/grid-search-model-selection.ipynb>`__.

Quick Links
-----------

- `Grid Search in R <#grid-search-in-r>`__
- `Grid Search in Python <#grid-search-in-python>`__
- `Java API <#grid-search-java-api>`__
- `REST API <#rest-api>`__
- `Supported Grid Search Hyperparameters <#supported-grid-search-hyperparameters>`__
- `Grid Testing <#grid-testing>`__
- `Caveats/In Progress <#caveats-in-progress>`__
- `Additional Documentation <#additional-documentation>`__


Grid Search in R and Python
---------------------------

.. tabs::
   .. group-tab:: R

    Grid search in R provides the following capabilities:

    -  ``H2OGrid class``: Represents the results of the grid search
    -  ``h2o.getGrid(<grid_id>, sort_by, decreasing)``: Displays the
       specified grid
    -  ``h2o.grid()``: Starts a new grid search parameterized by

       -  model builder name (e.g., ``gbm``)
       -  model parameters (e.g., ``ntrees = 100``)
       -  ``hyper_parameters`` attribute for passing a list of hyper
          parameters (e.g., ``list(ntrees = c(1,100), learn_rate = c(0.1, 0.001))``)
       -  ``search_criteria`` optional attribute for specifying a more
          advanced search strategy  
       -  ``parallelism`` (number of models built in parallel) allows 
          the "leader node" to search the hyperspace and build models in a parallel way which ultimately speeds up grid search on small data

    More about ``search_criteria``:  

    This is a named list of control parameters for smarter hyperparameter search.  The list can include values for: ``strategy``, ``max_models``, ``max_runtime_secs``, ``stopping_metric``, ``stopping_tolerance``, ``stopping_rounds`` and ``seed``. The default value for ``strategy``, "Cartesian", covers the entire space of hyperparameter combinations.  If you want to use cartesian grid search, you can leave the ``search_criteria`` argument unspecified.  Specify the "RandomDiscrete" strategy to perform a random search of all the combinations of your hyperparameters. RandomDiscrete should be usually combined with at least one early stopping criterion, ``max_models`` and/or ``max_runtime_secs``.  Some examples below:

    .. code-block:: r 

        list(strategy = "RandomDiscrete", max_models = 10, seed = 1)
        list(strategy = "RandomDiscrete", max_runtime_secs = 3600)
        list(strategy = "RandomDiscrete", max_models = 42, max_runtime_secs = 28800)
        list(strategy = "RandomDiscrete", stopping_tolerance = 0.001, stopping_rounds = 10)
        list(strategy = "RandomDiscrete", stopping_metric = "misclassification", stopping_tolerance = 0.0005, stopping_rounds = 5)

   .. group-tab:: Python

    -  Class is ``H2OGridSearch``
    -  ``<grid_name>.show()``: Display a list of models (including model
       IDs, hyperparameters, and MSE) explored by grid search (where
       ``<grid_name>`` is an instance of an ``H2OGridSearch`` class)
    -  ``grid_search = H2OGridSearch(<model_type), hyper_params=hyper_parameters)``:
       Start a new grid search parameterized by:

       -  ``model_type`` is the type of H2O estimator model with its
          unchanged parameters
       -  ``hyper_params`` in Python is a dictionary of string parameters
          (keys) and a list of values to be explored by grid search (values)
          (e.g., ``{'ntrees':[1,100], 'learn_rate':[0.1, 0.001]}``
       -  ``search_criteria`` is the optional dictionary for specifying more a
          advanced search strategy
       -  ``parallelism`` (number of models built in parallel) allows 
          the "leader node" to search the hyperspace and build models in a parallel way which ultimately speeds up grid search on small data


    More about ``search_criteria``:  

    This is a dictionary of control parameters for smarter hyperparameter search.  The dictionary can include values for: ``strategy``, ``max_models``, ``max_runtime_secs``, ``stopping_metric``, ``stopping_tolerance``, ``stopping_rounds`` and ``seed``. The default value for ``strategy``, "Cartesian", covers the entire space of hyperparameter combinations.  If you want to use cartesian grid search, you can leave the ``search_criteria`` argument unspecified.  Specify the "RandomDiscrete" strategy to perform a random search of all the combinations of your hyperparameters. RandomDiscrete should be usually combined with at least one early stopping criterion, ``max_models`` and/or ``max_runtime_secs``.  Some examples below:

    .. code-block:: python

        {'strategy': "RandomDiscrete", 'max_models': 10, 'seed': 1}
        {'strategy': "RandomDiscrete", 'max_runtime_secs': 3600}
        {'strategy': "RandomDiscrete", 'max_models': 42, 'max_runtime_secs': 28800}
        {'strategy': "RandomDiscrete", 'stopping_tolerance': 0.001, 'stopping_rounds': 10}
        {'strategy': "RandomDiscrete", 'stopping_metric': "misclassification", 'stopping_tolerance': 0.0005, 'stopping_rounds': 5}

Saving and Loading a Grid Search
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

H2O supports saving and loading grids even after a cluster wipe or complete cluster restart. The ``save_grid`` function will export a grid and its models into a given folder while the ``load_grid`` function loads a previously saved grid and all its models from the given folder.

There are two modes to save a grid (in both R and Python):

- Use auto-checkpointing and supply the ``export_checkpoitns_dir`` parameter
{'fraction_non_alphanumeric': 0.107506817855605, 'fraction_numerical': 0.016506387254198365, 'mean_word_length': 4.0071864893999285, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 29, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1349193', 'n_tokens_mistral': 4061, 'n_tokens_neox': 3718, 'n_words': 1598}