<jupyter_start><jupyter_text>Ray Tune -Ray Tune with Sklearn Hyperparameter Tuning© 2019-2022, Anyscale. All Rights Reserved![Anyscale Academy](../images/AnyscaleAcademyLogo.png) Scikit-Learn is one of the most widely used tools in the ML community for working with data, offering dozens of easy-to-use machine learning algorithms. However, to achieve high performance for these algorithms, you often need to perform **model selection**. Model selection is way to elect the best performant model, after tuning over a set of parameters.`tune-sklearn` is a module that integrates Ray Tune's hyperparameter tuning and scikit-learn's Classifier API. `tune-sklearn` has two APIs: [TuneSearchCV](https://docs.ray.io/en/latest/tune/api_docs/sklearn.htmltunesearchcv-docs) and [TuneGridSearchCV](https://docs.ray.io/en/latest/tune/api_docs/sklearn.htmltunesearchcv-docs). They are drop-in replacements for scikit-learn's [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html) and [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=gridsearchcvsklearn.model_selection.GridSearchCV), so you only need to change less than five lines in a standard scikit-Learn script to use Tune's replacement API.Let's compare Tune's scikit-learn APIs to the standard scikit-learn `GridSearchCV`. For this example, we'll be using `TuneGridSearchCV` with a stochastic gradient descent (SGD) [SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html).To start out, include the import statement to get tune-scikit-learn’s grid search cross validation interface.We need to install a few libraries.<jupyter_code># !pip install tune-sklearn
from sklearn.model_selection import GridSearchCV
# Import Tune's replacements
from ray.tune.sklearn import TuneGridSearchCV
from ray.tune.sklearn import TuneSearchCV

# Other relevant imports
from sklearn.model_selection import train_test_split

# Use the stochastic gradient descent (SGD) classifier
from sklearn.linear_model import SGDClassifier

# import the classification dataset
from sklearn import datasets
from sklearn.datasets import make_classification
import numpy as np<jupyter_output>/usr/local/anaconda3/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:7: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import (to_datetime, Int64Index, DatetimeIndex, Period,
/usr/local/anaconda3/lib/python3.8/site-packages/statsmodels/tsa/base/tsa_model.py:7: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  from pandas import (to_datetime, Int64Index, DatetimeIndex, Period,<jupyter_text>Create classification data using `sklearn.datasets`. To start with, with we using a small dataset of 11K rows and 1k columns. As an excercise you can increase the number and see the difference between using regular scikit-learn and tune-scikit-learn.<jupyter_code>def create_classification_data() -> (np.ndarray, np.ndarray):
    X, y = make_classification(
        n_samples=11000,
        n_features=1000,
        n_informative=50,
        n_redundant=0,
        n_classes=10,
        class_sep=2.5)
    return X, y<jupyter_output><empty_output><jupyter_text>Create the classifcation data, training and test data sets, and define our hyperparametergrid.<jupyter_code>X, y = create_classification_data()
# Split the dataset into train and test sets
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=1000)

# Example parameters grid to tune from SGDClassifier
parameter_grid = {"alpha": [1e-4, 1e-1, 1], "epsilon": [0.01, 0.1]}<jupyter_output><empty_output><jupyter_text>Use Sklearn to train the modelRun this on a single node<jupyter_code>%%time
# n_jobs=-1 enables use of all cores
sklearn_search = GridSearchCV(SGDClassifier(), parameter_grid, n_jobs=-1, verbose=True)
sklearn_search.fit(x_train, y_train)
print("Best hyperparameters found were: ", sklearn_search.best_params_)<jupyter_output>Best hyperparameters found were:  {'alpha': 1, 'epsilon': 0.01}<jupyter_text>Use Ray's Tune's drop-in replacement And from here, we proceed just like how we would in scikit-learn’s interface!The `SGDClassifier` has a `partial_fit` API, which enables it to stop fitting to the data for a certain hyperparameter configuration. If the estimator does not support early stopping, we would fall back to a parallel grid search. As you can see, the setup here is exactly how you would do it for scikit-learn, except we replace `GridSearchCV` with `TuneGridSearchCV`. Now, let's try fitting a model.  Start Ray on the local hostThis will start Ray on the localhost. If you have a cluster, then you can supply the arguments to `ray.init(...)`.Check the [documentation](https://docs.ray.io/en/latest/package-ref.html?highlight=ray.initray-init) for the specific arguments. Some examples: * `ray.init()`: Start Ray locally and all the relevant processes * `ray.init(address="localhost:6379")`: connect to the localhost cluster at a specified port (for the head node) * `ray.init(address="ray://192.168.3.11:10001")`: connect to an existing remote cluster, using the URI<jupyter_code>import ray
ray.init(ignore_reinit_error=True)<jupyter_output>2022-03-16 16:04:28,047	INFO services.py:1412 -- View the Ray dashboard at [1m[32mhttp://127.0.0.1:8266[39m[22m<jupyter_text>Note the slight differences we introduced below: * an `early_stopping`, and * a specification of `max_iters` parameterThe ``early_stopping`` parameter allows us to terminate unpromising configurations. If ``early_stopping=True``, ``TuneGridSearchCV`` will default to using Tune's [ASHAScheduler](https://docs.ray.io/en/latest/tune/api_docs/schedulers.htmltune-scheduler-hyperband). You can pass in a custom algorithm - see Tune's documentation on [schedulers](https://docs.ray.io/en/latest/tune/api_docs/schedulers.htmltune-schedulers) for a full list to choose from.``max_iters`` is the maximum number of iterations a given hyperparameter set could run for; it may run for fewer iterations if it is early stopped.<jupyter_code>%%time
tune_search = TuneGridSearchCV(
    SGDClassifier(), parameter_grid, early_stopping=True, 
    max_iters=10, name="AcademyTraining", verbose=1)
tune_search.fit(x_train, y_train)
print("Best hyperparameters found were: ", tune_search.best_params)<jupyter_output>Best hyperparameters found were:  {'alpha': 0.1, 'epsilon': 0.01}<jupyter_text>Using Bayesian OptimizationIn addition to the grid search interface, tune-sklearn also provides an interface, `TuneSearchCV`, for sampling from **distributions of hyperparameters**.In addition, you can easily enable Bayesian optimization over the distributions in only 2 lines of code:<jupyter_code>%%time
digits = datasets.load_digits()
x = digits.data
y = digits.target
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=.2)

clf = SGDClassifier()
parameter_grid = {"alpha": (1e-4, 1), "epsilon": (0.01, 0.1)}

bayopt_tune_search = TuneSearchCV(
    clf,
    parameter_grid,
    search_optimization="bayesian",
    n_trials=3,
    early_stopping=True,
    max_iters=10,
    verbose=1,
)
bayopt_tune_search.fit(x_train, y_train)
print("Best hyperparameters found were: ", bayopt_tune_search.best_params)
ray.shutdown()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09305721492101421, 'fraction_numerical': 0.018717642373556352, 'mean_word_length': 5.793507664562669, 'pattern_counts': {'":': 4, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '440729', 'license': 'Apache-2.0', 'n_tokens_mistral': 2427, 'n_tokens_neox': 2268, 'n_words': 731, 'path': 'ray-tune/03-Ray-Tune-with-Sklearn.ipynb', 'repo_name': 'dmatrix/academy', 'size': '7533'}