<jupyter_start><jupyter_text>Scale Scikit-Learn with Dask and Joblib  Many Scikit-Learn operations already support parallelism with [joblib](http://joblib.readthedocs.io/) for single-machine parallelism. This lets you train most estimators (anything that accepts an `n_jobs` parameter) using all the cores of your laptop or workstation.Dask can scale these operations by replacing Joblib's single-machine system with a distributed cluster.  This lets you train those estimators using all the cores of your *cluster*, by changing one line of code. This is most useful for training large models on medium-sized datasets. You may have a large model when searching over many hyper-parameters, or when using an ensemble method with many individual estimators. For too small datasets, training times will typically be small enough that cluster-wide parallelism isn't helpful. For too large datasets (larger than a single machine's memory), the scikit-learn estimators may not be able to cope (see below).<jupyter_code>from dask.distributed import Client, progress
client = Client()
client.restart()<jupyter_output><empty_output><jupyter_text>Create a Dataset, Model, and Grid Search<jupyter_code># import dask_ml.joblib  # register the distriubted backend
from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>We'll use scikit-learn to create a pair of small random arrays, one for the features `X`, and one for the target `y`.<jupyter_code>X, y = make_classification(n_samples=1000, random_state=0)
X[:5]<jupyter_output><empty_output><jupyter_text>We'll fit a [Support Vector Classifier](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html), using [grid search](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) to find the best value of the $C$ hyperparameter.<jupyter_code>param_grid = {"C": np.logspace(-3, 1, 30),
              "gamma": [0.05, 0.5, 2],
              "kernel": ['rbf', 'poly', 'sigmoid'],
              "shrinking": [True, False]}

grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                           param_grid=param_grid,
                           return_train_score=False,
                           iid=True,
                           cv=3,
                           n_jobs=-1)<jupyter_output><empty_output><jupyter_text>Train on a Cluster To fit that normally, we'd call```pythongrid_search.fit(X, y)```To fit it using the cluster, we just need to use a context manager provided by joblib.We'll pre-scatter the data to each worker, which can help with performance.<jupyter_code>%%time

import dask_ml.joblib
from sklearn.externals import joblib

with joblib.parallel_backend('dask', scatter=[X, y]):
    grid_search.fit(X, y)<jupyter_output><empty_output><jupyter_text>We fit 1620 different models, one for each hyper-parameter and CV-split combination. The training was distributed across the cluster. At this point, we have a regular scikit-learn model, which can be used for prediction, scoring, etc.<jupyter_code>res = pd.DataFrame(grid_search.cv_results_)
res.head()<jupyter_output><empty_output><jupyter_text>The result is just a Scikit-Learn objectDask didn't replace Scikti-Learn, so everything is the same as you are accustomed to.<jupyter_code>grid_search.score(X, y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08730614589316485, 'fraction_numerical': 0.006605399195864446, 'mean_word_length': 4.608695652173913, 'pattern_counts': {'":': 4, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '109571', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1057, 'n_tokens_neox': 1002, 'n_words': 350, 'path': 'notebook/examples/ml/sklearn-joblib.ipynb', 'repo_name': 'learningdymyr/dask-docker', 'size': '3482'}