<jupyter_start><jupyter_text>Out-of-core Prediction For some estimators, additional data don't improve performance past a certain point.The [learning curve](http://scikit-learn.org/stable/modules/learning_curve.htmllearning-curve) levels off.You may have additional data, but using it in the `fit` step won't make any difference.In these cases, you'll commonly fit a model on a dataset that fits in memory, and use it to predict for datasets that may not.Dask can make the prediction step easier and faster.<jupyter_code>import numpy as np
import dask.array as da
from sklearn.datasets import make_classification
X_train, y_train = make_classification(
    n_features=2, n_redundant=0, n_informative=2,
    random_state=1, n_clusters_per_class=1, n_samples=1000)
N = 100
X = da.concatenate([da.from_array(X_train, chunks=X_train.shape)
                    for _ in range(N)])
y = da.concatenate([da.from_array(y_train, chunks=y_train.shape)
                    for _ in range(N)])<jupyter_output><empty_output><jupyter_text>So `X_train` and `y_train` are regular numpy arrays that we'll use to fit the model.`X` and `y` are large dask arrays that may not fit in memory.<jupyter_code>from sklearn.linear_model import LogisticRegressionCV
clf = LogisticRegressionCV()

clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>With the model, we can make predictions for each observation by mapping the `clf.predict_proba` method over each block. This can then be scheduled to run on your single machine or your cluster.<jupyter_code>yhat = X.map_blocks(clf.predict_proba, dtype=np.float64)
yhat
yhat[:5].compute()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0852994555353902, 'fraction_numerical': 0.009074410163339383, 'mean_word_length': 5.386100386100386, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '439280', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 542, 'n_tokens_neox': 521, 'n_words': 167, 'path': 'docs/source/examples/predict.ipynb', 'repo_name': 'jrbourbeau/dask-ml', 'size': '1653'}