<jupyter_start><jupyter_text>Pickling cuML Models for PersistenceThis notebook demonstrates simple pickling of both single-GPU and multi-GPU cuML models for persistence<jupyter_code>import warnings
warnings.filterwarnings("ignore", category=FutureWarning)<jupyter_output><empty_output><jupyter_text>Single GPU Model PicklingAll single-GPU estimators are pickleable. The following example demonstrates the creation of a synthetic dataset, training, and pickling of the resulting model for storage. Trained single-GPU models can also be used to distribute the inference on a Dask cluster, which the `Distributed Model Pickling` section below demonstrates.<jupyter_code>from cuml.datasets import make_blobs

X, y = make_blobs(n_samples=50,
                  n_features=10,
                  centers=5,
                  cluster_std=0.4,
                  random_state=0)
from cuml.cluster import KMeans

model = KMeans(n_clusters=5)

model.fit(X)
import pickle

pickle.dump(model, open("kmeans_model.pkl", "wb"))
model = pickle.load(open("kmeans_model.pkl", "rb"))
model.cluster_centers_<jupyter_output><empty_output><jupyter_text>Distributed Model Pickling The distributed estimator wrappers inside of the `cuml.dask` are not intended to be pickled directly. The Dask cuML estimators provide a function `get_combined_model()`, which returns the trained single-GPU model for pickling. The combined model can be used for inference on a single-GPU, and the `ParallelPostFit` wrapper from the [Dask-ML](https://ml.dask.org/meta-estimators.html) library can be used to perform distributed inference on a Dask cluster.<jupyter_code>from dask.distributed import Client
from dask_cuda import LocalCUDACluster

cluster = LocalCUDACluster()
client = Client(cluster)
client
from cuml.dask.datasets import make_blobs

n_workers = len(client.scheduler_info()["workers"].keys())

X, y = make_blobs(n_samples=5000, 
                  n_features=30,
                  centers=5, 
                  cluster_std=0.4, 
                  random_state=0,
                  n_parts=n_workers*5)

X = X.persist()
y = y.persist()
from cuml.dask.cluster import KMeans

dist_model = KMeans(n_clusters=5)
dist_model.fit(X)
import pickle

single_gpu_model = dist_model.get_combined_model()
pickle.dump(single_gpu_model, open("kmeans_model.pkl", "wb"))
single_gpu_model = pickle.load(open("kmeans_model.pkl", "rb"))
single_gpu_model.cluster_centers_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08455882352941177, 'fraction_numerical': 0.00857843137254902, 'mean_word_length': 4.540723981900452, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '104982', 'license': 'Apache-2.0', 'n_tokens_mistral': 783, 'n_tokens_neox': 751, 'n_words': 207, 'path': 'docs/source/pickling_cuml_models.ipynb', 'repo_name': 'st071300/cuML', 'size': '2448'}