<jupyter_start><jupyter_text>LightGBM + Dask                                                                 This notebook shows how to use `lightgbm.dask` to train a LightGBM model on data stored as a [Dask DataFrame](https://docs.dask.org/en/latest/dataframe.html) or [Dask Array](https://docs.dask.org/en/latest/array.html).It uses `LocalCluster` to run on this machine only. If you want to try with a distributed cluster on AWS Fargate, see [the AWS notebook](./aws.ipynb).<jupyter_code>import dask.array as da

from dask.distributed import Client, LocalCluster, wait

from lightgbm.dask import DaskLGBMRegressor<jupyter_output><empty_output><jupyter_text>Create a cluster with 3 workers. Since this is a `LocalCluster`, those workers are just 3 local processes.<jupyter_code>n_workers = 3
cluster = LocalCluster(n_workers=n_workers)
client = Client(cluster)
client.wait_for_workers(n_workers)

print(f"View the dashboard: {cluster.dashboard_link}")<jupyter_output><empty_output><jupyter_text>Click the link above to view a diagnostic dashboard while you run the training code below.<jupyter_code>num_rows = 1e6
num_features = 1e2
num_partitions = 10
rows_per_chunk = num_rows / num_partitions

data = da.random.random((num_rows, num_features), (rows_per_chunk, num_features))

labels = da.random.random((num_rows, 1), (rows_per_chunk, 1))<jupyter_output><empty_output><jupyter_text>Right now, the Dask Arrays `data` and `labels` are lazy. Before training, you can force the cluster to compute them by running `.persist()` and then wait for that computation to finish by `wait()`-ing on them.<jupyter_code>data = data.persist()
labels = labels.persist()
_ = wait(data)
_ = wait(labels)<jupyter_output><empty_output><jupyter_text>With the data set up on the workers, train a model. `lightgbm.dask.DaskLGBMRegressor` has an interface that tries to stay as close as possible to the non-Dask scikit-learn interface to LightGBM (`lightgbm.sklearn.LGBMRegressor`).<jupyter_code>dask_reg = DaskLGBMRegressor(
    silent=False,
    max_depth=5,
    random_state=708,
    objective="regression_l2",
    learning_rate=0.1,
    tree_learner="data",
    n_estimators=10,
    min_child_samples=1,
    n_jobs=-1,
)

dask_reg.fit(
    client=client,
    X=data,
    y=labels,
)<jupyter_output><empty_output><jupyter_text>The model produced by this training run is an instance of `DaskLGBMRegressor`. To get a regular non-Dask model (which can be pickled and saved), run `.to_local()`.<jupyter_code>local_model = dask_reg.to_local()
type(local_model)<jupyter_output><empty_output><jupyter_text>You can visualize this model by looking at a data frame representation of it.<jupyter_code>local_model.booster_.trees_to_dataframe()
preds = dask_reg.predict(data[:100])
type(preds)
preds.compute()
pred_leaf = dask_reg.predict(data[:100], pred_leaf=True).compute()
pred_leaf<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10083160083160084, 'fraction_numerical': 0.009702009702009701, 'mean_word_length': 5.3034934497816595, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '443836', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 995, 'n_tokens_neox': 965, 'n_words': 253, 'path': 'recent-developments-in-lightgbm/local.ipynb', 'repo_name': 'jameslamb/talks', 'size': '2886'}