<jupyter_start><jupyter_text><NAME>Provide best practices, tips and tricks for single node Dask computing clusters, if you learn something valuable please add it here. This is likely going to be a constantly evolving document.<jupyter_code>import dask
import pandas as pd
import numpy as np
import dask.array as da
import dask.dataframe as dd<jupyter_output><empty_output><jupyter_text>Scheduler<jupyter_code># TODO (rav): document tradeoffs between process and thread pool based local scheduler<jupyter_output><empty_output><jupyter_text>Graph optimizations/changes `optimization.fuse.ave-width` is a useful optimization strategy to effectively control/trade task parallelism for processing locality, thus in some cases reducing communication overhead between workers, this can be particaully important for distributed client/cluster.<jupyter_code>with dask.config.set({"optimization.fuse.ave-width": ...}):
    ...<jupyter_output><empty_output><jupyter_text>Official doc:> Upper limit for width, where width = num_nodes / height, a good measure of parallelizabilityfrom [here](https://docs.dask.org/en/latest/configuration-reference.htmldask.optimization.fuse.ave-width) and [here](https://docs.dask.org/en/latest/optimize.htmldask.optimization.fuse)Example [notebook](https://github.com/dask/dask-examples/blob/4affee9d31bccd327205af90dd495347c8f2f7f7/applications/array-optimization.ipynb).  Data organization  client.rebalance Assuming you have a process-pool/distributed cluster, before computational expensive operation (if possible) it's worth to `persist` data, keep in mind though that `persist` returns `delayed` object, in some cases the chunk/partition/part data distribution among workers can be uneven, which might lead to communication overhead in steps following the `persist` operation. In these cases it's worth to `rebalance` data, to allow for better distribution among workers, for example:<jupyter_code>a = da.from_delayed(...)
a = a.persist() # this operation is asynchronized, returns immediately
client.rebalance(a) # this operation is synchronized, and returns only after persist and rebalance is done
# next operation is computationally expensive and embarrassingly parallel
...<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08139534883720931, 'fraction_numerical': 0.009838998211091235, 'mean_word_length': 6.4072847682119205, 'pattern_counts': {'":': 1, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '46900', 'license': 'Apache-2.0', 'n_tokens_mistral': 634, 'n_tokens_neox': 605, 'n_words': 214, 'path': 'configs/dask_single_node.ipynb', 'repo_name': 'tomwhite/gwas-analysis', 'size': '2243'}