<jupyter_start><jupyter_text>2.3. Dask ParallelismYou are now ready to parallelize your Dask computations with either the *multiprocessing* or *threaded* Dask schedulers.  However, in order to do this effectively, you need to know a bit about how Dask will parallelize your Task Graphs, depending upon available resources and the different schedulers.<jupyter_code>import time
import dask
import dask.multiprocessing<jupyter_output><empty_output><jupyter_text>Example: *Exploiting Parallelism*Let's consider another example where we want to apply two independent functions to a list of numbers.*This time, I will use the `time` package's `sleep` function, which releases the GIL, preventing multi-threading contention.*<jupyter_code>@dask.delayed
def inc(x):
    time.sleep(1)
    return x + 1
@dask.delayed
def dbl(x):
    time.sleep(1)
    return 2*x<jupyter_output><empty_output><jupyter_text>Now, let's consider a short list of data:<jupyter_code>data = [2,5,7,3]<jupyter_output><empty_output><jupyter_text>And from this data, let's construct odd numbers from each element of the list using the operation:    2*x + 1<jupyter_code>%time odd_data = [inc(dbl(x)) for x in data]
odd_data<jupyter_output><empty_output><jupyter_text>And, finally, let's sum up these values using Python's `sum` method:<jupyter_code>%time sum_odds = sum(odd_data)
sum_odds<jupyter_output><empty_output><jupyter_text>Notice!Since `Delayed` objects can be added (`+`), the standard Python `sum` function operates on `Delayed` objects, too!**  What does this Task Graph look like?<jupyter_code>sum_odds.visualize()<jupyter_output><empty_output><jupyter_text>Notice!The `_inner` functions are added to the graph automatically so that Dask can apply the `sum` operation on the `Delayed` object elements.  How long will this take to compute with 2 workers?  Multi-Threading:<jupyter_code>%time sum_odds.compute(scheduler='threads', num_workers=2)<jupyter_output><empty_output><jupyter_text>Multi-Processing:<jupyter_code>%time sum_odds.compute(scheduler='multiprocessing', num_workers=2)<jupyter_output><empty_output><jupyter_text>Did you guess correctly?  Was it obvious?If it was not obvious, what might you do to "clean up" the Task Graph? >  TIP:> > Make sure all of your "substantial" functions are `delayed`!<jupyter_code>delayed_sum = dask.delayed(sum)
%time delayed_sum_odds = delayed_sum(odd_data)
delayed_sum_odds
delayed_sum_odds.visualize()<jupyter_output><empty_output><jupyter_text>Multi-Threading with 2 Workers:<jupyter_code>%time delayed_sum_odds.compute(scheduler='threads', num_workers=2)<jupyter_output><empty_output><jupyter_text>Multi-Processing with 2 Workers:<jupyter_code>%time delayed_sum_odds.compute(scheduler='multiprocessing', num_workers=2)<jupyter_output><empty_output><jupyter_text>Now, let's scale up!  Multi-Threading with 4 Workers:<jupyter_code>%time delayed_sum_odds.compute(scheduler='threads', num_workers=4)<jupyter_output><empty_output><jupyter_text>Multi-Processing with 4 Workers:<jupyter_code>%time delayed_sum_odds.compute(scheduler='multiprocessing', num_workers=4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09980744544287548, 'fraction_numerical': 0.007381258023106547, 'mean_word_length': 8.53211009174312, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '291587', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1069, 'n_tokens_neox': 1036, 'n_words': 263, 'path': 'notebooks/2.3.dask_parallelism.ipynb', 'repo_name': 'pangeo-data/pangeo-tutorial-sea-2018', 'size': '3116'}