<jupyter_start><jupyter_text>2.4. Dask Profiling & DiagnosticsProfiling Dask operations is critical to understanding how they work and how to improve performance.<jupyter_code>import time
import dask
import dask.multiprocessing<jupyter_output><empty_output><jupyter_text>Example: *Adding Odds*Let's consider the example from the previous notebook, but with time delays determined by the input data:<jupyter_code>@dask.delayed
def add(x, y):
    time.sleep((x + y)/10.0)
    return x + y
@dask.delayed
def inc(x):
    time.sleep(x/10.0)
    return x + 1
@dask.delayed
def dbl(x):
    time.sleep(x/10.0)
    return 2*x
@dask.delayed
def dsum(*args):
    s = sum(*args)
    time.sleep(s/10.0)
    return s
data = [1,3,2,0]
sum_odds = dsum(inc(dbl(x)) for x in data)
sum_odds
sum_odds.visualize()<jupyter_output><empty_output><jupyter_text>Let's profile this with standard Python techniques  Multi-Threading Scheduler:<jupyter_code>%prun sum_odds.compute(scheduler='threads', num_workers=4)<jupyter_output><empty_output><jupyter_text>Was that helpful?  Do you know why it ran the way it did?  Local Scheduler:<jupyter_code>%prun sum_odds.compute(scheduler='single-threaded')<jupyter_output><empty_output><jupyter_text>Was that more helpful?  Do you know why it ran the way it did?  Dask ProfilerUsing the `Bokeh` package, Dask can visualize profiling (or diagnostic) information with the `diagnostics` subpackage.<jupyter_code>from dask.diagnostics import Profiler, visualize
from bokeh.io import output_notebook
output_notebook()

with Profiler() as p:
    sum_odds.compute(scheduler='threads', num_workers=4)
    
visualize(p)<jupyter_output><empty_output><jupyter_text>The ResourceProfiler & CacheProfilerSimilar to the `Profiler` above, you can also import the `ResourceProfiler` to display memory and CPU usage and the `CacheProfiler` to display different metrics of the scheduler.> ```python> from dask.diagnostics import Profiler, ResourceProfiler, CacheProfiler, visualize> from bokeh.io import output_notebook> output_notebook()> > with Profiler() as p, ResourceProfiler() as r, CacheProfiler() as c:>     sum_odds.compute()>     > visualize([r, p, c])```  Dask ProgressBarDask can visualize the progress of computation using the `diagnostics` subpackage's `ProgressBar`.<jupyter_code>from dask.diagnostics import ProgressBar

with ProgressBar():
    sum_odds.compute(scheduler='threads', num_workers=4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10131255127153405, 'fraction_numerical': 0.009433962264150943, 'mean_word_length': 6.258928571428571, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '104756', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 829, 'n_tokens_neox': 812, 'n_words': 225, 'path': 'notebooks/2.4.dask_profiling_and_diagnostics.ipynb', 'repo_name': 'pangeo-data/pangeo-tutorial-sea-2018', 'size': '2438'}