<jupyter_start><jupyter_text>Dask IO workload Specify Dask cluster,<jupyter_code>import dask.distributed
cluster = dask.distributed.LocalCluster(
    n_workers=1,
    threads_per_worker=2,
    memory_limit='10GB',
    ip='0.0.0.0'
)
client = dask.distributed.Client(cluster); client<jupyter_output><empty_output><jupyter_text>Connect pi workload zarr store,<jupyter_code>import dask.array
zarr_store_base_path = 'gs://<bucket>'
xy_zarr_store = dask.array.from_zarr(zarr_store_base_path+'/pi-workload.zarr')
xy_zarr_store<jupyter_output><empty_output><jupyter_text>Calculate pi,<jupyter_code>def calculate_pi(xy_zarr_store):
    """Calculate pi using a Monte Carlo method."""
    xy_inside_circle = (xy_zarr_store ** 2).sum(axis=1) < 1
    pi = 4 * xy_inside_circle.mean()
    return pi
pi = calculate_pi(xy_zarr_store)
%%time
pi.compute()<jupyter_output>CPU times: user 2.29 s, sys: 305 ms, total: 2.6 s
Wall time: 18.5 s<jupyter_text>Python environment,<jupyter_code>pip list<jupyter_output>Package                       Version
----------------------------- --------------------------
adal                          1.2.7
adlfs                         2021.7.1
affine                        2.3.0
aiobotocore                   1.3.3
aiohttp                       3.7.4.post0
aioitertools                  0.7.1
alembic                       1.6.5
amqp                          5.0.6
anyio                         3.2.1
appdirs                       1.4.4
argon2-cffi                   20.1.0
asciitree                     0.3.3
asgiref                       3.4.1
asn1crypto                    1.4.0
astropy                       4.2.1
async-generator               1.10
async-timeout                 3.0.1
attrs                         21.2.0
av                            8.0.3
awscli                        1.19.106
azure-core                    1.14.0
azure-datalake-store          0.0.51
azure-identity                1.6.0
azure-storage-blob            12.8.1
Babel                    [...]
{'fraction_non_alphanumeric': 0.11967951927891837, 'fraction_numerical': 0.05458187280921382, 'mean_word_length': 1.4365853658536585, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '378693', 'license': 'MIT', 'n_tokens_mistral': 703, 'n_tokens_neox': 648, 'n_words': 114, 'path': 'dask-test-workloads/test-dask-io-workload-at-jupyterhub.ipynb', 'repo_name': 'geomar-od/NEMO-cloud-project', 'size': '1997'}