<jupyter_start><jupyter_text>Dask Arrays interface is similar to Numpy API<jupyter_code>np_arr = np.random.randint(20, size=20)
np_arr<jupyter_output><empty_output><jupyter_text>In this case, we just need to add an additional attribute `chunks`<jupyter_code>dask_arr = da.random.randint(20, size=20, chunks=5)<jupyter_output><empty_output><jupyter_text>You will see that the array has not been computed till now<jupyter_code>dask_arr<jupyter_output><empty_output><jupyter_text>This is simply because Dask does lazy evaluaion.    You need to call `compute()` to start the execution<jupyter_code>dask_arr.compute()<jupyter_output><empty_output><jupyter_text>Chunks size<jupyter_code>dask_arr.chunks<jupyter_output><empty_output><jupyter_text>Creating arrays from existing data<jupyter_code>dask_arr_from_np = da.from_array(np_arr, chunks=5)
dask_arr_from_np
dask_arr_from_np.compute()<jupyter_output><empty_output><jupyter_text>Dask translates your array operations into a graph to tasks  See : http://docs.dask.org/en/latest/graphviz.html<jupyter_code>dask_arr_from_np.sum().visualize()
dask_arr_from_np.sum().visualize(rankdir="LR")
(dask_arr_from_np+1).visualize(rankdir="LR")
dask_arr_mean = da.mean(dask_arr_from_np)
dask_arr_mean.compute()
dask_arr_mean.visualize(rankdir="LR")<jupyter_output><empty_output><jupyter_text>Another Example<jupyter_code>x = da.random.random(10, chunks=2)
y = da.random.random(10, chunks=2)

sum_x_y = da.add(x, y) #similar to numpy.add
mean_x_y = da.mean(sum_x_y)
sum_x_y.compute()
sum_x_y.visualize()
mean_x_y.visualize()<jupyter_output><empty_output><jupyter_text>An example of out-of-memory computation<jupyter_code>da_arr_large = da.random.randint(10000, size=(50000, 50000),
                                 chunks=(5000, 1000))   
da_sum_large = da_arr_large.sum()<jupyter_output><empty_output><jupyter_text>Get no. bytes using `nbytes` : http://docs.dask.org/en/latest/array-api.htmldask.array.Array.nbytes<jupyter_code>da_arr_large.nbytes<jupyter_output><empty_output><jupyter_text>Convert bytes to GB, 1Gb = 1e+9 bytes<jupyter_code>da_arr_large.nbytes/1e+9
da_sum_large.compute()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11895910780669144, 'fraction_numerical': 0.020910780669144983, 'mean_word_length': 8.40174672489083, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '427254', 'license': 'MIT', 'n_tokens_mistral': 894, 'n_tokens_neox': 852, 'n_words': 118, 'path': 'Section 2/Introduction to Dask Arrays.ipynb', 'repo_name': 'akashmavle5/-Scalable-Data-Analysis-in-Python-with-Dask', 'size': '2152'}