<jupyter_start><jupyter_text>分布式数据结构dask提供的分布式数据结构有3种+ [array](https://docs.dask.org/en/latest/array-api.html)类似numpy的多维数组+ [bag](https://docs.dask.org/en/latest/bag-api.html)类似python的list+ [DataFrame](https://docs.dask.org/en/latest/dataframe-api.html)类似pandas的DataFrame这三种数据结构基本已经可以满足多有需求.更可贵的是他们的接口和对照版基本一致这边只简单介绍,DataFrame会在后面数据科学部分介绍好pandas后再详细介绍<jupyter_code>from dask.distributed import Client
client = Client('localhost:8786')<jupyter_output><empty_output><jupyter_text>Arrayarray的接口参考的是numpy的ndarray,实现的接口范围有:+ 基本的运算符+ reduce相关的操作函数如:`sum()`, `mean()`,`std()`...+ 矩阵计算+ 切片操作+ Array protocols比如`__array__`和 `__array_ufunc__`+ 一些线性代数运算比如:`svd`,`qr`,`solve`,`solve_triangular`,`lstsq`未实现的接口包括:+ 全局排序,但有部分排序+ tolist这样的操作+ shape不确定的序列无法操作+ `np.linalg`中的多数内容<jupyter_code>import dask.array as da
x = da.random.random((100000, 100000), chunks=(1000, 1000))
y = x + x.T - x.mean(axis=0)
y
print(y[0,0].compute())<jupyter_output>1.4348119494257066<jupyter_text>Bag[Bag](https://docs.dask.org/en/latest/bag-api.html)的接口参考的是list,但比python的list接口更丰富,它是最通用的数据类型,类似spark中的rdd,但代价就是性能比不上array和dataframe.<jupyter_code>import dask.bag as db
graph = db.from_sequence(range(10000), partition_size=2).map(lambda x: x**2).sum()
graph.compute()<jupyter_output><empty_output><jupyter_text>DataFame和pandas一样,[DataFame](https://docs.dask.org/en/latest/dataframe-api.html)数据结构通常作为dask的交互层,因此接口最丰富.它可以无缝和pandas交互<jupyter_code>import pandas as pd
import dask.dataframe as dd
pdf1 = pd.DataFrame([{"a":1,"b":2},{"a":3,"b":4}])
pdf2 = pd.DataFrame([{"a":1,"b":2},{"a":2,"b":5}])
ddf1 = dd.from_pandas(pdf1,npartitions=2)
ddf2 = dd.from_pandas(pdf2,npartitions=2)
ddf =  dd.concat([ddf1, ddf2], axis=0)
ddf
z = ddf.groupby(["a"]).count()
z.compute()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1788154897494305, 'fraction_numerical': 0.040432801822323464, 'mean_word_length': 10.262820512820513, 'pattern_counts': {'":': 8, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '338177', 'license': 'MIT', 'n_tokens_mistral': 939, 'n_tokens_neox': 948, 'n_words': 68, 'path': 'src/数据清洗篇/工具介绍/dask/分布式数据结构.ipynb', 'repo_name': 'hsz1273327/TutorialForDataScience', 'size': '1756'}