Adjusting chunk size for streaming datasets
{'login': 'JoelNiklaus', 'id': 3775944, 'node_id': 'MDQ6VXNlcjM3NzU5NDQ=', 'avatar_url': 'https://avatars.githubusercontent.com/u/3775944?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/JoelNiklaus', 'html_url': 'https://github.com/JoelNiklaus', 'followers_url': 'https://api.github.com/users/JoelNiklaus/followers', 'following_url': 'https://api.github.com/users/JoelNiklaus/following{/other_user}', 'gists_url': 'https://api.github.com/users/JoelNiklaus/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/JoelNiklaus/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/JoelNiklaus/subscriptions', 'organizations_url': 'https://api.github.com/users/JoelNiklaus/orgs', 'repos_url': 'https://api.github.com/users/JoelNiklaus/repos', 'events_url': 'https://api.github.com/users/JoelNiklaus/events{/privacy}', 'received_events_url': 'https://api.github.com/users/JoelNiklaus/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
['Hi ! Data streaming uses `fsspec` to read the data files progressively. IIRC the block size for buffering is 5MiB by default. So every time you finish iterating over a block, it downloads the next one. You can still try to increase the `fsspec` block size for buffering if it can help. To do so you just need to increase `fsspec.spec.AbstractBufferedFile.DEFAULT_BLOCK_SIZE `\r\n\r\nCurrently this is unfortunately done in a single thread, so it blocks the processing to download and uncompress the next block. At one point it would be nice to be able to do that in parallel !'
 'Hi! Thanks for the help, I will try it :)']
**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing. 

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.


{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3499/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}