Shard generator
{'login': 'marianna13', 'id': 43296932, 'node_id': 'MDQ6VXNlcjQzMjk2OTMy', 'avatar_url': 'https://avatars.githubusercontent.com/u/43296932?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/marianna13', 'html_url': 'https://github.com/marianna13', 'followers_url': 'https://api.github.com/users/marianna13/followers', 'following_url': 'https://api.github.com/users/marianna13/following{/other_user}', 'gists_url': 'https://api.github.com/users/marianna13/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/marianna13/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/marianna13/subscriptions', 'organizations_url': 'https://api.github.com/users/marianna13/orgs', 'repos_url': 'https://api.github.com/users/marianna13/repos', 'events_url': 'https://api.github.com/users/marianna13/events{/privacy}', 'received_events_url': 'https://api.github.com/users/marianna13/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi, thanks!\r\n\r\n> I was using Hugging Face datasets to process some very large datasets and found that it would be quite handy to have a feature that will allow to "split" these large datasets into chunks with equal size\r\n\r\n`map`, the method we use for processing in `datasets`, already does that if `batched=True`. And you can control the batch size with `batch_size`.\r\n\r\n> Even better - be able to run through these chunks one by one in simple and convenient way\r\n\r\nIt\'s not hard to do this "manually" with the existing API:\r\n```python\r\nbatch_size = <BATCH_SIZE>\r\nfor i in range(len(dset) // batch_size)\r\n    shard = dset[i * batch_size:(i+1) * batch_size] # a dict of lists\r\n    shard = Dataset.from_dict(shard)\r\n```\r\n(should be of similar performance to your implementation)\r\n\r\nStill, I think an API like that could be useful if implemented efficiently (see [this](https://discuss.huggingface.co/t/why-is-it-so-slow-to-access-data-through-iteration-with-hugginface-dataset/20385) discussion to understand what\'s the issue with `select`/`__getitem__` on which your implementation relies on), which can be done with `pa.Table.to_reader` in PyArrow 8.0.0+, .\r\n\r\n@lhoestq @albertvillanova wdyt? We could use such API to efficiently iterate over the batches in `map` before processing them.'
 'The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4798). All of your documentation changes will be reflected on that endpoint.'
 "This is more efficient since it doesn't bring the data in memory:\r\n```python\r\nfor i in range(len(dset) // batch_size)\r\n    start = i * batch_size\r\n    end = min((i+1) * batch_size, len(dset))\r\n    shard = dset.select(range(start, end))\r\n```\r\n\r\n@marianna13 can you give more details on when it would be handy to have this shard generator ?"
 "> This is more efficient since it doesn't bring the data in memory:\r\n> \r\n> ```python\r\n> for i in range(len(dset) // batch_size)\r\n>     start = i * batch_size\r\n>     end = min((i+1) * batch_size, len(dset))\r\n>     shard = dset.select(range(start, end))\r\n> ```\r\n> \r\n> @marianna13 can you give more details on when it would be handy to have this shard generator ?\r\n\r\nSure! I used such generator when I needed to process a very large dataset  (>1TB) in parallel, I've found out empirically that it's much more efficient to do that by processing only one part of the dataset with the shard generator. I tried to use a map with batching but it causesd oom errors, I tried to use the normal shard and here's what I came up with. So I thought it might be helpful to someone else!"
 "I see thanks ! `map` should work just fine even at this scale, feel free to open an issue if you'd like to discuss your OOM issue.\r\n\r\nRegarding `shard_generator`, since it is pretty straightforward to get shards I'm not sure we need that extra Dataset method"]
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/4798', 'html_url': 'https://github.com/huggingface/datasets/pull/4798', 'diff_url': 'https://github.com/huggingface/datasets/pull/4798.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/4798.patch', 'merged_at': None}
Hi everyone! I was using Hugging Face datasets to process some very large datasets and found that it would be quite handy to have a feature that will allow to "split" these large datasets into chunks with equal size. Even better - be able to run through these chunks one by one in simple and convenient way. So I decided to add the method called shard_generator() to the main Dataset class. It works similar to shard method but it returns a generator of datasets with equal size (defined by shard_size attribute).
Example:
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="validation")
>>> ds
Dataset({
      features: ['text', 'label'],
      num_rows: 1066
})
>>> next(ds.shard_generator(300))
Dataset({
      features: ['text', 'label'],
      num_rows: 300
})
```
I hope it can be helpful to someone. Thanks!
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4798/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}