Distributed data parallel training for streaming datasets
{'login': 'cyk1337', 'id': 13767887, 'node_id': 'MDQ6VXNlcjEzNzY3ODg3', 'avatar_url': 'https://avatars.githubusercontent.com/u/13767887?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/cyk1337', 'html_url': 'https://github.com/cyk1337', 'followers_url': 'https://api.github.com/users/cyk1337/followers', 'following_url': 'https://api.github.com/users/cyk1337/following{/other_user}', 'gists_url': 'https://api.github.com/users/cyk1337/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/cyk1337/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/cyk1337/subscriptions', 'organizations_url': 'https://api.github.com/users/cyk1337/orgs', 'repos_url': 'https://api.github.com/users/cyk1337/repos', 'events_url': 'https://api.github.com/users/cyk1337/events{/privacy}', 'received_events_url': 'https://api.github.com/users/cyk1337/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)

<s> EDIT: here is a code example </s>
```python
# ds = ds.with_format("torch")
# dataloader = DataLoader(ds, num_workers=num_workers)
```

EDIT: `with_format("torch")` is not required, now you can just do
```python
dataloader = DataLoader(ds, num_workers=num_workers)
``` @cyk1337 does streaming datasets with multi-gpu works for you? I am testing on one node with multiple gpus, but this is freezing, https://github.com/huggingface/datasets/issues/5123 
In case you could make this work, could you share with me your data-loading codes?
thank you +1 This has been implemented in `datasets` 2.8:
```python
from datasets.distributed import split_dataset_by_node

ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

docs: https://huggingface.co/docs/datasets/use_with_pytorch#distributed
### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training? 

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4694/reactions', 'total_count': 1, '+1': 1, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}