Ability to split a dataset in multiple files.
{'login': 'Dref360', 'id': 8976546, 'node_id': 'MDQ6VXNlcjg5NzY1NDY=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8976546?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/Dref360', 'html_url': 'https://github.com/Dref360', 'followers_url': 'https://api.github.com/users/Dref360/followers', 'following_url': 'https://api.github.com/users/Dref360/following{/other_user}', 'gists_url': 'https://api.github.com/users/Dref360/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/Dref360/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/Dref360/subscriptions', 'organizations_url': 'https://api.github.com/users/Dref360/orgs', 'repos_url': 'https://api.github.com/users/Dref360/repos', 'events_url': 'https://api.github.com/users/Dref360/events{/privacy}', 'received_events_url': 'https://api.github.com/users/Dref360/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3544/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}