Download and prepare as Parquet for cloud storage
{'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4724). All of your documentation changes will be reflected on that endpoint.'
 'Added some docs for dask and took your comments into account\r\n\r\ncc @philschmid if you also want to take a look :)'
 'Just noticed that it would be more convenient to pass the output dir to download_and_prepare directly, to bypass the caching logic which prepares the dataset at `<cache_dir>/<name>/<version>/<hash>/`. And this way the cache is only used for the downloaded files. What do you think ?\r\n\r\n```python \r\n\r\nbuilder = load_datadet_builder("squad")\r\n# or with a custom cache\r\nbuilder = load_datadet_builder("squad", cache_dir="path/to/local/cache/for/downloaded/files")\r\n\r\n# download and prepare to s3\r\nbuilder.download_and_prepare("s3://my_bucket/squad")\r\n```'
 'Might be of interest: \r\nPyTorch and AWS introduced better support for S3 streaming in `torchtext`. \r\n![image](https://user-images.githubusercontent.com/32632186/183354186-a7f005e3-4167-4d80-ad1a-c62dd51ad7b6.png)\r\n'
 'Having thought about it a bit more, I also agree with @philschmid in that it\'s important to follow the existing APIs (pandas/dask), which means we should support the following at some point:\r\n\r\n* remote data files resolution for the packaged modules to support `load_dataset("<format>", data_files="<fs_url>")`\r\n* `to_<format>("<fs_url>")`\r\n* `load_from_disk` and `save_to_disk` already expose the `fs` param, but it would be cool to support specifying `fsspec` URLs directly as the source/destination path (perhaps we can then deprecate `fs` to be fully aligned with pandas/dask)\r\n\r\nIMO these are the two main issues with the current approach:\r\n* relying on the builder API to generate the formatted files results in a non-friendly format due to how our caching works (a lot of nested subdirectories)\r\n* this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running `to_<format>` on streamed/iterable datasets']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/4724', 'html_url': 'https://github.com/huggingface/datasets/pull/4724', 'diff_url': 'https://github.com/huggingface/datasets/pull/4724.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/4724.patch', 'merged_at': None}
Download a dataset as Parquet in a cloud storage can be useful for streaming mode and to use with spark/dask/ray.

This PR adds support for `fsspec` URIs like `s3://...`, `gcs://...` etc. and ads the `file_format` to save as parquet instead of arrow:
```python
from datasets import *

cache_dir = "s3://..."
builder = load_dataset_builder("crime_and_punish", cache_dir=cache_dir)
builder.download_and_prepare(file_format="parquet")
```

credentials to cloud storage can be passed using the `storage_options` argument in `load_dataset_builder`

For consistency with the BeamBasedBuilder, I name the parquet files `{builder.name}-{split}-xxxxx-of-xxxxx.parquet`. I think this is fine since we'll need to implement parquet sharding after this PR, so that a dataset can be used efficiently with dask for example.

Note that images/audio files are not embedded yet in the parquet files, this will added in a subsequent PR

TODO:
- [x] docs
- [x] tests
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4724/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}