Make `pyarrow` optional
{'login': 'KOLANICH', 'id': 240344, 'node_id': 'MDQ6VXNlcjI0MDM0NA==', 'avatar_url': 'https://avatars.githubusercontent.com/u/240344?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/KOLANICH', 'html_url': 'https://github.com/KOLANICH', 'followers_url': 'https://api.github.com/users/KOLANICH/followers', 'following_url': 'https://api.github.com/users/KOLANICH/following{/other_user}', 'gists_url': 'https://api.github.com/users/KOLANICH/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/KOLANICH/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/KOLANICH/subscriptions', 'organizations_url': 'https://api.github.com/users/KOLANICH/orgs', 'repos_url': 'https://api.github.com/users/KOLANICH/repos', 'events_url': 'https://api.github.com/users/KOLANICH/events{/privacy}', 'received_events_url': 'https://api.github.com/users/KOLANICH/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639 

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?) Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached.  Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.

**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4980/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}