[Question] Create Apache Arrow dataset from raw text file
{'login': 'mrm8488', 'id': 3653789, 'node_id': 'MDQ6VXNlcjM2NTM3ODk=', 'avatar_url': 'https://avatars.githubusercontent.com/u/3653789?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/mrm8488', 'html_url': 'https://github.com/mrm8488', 'followers_url': 'https://api.github.com/users/mrm8488/followers', 'following_url': 'https://api.github.com/users/mrm8488/following{/other_user}', 'gists_url': 'https://api.github.com/users/mrm8488/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/mrm8488/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/mrm8488/subscriptions', 'organizations_url': 'https://api.github.com/users/mrm8488/orgs', 'repos_url': 'https://api.github.com/users/mrm8488/repos', 'events_url': 'https://api.github.com/users/mrm8488/events{/privacy}', 'received_events_url': 'https://api.github.com/users/mrm8488/received_events', 'type': 'User', 'site_admin': False}
[]
[]
We store every dataset in the Arrow format. This is convenient as it supports nested types and memory mapping. If you are curious feel free to check the [pyarrow documentation](https://arrow.apache.org/docs/python/)

You can use this library to load your covid papers by creating a dataset script. You can find inspiration from the ones we've already written in `/datasets`. Here is a link to the steps to [add a dataset](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset) Hello @mrm8488 and @lhoestq 

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Thanks :) > Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Sure, to get a dataset in arrow format you can either:
- [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
- OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
- OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)
 > > Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?
> 
> Sure, to get a dataset in arrow format you can either:
> 
>     * [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
> 
>     * OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
> 
>     * OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)

Links were broken. 

Updated links provided as below
- [load from local files (txt, json, csv)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-or-remote-files)
- [load from python data (dict, pandas)](https://huggingface.co/docs/datasets/loading_datasets.html#from-in-memory-data)
- [create your own dataset script](https://huggingface.co/docs/datasets/loading_datasets.html#using-a-custom-dataset-loading-script)

Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/192/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}