add draft of registering function for pandas
{'login': 'lvwerra', 'id': 8264887, 'node_id': 'MDQ6VXNlcjgyNjQ4ODc=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8264887?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lvwerra', 'html_url': 'https://github.com/lvwerra', 'followers_url': 'https://api.github.com/users/lvwerra/followers', 'following_url': 'https://api.github.com/users/lvwerra/following{/other_user}', 'gists_url': 'https://api.github.com/users/lvwerra/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lvwerra/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lvwerra/subscriptions', 'organizations_url': 'https://api.github.com/users/lvwerra/orgs', 'repos_url': 'https://api.github.com/users/lvwerra/repos', 'events_url': 'https://api.github.com/users/lvwerra/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lvwerra/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3912). All of your documentation changes will be reflected on that endpoint.'
 'That\'s cool ! Though I would expect such an integration to only require `huggingface_hub`, not the full `datasets` library. \r\n Indeed if users want to use the `datasets` lib they could just to `Dataset.from_pandas(df).push_to_hub()` already. Therefore I would explore something that doesn\'t not necessarily requires `datasets`.\r\n\r\nFor other could storage solutions (S3, GCS, etc.), pandas allows users to pass URIs like `s3://bucket-name/path/data.csv` to the `read_xxx` and `to_xxx` (for csv, parquet, json, etc). It also support passing the **root directory** like `s3://bucket-name/dataset-dir` instead of a single file name.\r\n\r\nIn the Hugging Face Hub case, we have one dataset = one repository. We can enter pandas\' paradigm by saying one dataset = one repository = one root directory. Here is what we could have:\r\n\r\n### push to Hub:\r\n```python\r\n"""\r\nDemo script for writing a pandas data frame to a CSV file on HF using fsspec-supported pandas APIs\r\n"""\r\nimport pandas as pd\r\n\r\nHF_USER = os.getenv("HF_USER")\r\nHF_TOKEN = os.getenv("HF_TOKEN")\r\n\r\nbooks_df = pd.DataFrame(\r\n    data={"Title": ["Book I", "Book II", "Book III"], "Price": [56.6, 59.87, 74.54]},\r\n    columns=["Title", "Price"],\r\n)\r\n\r\ndataset_name = "books1"\r\n\r\nbooks_df.to_csv(\r\n    f"hf://{HF_USER}/{dataset_name}",\r\n    index=False,\r\n    storage_options={\r\n        "repo_type": "dataset",\r\n        "token": HF_TOKEN,\r\n    },\r\n)\r\n\r\n```\r\n\r\n### load from Hub:\r\n```python\r\n"""\r\nDemo script for reading a CSV file from HF into a pandas data frame using fsspec-supported pandas\r\nAPIs\r\n"""\r\nimport pandas as pd\r\n\r\nHF_USER = os.getenv("HF_USER")\r\nHF_TOKEN = os.getenv("HF_TOKEN")\r\n\r\ndataset_name = "books1"\r\n\r\nbooks_df = pd.read_csv(\r\n    f"hf://{HF_USER}/{dataset_name}",\r\n    storage_options={\r\n        "repo_type": "dataset",\r\n        "token": HF_TOKEN,\r\n    },\r\n)\r\n\r\nprint(books_df)\r\n```\r\n\r\nAnd you could do the same with Parquet data using `read/to_parquet` or other formats. Formats like CSV, Parquet or JSON Lines would work out of the box with `datasets`. This API would also allow anyone to use Dask with the Hugging Face Hub for example.\r\n\r\nWhat do you think ?']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/3912', 'html_url': 'https://github.com/huggingface/datasets/pull/3912', 'diff_url': 'https://github.com/huggingface/datasets/pull/3912.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/3912.patch', 'merged_at': None}
This PR adds a register function for `pandas`. It allows to directly push `DataFrame` objects to the hub and in return loading datasets on the hub from `DataFrame`. The motivation for this integration is to enable the vast number of `pandas` users to be able to easily push `DataFrames` to the hub. 

Here is an example:

```python
import pandas as pd
from datasets import register_pandas

register_pandas()

# push to hub
df = pd.DataFrame.from_dict({"test": [1,2,3]})
df.push_to_hub("my_test")

# load from hub
df_retrieved = pd.DataFrame.load_from_hub("lvwerra/my_test")
```

It follows a similar philosophy as the `tqdm` [integration](https://github.com/tqdm/tqdm#pandas-integration). Also see [this issue](https://github.com/pandas-dev/pandas/issues/46000) on the `pandas` repository.

This is just a rough draft of what such integration could look like but I would like appreciate some feedback on this: is this something you would like to add the library and is this the way to go? cc @lhoestq @albertvillanova @julien-c 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3912/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}