Merge DatasetDict and Dataset
{'login': 'albertvillanova', 'id': 8515462, 'node_id': 'MDQ6VXNlcjg1MTU0NjI=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8515462?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/albertvillanova', 'html_url': 'https://github.com/albertvillanova', 'followers_url': 'https://api.github.com/users/albertvillanova/followers', 'following_url': 'https://api.github.com/users/albertvillanova/following{/other_user}', 'gists_url': 'https://api.github.com/users/albertvillanova/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/albertvillanova/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/albertvillanova/subscriptions', 'organizations_url': 'https://api.github.com/users/albertvillanova/orgs', 'repos_url': 'https://api.github.com/users/albertvillanova/repos', 'events_url': 'https://api.github.com/users/albertvillanova/events{/privacy}', 'received_events_url': 'https://api.github.com/users/albertvillanova/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}, {'id': 2067400324, 'node_id': 'MDU6TGFiZWwyMDY3NDAwMzI0', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/generic%20discussion', 'name': 'generic discussion', 'color': 'c5def5', 'default': False, 'description': 'Generic discussion on the library'}]
[]
As discussed in #2424 and #2437 (please see there for detailed conversation):
- It would be desirable to improve UX with respect the confusion between DatasetDict and Dataset.
- The difference between Dataset and DatasetDict is an additional abstraction complexity that confuses "typical" end users. 
- A user expects a "Dataset" (whatever it contains multiple or a single split) and maybe it could be interesting to try to simplify the user-facing API as much as possible to hide this complexity from the end user.

Here is a proposal for discussion and refined (and potential abandon if it's not good enough):
- let's consider that a DatasetDict is also a Dataset with the various split concatenated one after the other
- let's disallow the use of integers in split names (probably not a very big breaking change)
- when you index with integers you access the examples progressively in split after the other is finished (in a deterministic order)
- when you index with strings/split name you have the same behavior as now (full backward compat)
- let's then also have all the methods of a Dataset on the DatasetDict

The end goal would be to merge both Dataset and DatasetDict object in a single object that would be (pretty much totally) backward compatible with both.


There are a few things that we could discuss if we want to merge Dataset and DatasetDict:

1. what happens if you index by a string ? Does it return the column or the split ? We could disallow conflicts between column names and split names to avoid ambiguities. It can be surprising to be able to get a column or a split using the same indexing feature
   ```
   from datasets import load_dataset

   dataset = load_dataset(...)
   dataset["train"]
   dataset["input_ids"]
   ```
2. what happens when you iterate over the object ? I guess it should iterate over the examples as a Dataset object, but a DatasetDict used to iterate over the splits as they are the dictionary keys. This is a breaking change that we can discuss.

Moreover regarding your points:

- integers are not allowed as split names already
- it's definitely doable to have all the methods. Maybe some of them like train_test_split that is currently only available for Dataset can be tweaked to work for a split dataset


cc: @thomwolf @lhoestq 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2462/reactions', 'total_count': 2, '+1': 2, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}