Better error message when using the wrong load_from_disk
{'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["We also have other cases where people are lost between Dataset and DatasetDict, maybe let's gather and solve them all here?\r\n\r\nFor instance, I remember that some people thought they would request a single element of a split but are calling this on a DatasetDict. Maybe here also a better error message when the split requested in not in the dict? pointing to the list of split and the fact that this is a datasetdict containing several datasets?"
 'Good idea, let me add a better error message for this case too'
 'As a digression from the topic of this PR, IMHO I think that the difference between Dataset and DatasetDict is an additional abstraction complexity that confuses "typical" end users. I think a user expects a "Dataset" (whatever it contains multiple or a single split) and maybe it could be interesting to try to simplify the user-facing API as much as possible to hide this complexity from the end user.\r\n\r\nI don\'t know your opinion about this, but it might be worth discussing...\r\n\r\nFor example, I really like the line of the solution of using the function `load_from_disk`, which hides the previous mentioned complexity and handles under the hood whether Dataset/DatasetDict instances should be created...'
 "I totally agree, I just haven't found a solution that doesn't imply major breaking changes x)"
 "Yes I would also like to find a better solution. Do we have any solution actually? (even implying breaking changes)\r\n\r\nHere is a proposal for discussion and refined (and potential abandon if it's not good enough):\r\n- let's consider that a DatasetDict is also a Dataset with the various split concatenated one after the other\r\n- let's disallow the use of integers in split names (probably not a very big breaking change)\r\n- when you index with integers you access the examples progressively in split after the other is finished (in a deterministic order)\r\n- when you index with strings/split name you have the same behavior as now (full backward compat)\r\n- let's then also have all the methods of a Dataset on the DatasetDict"
 'The end goal would be to merge both `Dataset` and `DatasetDict` object in a single object that would be (pretty much totally) backward compatible with both.'
 'I like the direction :) I think it can make sense to concatenate them.\r\n\r\nThere are a few things that I we could discuss if we want to merge Dataset and DatasetDict:\r\n1. what happens if you index by a string ? Does it return the column or the split ? We could disallow conflicts between column names and split names to avoid ambiguities. It can be surprising to be able to get a column or a split using the same indexing feature\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(...)\r\ndataset["train"]\r\ndataset["input_ids"]\r\n```\r\n2. what happens when you iterate over the object ? I guess it should iterate over the examples as a Dataset object, but a DatasetDict used to iterate over the splits as they are the dictionary keys. This is a breaking change that we can discuss.\r\n\r\nMoreover regarding your points:\r\n- integers are not allowed as split names already\r\n- it\'s definitely doable to have all the methods. Maybe some of them like `train_test_split` that is currently only available for Dataset can be tweaked to work for a split dataset'
 'Instead of suggesting the use of `Dataset.load_from_disk` and `DatasetDict.load_from_disk`, the error message now suggests to use `datasets.load_from_disk` directly'
 'Merging the error message improvement, feel free to continue the discussion here or in a github issue']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/2437', 'html_url': 'https://github.com/huggingface/datasets/pull/2437', 'diff_url': 'https://github.com/huggingface/datasets/pull/2437.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/2437.patch', 'merged_at': datetime.datetime(2021, 6, 8, 18, 3, 49)}
As mentioned in #2424, the error message when one tries to use `Dataset.load_from_disk` to load a DatasetDict object (or _vice versa_) can be improved. I added a suggestion in the error message to let users know that they should use the other one.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2437/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}