Easily create loading script for benchmark comprising multiple huggingface datasets
{'login': 'JoelNiklaus', 'id': 3775944, 'node_id': 'MDQ6VXNlcjM3NzU5NDQ=', 'avatar_url': 'https://avatars.githubusercontent.com/u/3775944?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/JoelNiklaus', 'html_url': 'https://github.com/JoelNiklaus', 'followers_url': 'https://api.github.com/users/JoelNiklaus/followers', 'following_url': 'https://api.github.com/users/JoelNiklaus/following{/other_user}', 'gists_url': 'https://api.github.com/users/JoelNiklaus/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/JoelNiklaus/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/JoelNiklaus/subscriptions', 'organizations_url': 'https://api.github.com/users/JoelNiklaus/orgs', 'repos_url': 'https://api.github.com/users/JoelNiklaus/repos', 'events_url': 'https://api.github.com/users/JoelNiklaus/events{/privacy}', 'received_events_url': 'https://api.github.com/users/JoelNiklaus/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi ! I think the simplest is to copy paste the `_split_generators` code from the other datasets and do a bunch of if-else, as in the glue dataset: https://huggingface.co/datasets/glue/blob/main/glue.py#L467 Ok, I see. Thank you
Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets. 
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4750/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}