Autoeval config
{'login': 'nazneenrajani', 'id': 3278583, 'node_id': 'MDQ6VXNlcjMyNzg1ODM=', 'avatar_url': 'https://avatars.githubusercontent.com/u/3278583?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/nazneenrajani', 'html_url': 'https://github.com/nazneenrajani', 'followers_url': 'https://api.github.com/users/nazneenrajani/followers', 'following_url': 'https://api.github.com/users/nazneenrajani/following{/other_user}', 'gists_url': 'https://api.github.com/users/nazneenrajani/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/nazneenrajani/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/nazneenrajani/subscriptions', 'organizations_url': 'https://api.github.com/users/nazneenrajani/orgs', 'repos_url': 'https://api.github.com/users/nazneenrajani/repos', 'events_url': 'https://api.github.com/users/nazneenrajani/events{/privacy}', 'received_events_url': 'https://api.github.com/users/nazneenrajani/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['_The documentation is not available anymore as the PR was closed or merged._'
 'Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424'
 "The tests are failing due to the changed metadata:\r\n\r\n```\r\ngot an unexpected keyword argument 'train-eval-index'\r\n```\r\n\r\nI think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function\r\n\r\n@lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414\r\nI need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this?"
 "On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).\r\n\r\nFor me, the required fields include:\r\n\r\n* `config`\r\n* `task`\r\n* `task_id`\r\n* `splits` (train / validation / eval)\r\n* `col_mapping`\r\n* `metrics` (checking that each one has `type`, `name`) \r\n\r\nHere I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.\r\n\r\nWDYT @lhoestq ?"
 "Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it"
 'I ended up improving the metadata validation in this PR x)\r\n\r\nIn particular:\r\n- I added support YAML keys with dashes instead of underscores for `train-eval-index`\r\n- I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want\r\n\r\nLet me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR'
 "Come on windows... I didn't do anything advanced...\r\n\r\nAnyway, will try to fix this when I get back home x)"
 "> Come on windows... I didn't do anything advanced...\r\n> \r\n> Anyway, will try to fix this when I get back home x)\r\n\r\nHehe, thanks!"
 'Thanks, @lhoestq this is great! '
 'Did I just fix it for windows and now it fails on linux ? xD'
 '> Did I just fix it for windows and now it fails on linux ? xD\r\n\r\nLooks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅 '
 'The worst is that the tests pass locally both on my windows and my linux x)'
 "Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types"
 '> Alright thanks for adding the first Autoeval config ! :D\r\n\r\nWoohoo! Thank you so much 🤗 '
 'This is cool!']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/4234', 'html_url': 'https://github.com/huggingface/datasets/pull/4234', 'diff_url': 'https://github.com/huggingface/datasets/pull/4234.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/4234.patch', 'merged_at': datetime.datetime(2022, 5, 5, 18, 20, 58)}
Added autoeval config to imdb as pilot
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4234/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}