Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations
{'login': 'thomwolf', 'id': 7353373, 'node_id': 'MDQ6VXNlcjczNTMzNzM=', 'avatar_url': 'https://avatars.githubusercontent.com/u/7353373?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/thomwolf', 'html_url': 'https://github.com/thomwolf', 'followers_url': 'https://api.github.com/users/thomwolf/followers', 'following_url': 'https://api.github.com/users/thomwolf/following{/other_user}', 'gists_url': 'https://api.github.com/users/thomwolf/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/thomwolf/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/thomwolf/subscriptions', 'organizations_url': 'https://api.github.com/users/thomwolf/orgs', 'repos_url': 'https://api.github.com/users/thomwolf/repos', 'events_url': 'https://api.github.com/users/thomwolf/events{/privacy}', 'received_events_url': 'https://api.github.com/users/thomwolf/received_events', 'type': 'User', 'site_admin': False}
[{'id': 2067400324, 'node_id': 'MDU6TGFiZWwyMDY3NDAwMzI0', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/generic%20discussion', 'name': 'generic discussion', 'color': 'c5def5', 'default': False, 'description': 'Generic discussion on the library'}]
[]
['As @yoavgo suggested, there should be the possibility to call a function like  nlp.bib that outputs all bibtex ref from the datasets and models actually used and eventually nlp.bib.forreadme that would output the same info + versions numbers so they can be included in a readme.md file.'
 "Actually, double checking with @mariamabarham, we already have this feature I think.\r\n\r\nIt's like this currently:\r\n```python\r\n>>> from nlp import load_dataset\r\n>>> \r\n>>> dataset = load_dataset('glue', 'cola', split='train')\r\n>>> print(dataset.info.citation)\r\n@article{warstadt2018neural,\r\n  title={Neural Network Acceptability Judgments},\r\n  author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R},\r\n  journal={arXiv preprint arXiv:1805.12471},\r\n  year={2018}\r\n}\r\n@inproceedings{wang2019glue,\r\n  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},\r\n  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},\r\n  note={In the Proceedings of ICLR.},\r\n  year={2019}\r\n}\r\n\r\nNote that each GLUE dataset has its own citation. Please see the source to see\r\nthe correct citation for each contained dataset.\r\n```\r\n\r\nWhat do you think @dseddah?"
 'Looks good but why would there be a difference between the ref in the source and the one to be printed? '
 "Yes, I think we should remove this warning @mariamabarham.\r\n\r\nIt's probably a relic of tfds which didn't have the same way to access citations. "]
Meta-datasets are interesting in terms of standardized benchmarks but they also have specific behaviors, in particular in terms of attribution and authorship. It's very important that each specific dataset inside a meta dataset is properly referenced and the citation/specific homepage/etc are very visible and accessible and not only the generic citation of the meta-dataset itself.

Let's take GLUE as an example:

The configuration has the citation for each dataset included (e.g. [here](https://github.com/huggingface/nlp/blob/master/datasets/glue/glue.py#L154-L161)) but it should be copied inside the dataset info so that, when people access `dataset.info.citation` they get both the citation for GLUE and the citation for the specific datasets inside GLUE that they have loaded.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/153/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}