Fix XTREME PAN-X loading
{'login': 'lvwerra', 'id': 8264887, 'node_id': 'MDQ6VXNlcjgyNjQ4ODc=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8264887?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lvwerra', 'html_url': 'https://github.com/lvwerra', 'followers_url': 'https://api.github.com/users/lvwerra/followers', 'following_url': 'https://api.github.com/users/lvwerra/following{/other_user}', 'gists_url': 'https://api.github.com/users/lvwerra/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lvwerra/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lvwerra/subscriptions', 'organizations_url': 'https://api.github.com/users/lvwerra/orgs', 'repos_url': 'https://api.github.com/users/lvwerra/repos', 'events_url': 'https://api.github.com/users/lvwerra/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lvwerra/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["There is an interesting design question here (cc @lhoestq).\r\n\r\nI guess the labels form a closed set so we could also use a [nlp.ClassLabel](https://huggingface.co/nlp/package_reference/main_classes.html#nlp.ClassLabel) instead of a string. The differences will be mainly that:\r\n- the labels are stored as integers and thus ready for training a model\r\n- the string to int conversion methods are handled by the `nlp.ClassLabel` feature (see the [doc](https://huggingface.co/nlp/package_reference/main_classes.html#nlp.ClassLabel)  and [here](https://huggingface.co/nlp/features.html)  and [here](https://huggingface.co/nlp/quicktour.html#fine-tuning-a-deep-learning-model)).\r\n\r\nIn my opinion, storing the labels as integers instead of strings makes it:\r\n- slightly less readable when accessing a dataset example (e.g. with `dataset[0]`)\r\n- force you with a specific mapping from string to integers\r\n- more clear that there is a fixed and predefined list of labels\r\n- easier to list all the labels (directly visible in the features).\r\n\r\n=> overall I'm pretty neutral about using one or the other option (`nlp.string` or `nlp.ClassLabel`).\r\n\r\nNote that we can now rather easily convert from one to the other with the map function and something like:\r\n```python\r\ndataset = dataset.map(lambda x: x, features=nlp.Features({'labels': nlp.ClassLabel(MY_LABELS_NAMES)}))\r\ndataset = dataset.map(lambda x: {'labels': dataset.features['labels'].int2str(x['labels'])}, features=nlp.Features({'labels': nlp.Value('string')}))\r\n```\r\n^^ this could probably be made even simpler (in particular for the second case)"
 'I see. This is an interesting question.\r\nMaybe as the dataset doesn\'t provide the mapping we shouldn\'t force an arbitrary one, and keep them as strings ?\r\nMoreover for NER the labels are often different from a dataset to the other so it\'s probably good to keep strings (there is no conventional mapping).\r\nAlso as the column is called "ner_tags" (or "langs"), you can already assume that there is a fixed and predefined list of labels.'
 'Yes sounds good to me.\r\nThis make me wonder if we don’t want to have a default identity function in `map` so this method could also be used to easily cast features. What do you think?'
 'Yes sounds good. I also noticed that people use map with identity to write a dataset into a specified cache file.']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/437', 'html_url': 'https://github.com/huggingface/datasets/pull/437', 'diff_url': 'https://github.com/huggingface/datasets/pull/437.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/437.patch', 'merged_at': datetime.datetime(2020, 7, 30, 8, 28, 15)}
Hi 🤗 
In response to the discussion in #425 @lewtun and I made some fixes to the repo. In the original XTREME implementation the PAN-X dataset for named entity recognition loaded each word/tag pair as a single row and the sentence relation was lost. With the fix each row contains the list of all words in a single sentence and their NER tags. This is also in agreement with the [NER example](https://github.com/huggingface/transformers/tree/master/examples/token-classification) in the transformers repo.

With the fix the output of the dataset should look as follows:
```python
>>> dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
>>> dataset['train'][0]
{'words': ['R.H.', 'Saunders', '(', 'St.', 'Lawrence', 'River', ')', '(', '968', 'MW', ')'],
 'ner_tags': ['B-ORG', 'I-ORG', 'O', 'B-ORG', 'I-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O'],
 'langs': ['en', 'en', 'en', 'en', 'en', 'en', 'en', 'en', 'en', 'en', 'en']}
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/437/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}