Correct data structure for PAN-X task in XTREME dataset?
Thanks, I am making progress. I got a new error `NonMatchingSplitsSizesError ` (see traceback below), which I suspect is due to the fact that number of rows in the dataset changed (one row per word --> one row per sentence) as well as the number of bytes due to the slightly updated data structure. 

```python
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=1756492, num_examples=80536, dataset_name='xtreme'), 'recorded': SplitInfo(name='validation', num_bytes=1837109, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='test', num_bytes=1752572, num_examples=80326, dataset_name='xtreme'), 'recorded': SplitInfo(name='test', num_bytes=1833214, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='train', num_bytes=3496832, num_examples=160394, dataset_name='xtreme'), 'recorded': SplitInfo(name='train', num_bytes=3658428, num_examples=20000, dataset_name='xtreme')}]
```
I can fix the error by replacing the values in the `datasets_infos.json` file, which I tested for English. However, to update this for all 40 datasets manually is slightly painful. Is there a better way to update the expected values for all datasets?
Hi 🤗  team!

## Description of the problem
Thanks to the fix from #416 I am now able to load the NER task in the XTREME dataset as follows:

```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
dataset_train = dataset['train']
```

However, I am not sure that `load_dataset()` is returning the correct data structure for NER. 

Currently, every row in `dataset_train` is of the form
```python
{'word': str, 'ner_tag': str, 'lang': str}
```
but I think we actually want something like
```python
{'words': List[str], 'ner_tags': List[str], 'langs': List[str]}
```
so that each row corresponds to a _sequence_ of words associated with each example. With the current data structure I do not think it is possible to transform `dataset_train` into a form suitable for training because we do not know the boundaries between examples.

Indeed, [this line](https://github.com/google-research/xtreme/blob/522434d1aece34131d997a97ce7e9242a51a688a/third_party/utils_tag.py#L58) in the XTREME repo, processes the texts as lists of sentences, tags, and languages.

## Proposed solution
Replace
```python
with open(filepath) as f:
    data = csv.reader(f, delimiter="\t", quoting=csv.QUOTE_NONE)
    for id_, row in enumerate(data):
        if row:
            lang, word = row[0].split(":")[0], row[0].split(":")[1]
            tag = row[1]
            yield id_, {"word": word, "ner_tag": tag, "lang": lang}
```
from  [these lines](https://github.com/huggingface/nlp/blob/ce7d3a1d630b78fe27188d1706f3ea980e8eec43/datasets/xtreme/xtreme.py#L881-L887) of the `_generate_examples()` function with something like

```python
guid_index = 1
with open(filepath, encoding="utf-8") as f:
    words = []
    ner_tags = []
    langs = []
    for line in f:
        if line.startswith("-DOCSTART-") or line == "" or line == "\n":
            if words:
                yield guid_index, {"words": words, "ner_tags": ner_tags, "langs": langs}
                guid_index += 1
                words = []
                ner_tags = []
        else:
            # pan-x data is tab separated
            splits = line.split("\t")
            # strip out en: prefix
            langs.append(splits[0][:2])
            words.append(splits[0][3:])
            if len(splits) > 1:
                labels.append(splits[-1].replace("\n", ""))
            else:
                # examples have no label in test set
                labels.append("O")
```
If you agree, me or @lvwerra would be happy to implement this and create a PR.
Correct data structure for PAN-X task in XTREME dataset? 
 Hi 🤗  team!

## Description of the problem
Thanks to the fix from #416 I am now able to load the NER task in the XTREME dataset as follows:

```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
dataset_train = dataset['train']
```

However, I am not sure that `load_dataset()` is returning the correct data structure for NER. 

Currently, every row in `dataset_train` is of the form
```python
{'word': str, 'ner_tag': str, 'lang': str}
```
but I think we actually want something like
```python
{'words': List[str], 'ner_tags': List[str], 'langs': List[str]}
```
so that each row corresponds to a _sequence_ of words associated with each example. With the current data structure I do not think it is possible to transform `dataset_train` into a form suitable for training because we do not know the boundaries between examples.

Indeed, [this line](https://github.com/google-research/xtreme/blob/522434d1aece34131d997a97ce7e9242a51a688a/third_party/utils_tag.py#L58) in the XTREME repo, processes the texts as lists of sentences, tags, and languages.

## Proposed solution
Replace
```python
with open(filepath) as f:
    data = csv.reader(f, delimiter="\t", quoting=csv.QUOTE_NONE)
    for id_, row in enumerate(data):
        if row:
            lang, word = row[0].split(":")[0], row[0].split(":")[1]
            tag = row[1]
            yield id_, {"word": word, "ner_tag": tag, "lang": lang}
```
from  [these lines](https://github.com/huggingface/nlp/blob/ce7d3a1d630b78fe27188d1706f3ea980e8eec43/datasets/xtreme/xtreme.py#L881-L887) of the `_generate_examples()` function with something like

```python
guid_index = 1
with open(filepath, encoding="utf-8") as f:
    words = []
    ner_tags = []
    langs = []
    for line in f:
        if line.startswith("-DOCSTART-") or line == "" or line == "\n":
            if words:
                yield guid_index, {"words": words, "ner_tags": ner_tags, "langs": langs}
                guid_index += 1
                words = []
                ner_tags = []
        else:
            # pan-x data is tab separated
            splits = line.split("\t")
            # strip out en: prefix
            langs.append(splits[0][:2])
            words.append(splits[0][3:])
            if len(splits) > 1:
                labels.append(splits[-1].replace("\n", ""))
            else:
                # examples have no label in test set
                labels.append("O")
```
If you agree, me or @lvwerra would be happy to implement this and create a PR. 
 Thanks, I am making progress. I got a new error `NonMatchingSplitsSizesError ` (see traceback below), which I suspect is due to the fact that number of rows in the dataset changed (one row per word --> one row per sentence) as well as the number of bytes due to the slightly updated data structure. 

```python
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=1756492, num_examples=80536, dataset_name='xtreme'), 'recorded': SplitInfo(name='validation', num_bytes=1837109, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='test', num_bytes=1752572, num_examples=80326, dataset_name='xtreme'), 'recorded': SplitInfo(name='test', num_bytes=1833214, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='train', num_bytes=3496832, num_examples=160394, dataset_name='xtreme'), 'recorded': SplitInfo(name='train', num_bytes=3658428, num_examples=20000, dataset_name='xtreme')}]
```
I can fix the error by replacing the values in the `datasets_infos.json` file, which I tested for English. However, to update this for all 40 datasets manually is slightly painful. Is there a better way to update the expected values for all datasets?