Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task
{'login': 'jerryIsHere', 'id': 50871412, 'node_id': 'MDQ6VXNlcjUwODcxNDEy', 'avatar_url': 'https://avatars.githubusercontent.com/u/50871412?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/jerryIsHere', 'html_url': 'https://github.com/jerryIsHere', 'followers_url': 'https://api.github.com/users/jerryIsHere/followers', 'following_url': 'https://api.github.com/users/jerryIsHere/following{/other_user}', 'gists_url': 'https://api.github.com/users/jerryIsHere/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/jerryIsHere/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/jerryIsHere/subscriptions', 'organizations_url': 'https://api.github.com/users/jerryIsHere/orgs', 'repos_url': 'https://api.github.com/users/jerryIsHere/repos', 'events_url': 'https://api.github.com/users/jerryIsHere/events{/privacy}', 'received_events_url': 'https://api.github.com/users/jerryIsHere/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892857, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODU3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/bug', 'name': 'bug', 'color': 'd73a4a', 'default': True, 'description': "Something isn't working"}]
[]
Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**? > Hi @jerryIsHere, thanks for reporting the issue. But are you sure this is a bug in HuggingFace **Datasets**?

Oh, I am sorry
I would reopen the post on huggingface/transformers
[This colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) implements a token classification input pipeline extending the logic from [this hugging example](https://huggingface.co/transformers/custom_datasets.html#tok-ner).

The pipeline works fine with most instance in different languages, but unfortunately, [the Japanese Kana ligature (a form of abbreviation? I don't know Japanese well)](https://en.wikipedia.org/wiki/Kana_ligature) break the alignment of `return_offsets_mapping`:
![image](https://user-images.githubusercontent.com/50871412/122904371-db192700-d382-11eb-8917-1775db76db69.png)

Without the try catch block, it riase `ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true`, example shown here [(another colab notebook)](https://colab.research.google.com/drive/1MmOqf3ppzzdKKyMWkn0bJy6DqzOO0SSm?usp=sharing)

It is clear that the normalizer is the process that break the alignment, as it is observed that `tokenizer._tokenizer.normalizer.normalize_str('ヿ')` return 'コト'.

One workaround is to include `tokenizer._tokenizer.normalizer.normalize_str` before the tokenizer preprocessing pipeline, which is also provided in the [first colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) with the name `udposTestDatasetWorkaround`.

I guess similar logics should be included inside the tokenizer and the offsets_mapping generation process such that user don't need to include them in their code. But I don't understand the code of tokenizer well that I think I am not able to do this.

p.s.
**I am using my own dataset building script in the provided example, but the script should be equivalent to the changes made by this [update](https://github.com/huggingface/datasets/pull/2466)**
`get_dataset `is just a simple wrapping for `load_dataset`
and the `tokenizer` is just `XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-large")`
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2532/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}