Add XLM-R tokenizer files (#2)

- Copy XLM-R tokenizer to this repo (007e82656cdb52dda51a3f6b674aa57df98ba058)


Co-authored-by: Jannis Vamvas <[email protected]>

README.md

@@ -93,13 +93,7 @@ Because it has been pre-trained with language-specific modular components (_lang
 # Usage
 
 ## Tokenizer
-This model reuses the tokenizer of [XLM-R](https://huggingface.co/xlm-roberta-base), so you can load the tokenizer as follows:
-
-```python
-from transformers import AutoTokenizer
-
-tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
-```
+This model reuses the tokenizer of [XLM-R](https://huggingface.co/xlm-roberta-base).
 
 ## Input Language
 Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:
@@ -107,7 +101,7 @@ Because this model uses language adapters, you need to specify the language of y
 ```python
 from transformers import XmodModel
 
-model = XmodModel.from_pretrained("jvamvas/xmod-base")
+model = XmodModel.from_pretrained("facebook/xmod-base")
 model.set_default_language("en_XX")
 ```
 

tokenizer_config.json

@@ -0,0 +1,4 @@
+{
+  "tokenizer_class": "XLMRobertaTokenizer"
+}
+