Add model card

README.md

@@ -83,3 +83,165 @@ language:
   - zh
 license: mit
 ---
+
+# xmod-base
+
+X-MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](http://dx.doi.org/10.18653/v1/2022.naacl-main.255) (Pfeiffer et al., NAACL 2022) and first released in [this repository](https://github.com/facebookresearch/fairseq/tree/main/examples/xmod).
+
+Because it has been pre-trained with language-specific modular components (_language adapters_), X-MOD differs from previous multilingual models like [XLM-R](https://huggingface.co/xlm-roberta-base). For fine-tuning, the language adapters in each transformer layer are frozen.
+
+# Usage
+
+## Tokenizer
+This model reuses the tokenizer of [XLM-R](https://huggingface.co/xlm-roberta-base), so you can load the tokenizer as follows:
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
+```
+
+## Input Language
+Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:
+
+```python
+from transformers import XMODModel
+
+model = XMODModel.from_pretrained("jvamvas/xmod-base")
+model.set_default_language("en_XX")
+```
+
+A directory of the language adapters in this model is found at the bottom of this model card.
+
+## Fine-tuning
+The paper recommends that the embedding layer and the language adapters are frozen during fine-tuning. A method for doing this is provided in the code:
+
+```python
+model.freeze_embeddings_and_language_adapters()
+# Fine-tune the model ...
+```
+
+## Cross-lingual Transfer
+After fine-tuning, zero-shot cross-lingual transfer can be tested by activating the language adapter of the target language:
+```python
+model.set_default_language("de_DE")
+# Evaluate the model on German examples ...
+```
+
+# Bias, Risks, and Limitations
+
+Please refer to the model card of [XLM-R](https://huggingface.co/xlm-roberta-base), because X-MOD has a similar architecture and has been trained on similar training data.
+
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@inproceedings{pfeiffer-etal-2022-lifting,
+    title = "Lifting the Curse of Multilinguality by Pre-training Modular Transformers",
+    author = "Pfeiffer, Jonas  and
+      Goyal, Naman  and
+      Lin, Xi  and
+      Li, Xian  and
+      Cross, James  and
+      Riedel, Sebastian  and
+      Artetxe, Mikel",
+    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
+    month = jul,
+    year = "2022",
+    address = "Seattle, United States",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2022.naacl-main.255",
+    doi = "10.18653/v1/2022.naacl-main.255",
+    pages = "3479--3495"
+}
+```
+
+# Languages
+
+This model contains the following language adapters:
+
+| Language code | Language              |
+|---------------|-----------------------|
+| af_ZA         | Afrikaans             |
+| am_ET         | Amharic               |
+| ar_AR         | Arabic                |
+| az_AZ         | Azerbaijani           |
+| be_BY         | Belarusian            |
+| bg_BG         | Bulgarian             |
+| bn_IN         | Bengali               |
+| ca_ES         | Catalan               |
+| cs_CZ         | Czech                 |
+| cy_GB         | Welsh                 |
+| da_DK         | Danish                |
+| de_DE         | German                |
+| el_GR         | Greek                 |
+| en_XX         | English               |
+| eo_EO         | Esperanto             |
+| es_XX         | Spanish               |
+| et_EE         | Estonian              |
+| eu_ES         | Basque                |
+| fa_IR         | Persian               |
+| fi_FI         | Finnish               |
+| fr_XX         | French                |
+| ga_IE         | Irish                 |
+| gl_ES         | Galician              |
+| gu_IN         | Gujarati              |
+| ha_NG         | Hausa                 |
+| he_IL         | Hebrew                |
+| hi_IN         | Hindi                 |
+| hr_HR         | Croatian              |
+| hu_HU         | Hungarian             |
+| hy_AM         | Armenian              |
+| id_ID         | Indonesian            |
+| is_IS         | Icelandic             |
+| it_IT         | Italian               |
+| ja_XX         | Japanese              |
+| ka_GE         | Georgian              |
+| kk_KZ         | Kazakh                |
+| km_KH         | Central Khmer         |
+| kn_IN         | Kannada               |
+| ko_KR         | Korean                |
+| ku_TR         | Kurdish               |
+| ky_KG         | Kirghiz               |
+| la_VA         | Latin                 |
+| lo_LA         | Lao                   |
+| lt_LT         | Lithuanian            |
+| lv_LV         | Latvian               |
+| mk_MK         | Macedonian            |
+| ml_IN         | Malayalam             |
+| mn_MN         | Mongolian             |
+| mr_IN         | Marathi               |
+| ms_MY         | Malay                 |
+| my_MM         | Burmese               |
+| ne_NP         | Nepali                |
+| nl_XX         | Dutch                 |
+| no_XX         | Norwegian             |
+| or_IN         | Oriya                 |
+| pa_IN         | Punjabi               |
+| pl_PL         | Polish                |
+| ps_AF         | Pashto                |
+| pt_XX         | Portuguese            |
+| ro_RO         | Romanian              |
+| ru_RU         | Russian               |
+| sa_IN         | Sanskrit              |
+| si_LK         | Sinhala               |
+| sk_SK         | Slovak                |
+| sl_SI         | Slovenian             |
+| so_SO         | Somali                |
+| sq_AL         | Albanian              |
+| sr_RS         | Serbian               |
+| sv_SE         | Swedish               |
+| sw_KE         | Swahili               |
+| ta_IN         | Tamil                 |
+| te_IN         | Telugu                |
+| th_TH         | Thai                  |
+| tl_XX         | Tagalog               |
+| tr_TR         | Turkish               |
+| uk_UA         | Ukrainian             |
+| ur_PK         | Urdu                  |
+| uz_UZ         | Uzbek                 |
+| vi_VN         | Vietnamese            |
+| zh_CN         | Chinese (simplified)  |
+| zh_TW         | Chinese (traditional) |