|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- akoksal/muri-it |
|
language: |
|
- afr |
|
- amh |
|
- ara |
|
- aze |
|
- bel |
|
- ben |
|
- bul |
|
- cat |
|
- ceb |
|
- ces |
|
- cos |
|
- cym |
|
- dan |
|
- deu |
|
- ell |
|
- eng |
|
- epo |
|
- est |
|
- eus |
|
- fas |
|
- fin |
|
- fra |
|
- fry |
|
- gla |
|
- gle |
|
- glg |
|
- guj |
|
- hat |
|
- hau |
|
- haw |
|
- hbs |
|
- heb |
|
- hin |
|
- hun |
|
- hye |
|
- ibo |
|
- isl |
|
- ita |
|
- jav |
|
- jpn |
|
- kan |
|
- kat |
|
- kaz |
|
- khm |
|
- kir |
|
- kor |
|
- kur |
|
- lao |
|
- lat |
|
- lav |
|
- lit |
|
- ltz |
|
- mal |
|
- mar |
|
- mkd |
|
- mlg |
|
- mlt |
|
- mon |
|
- mri |
|
- msa |
|
- msa |
|
- mya |
|
- nep |
|
- nld |
|
- nor |
|
- nya |
|
- pan |
|
- pol |
|
- por |
|
- pus |
|
- ron |
|
- rus |
|
- sin |
|
- slk |
|
- slv |
|
- smo |
|
- sna |
|
- snd |
|
- som |
|
- sot |
|
- spa |
|
- sqi |
|
- sun |
|
- swa |
|
- swe |
|
- tam |
|
- tel |
|
- tgk |
|
- tha |
|
- tur |
|
- ukr |
|
- urd |
|
- uzb |
|
- vie |
|
- xho |
|
- yid |
|
- yor |
|
- zho |
|
- zul |
|
base_model: |
|
- google/mt5-xxl |
|
pipeline_tag: text2text-generation |
|
--- |
|
|
|
# MURI-101: Multilingual Instruction-Following Model for 101 languages (mT5-XXL) |
|
|
|
MURI-101 is a multilingual instruction-following model, fine-tuned using a subset of the [**MURI-IT**](https://huggingface.co/datasets/akoksal/muri-it) dataset. It supports **101 languages** and outperforms most multilingual models in both **Natural Language Understanding (NLU)** and **Natural Language Generation (NLG)** tasks, especially in low-resource settings. |
|
|
|
This model was trained on a dataset with multilingual reverse instructions, ensuring that outputs are culturally and linguistically appropriate for the target language, thus reducing translation artifacts. |
|
|
|
[Paper](https://arxiv.org/abs/2409.12958) |
|
|
|
### Model Architecture |
|
- **Base Model**: mT5-XXL |
|
- **Training Data**: Subset of MURI-IT |
|
- **Training Setup**: Trained with [t5x](https://github.com/google-research/t5x) on 32 TPU v4-32. Batch size: 64, data packing enabled, learning rate: 3e-4 without a scheduler, 5 epochs. |
|
|
|
## Results |
|
We compare **MURI-101** against state-of-the-art models for multilingual instruction following. MURI-101 outperforms most multilingual models, except for Aya, across both NLU and NLG datasets. |
|
|
|
|
|
| | Okapi | mT0 | mT0x | Aya-101 | MURI-101 | |
|
|-------------------|----------------|--------------|---------------|------------------|---------------------------| |
|
| arb | 27.7 | 31.5 | 31.6 | 38.2 | 36.5 | |
|
| ben | 26.8 | 31.6 | 30.2 | 35.8 | 33.0 | |
|
| cat | 30.5 | 32.8 | 32.6 | 39.6 | 38.8 | |
|
| dan | 31.8 | 33.0 | 32.0 | 39.7 | 38.4 | |
|
| deu | 31.7 | 32.7 | 32.5 | 39.7 | 38.9 | |
|
... |
|
| vie | 27.5 | 30.9 | 31.1 | 34.8 | 36.8 | |
|
| zho | 28.2 | 32.5 | 31.6 | 38.3 | 36.9 | |
|
| Avg. | 28.8 | 31.5 | 30.8 | 37.3 | 36.0 | |
|
|
|
Additionally, our model complements Aya effectively, especially in low-resource settings. |
|
|
|
| Language | mT5 | Aya_1 | Aya_1 + MURI_1 | |
|
|-------------------|------|-------|----------------| |
|
| aze | 20.4 | 37.0 | 39.5 | |
|
| bel | 22.4 | 32.1 | 33.7 | |
|
| bul | 20.7 | 34.4 | 38.1 | |
|
| cym | 18.4 | 33.0 | 35.5 | |
|
| gla | 19.3 | 28.7 | 35.2 | |
|
| kaz | 19.8 | 44.7 | 46.7 | |
|
| khm | 16.5 | 30.0 | 31.3 | |
|
| lao | 21.3 | 32.7 | 33.0 | |
|
| slk | 19.2 | 38.1 | 39.1 | |
|
| slv | 18.9 | 40.3 | 39.6 | |
|
| Avg. | 19.7 | 35.1 | **37.2** | |
|
|
|
|
|
|
|
## Use |
|
To load and use the model, you can use the following: |
|
|
|
### AutoModelForSeq2SeqLM |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
|
|
|
muri = AutoModelForSeq2SeqLM.from_pretrained("akoksal/muri-101") |
|
tokenizer = AutoTokenizer.from_pretrained("akoksal/muri-101") |
|
|
|
instruction = "Verilen cümlenin pozitif mi negatif mi olduğunu tahmin edin: Hayatta kesinlikle izlenmemesi gereken filmler kategorisindeki listemin en başına bu filmi koyarım." |
|
# Turkish to English translation: Guess whether the given sentence is positive or negative: I would put this movie at the very top of the list of movies that absolutely should not be watched in life. |
|
inputs = tokenizer(instruction, return_tensors="pt").to(device) |
|
outputs = muri.generate(**inputs, max_new_tokens=5) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
# > negatif |
|
# (negative) |
|
``` |
|
|
|
### Pipeline |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
muri = pipeline("text2text-generation", |
|
model="akoksal/muri-101") |
|
|
|
muri("""این مقاله را خلاصه کنید |
|
...تیم دانش‌آموزی کاوش باستانی یک بطری حاوی پیغام ۲۰۰ ساله در شمال فرانسه پیدا کردند""", |
|
max_new_tokens=150, |
|
do_sample=True, |
|
temperature=0.9, |
|
top_p=0.8) |
|
# Summarize this article |
|
# A student team of archeologists found a bottle containing a 200-year-old message in northern France ... [300 words] |
|
|
|
# > در طول سالیان متمادی باستان شناسان فرانسوی تلاش زیادی برای پیدا کردن آثار و اشیای باستانی انجام داده اند اما این بار پیدا شدن بطری حاوی پیغامی به بیش از دو قرن پیش از آن تاریخ نشان می دهد. |
|
# > Over the years, French archaeologists have made great efforts to find ancient works and objects, but this time, the discovery of a bottle containing a message shows that date more than two centuries ago. |
|
``` |
|
|
|
Thanks to [Google's TRC program](https://sites.research.google/trc/about/) for supporting the training of this model. |
|
|
|
Check out [the paper](https://arxiv.org/abs/2409.12958) for more detailed information on the experiments and results. |
|
|
|
## Citation |
|
``` |
|
@misc{koksal2024muri, |
|
title={MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions}, |
|
author={Abdullatif Köksal and Marion Thaler and Ayyoob Imani and Ahmet Üstün and Anna Korhonen and Hinrich Schütze}, |
|
year={2024}, |
|
eprint={2409.12958}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2409.12958}, |
|
} |
|
``` |
