|
--- |
|
library_name: transformers |
|
license: cc-by-nc-4.0 |
|
language: |
|
- myv |
|
- ru |
|
- ar |
|
- en |
|
- et |
|
- fr |
|
- de |
|
- kk |
|
- ch |
|
- zh |
|
- mn |
|
- es |
|
- tr |
|
- uk |
|
- uz |
|
base_model: |
|
- facebook/nllb-200-distilled-600M |
|
datasets: |
|
- slone/myv_ru_2022 |
|
- slone/e-mordovia-articles-2023 |
|
pipeline_tag: translation |
|
--- |
|
|
|
# Model Card for NLLB-with-myv-v2024 (a translation model for Erzya) |
|
|
|
This is a version of the [nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) machine translation model |
|
with one added language: Erzya (the new language code is `myv_Cyrl`). |
|
It can probably translate from all 202 NLLB languages, but it fine-tuned with the focus on Erzya, Russian, and, to a lesser extent, |
|
on Arabic, English, Estonian, Finnish, French, German, Kazakh, Mandarin, Mongolian, Spanish, Turkish, Ukrainian, and Uzbek. |
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
|
|
- **Developed by:** Isai Gordeev, Sergey Kuldin and David Dale |
|
- **Model type:** Encoder-decoder transformer |
|
- **Language(s) (NLP):** Erzya, Russian, and all the 202 NLLB languages. |
|
- **License:** CC-BY-NC-4.0 |
|
- **Finetuned from model:** [nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** will be published later |
|
- **Paper:** will be published later |
|
- **Demo:** https://lango.to/ (it is powered by a similar model) |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
Translation between Erzya, Russian, and potentially other languages. The model seems to be SOTA for translating into Erzya. |
|
|
|
### Out-of-Scope Use |
|
Translation between other NLLB languages, not inclusing Erzya as source or target. |
|
|
|
## Bias, Risks, and Limitations |
|
The model is not producing the most fluent translations into Russian and other high-resourced languages. |
|
|
|
Its translations into Erzya seem to be better than anything else, but may still include inaccurate or ungrammatical translations, |
|
so they should be always manually reviewed before any high-responsibility use. |
|
|
|
### Recommendations |
|
Please contact the authors for any substantial recommendation. |
|
|
|
## How to Get Started with the Model |
|
|
|
See the NLLB generation code: https://huggingface.co/docs/transformers/v4.44.2/en/model_doc/nllb#generating-with-nllb. |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
- https://huggingface.co/datasets/slone/myv_ru_2022 |
|
- https://huggingface.co/datasets/slone/e-mordovia-articles-2023 |
|
|
|
### Training Procedure |
|
|
|
|
|
#### Preprocessing [optional] |
|
|
|
The preprocessing code is adapted from the Stopes repo of the NLLB team: |
|
https://github.com/facebookresearch/stopes/blob/main/stopes/pipelines/monolingual/monolingual_line_processor.py#L214 |
|
|
|
It performs punctuation normalization, nonprintable character removal and Unicode normalization. |
|
|
|
#### Training Hyperparameters |
|
|
|
The tokenizer of the model was updated with 6209 new Erzya tokens. They were initialized with the average embeddings of the old tokens from which they are combined. |
|
|
|
- training regime: `fp32` |
|
- batch_size: 6 |
|
- grad_acc_steps: 4 |
|
- max_length: 128 |
|
- optimizer: Adafactor |
|
- lr: 1e-4 |
|
- clip_threshold=1.0 |
|
- weight_decay: 1e-3 |
|
- warmup_steps: 3_000 (with a linear warmup from 0) |
|
- training_steps: 220_000 |
|
- weight_loss_coef: 100 (a coefficient for the additional penalty, MSE between the embeddings of old tokens and their values for NLLB-200) |
|
|
|
|
|
## Technical Specifications |
|
|
|
### Model Architecture and Objective |
|
|
|
A standard encoder-decoder translation model with cross-entropy loss. |
|
|
|
### Compute Infrastructure |
|
|
|
Google Colab with a T4 GPU. |
|
|
|
``` |
|
pip install --upgrade sentencepiece transformers==4.40 datasets sacremoses editdistance sacrebleu razdel ctranslate2 |
|
``` |
|
|
|
## Model Card Contact |
|
|
|
@cointegrated |
