Model Card for NLLB-with-myv-v2024 (a translation model for Erzya)
This is a version of the nllb-200-distilled-600M machine translation model
with one added language: Erzya (the new language code is myv_Cyrl
).
It can probably translate from all 202 NLLB languages, but it fine-tuned with the focus on Erzya, Russian, and, to a lesser extent,
on Arabic, English, Estonian, Finnish, French, German, Kazakh, Mandarin, Mongolian, Spanish, Turkish, Ukrainian, and Uzbek.
Model Details
Model Description
- Developed by: Isai Gordeev, Sergey Kuldin and David Dale
- Model type: Encoder-decoder transformer
- Language(s) (NLP): Erzya, Russian, and all the 202 NLLB languages.
- License: CC-BY-NC-4.0
- Finetuned from model: nllb-200-distilled-600M
Model Sources [optional]
- Repository: will be published later
- Paper: will be published later
- Demo: https://lango.to/ (it is powered by a similar model)
Uses
Direct Use
Translation between Erzya, Russian, and potentially other languages. The model seems to be SOTA for translating into Erzya.
Out-of-Scope Use
Translation between other NLLB languages, not inclusing Erzya as source or target.
Bias, Risks, and Limitations
The model is not producing the most fluent translations into Russian and other high-resourced languages.
Its translations into Erzya seem to be better than anything else, but may still include inaccurate or ungrammatical translations, so they should be always manually reviewed before any high-responsibility use.
Recommendations
Please contact the authors for any substantial recommendation.
How to Get Started with the Model
See the NLLB generation code: https://huggingface.co/docs/transformers/v4.44.2/en/model_doc/nllb#generating-with-nllb.
Training Details
Training Data
- https://huggingface.co/datasets/slone/myv_ru_2022
- https://huggingface.co/datasets/slone/e-mordovia-articles-2023
Training Procedure
Preprocessing [optional]
The preprocessing code is adapted from the Stopes repo of the NLLB team: https://github.com/facebookresearch/stopes/blob/main/stopes/pipelines/monolingual/monolingual_line_processor.py#L214
It performs punctuation normalization, nonprintable character removal and Unicode normalization.
Training Hyperparameters
The tokenizer of the model was updated with 6209 new Erzya tokens. They were initialized with the average embeddings of the old tokens from which they are combined.
- training regime:
fp32
- batch_size: 6
- grad_acc_steps: 4
- max_length: 128
- optimizer: Adafactor
- lr: 1e-4
- clip_threshold=1.0
- weight_decay: 1e-3
- warmup_steps: 3_000 (with a linear warmup from 0)
- training_steps: 220_000
- weight_loss_coef: 100 (a coefficient for the additional penalty, MSE between the embeddings of old tokens and their values for NLLB-200)
Technical Specifications
Model Architecture and Objective
A standard encoder-decoder translation model with cross-entropy loss.
Compute Infrastructure
Google Colab with a T4 GPU.
pip install --upgrade sentencepiece transformers==4.40 datasets sacremoses editdistance sacrebleu razdel ctranslate2
Model Card Contact
@cointegrated
- Downloads last month
- 6
Model tree for slone/nllb-with-myv-v2024
Base model
facebook/nllb-200-distilled-600M