Translation
Transformers
Safetensors
m2m_100
text2text-generation
Inference Endpoints
Edit model card

Model Card for NLLB-with-myv-v2024 (a translation model for Erzya)

This is a version of the nllb-200-distilled-600M machine translation model with one added language: Erzya (the new language code is myv_Cyrl). It can probably translate from all 202 NLLB languages, but it fine-tuned with the focus on Erzya, Russian, and, to a lesser extent, on Arabic, English, Estonian, Finnish, French, German, Kazakh, Mandarin, Mongolian, Spanish, Turkish, Ukrainian, and Uzbek.

Model Details

Model Description

  • Developed by: Isai Gordeev, Sergey Kuldin and David Dale
  • Model type: Encoder-decoder transformer
  • Language(s) (NLP): Erzya, Russian, and all the 202 NLLB languages.
  • License: CC-BY-NC-4.0
  • Finetuned from model: nllb-200-distilled-600M

Model Sources [optional]

  • Repository: will be published later
  • Paper: will be published later
  • Demo: https://lango.to/ (it is powered by a similar model)

Uses

Direct Use

Translation between Erzya, Russian, and potentially other languages. The model seems to be SOTA for translating into Erzya.

Out-of-Scope Use

Translation between other NLLB languages, not inclusing Erzya as source or target.

Bias, Risks, and Limitations

The model is not producing the most fluent translations into Russian and other high-resourced languages.

Its translations into Erzya seem to be better than anything else, but may still include inaccurate or ungrammatical translations, so they should be always manually reviewed before any high-responsibility use.

Recommendations

Please contact the authors for any substantial recommendation.

How to Get Started with the Model

See the NLLB generation code: https://huggingface.co/docs/transformers/v4.44.2/en/model_doc/nllb#generating-with-nllb.

Training Details

Training Data

Training Procedure

Preprocessing [optional]

The preprocessing code is adapted from the Stopes repo of the NLLB team: https://github.com/facebookresearch/stopes/blob/main/stopes/pipelines/monolingual/monolingual_line_processor.py#L214

It performs punctuation normalization, nonprintable character removal and Unicode normalization.

Training Hyperparameters

The tokenizer of the model was updated with 6209 new Erzya tokens. They were initialized with the average embeddings of the old tokens from which they are combined.

  • training regime: fp32
  • batch_size: 6
  • grad_acc_steps: 4
  • max_length: 128
  • optimizer: Adafactor
  • lr: 1e-4
  • clip_threshold=1.0
  • weight_decay: 1e-3
  • warmup_steps: 3_000 (with a linear warmup from 0)
  • training_steps: 220_000
  • weight_loss_coef: 100 (a coefficient for the additional penalty, MSE between the embeddings of old tokens and their values for NLLB-200)

Technical Specifications

Model Architecture and Objective

A standard encoder-decoder translation model with cross-entropy loss.

Compute Infrastructure

Google Colab with a T4 GPU.

pip install --upgrade sentencepiece transformers==4.40 datasets sacremoses editdistance sacrebleu razdel ctranslate2

Model Card Contact

@cointegrated

Downloads last month
6
Safetensors
Model size
621M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for slone/nllb-with-myv-v2024

Finetuned
(77)
this model

Datasets used to train slone/nllb-with-myv-v2024