Hugging Face's logo

cartesinus
/
iva_mt_wslot-m2m100_418M-en-pl

Text2Text Generation
Transformers
PyTorch
TensorBoard
Safetensors
Polish
English
m2m_100
machine translation
iva
virtual assistants
natural-language-understanding
nlu
Inference Endpoints
Model card Files Metrics Community
1
iva_mt_wslot-m2m100_418M-en-pl / README.md
cartesinus's picture
cartesinus
Update README.md
24ea82e
preview code
|
raw
history blame contribute delete
7.98 kB
metadata
license: mit
tags:
  - machine translation
  - iva
  - virtual assistants
  - natural-language-understanding
  - nlu
metrics:
  - bleu
model-index:
  - name: iva_mt_wslot-m2m100_418M-0.1.0
    results: []
datasets:
  - cartesinus/iva_mt_wslot
language:
  - pl
  - en
co2_eq_emissions:
  emissions: 0.68

iva_mt_wslot-m2m100_418M-0.1.0 en-pl

This model is a fine-tuned version of facebook/m2m100_418M on the iva_mt_wslot dataset. It achieves the following results:

  1. On the test set (iva_mt):
  • BLEU (plain text): 39.1560
  • BLEU (with slots): 63.8767
  • F1 score: (in preparation)

For reference BLEU for baseline m2m100-418M (plain text) was 21.9468. Second result (63.8767) is when tags are treated as "normal" words in sentence. Therefore that result might be a bit misleading. Please refer to plain text results if you are not sure how to interpret them.

  1. WMT20 (en2pl):
  • BLEU (lowercased, tags removed): 15.0863
  • BLEU for baseline m2m100-418M (plain text): 20.2750

For reference WMT20 submission systems in en-pl direction had between 25 and 30 BLEU

  1. BLEU on the evaluation set (same as in below table 'Training results'): 61.6249

  2. On the training set (to see how it adjusted to train):

  • BLEU (plain text): 70.5597
  • BLEU (with slots): 93.8200

BLEU was measured with sacreBLEU library.

Model description, intended uses & limitations

Model is biased towards virtual assistant (IVA) sentences in prediction/translation. These sentences are short, imperatives with a lot of name entities (slots) and particular vocabulary (for example settings name). It can be observed in above results where WMT results are very low while in-domain test is very high.

This model will most probably force IVA translations on your text. As long as sentences that you are translating are more or less similar to massive and leyzer domains it will be ok. If you will translate out-of-domain sentenences (such as for example News, Medical) that are not very similar then results will drop significantly up to the point where baseline m2m100-418M will be better than this model.

This model will generate tags in output even if there is not tag in input sentence. Frequency of this depends on input text origin. When testing IVA utterances this occurs between 3 and 5%. When WMT20 was translated it happened in 40% cases (input text was from News domain). This is not very severe problem and it can be fixed easily in post-processing (simple sed 's/<[a-z]>//g' should be enough in most cases).

Translations with slot annotation very often differ from same sentences when slots are removed. This is quite frequent and it happens between 30 and 50% of translated utterances. For example there will be a difference between "is it raining in barcelona" and "is it raining in <a>barcelona<a>". In second case model will more likely localize name of city to some Polish name (here Lublin, because such city was given in Massive train set). This might be useful if you want to generate more variants.

One last thing that needs to be mentioned is that BLEU is not particulary good metric to evaluate IVA sentences due to their length and it should be evalued with other metrices (e.g. GLEU).

How to use

First please make sure to install pip install transformers. First download model: 

from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import torch

def translate(input_text, lang):
    input_ids = tokenizer(input_text, return_tensors="pt")
    generated_tokens = model.generate(**input_ids, forced_bos_token_id=tokenizer.get_lang_id(lang))
    return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)

model_name = "cartesinus/iva_mt_wslot-m2m100_418M-0.1.0"
tokenizer = M2M100Tokenizer.from_pretrained(model_name, src_lang="en", tgt_lang="pl")
model = M2M100ForConditionalGeneration.from_pretrained(model_name)

Then you can translate either plan text like this:

print(translate("set the temperature on my thermostat", "pl"))

or you can translate with slot annotations that will be restored in tgt language:

print(translate("wake me up at <a>nine am<a> on <b>friday<b>", "pl")) #translation: obudź mnie o <a>piątej rano<a> <b>w tym tygodniu<b>

Limitations of translation with slot transfer:

  1. Annotated words must be placed between semi-xml tags like this "this is <a>example<a>"
  2. There is no closing tag for example "<\a>" in above example - this is done on purpose to ommit problems with backslash escape
  3. If sentence consists of more than one slot then simply use next alphabet letter. For example "this is <a>example<a> with more than <b>one<b> slot"
  4. Please do not add space before first or last annotated word because this particular model was trained this way and it most probably will lower it's results

Training and evaluation data

Dataset Composition (en-pl)

Corpus Train Dev Test
Massive 1.1 11514 2033 2974
Leyzer 0.2.0 3974 701 1380
OpenSubtitles from OPUS 2329 411 500
KDE from OPUS 1154 241 241
CCMatrix from Opus 1096 232 237
Ubuntu from OPUS 281 60 59
Gnome from OPUS 14 3 3
total 20362 3681 5394

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 10
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss BLEU Gen Len
0.0178 1.0 5091 0.0171 57.4439 21.1396
0.013 2.0 10182 0.0159 58.886 21.2285
0.0091 3.0 15273 0.0157 60.159 21.1222
0.0073 4.0 20364 0.0159 60.5893 21.1212
0.0054 5.0 25455 0.0161 60.6484 21.0679
0.004 6.0 30546 0.0166 61.5283 21.0875
0.0031 7.0 35637 0.0169 61.0439 21.1562
0.0024 8.0 40728 0.0172 61.9427 21.2203
0.0018 9.0 45819 0.0175 61.7325 21.1478
0.0014 10.0 50910 0.0176 61.6249 21.157

Framework versions

  • Transformers 4.26.1
  • Pytorch 1.13.1+cu116
  • Datasets 2.10.1
  • Tokenizers 0.13.2

Citation

If you use this model, please cite the following:

@article{Sowanski2023SlotLI,
  title={Slot Lost in Translation? Not Anymore: A Machine Translation Model for Virtual Assistants with Type-Independent Slot Transfer},
  author={Marcin Sowanski and Artur Janicki},
  journal={2023 30th International Conference on Systems, Signals and Image Processing (IWSSIP)},
  year={2023},
  pages={1-5}
}