cartesinus
commited on
Commit
•
6dce599
1
Parent(s):
5de9e43
Update README.md
Browse files
README.md
CHANGED
@@ -3,7 +3,7 @@ license: mit
|
|
3 |
tags:
|
4 |
- generated_from_trainer
|
5 |
datasets:
|
6 |
-
- iva_mt_wslot
|
7 |
metrics:
|
8 |
- bleu
|
9 |
model-index:
|
@@ -22,6 +22,10 @@ model-index:
|
|
22 |
- name: Bleu
|
23 |
type: bleu
|
24 |
value: 67.0512
|
|
|
|
|
|
|
|
|
25 |
---
|
26 |
|
27 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
@@ -39,13 +43,37 @@ It achieves the following results on the evaluation set:
|
|
39 |
|
40 |
More information needed
|
41 |
|
42 |
-
##
|
43 |
-
|
44 |
-
|
45 |
-
|
46 |
-
|
47 |
-
|
48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
49 |
|
50 |
## Training procedure
|
51 |
|
|
|
3 |
tags:
|
4 |
- generated_from_trainer
|
5 |
datasets:
|
6 |
+
- cartesinus/iva_mt_wslot
|
7 |
metrics:
|
8 |
- bleu
|
9 |
model-index:
|
|
|
22 |
- name: Bleu
|
23 |
type: bleu
|
24 |
value: 67.0512
|
25 |
+
language:
|
26 |
+
- en
|
27 |
+
- pt
|
28 |
+
pipeline_tag: translation
|
29 |
---
|
30 |
|
31 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
|
|
43 |
|
44 |
More information needed
|
45 |
|
46 |
+
## How to use
|
47 |
+
|
48 |
+
First please make sure to install `pip install transformers`. First download model:
|
49 |
+
|
50 |
+
```python
|
51 |
+
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
|
52 |
+
import torch
|
53 |
+
|
54 |
+
def translate(input_text, lang):
|
55 |
+
input_ids = tokenizer(input_text, return_tensors="pt")
|
56 |
+
generated_tokens = model.generate(**input_ids, forced_bos_token_id=tokenizer.get_lang_id(lang))
|
57 |
+
return tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
|
58 |
+
|
59 |
+
model_name = "cartesinus/iva_mt_wslot-m2m100_418M-0.1.0-en-pt"
|
60 |
+
tokenizer = M2M100Tokenizer.from_pretrained(model_name, src_lang="en", tgt_lang="pt")
|
61 |
+
model = M2M100ForConditionalGeneration.from_pretrained(model_name)
|
62 |
+
```
|
63 |
+
|
64 |
+
Then you can translate either plain text like this:
|
65 |
+
```python
|
66 |
+
print(translate("set the temperature on my thermostat", "pt"))
|
67 |
+
```
|
68 |
+
or you can translate with slot annotations that will be restored in tgt language:
|
69 |
+
```python
|
70 |
+
print(translate("wake me up at <a>nine am<a> on <b>friday<b>", "pt"))
|
71 |
+
```
|
72 |
+
Limitations of translation with slot transfer:
|
73 |
+
1) Annotated words must be placed between semi-xml tags like this "this is \<a\>example\<a\>"
|
74 |
+
2) There is no closing tag for example "\<\a\>" in the above example - this is done on purpose to omit problems with backslash escape
|
75 |
+
3) If the sentence consists of more than one slot then simply use the next alphabet letter. For example "this is \<a\>example\<a\> with more than \<b\>one\<b\> slot"
|
76 |
+
4) Please do not add space before the first or last annotated word because this particular model was trained this way and it most probably will lower its results
|
77 |
|
78 |
## Training procedure
|
79 |
|