Defts-lab commited on
Commit
d56186a
1 Parent(s): 762e19d

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +278 -0
README.md ADDED
@@ -0,0 +1,278 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets:
3
+ - flores-200
4
+ language:
5
+ - ace
6
+ - acm
7
+ - acq
8
+ - aeb
9
+ - af
10
+ - ajp
11
+ - ak
12
+ - als
13
+ - am
14
+ - apc
15
+ - ar
16
+ - ars
17
+ - ary
18
+ - arz
19
+ - as
20
+ - ast
21
+ - awa
22
+ - ayr
23
+ - azb
24
+ - azj
25
+ - ba
26
+ - bm
27
+ - ban
28
+ - be
29
+ - bem
30
+ - bn
31
+ - bho
32
+ - bjn
33
+ - bo
34
+ - bs
35
+ - bug
36
+ - bg
37
+ - ca
38
+ - ceb
39
+ - cs
40
+ - cjk
41
+ - ckb
42
+ - crh
43
+ - cy
44
+ - da
45
+ - de
46
+ - dik
47
+ - dyu
48
+ - dz
49
+ - el
50
+ - en
51
+ - eo
52
+ - et
53
+ - eu
54
+ - ee
55
+ - fo
56
+ - fj
57
+ - fi
58
+ - fon
59
+ - fr
60
+ - fur
61
+ - fuv
62
+ - gaz
63
+ - gd
64
+ - ga
65
+ - gl
66
+ - gn
67
+ - gu
68
+ - ht
69
+ - ha
70
+ - he
71
+ - hi
72
+ - hne
73
+ - hr
74
+ - hu
75
+ - hy
76
+ - ig
77
+ - ilo
78
+ - id
79
+ - is
80
+ - it
81
+ - jv
82
+ - ja
83
+ - kab
84
+ - kac
85
+ - kam
86
+ - kn
87
+ - ks
88
+ - ka
89
+ - kk
90
+ - kbp
91
+ - kea
92
+ - khk
93
+ - km
94
+ - ki
95
+ - rw
96
+ - ky
97
+ - kmb
98
+ - kmr
99
+ - knc
100
+ - kg
101
+ - ko
102
+ - lo
103
+ - lij
104
+ - li
105
+ - ln
106
+ - lt
107
+ - lmo
108
+ - ltg
109
+ - lb
110
+ - lua
111
+ - lg
112
+ - luo
113
+ - lus
114
+ - lvs
115
+ - mag
116
+ - mai
117
+ - ml
118
+ - mar
119
+ - min
120
+ - mk
121
+ - mt
122
+ - mni
123
+ - mos
124
+ - mi
125
+ - my
126
+ - nl
127
+ - nn
128
+ - nb
129
+ - npi
130
+ - nso
131
+ - nus
132
+ - ny
133
+ - oc
134
+ - ory
135
+ - pag
136
+ - pa
137
+ - pap
138
+ - pbt
139
+ - pes
140
+ - plt
141
+ - pl
142
+ - pt
143
+ - prs
144
+ - quy
145
+ - ro
146
+ - rn
147
+ - ru
148
+ - sg
149
+ - sa
150
+ - sat
151
+ - scn
152
+ - shn
153
+ - si
154
+ - sk
155
+ - sl
156
+ - sm
157
+ - sn
158
+ - sd
159
+ - so
160
+ - st
161
+ - es
162
+ - sc
163
+ - sr
164
+ - ss
165
+ - su
166
+ - sv
167
+ - swh
168
+ - szl
169
+ - ta
170
+ - taq
171
+ - tt
172
+ - te
173
+ - tg
174
+ - tl
175
+ - th
176
+ - ti
177
+ - tpi
178
+ - tn
179
+ - ts
180
+ - tk
181
+ - tum
182
+ - tr
183
+ - tw
184
+ - tzm
185
+ - ug
186
+ - uk
187
+ - umb
188
+ - ur
189
+ - uzn
190
+ - vec
191
+ - vi
192
+ - war
193
+ - wo
194
+ - xh
195
+ - ydd
196
+ - yo
197
+ - yue
198
+ - zh
199
+ - zsm
200
+ - zu
201
+ license: cc-by-nc-4.0
202
+ metrics:
203
+ - bleu
204
+ - spbleu
205
+ - chrf++
206
+ tags:
207
+ - nllb
208
+ - translation
209
+ - autoquant
210
+ - gguf
211
+ language_details: ace_Arab, ace_Latn, acm_Arab, acq_Arab, aeb_Arab, afr_Latn, ajp_Arab,
212
+ aka_Latn, amh_Ethi, apc_Arab, arb_Arab, ars_Arab, ary_Arab, arz_Arab, asm_Beng,
213
+ ast_Latn, awa_Deva, ayr_Latn, azb_Arab, azj_Latn, bak_Cyrl, bam_Latn, ban_Latn,bel_Cyrl,
214
+ bem_Latn, ben_Beng, bho_Deva, bjn_Arab, bjn_Latn, bod_Tibt, bos_Latn, bug_Latn,
215
+ bul_Cyrl, cat_Latn, ceb_Latn, ces_Latn, cjk_Latn, ckb_Arab, crh_Latn, cym_Latn,
216
+ dan_Latn, deu_Latn, dik_Latn, dyu_Latn, dzo_Tibt, ell_Grek, eng_Latn, epo_Latn,
217
+ est_Latn, eus_Latn, ewe_Latn, fao_Latn, pes_Arab, fij_Latn, fin_Latn, fon_Latn,
218
+ fra_Latn, fur_Latn, fuv_Latn, gla_Latn, gle_Latn, glg_Latn, grn_Latn, guj_Gujr,
219
+ hat_Latn, hau_Latn, heb_Hebr, hin_Deva, hne_Deva, hrv_Latn, hun_Latn, hye_Armn,
220
+ ibo_Latn, ilo_Latn, ind_Latn, isl_Latn, ita_Latn, jav_Latn, jpn_Jpan, kab_Latn,
221
+ kac_Latn, kam_Latn, kan_Knda, kas_Arab, kas_Deva, kat_Geor, knc_Arab, knc_Latn,
222
+ kaz_Cyrl, kbp_Latn, kea_Latn, khm_Khmr, kik_Latn, kin_Latn, kir_Cyrl, kmb_Latn,
223
+ kon_Latn, kor_Hang, kmr_Latn, lao_Laoo, lvs_Latn, lij_Latn, lim_Latn, lin_Latn,
224
+ lit_Latn, lmo_Latn, ltg_Latn, ltz_Latn, lua_Latn, lug_Latn, luo_Latn, lus_Latn,
225
+ mag_Deva, mai_Deva, mal_Mlym, mar_Deva, min_Latn, mkd_Cyrl, plt_Latn, mlt_Latn,
226
+ mni_Beng, khk_Cyrl, mos_Latn, mri_Latn, zsm_Latn, mya_Mymr, nld_Latn, nno_Latn,
227
+ nob_Latn, npi_Deva, nso_Latn, nus_Latn, nya_Latn, oci_Latn, gaz_Latn, ory_Orya,
228
+ pag_Latn, pan_Guru, pap_Latn, pol_Latn, por_Latn, prs_Arab, pbt_Arab, quy_Latn,
229
+ ron_Latn, run_Latn, rus_Cyrl, sag_Latn, san_Deva, sat_Beng, scn_Latn, shn_Mymr,
230
+ sin_Sinh, slk_Latn, slv_Latn, smo_Latn, sna_Latn, snd_Arab, som_Latn, sot_Latn,
231
+ spa_Latn, als_Latn, srd_Latn, srp_Cyrl, ssw_Latn, sun_Latn, swe_Latn, swh_Latn,
232
+ szl_Latn, tam_Taml, tat_Cyrl, tel_Telu, tgk_Cyrl, tgl_Latn, tha_Thai, tir_Ethi,
233
+ taq_Latn, taq_Tfng, tpi_Latn, tsn_Latn, tso_Latn, tuk_Latn, tum_Latn, tur_Latn,
234
+ twi_Latn, tzm_Tfng, uig_Arab, ukr_Cyrl, umb_Latn, urd_Arab, uzn_Latn, vec_Latn,
235
+ vie_Latn, war_Latn, wol_Latn, xho_Latn, ydd_Hebr, yor_Latn, yue_Hant, zho_Hans,
236
+ zho_Hant, zul_Latn
237
+ inference: false
238
+ ---
239
+
240
+ # NLLB-200
241
+
242
+ This is the model card of NLLB-200's 3.3B variant.
243
+
244
+ Here are the [metrics](https://tinyurl.com/nllb200dense3bmetrics) for that particular checkpoint.
245
+
246
+ - Information about training algorithms, parameters, fairness constraints or other applied approaches, and features. The exact training algorithm, data and the strategies to handle data imbalances for high and low resource languages that were used to train NLLB-200 is described in the paper.
247
+ - Paper or other resource for more information NLLB Team et al, No Language Left Behind: Scaling Human-Centered Machine Translation, Arxiv, 2022
248
+ - License: CC-BY-NC
249
+ - Where to send questions or comments about the model: https://github.com/facebookresearch/fairseq/issues
250
+
251
+
252
+
253
+ ## Intended Use
254
+ - Primary intended uses: NLLB-200 is a machine translation model primarily intended for research in machine translation, - especially for low-resource languages. It allows for single sentence translation among 200 languages. Information on how to - use the model can be found in Fairseq code repository along with the training code and references to evaluation and training data.
255
+ - Primary intended users: Primary users are researchers and machine translation research community.
256
+ - Out-of-scope use cases: NLLB-200 is a research model and is not released for production deployment. NLLB-200 is trained on general domain text data and is not intended to be used with domain specific texts, such as medical domain or legal domain. The model is not intended to be used for document translation. The model was trained with input lengths not exceeding 512 tokens, therefore translating longer sequences might result in quality degradation. NLLB-200 translations can not be used as certified translations.
257
+
258
+ ## Metrics
259
+ • Model performance measures: NLLB-200 model was evaluated using BLEU, spBLEU, and chrF++ metrics widely adopted by machine translation community. Additionally, we performed human evaluation with the XSTS protocol and measured the toxicity of the generated translations.
260
+
261
+
262
+ ## Evaluation Data
263
+ - Datasets: Flores-200 dataset is described in Section 4
264
+ - Motivation: We used Flores-200 as it provides full evaluation coverage of the languages in NLLB-200
265
+ - Preprocessing: Sentence-split raw text data was preprocessed using SentencePiece. The
266
+ SentencePiece model is released along with NLLB-200.
267
+
268
+ ## Training Data
269
+ • We used parallel multilingual data from a variety of sources to train the model. We provide detailed report on data selection and construction process in Section 5 in the paper. We also used monolingual data constructed from Common Crawl. We provide more details in Section 5.2.
270
+
271
+ ## Ethical Considerations
272
+ • In this work, we took a reflexive approach in technological development to ensure that we prioritize human users and minimize risks that could be transferred to them. While we reflect on our ethical considerations throughout the article, here are some additional points to highlight. For one, many languages chosen for this study are low-resource languages, with a heavy emphasis on African languages. While quality translation could improve education and information access in many in these communities, such an access could also make groups with lower levels of digital literacy more vulnerable to misinformation or online scams. The latter scenarios could arise if bad actors misappropriate our work for nefarious activities, which we conceive as an example of unintended use. Regarding data acquisition, the training data used for model development were mined from various publicly available sources on the web. Although we invested heavily in data cleaning, personally identifiable information may not be entirely eliminated. Finally, although we did our best to optimize for translation quality, mistranslations produced by the model could remain. Although the odds are low, this could have adverse impact on those who rely on these translations to make important decisions (particularly when related to health and safety).
273
+
274
+ ## Caveats and Recommendations
275
+ • Our model has been tested on the Wikimedia domain with limited investigation on other domains supported in NLLB-MD. In addition, the supported languages may have variations that our model is not capturing. Users should make appropriate assessments.
276
+
277
+ ## Carbon Footprint Details
278
+ • The carbon dioxide (CO2e) estimate is reported in Section 8.8.