README.md

---
license: cc-by-sa-4.0
datasets:
- Mitsua/wikidata-parallel-descriptions-en-ja
language:
- ja
- en
metrics:
- bleu
- chrf
library_name: transformers
pipeline_tag: translation
---
# ElanMT
[**ElanMT-BT-ja-en**](https://huggingface.co/Mitsua/elan-mt-bt-ja-en) is a Japanese to English translation model developed by [ELAN MITSUA Project](https://elanmitsua.com/en/) / Abstract Engine. 
- [**ElanMT-base-ja-en**](https://huggingface.co/Mitsua/elan-mt-base-ja-en) and [**ElanMT-base-en-ja**](https://huggingface.co/Mitsua/elan-mt-base-en-ja) are trained from scratch, exclusively on openly licensed corpora such as CC0, CC BY and CC BY-SA.
- This model is a fine-tuned checkpoint of **ElanMT-base-ja-en** and is trained exclusively on openly licensed data and Wikipedia back translated data using **ElanMT-base-en-ja**.
- Web crawled or other machine translated corpora are **not** used during the entire training procedure for the **ElanMT** models.

Despite the relatively low resource training, thanks to back-translation and [a newly built CC0 corpus](https://huggingface.co/datasets/Mitsua/wikidata-parallel-descriptions-en-ja),
the model achieved comparable performance to the currently available open translation models.

## Model Details
This is a translation model based on [Marian MT](https://marian-nmt.github.io/) 6-layer encoder-decoder transformer architecture with sentencepiece tokenizer.
- **Developed by**: [ELAN MITSUA Project](https://elanmitsua.com/en/) / Abstract Engine
- **Model type**: Translation
- **Source Language**: Japanese
- **Target Language**: English
- **License**: [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)

## Usage
1. Install the python packages

`pip install transformers accelerate sentencepiece`

* This model is verified on `transformers==4.40.2`

2. Run

```python
from transformers import pipeline
translator = pipeline('translation', model='Mitsua/elan-mt-bt-ja-en')
translator('こんにちは。私はAIです。')
```

3. For longer multiple sentences, using [pySBD](https://github.com/nipunsadvilkar/pySBD) is recommended.

`pip install transformers accelerate sentencepiece pysbd`
```python
import pysbd
seg = pysbd.Segmenter(language="ja", clean=False)
txt = 'こんにちは。私はAIです。お元気ですか？'
print(translator(seg.segment(txt)))
```
This idea is from [FuguMT](https://huggingface.co/staka/fugumt-ja-en) repo.

## Training Data
We heavily referred [FuguMT author's blog post](https://staka.jp/wordpress/?p=413) for dataset collection.

- [Mitsua/wikidata-parallel-descriptions-en-ja](https://huggingface.co/datasets/Mitsua/wikidata-parallel-descriptions-en-ja) (CC0 1.0)  
  - We newly built this 1.5M lines wikidata parallel corpus to augment the training data. This greatly improved the vocabulary on a word basis.
- [The Kyoto Free Translation Task (KFTT)](https://www.phontron.com/kftt/) (CC BY-SA 3.0)
  - Graham Neubig, "The Kyoto Free Translation Task," http://www.phontron.com/kftt, 2011.
- [Tatoeba](https://tatoeba.org/en/downloads) (CC BY 2.0 FR / CC0 1.0)
  - https://tatoeba.org/
- [wikipedia-interlanguage-titles](https://github.com/bhaddow/wikipedia-interlanguage-titles) (The MIT License / CC BY-SA 4.0)
  - We built parallel titles based on 2024-05-06 wikipedia dump.
- [WikiMatrix](https://github.com/facebookresearch/LASER/tree/main/tasks/WikiMatrix) (CC BY-SA 4.0)
  - Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong and Francisco Guzmán, "WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia"
- [MDN Web Docs](https://github.com/mdn/translated-content) (The MIT / CC0 1.0 / CC BY-SA 2.5)
  - https://github.com/mdn/translated-content
- [Wikimedia contenttranslation dump](https://dumps.wikimedia.org/other/contenttranslation/) (CC BY-SA 4.0)
  - 2024-5-10 dump is used.
 
*Even if the dataset itself is CC-licensed, we did not use it if the corpus contained in the dataset is based on web crawling, is based on unauthorized use of copyrighted works, or is based on the machine translation output of other translation models.

## Training Procedure
We heavily referred "[Beating Edinburgh's WMT2017 system for en-de with Marian's Transformer model](https://github.com/marian-nmt/marian-examples/tree/master/wmt2017-transformer)"
for training process and hyperparameter tuning.

1. Trains a sentencepiece tokenizer 32k vocab on 4M lines openly licensed corpus.
2. Trains `en-ja` back-translation model on 4M lines openly licensed corpus for 6 epochs. = **ElanMT-base-en-ja**
3. Trains `ja-en` base translation model on 4M lines openly licensed corpus for 6 epochs. = **ElanMT-base-ja-en**
4. Translates 20M lines `en` Wikipedia to `ja` using back-translation model.
5. Trains 4 `ja-en` models, which is finetuned from **ElanMT-base-ja-en** checkpoint, on 24M lines training data augmented with back-translated data for 6 epochs.
6. Merges 4 trained models that produces the best validation score on FLORES+ dev split.
7. Finetunes the merged model on 1M lines high quality corpus subset for 5 epochs.

## Evaluation
### Dataset
- [FLORES+](https://github.com/openlanguagedata/flores) (CC BY-SA 4.0) devtest split is used for evaluation.
- [NTREX](https://github.com/MicrosoftTranslator/NTREX) (CC BY-SA 4.0)

### Result
| **Model** | **Params** | **FLORES+ BLEU** | **FLORES+ chrf** | **NTREX BLEU** | **NTREX chrf** |  
|:---|---:|---:|---:|---:|---:|
| [**ElanMT-BT**](https://huggingface.co/Mitsua/elan-mt-bt-ja-en) | 61M | 24.87 | 55.02 | 22.57 | 52.48|
| [**ElanMT-base**](https://huggingface.co/Mitsua/elan-mt-base-ja-en) | 61M | 21.61 | 52.53 | 18.43 | 49.09|
| [**ElanMT-tiny**](https://huggingface.co/Mitsua/elan-mt-tiny-ja-en) | 15M | 20.40 | 51.81 | 18.43 | 49.39|
| [staka/fugumt-ja-en](https://huggingface.co/staka/fugumt-ja-en) | 61M | 24.10 | 54.97 | 22.33 | 51.84|
| [facebook/mbart-large-50-many-to-many-mmt](https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt) | 610M | 23.88 | 53.98 | 22.59 | 51.57|
| [facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) | 615M | 22.92 | 52.13 | 22.59 | 51.36|
| [facebook/nllb-200-3.3B](https://huggingface.co/facebook/nllb-200-3.3B) | 3B | 28.13 | 56.86 | 27.65 | 55.60|
| [google/madlad400-3b-mt](https://huggingface.co/google/madlad400-3b-mt) | 3B | 26.95 | 56.62 | 26.11 | 54.61|
| [google/madlad400-7b-mt](https://huggingface.co/google/madlad400-7b-mt) | 7B | 28.84 | 57.46 | 28.19 | 55.85|

- *1 tested on `transformers==4.29.2` and `num_beams=4`
- *2 BLEU score is calculated by `sacreBLEU`

## Disclaimer
- The translated result may be very incorrect, harmful or biased. The model was developed to investigate achievable performance with only a relatively small, licensed corpus, and is not suitable for use cases requiring high translation accuracy. Under Section 5 of the CC BY-SA 4.0 License, ELAN MITSUA Project / Abstract Engine is not responsible for any direct or indirect loss caused by the use of the model.
- 免責事項：翻訳結果は不正確で、有害であったりバイアスがかかっている可能性があります。本モデルは比較的小規模でライセンスされたコーパスのみで達成可能な性能を調査するために開発されたモデルであり、翻訳の正確性が必要なユースケースでの使用には適していません。絵藍ミツアプロジェクト及び株式会社アブストラクトエンジンはCC BY-SA 4.0ライセンス第5条に基づき、本モデルの使用によって生じた直接的または間接的な損失に対して、一切の責任を負いません。