# # Fast-Inference with Ctranslate2
Speedup inference by 2x-8x using int8 inference in C++

quantized version of [Helsinki-NLP/opus-mt-en-es](https://huggingface.co/Helsinki-NLP/opus-mt-en-es)
```bash
pip install hf-hub-ctranslate2>=1.0.0 ctranslate2>=3.13.0
```
Converted using
```
ct2-transformers-converter --model Helsinki-NLP/opus-mt-en-es --output_dir /home/michael/tmp-ct2fast-opus-mt-en-es --force --copy_files README.md generation_config.json tokenizer_config.json vocab.json source.spm .gitattributes target.spm --quantization float16
```

Checkpoint compatible to [ctranslate2](https://github.com/OpenNMT/CTranslate2) and [hf-hub-ctranslate2](https://github.com/michaelfeil/hf-hub-ctranslate2)
- `compute_type=int8_float16` for `device="cuda"` 
- `compute_type=int8`  for `device="cpu"`

```python
from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
from transformers import AutoTokenizer

model_name = "michaelfeil/ct2fast-opus-mt-en-es"
# use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model.
model = TranslatorCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name, 
        device="cuda",
        compute_type="int8_float16",
        tokenizer=AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-es")
)
outputs = model.generate(
    text=["How do you call a fast Flan-ingo?", "User: How are you doing?"],
)
print(outputs)
```

# Licence and other remarks:
This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.

# Original description
    

### eng-spa

* source group: English 
* target group: Spanish 
*  OPUS readme: [eng-spa](https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/eng-spa/README.md)

*  model: transformer
* source language(s): eng
* target language(s): spa
* model: transformer
* pre-processing: normalization + SentencePiece (spm32k,spm32k)
* download original weights: [opus-2020-08-18.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.zip)
* test set translations: [opus-2020-08-18.test.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.test.txt)
* test set scores: [opus-2020-08-18.eval.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.eval.txt)

## Benchmarks

| testset               | BLEU  | chr-F |
|-----------------------|-------|-------|
| newssyscomb2009-engspa.eng.spa 	| 31.0 	| 0.583 |
| news-test2008-engspa.eng.spa 	| 29.7 	| 0.564 |
| newstest2009-engspa.eng.spa 	| 30.2 	| 0.578 |
| newstest2010-engspa.eng.spa 	| 36.9 	| 0.620 |
| newstest2011-engspa.eng.spa 	| 38.2 	| 0.619 |
| newstest2012-engspa.eng.spa 	| 39.0 	| 0.625 |
| newstest2013-engspa.eng.spa 	| 35.0 	| 0.598 |
| Tatoeba-test.eng.spa 	| 54.9 	| 0.721 |


### System Info: 
- hf_name: eng-spa

- source_languages: eng

- target_languages: spa

- opus_readme_url: https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/eng-spa/README.md

- original_repo: Tatoeba-Challenge

- tags:
- ctranslate2 ['translation']

- languages: ['en', 'es']

- src_constituents: {'eng'}

- tgt_constituents: {'spa'}

- src_multilingual: False

- tgt_multilingual: False

- prepro:  normalization + SentencePiece (spm32k,spm32k)

- url_model: https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.zip

- url_test_set: https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.test.txt

- src_alpha3: eng

- tgt_alpha3: spa

- short_pair: en-es

- chrF2_score: 0.721

- bleu: 54.9

- brevity_penalty: 0.978

- ref_len: 77311.0

- src_name: English

- tgt_name: Spanish

- train_date: 2020-08-18 00:00:00

- src_alpha2: en

- tgt_alpha2: es

- prefer_old: False

- long_pair: eng-spa

- helsinki_git_sha: d2f0910c89026c34a44e331e785dec1e0faa7b82

- transformers_git_sha: f7af09b4524b784d67ae8526f0e2fcc6f5ed0de9

- port_machine: brutasse

- port_time: 2020-08-24-18:20
{"language": ["en", "es"], "license": "apache-2.0", "tags": ["ctranslate2", "translation"]}
transformers ctranslate2 translation en es license:apache-2.0 endpoints_compatible region:us
TAGS
#transformers #ctranslate2 #translation #en #es #license-apache-2.0 #endpoints_compatible #region-us 

# Fast-Inference with Ctranslate2
=================================


Speedup inference by 2x-8x using int8 inference in C++


quantized version of Helsinki-NLP/opus-mt-en-es


Converted using


Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2


* 'compute\_type=int8\_float16' for 'device="cuda"'
* 'compute\_type=int8' for 'device="cpu"'


Licence and other remarks:
==========================


This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.


Original description
====================


### eng-spa


* source group: English
* target group: Spanish
* OPUS readme: eng-spa
* model: transformer
* source language(s): eng
* target language(s): spa
* model: transformer
* pre-processing: normalization + SentencePiece (spm32k,spm32k)
* download original weights: URL
* test set translations: URL
* test set scores: URL


Benchmarks
----------


testset: URL, BLEU: 31.0, chr-F: 0.583
testset: URL, BLEU: 29.7, chr-F: 0.564
testset: URL, BLEU: 30.2, chr-F: 0.578
testset: URL, BLEU: 36.9, chr-F: 0.620
testset: URL, BLEU: 38.2, chr-F: 0.619
testset: URL, BLEU: 39.0, chr-F: 0.625
testset: URL, BLEU: 35.0, chr-F: 0.598
testset: URL, BLEU: 54.9, chr-F: 0.721


### System Info:


* hf\_name: eng-spa
* source\_languages: eng
* target\_languages: spa
* opus\_readme\_url: URL
* original\_repo: Tatoeba-Challenge
* tags:
* ctranslate2 ['translation']
* languages: ['en', 'es']
* src\_constituents: {'eng'}
* tgt\_constituents: {'spa'}
* src\_multilingual: False
* tgt\_multilingual: False
* prepro: normalization + SentencePiece (spm32k,spm32k)
* url\_model: URL
* url\_test\_set: URL
* src\_alpha3: eng
* tgt\_alpha3: spa
* short\_pair: en-es
* chrF2\_score: 0.721
* bleu: 54.9
* brevity\_penalty: 0.978
* ref\_len: 77311.0
* src\_name: English
* tgt\_name: Spanish
* train\_date: 2020-08-18 00:00:00
* src\_alpha2: en
* tgt\_alpha2: es
* prefer\_old: False
* long\_pair: eng-spa
* helsinki\_git\_sha: d2f0910c89026c34a44e331e785dec1e0faa7b82
* transformers\_git\_sha: f7af09b4524b784d67ae8526f0e2fcc6f5ed0de9
* port\_machine: brutasse
* port\_time: 2020-08-24-18:20



# Fast-Inference with Ctranslate2
=================================


Speedup inference by 2x-8x using int8 inference in C++


quantized version of Helsinki-NLP/opus-mt-en-es


Converted using


Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2


* 'compute\_type=int8\_float16' for 'device="cuda"'
* 'compute\_type=int8' for 'device="cpu"'


Licence and other remarks:
==========================


This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.


Original description
==================== ### eng-spa


* source group: English
* target group: Spanish
* OPUS readme: eng-spa
* model: transformer
* source language(s): eng
* target language(s): spa
* model: transformer
* pre-processing: normalization + SentencePiece (spm32k,spm32k)
* download original weights: URL
* test set translations: URL
* test set scores: URL


Benchmarks
----------


testset: URL, BLEU: 31.0, chr-F: 0.583
testset: URL, BLEU: 29.7, chr-F: 0.564
testset: URL, BLEU: 30.2, chr-F: 0.578
testset: URL, BLEU: 36.9, chr-F: 0.620
testset: URL, BLEU: 38.2, chr-F: 0.619
testset: URL, BLEU: 39.0, chr-F: 0.625
testset: URL, BLEU: 35.0, chr-F: 0.598
testset: URL, BLEU: 54.9, chr-F: 0.721 ### System Info:


* hf\_name: eng-spa
* source\_languages: eng
* target\_languages: spa
* opus\_readme\_url: URL
* original\_repo: Tatoeba-Challenge
* tags:
* ctranslate2 ['translation']
* languages: ['en', 'es']
* src\_constituents: {'eng'}
* tgt\_constituents: {'spa'}
* src\_multilingual: False
* tgt\_multilingual: False
* prepro: normalization + SentencePiece (spm32k,spm32k)
* url\_model: URL
* url\_test\_set: URL
* src\_alpha3: eng
* tgt\_alpha3: spa
* short\_pair: en-es
* chrF2\_score: 0.721
* bleu: 54.9
* brevity\_penalty: 0.978
* ref\_len: 77311.0
* src\_name: English
* tgt\_name: Spanish
* train\_date: 2020-08-18 00:00:00
* src\_alpha2: en
* tgt\_alpha2: es
* prefer\_old: False
* long\_pair: eng-spa
* helsinki\_git\_sha: d2f0910c89026c34a44e331e785dec1e0faa7b82
* transformers\_git\_sha: f7af09b4524b784d67ae8526f0e2fcc6f5ed0de9
* port\_machine: brutasse
* port\_time: 2020-08-24-18:20
TAGS
#transformers #ctranslate2 #translation #en #es #license-apache-2.0 #endpoints_compatible #region-us 
 # Fast-Inference with Ctranslate2
=================================


Speedup inference by 2x-8x using int8 inference in C++


quantized version of Helsinki-NLP/opus-mt-en-es


Converted using


Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2


* 'compute\_type=int8\_float16' for 'device="cuda"'
* 'compute\_type=int8' for 'device="cpu"'


Licence and other remarks:
==========================


This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.


Original description
==================== ### eng-spa


* source group: English
* target group: Spanish
* OPUS readme: eng-spa
* model: transformer
* source language(s): eng
* target language(s): spa
* model: transformer
* pre-processing: normalization + SentencePiece (spm32k,spm32k)
* download original weights: URL
* test set translations: URL
* test set scores: URL


Benchmarks
----------


testset: URL, BLEU: 31.0, chr-F: 0.583
testset: URL, BLEU: 29.7, chr-F: 0.564
testset: URL, BLEU: 30.2, chr-F: 0.578
testset: URL, BLEU: 36.9, chr-F: 0.620
testset: URL, BLEU: 38.2, chr-F: 0.619
testset: URL, BLEU: 39.0, chr-F: 0.625
testset: URL, BLEU: 35.0, chr-F: 0.598
testset: URL, BLEU: 54.9, chr-F: 0.721 ### System Info:


* hf\_name: eng-spa
* source\_languages: eng
* target\_languages: spa
* opus\_readme\_url: URL
* original\_repo: Tatoeba-Challenge
* tags:
* ctranslate2 ['translation']
* languages: ['en', 'es']
* src\_constituents: {'eng'}
* tgt\_constituents: {'spa'}
* src\_multilingual: False
* tgt\_multilingual: False
* prepro: normalization + SentencePiece (spm32k,spm32k)
* url\_model: URL
* url\_test\_set: URL
* src\_alpha3: eng
* tgt\_alpha3: spa
* short\_pair: en-es
* chrF2\_score: 0.721
* bleu: 54.9
* brevity\_penalty: 0.978
* ref\_len: 77311.0
* src\_name: English
* tgt\_name: Spanish
* train\_date: 2020-08-18 00:00:00
* src\_alpha2: en
* tgt\_alpha2: es
* prefer\_old: False
* long\_pair: eng-spa
* helsinki\_git\_sha: d2f0910c89026c34a44e331e785dec1e0faa7b82
* transformers\_git\_sha: f7af09b4524b784d67ae8526f0e2fcc6f5ed0de9
* port\_machine: brutasse
* port\_time: 2020-08-24-18:20