# # Fast-Inference with Ctranslate2
Speedup inference by 2x-8x using int8 inference in C++

quantized version of [Helsinki-NLP/opus-mt-en-de](https://huggingface.co/Helsinki-NLP/opus-mt-en-de)
```bash
pip install hf-hub-ctranslate2>=1.0.0 ctranslate2>=3.13.0
```
Converted using
```
ct2-transformers-converter --model Helsinki-NLP/opus-mt-en-de --output_dir /home/michael/tmp-ct2fast-opus-mt-en-de --force --copy_files README.md generation_config.json tokenizer_config.json vocab.json source.spm .gitattributes target.spm --quantization float16
```

Checkpoint compatible to [ctranslate2](https://github.com/OpenNMT/CTranslate2) and [hf-hub-ctranslate2](https://github.com/michaelfeil/hf-hub-ctranslate2)
- `compute_type=int8_float16` for `device="cuda"` 
- `compute_type=int8`  for `device="cpu"`

```python
from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub
from transformers import AutoTokenizer

model_name = "michaelfeil/ct2fast-opus-mt-en-de"
# use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model.
model = TranslatorCT2fromHfHub(
        # load in int8 on CUDA
        model_name_or_path=model_name, 
        device="cuda",
        compute_type="int8_float16",
        tokenizer=AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
)
outputs = model.generate(
    text=["How do you call a fast Flan-ingo?", "User: How are you doing?"],
)
print(outputs)
```

# Licence and other remarks:
This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.

# Original description
    

### opus-mt-en-de


## Table of Contents
- [Model Details](#model-details)
- [Uses](#uses)
- [Risks, Limitations and Biases](#risks-limitations-and-biases)
- [Training](#training)
- [Evaluation](#evaluation)
- [Citation Information](#citation-information)
- [How to Get Started With the Model](#how-to-get-started-with-the-model)

## Model Details
**Model Description:**
- **Developed by:** Language Technology Research Group at the University of Helsinki
- **Model Type:** Translation
- **Language(s):**  
  - Source Language: English
  - Target Language: German 
- **License:** CC-BY-4.0
- **Resources for more information:**
  - [GitHub Repo](https://github.com/Helsinki-NLP/OPUS-MT-train)
  

## Uses

#### Direct Use

This model can be used for translation and text-to-text generation.


## Risks, Limitations and Biases



**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**

Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).

Further details about the dataset for this model can be found in the OPUS readme: [en-de](https://github.com/Helsinki-NLP/OPUS-MT-train/blob/master/models/en-de/README.md)


#### Training Data
##### Preprocessing
* pre-processing: normalization + SentencePiece

* dataset: [opus](https://github.com/Helsinki-NLP/Opus-MT)
* download original weights: [opus-2020-02-26.zip](https://object.pouta.csc.fi/OPUS-MT-models/en-de/opus-2020-02-26.zip)

* test set translations: [opus-2020-02-26.test.txt](https://object.pouta.csc.fi/OPUS-MT-models/en-de/opus-2020-02-26.test.txt)

## Evaluation

#### Results

* test set scores: [opus-2020-02-26.eval.txt](https://object.pouta.csc.fi/OPUS-MT-models/en-de/opus-2020-02-26.eval.txt)


#### Benchmarks

| testset               | BLEU  | chr-F |
|-----------------------|-------|-------|
| newssyscomb2009.en.de 	| 23.5 	| 0.540 |
| news-test2008.en.de 	| 23.5 	| 0.529 |
| newstest2009.en.de 	| 22.3 	| 0.530 |
| newstest2010.en.de 	| 24.9 	| 0.544 |
| newstest2011.en.de 	| 22.5 	| 0.524 |
| newstest2012.en.de 	| 23.0 	| 0.525 |
| newstest2013.en.de 	| 26.9 	| 0.553 |
| newstest2015-ende.en.de 	| 31.1 	| 0.594 |
| newstest2016-ende.en.de 	| 37.0 	| 0.636 |
| newstest2017-ende.en.de 	| 29.9 	| 0.586 |
| newstest2018-ende.en.de 	| 45.2 	| 0.690 |
| newstest2019-ende.en.de 	| 40.9 	| 0.654 |
| Tatoeba.en.de 	| 47.3 	| 0.664 |



## Citation Information

```bibtex
@InProceedings{TiedemannThottingal:EAMT2020,
  author = {J{\"o}rg Tiedemann and Santhosh Thottingal},
  title = {{OPUS-MT} — {B}uilding open translation services for the {W}orld},
  booktitle = {Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT)},
  year = {2020},
  address = {Lisbon, Portugal}
 }
```

## How to Get Started With the Model
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")

model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-de")

```





{"license": "cc-by-4.0", "tags": ["ctranslate2", "translation"]}
transformers ctranslate2 translation license:cc-by-4.0 endpoints_compatible region:us
TAGS
#transformers #ctranslate2 #translation #license-cc-by-4.0 #endpoints_compatible #region-us 

# Fast-Inference with Ctranslate2
=================================


Speedup inference by 2x-8x using int8 inference in C++


quantized version of Helsinki-NLP/opus-mt-en-de


Converted using


Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2


* 'compute\_type=int8\_float16' for 'device="cuda"'
* 'compute\_type=int8' for 'device="cpu"'


Licence and other remarks:
==========================


This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.


Original description
====================


### opus-mt-en-de


Table of Contents
-----------------


* Model Details
* Uses
* Risks, Limitations and Biases
* Training
* Evaluation
* Citation Information
* How to Get Started With the Model


Model Details
-------------


Model Description:


* Developed by: Language Technology Research Group at the University of Helsinki
* Model Type: Translation
* Language(s):
	+ Source Language: English
	+ Target Language: German
* License: CC-BY-4.0
* Resources for more information:
	+ GitHub Repo


Uses
----


#### Direct Use


This model can be used for translation and text-to-text generation.


Risks, Limitations and Biases
-----------------------------


CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.


Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).


Further details about the dataset for this model can be found in the OPUS readme: en-de


#### Training Data


##### Preprocessing


* pre-processing: normalization + SentencePiece
* dataset: opus
* download original weights: URL
* test set translations: URL


Evaluation
----------


#### Results


* test set scores: URL


#### Benchmarks


testset: URL, BLEU: 23.5, chr-F: 0.540
testset: URL, BLEU: 23.5, chr-F: 0.529
testset: URL, BLEU: 22.3, chr-F: 0.530
testset: URL, BLEU: 24.9, chr-F: 0.544
testset: URL, BLEU: 22.5, chr-F: 0.524
testset: URL, BLEU: 23.0, chr-F: 0.525
testset: URL, BLEU: 26.9, chr-F: 0.553
testset: URL, BLEU: 31.1, chr-F: 0.594
testset: URL, BLEU: 37.0, chr-F: 0.636
testset: URL, BLEU: 29.9, chr-F: 0.586
testset: URL, BLEU: 45.2, chr-F: 0.690
testset: URL, BLEU: 40.9, chr-F: 0.654
testset: URL, BLEU: 47.3, chr-F: 0.664


How to Get Started With the Model
---------------------------------



# Fast-Inference with Ctranslate2
=================================


Speedup inference by 2x-8x using int8 inference in C++


quantized version of Helsinki-NLP/opus-mt-en-de


Converted using


Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2


* 'compute\_type=int8\_float16' for 'device="cuda"'
* 'compute\_type=int8' for 'device="cpu"'


Licence and other remarks:
==========================


This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.


Original description
==================== ### opus-mt-en-de


Table of Contents
-----------------


* Model Details
* Uses
* Risks, Limitations and Biases
* Training
* Evaluation
* Citation Information
* How to Get Started With the Model


Model Details
-------------


Model Description:


* Developed by: Language Technology Research Group at the University of Helsinki
* Model Type: Translation
* Language(s):
	+ Source Language: English
	+ Target Language: German
* License: CC-BY-4.0
* Resources for more information:
	+ GitHub Repo


Uses
---- #### Direct Use


This model can be used for translation and text-to-text generation.


Risks, Limitations and Biases
-----------------------------


CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.


Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).


Further details about the dataset for this model can be found in the OPUS readme: en-de #### Training Data ##### Preprocessing


* pre-processing: normalization + SentencePiece
* dataset: opus
* download original weights: URL
* test set translations: URL


Evaluation
---------- #### Results


* test set scores: URL #### Benchmarks


testset: URL, BLEU: 23.5, chr-F: 0.540
testset: URL, BLEU: 23.5, chr-F: 0.529
testset: URL, BLEU: 22.3, chr-F: 0.530
testset: URL, BLEU: 24.9, chr-F: 0.544
testset: URL, BLEU: 22.5, chr-F: 0.524
testset: URL, BLEU: 23.0, chr-F: 0.525
testset: URL, BLEU: 26.9, chr-F: 0.553
testset: URL, BLEU: 31.1, chr-F: 0.594
testset: URL, BLEU: 37.0, chr-F: 0.636
testset: URL, BLEU: 29.9, chr-F: 0.586
testset: URL, BLEU: 45.2, chr-F: 0.690
testset: URL, BLEU: 40.9, chr-F: 0.654
testset: URL, BLEU: 47.3, chr-F: 0.664


How to Get Started With the Model
---------------------------------
TAGS
#transformers #ctranslate2 #translation #license-cc-by-4.0 #endpoints_compatible #region-us 
 # Fast-Inference with Ctranslate2
=================================


Speedup inference by 2x-8x using int8 inference in C++


quantized version of Helsinki-NLP/opus-mt-en-de


Converted using


Checkpoint compatible to ctranslate2 and hf-hub-ctranslate2


* 'compute\_type=int8\_float16' for 'device="cuda"'
* 'compute\_type=int8' for 'device="cpu"'


Licence and other remarks:
==========================


This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.


Original description
==================== ### opus-mt-en-de


Table of Contents
-----------------


* Model Details
* Uses
* Risks, Limitations and Biases
* Training
* Evaluation
* Citation Information
* How to Get Started With the Model


Model Details
-------------


Model Description:


* Developed by: Language Technology Research Group at the University of Helsinki
* Model Type: Translation
* Language(s):
	+ Source Language: English
	+ Target Language: German
* License: CC-BY-4.0
* Resources for more information:
	+ GitHub Repo


Uses
---- #### Direct Use


This model can be used for translation and text-to-text generation.


Risks, Limitations and Biases
-----------------------------


CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.


Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).


Further details about the dataset for this model can be found in the OPUS readme: en-de #### Training Data ##### Preprocessing


* pre-processing: normalization + SentencePiece
* dataset: opus
* download original weights: URL
* test set translations: URL


Evaluation
---------- #### Results


* test set scores: URL #### Benchmarks


testset: URL, BLEU: 23.5, chr-F: 0.540
testset: URL, BLEU: 23.5, chr-F: 0.529
testset: URL, BLEU: 22.3, chr-F: 0.530
testset: URL, BLEU: 24.9, chr-F: 0.544
testset: URL, BLEU: 22.5, chr-F: 0.524
testset: URL, BLEU: 23.0, chr-F: 0.525
testset: URL, BLEU: 26.9, chr-F: 0.553
testset: URL, BLEU: 31.1, chr-F: 0.594
testset: URL, BLEU: 37.0, chr-F: 0.636
testset: URL, BLEU: 29.9, chr-F: 0.586
testset: URL, BLEU: 45.2, chr-F: 0.690
testset: URL, BLEU: 40.9, chr-F: 0.654
testset: URL, BLEU: 47.3, chr-F: 0.664


How to Get Started With the Model
---------------------------------