---
abstract: 'Domain adaptation is a key feature in Machine Translation. It generally encompasses terminology, domain and style adaptation, especially for human post-editing workflows in Computer Assisted Translation (CAT). With Neural Machine Translation (NMT), we introduce a new notion of domain adaptation that we call “specialization” and which is showing promising results both in the learning speed and in adaptation accuracy. In this paper, we propose to explore this approach under several perspectives.'
author:
- |
    Christophe Servan    [and]{}    Josep Crego   [and]{}   Jean Senellart\
    [firstname.lastname@systrangroup.com]{}\
    SYSTRAN / 5 rue Feydeau, 75002 Paris, France\
bibliography:
- 'eacl2017.bib'
title: |
    Domain specialization: a post-training domain adaptation\
    for Neural Machine Translation
---

Introduction
============

Domain adaptation techniques have successfully been used in Statistical Machine Translation. It is well known that an optimized model on a specific genre (litterature, speech, IT, patent...) obtains higher accuracy results than a “generic” system. The adaptation process can be done before, during or after the training process. We propose to explore a new post-process approach, which incrementally adapt a “generic” model to a specific domain by running additional training epochs over newly available in-domain data.

In this way, adaptation proceeds incrementally when new in-domain data becomes available, generated by human translators in a post-edition context. Similar to the Computer Assisted Translation (CAT) framework described in [@Cettolo2014].

#### Contributions

The main contribution of this paper is a study of the new “specialization” approach, which aims to adapt generic NMT model without a full retraining process. Actually, it consist in using the generic model in a retraining phase, which only involves additional in-domain data. Results show this approach can reach good performances in a far less time than full-retraining, which is a key feature to adapt rapidly models in a CAT framework.

Approach
========

Following the framework proposed by [@Cettolo2014], we seek to adapt incrementally a generic model to a specific task or domain. They show incremental adaptation brings new information in a Phrase-Based Statistical Machine Translation like terminology or style, which can also belong to the human translator. Recent advances in Machine Translation focuses on Neural Machine Translation approaches, for which we propose a method to adapt incrementally to a specific domain, in this specific framework.

The main idea of the approach is to specialize a generic model already trained on generic data. Hence, we propose to retrain the generic model on specific data, though several training iterations (see figure \[fig:selection\]). The retraining process consist in re-estimating the conditional probability $p(y_1, \ldots , y_{m} |x_1, \ldots , x_n )$ where $(x_1, \ldots , x_n )$ is an input sequence of length $n$ and $(y_1, \ldots , y_{m})$ is its corresponding output sequence whose length $m$ may differ from $n$. This is done without dropping the previous learning states of the Recurrent Neural Network.

The resulting model is considered as adapted or specialized to a specific domain.

![\[fig:model\]The generic model is trained with generic data, then the generic model obtained is retrained with in-domain data to generate an specialized model.[]{data-label="fig:selection"}](figure_training){width="\linewidth"}

Experiment framework
====================

We create our own data framework described in the next section and we evaluate our results using the BLEU score [@papineni02bleu] and the TER [@Snover2006].

The Neural Machine Translation system combines the attention model approach [@Luong2015effective] jointly with the sequence-to-sequence approach [@Sutskever2014]. According to our approach, we propose to compare several configurations, in which main difference is the training corpus. On one hand, we consider the generic data and several amounts of in-domain data for the training process. On the other hand, only the generic data are considered for the training process, then several amounts of in-domain data are used only for the specialization process in a retraining phase. The main idea behind these experiment is to simulate an incremental adaptation framework, which enables the adaptation process only when data are available (e.g.: translation post-editions done by a human translator.)

The approach is studied in the light of two experiments and a short linguistic study. The first experiment concerns the impact of “specialization” approach among several additional epochs; then, the second one, focuses on the amount of data needed to observe a significant impact on the translation scores. Finally, we propose to compare some translation examples from several outputs.

Training data {#sec:data}
-------------

The table \[tab:alldata\] presents all data used in our experiments. We propose to create a generic model with comparable amount of several corpora, which each of them belong to a specific domain (IT, literature, news, parliament). All corpora are available from the OPUS repository [@Tiedemann2012parallel].

We propose to specialize the generic model using a last corpus, which is a corpus extracted from the European Medical Agency (*emea*). The corpus is composed of more than 650 documents, which are medicine manuals.

We took apart a $2K$ lines as test corpus, then, to simulate the incremental adding of data, we created four training corpora corresponding to several amount of documents: $500$, $5K$, $50K$ and all the lines of the training corpus. These amount of data are corresponding roughly to $10\%$ of a document, one document and ten documents, respectively.

  Type   Domain          \#lines   \#src tokens   \#tgt tokens
  ------ ------------- --------- -------------- --------------
         *generic*          3.4M            73M            86M
         *emea-0.5K*         500           5.6K           6.6K
         *emea-5K*            5K          56.1K          66.4K
         *emea-50K*          50K           568K           670K
         *emea-full*        922K          10.5M          12.3M
  dev.   *generic*            2K          43.7K          51.3K
  test   *emea*               2K          35.6K          42.9K

  : \[tab:alldata\] details of corpora used in this paper.

Training Details {#ssec:training}
----------------

The Neural Machine Translation approach we use is following the sequence-to-sequence approach [@Sutskever2014] combined with attentional architecture [@Luong2015effective]. In addition, all the generic and in-domain data are pre-processed using the *byte pair encoding* compression algorithm [@sennrich2016improving] with 30K operations, to avoid Out-of-Vocabulary words.

We keep the most frequent $32K$ words for both source and target languages with $4$ hidden layers with $500$-dimensional embeddings and $800$ bidirectional Long-Short Term Memory (bi-LSTM) cells. During training we use a mini-batch size of $64$ with dropout probability set to $0.3$. We train our models for $18$ epochs and the learning rate is set to $1$ and start decay after epoch $10$ by $0.5$. It takes about $8$ days to train the generic model on our NVidia GeForce GTX 1080. The models were trained with the open-source toolkit `seq2seq-attn`[^1] [@kim2016sequence].

Experiments
-----------

  Models                     BLEU     TER
  ------------------------- ------- -------
  *generic*                  26.23   62.47
  *generic*$+$*emea-0.5K*    26.48   63.09
  *generic*$+$*emea-5K*      28.99   58.98
  *generic*$+$*emea-50K*     33.76   53.87
  *generic*$+$*emea-full*    41.97   47.07

  : \[tab:resultsFull\] BLEU score of full trained systems.

As a baseline, we fully trained five systems, one with the generic data (*generic*) and the other with generic and various amount of in-domain data: $500$ lines (*emea-0.5K*), $5K$ lines (*emea-5K*) and $50K$ lines (*emea-50K*). The evaluation is done on the in-domain test (*emea-tst*) and presented in the table \[tab:resultsFull\]. Without surprises, the more the model is trained with in-domain data the more BLEU and TER scores are improved. These models are baselines in incremental adaptation experiments.

### Performances among training iterations {#sec:stuEpochs}

![\[fig:incEpohcs\]Curve of “specialization” performances among epochs.[]{data-label="fig:selection"}](emea_curves){width="\linewidth"}

The first study aims to evaluate the approach among additional training iterations (also called “epochs”). Figure \[fig:incEpohcs\] presents the curve of performances when the specialization approach is applied to the *generic* model by using all in-domain data (*emea-full*).

We compare the results with two baselines: on the top of the graphic, we show a line which corresponds to the score obtained by the model trained with both generic and in-domain data (noted *generic*$+$*emea-full*); On the bottom, the line is associated to the generic model score, which is trained with only generic data (noted *generic*). The curve is done with the generic model specialized with five epochs additional epochs on all in-domain data (noted *specialized model with emea*). In the graphic, we can observe that a gap obtained with the first additional epoch with more than $13$ points, but then the BLEU score improves around $0.15$ points with each additional epoch and tend to stall after $10$ epochs (not shown).

So far, the specialization approach does not replace a full retraining, while the specialization curve does not reach the *generic*$+$*emea-full* model. But, the retraining time of one additional epoch with all in-domain data is around $1$ hour and $45$ minutes, while a full retraining would takes more than $8$ days.

In our CAT framework, even $1$ hour and $45$ minutes is too much, the adaptation process need to be performed faster with smaller amount of data like a part of a document ($500$ lines) or a full document ($5K$ lines). Considering the time constraint, the approach tends to be performed though one additional epoch.

### Performances among data size

The second experiment concerns the observation of specialization performances when we vary the amount of data. Using the data presented Table \[tab:alldata\], we apply the specialization process on the generic corpus by taking $0.5K$, $5K$, $50K$ and all the in-domain data (as presented in section \[sec:data\]). According to our previous study (see section \[sec:stuEpochs\]), we focuses on the results obtained with only one additional epoch.

  Training corpus           Specialization corpus    BLEU     TER
  ------------------------- ----------------------- ------- -------
  *generic*                 N/A                      26.23   62.47
  *generic*$+$*emea-0.5K*   N/A                      26.48   63.09
  *generic*$+$*emea-5K*     N/A                      28.99   58.98
  *generic*$+$*emea-50K*    N/A                      33.76   53.87
  *generic*$+$*emea-full*   N/A                      41.97   47.07
  *generic*                 *emea-0.5K*              27.33   60.92
  *generic*                 *emea-5K*                28.41   58.84
  *generic*                 *emea-50K*               34.25   53.47
  *generic*                 *emea-full*              39.44   49.24

  : \[tab:results\] BLEU and TER scores of the specialization approach on the in-domain test set.

  ----------- ------------- --------- -------- -------- ---------------- --
  Process     Corpus          \#lines    \#src    \#tgt          Process 
                                        tokens   tokens             time 
  Train       *generic*          3.4M      73M      86M           8 days 
              *emea-0.5K*         500     5.6K     6.6K         $<$1 min 
  Speciali-   *emea-5K*            5K    56.1K    66.4K   $\approx$1 min 
  zation      *emea-50K*          50K     568K     670K   $\approx$6 min 
              *emea-full*        922K    10.5M    12.3M          105 min 
  ----------- ------------- --------- -------- -------- ---------------- --

  : \[tab:timeresults\] Time spent for each process, the training and the specialization process, according to the amount of data we have.

  Source:                      What benefit has SonoVue shown during the studies ?
  ---------------------------- ----------------------------------------------------------------------------
  Reference:                   Quel est le bénéfice démontré par SonoVue au cours des études ?
  *generic model*              Quel [ avantage SSonVue]{} [a-t-il montré pendant les études]{} ?
  specialization *emea-0.5K*   Quel [ bénéfice]{} [ SSonVue]{} [a-t-il montré lors des études]{} ?
  specialization *emea-5K*     Quel [ bénéfice]{} [ SSonVue a-il]{} [montré pendant les études]{} ?
  specialization *emea-50K*    [ Quels]{} [ est le bénéfice démontré par SonoVue au cours des études]{} ?

We can observe that with only 500 lines, the improvements reaches more than $1$ BLEU points and $2$ TER points. Then, with 10 time more additional data, BLEU and TER scores improved the baseline of $2$ and nearly $4$ points, respectively. With more additional data ($10$ documents), improvements reach $8$ points of BLEU and $9$ points of TER. Finally with all the in-domain data available, the specialization increase the baseline of $13$ points of both BLEU and TER scores.

Comparing the approach with retraining all the generic data, with the same amount of in-domain data, it appears our approach reaches nearly the same results. Moreover, with $50K$ of in-domain data, the specialization approach performs better of $0.5$ of BLEU and TER points. But, when we have much more in-domain data available, the specialization approach does not outperforms the full retraining ($39.44$ against $41.97$ BLEU points).

Discussion
----------

Focussing on the time constraint of the CAT framework, the table \[tab:timeresults\] presents the time taken to process our specialization approach. It goes from less than one minutes to more than $1$ hour and $45$ minutes. If we compare this table with the table \[tab:results\], we observe that this approach enables to gain $1$ BLEU point in less than $1$ minute, $2$ points in $1$ minute and more than $6$ BLEU points in $6$ minutes. The ratio of “time spent” to “score gained” seems impressive. The table \[tab:ex\] shows an example of the outputs obtained with the specialization approach. We compare the generic model compared to the specialized models with respectively $0.5K$, $5K$ and $50K$ lines of in-domain data.

We can clearly see the improvements obtained on the translation outputs. Even if the last one does not stick strictly to the reference, the translation output can be considered as a good translation (syntactically well formed and semantically equivalent).

This specialization approach can be seen as an optimization process (like in classical Phrase-Based approach), which aims to tune the model [@Och2003].

Related work
============

Last years, domain adaptation for machine translation has received lot of attention and studies. These approaches can be processed at three levels: the pre-processing, the training, the post-processing. In a CAT framework, most of the approaches focuses on the pre-processing or on the post-processing to adapt models.

Such pre-processing approaches like data selection introduced by [@Lue2007] and improved by [@Gao2002improving] and many others [@moore2010dataSelection; @Axelrod2011] are effective and their impact studied [@Lambert2011; @Cettolo2014; @Wuebker2014]. But, the main draw back of these approaches is they need a full retrain to be effective.

The post-training family concerns methods which aims to update the model or to optimize the model to a specific domain. Our approach belongs to this category.

This approach is inspired by [@Luong2015], they propose to train a generic model and, then, they further a training over a dozen of epochs on a full in-domain data (the TED corpus). We do believe this approach is under estimated and we propose to study its efficiency in a specific CAT framework with a few data. On one hand, we propose to follow this approach by proposing to use a fully trained generic model. But, on the other hand, we propose to train further only on small specific data over a few additional epochs (from 1 to 5). In this way, our approach is slightly different and can be equated to a tuning process [@Och2003].

Conclusion
==========

In this paper we propose a study of the “specialization” approach. This domain adaptation approach shows good improvements with few in-domain data in a very short time. For instance, to gain 2 BLEU points, we used 5K lines of in-domain data, which takes 1 minute to be performed.

Moreover, this approach reaches the same results as a full retraining, when $10$ documents are available. Within a CAT framework, this approach could be a solution for incremental adaptation of NMT models, and could be performed between two rounds of post-edition. In this way, we propose as future work to evaluate our approach in a real CAT framework.

[^1]: <https://github.com/harvardnlp/seq2seq-attn>
