Word embedding is central to neural machine translation ( NMT ) , which has attracted intensive research interest in recent years . In NMT , the source embedding plays the role of the entrance while the target embedding acts as the terminal . These layers occupy most of the model parameters for representation learning . Furthermore , they indirectly interface via a soft-attention mechanism , which makes them comparatively isolated . In this paper , we propose shared-private bilingual word embeddings , which give a closer relationship between the source and target embeddings , and which also reduce the number of model parameters . For similar source and target words , their embeddings tend to share a part of the features and they cooperatively learn these common representation units . Experiments on 5 language pairs belonging to 6 different language families and written in 5 different alphabets demonstrate that the proposed model provides a significant performance boost over the strong baselines with dramatically fewer model parameters . With the introduction of ever more powerful architectures , neural machine translation ( NMT ) has become the most promising machine translation method ( Kalchbrenner and Blunsom , 2013 ; Sutskever et al . , 2014 ; Bahdanau et al . , 2015 ) . For word representation , different architecturesincluding , but not limited to , recurrence-based ( Chen et al . , 2018 ) , convolution-based ( Gehring et al . , 2017 ) and transformation-based ( Vaswani et al . , 2017 ) NMT models-have been taking advantage of the distributed word embeddings to capture the syntactic and semantic properties of words ( Turian et al . , 2010 ) . Figure 1 : Comparison between ( a ) standard word embeddings and ( b ) shared-private word embeddings . In ( a ) , the English word " Long " and the German word " Lange " , which have similar lexical meanings , are represented by two private d-dimension vectors . While in ( b ) , the two word embeddings are made up of two parts , indicating the shared ( lined nodes ) and the private ( unlined nodes ) features . This enables the two words to make use of common representation units , leading to a closer relationship between them . NMT usually utilizes three matrices to represent source embeddings , target input embeddings , and target output embeddings ( also known as pre-softmax weight ) , respectively . These embeddings occupy most of the model parameters , which constrains the improvements of NMT because the recent methods become increasingly memory-hungry ( Vaswani et al . , 2017 ; Chen et al . , 2018 ) . 1 Even though converting words into subword units ( Sennrich et al . , 2016b ) , nearly 55 % of model parameters are used for word representation in the Transformer model ( Vaswani et al . , 2017 ) . To overcome this difficulty , several methods are proposed to reduce the parameters used for word representation of NMT . Press and Wolf ( 2017 ) propose two weight tying ( WT ) methods , called decoder WT and three-way WT , to substantially reduce the parameters of the word embeddings . Decoder WT ties the target input embedding and target output embedding , which has become the new de facto standard of practical NMT ( Sen- Figure 2 : Shared-private bilingual word embeddings perform between the source and target words or sub-words ( a ) with similar lexical meaning , ( b ) with same word form , and ( c ) without any relationship . Different sharing mechanisms are adapted into different relationship categories . This strikes the right balance between capturing monolingual and bilingual characteristics . The closeness of relationship decides the portion of features to be used for sharing . Words with similar lexical meaning tend to share more features , followed by the words with the same word form , and then the unrelated words , as illustrated by the lined nodes . nrich et al . , 2017 ) . Three-way WT uses only one matrix to represent the three word embeddings , where the source and target words that have the same word form tend to share a word vector . This method can also be adapted to sub-word NMT with a shared source-target sub-word vocabulary and it performs well in language pairs with many of the same characters , such as English-German and English-French ( Vaswani et al . , 2017 ) . Unfortunately , this method is not applicable to languages that are written in different alphabets , such as Chinese-English ( Hassan et al . , 2018 ) . Another challenge facing the source and target word embeddings of NMT is the lack of interactions . This degrades the attention performance , leading to some unaligned translations that hurt the translation quality . Hence , Kuang et al . ( 2018 ) propose to bridge the source and target embeddings , which brings better attention to the related source and target words . Their method is applicable to any language pairs , providing a tight interaction between the source and target word pairs . However , their method requires additional components and model parameters . In this work , we aim to enhance the word representations and the interactions between the source and target words , while using even fewer parameters . To this end , we present a languageindependent method , which is called sharedprivate bilingual word embeddings , to share a part of the embeddings of a pair of source and target words that have some common characteristics ( i.e. similar words should have similar vectors ) . Figure 1 illustrates the difference between the standard word embeddings and shared-private word embeddings of NMT . In the proposed method , each source ( or target ) word is represented by a word embedding that consists of the shared features and the private features . The shared features can also be regarded as the prior alignments connecting the source and target words . The private features allow the words to better learn the monolingual characteristics . Meanwhile , the features shared by the source and target embeddings result in a significant reduction of the number of parameters used for word representations . The experimental results on 6 translation datasets of different scales show that our model with fewer parameters yields consistent improvements over the strong Transformer baselines . In this work , we propose a novel sharing technique to improve the learning of word embeddings for NMT . Each word embedding is composed of shared and private features . The shared features act as a prior alignment guidance for the attention model to improve the quality of attention . Meanwhile , the private features enable the words to better capture the monolingual characteristics , result in an improvement of the overall translation quality . According to the degree of relevance between a parallel word pair , the word pairs are categorized into three different groups and the number of shared features is different . Our experimental results show that the proposed method outperforms the strong Transformer baselines while using fewer model parameters .
Word embeddings occupy a large amount of memory, and weight tying does not mitigate this issue for distant language pairs on translation tasks.
They propose a language independet method where a model shares embeddings between source and target only when words have some common characteristics.
Experiments on machine translation datasets involving multiple language families and scripts show that the proposed model outperforms baseline models while using fewer parameters.