Transformers have achieved state-of-the-art results on a variety of natural language processing tasks. 
 Despite good performance, Transformers are still weak in long sentence modeling where the global attention map is too dispersed to capture valuable information.
 In such case, the local/token features that are also significant to sequence modeling are omitted to some extent.
 To address this problem, we propose a Multi-scale attention model (MUSE) by concatenating attention networks with convolutional networks and position-wise feed-forward networks to explicitly capture local and token features. Considering the parameter size and computation efficiency, we re-use the feed-forward layer in the original Transformer and adopt a lightweight dynamic convolution as implementation. 
 Experimental results show that the proposed model achieves substantial performance improvements over Transformer, especially on long sentences, and pushes the state-of-the-art from 35.6 to 36.2 on IWSLT 2014  German to English translation task,  from 30.6 to 31.3 on  IWSLT 2015 English to Vietnamese translation task. We also reach the state-of-art performance on  WMT 2014 English to French translation dataset, with a BLEU score of 43.2. In recent years, Transformer has been remarkably adept at sequence learning tasks like machine translation (Vaswani et al., 2017; Dehghani et al., 2018 ), text classification (Devlin et al., 2018; , language modeling (Sukhbaatar et al., 2019b; , etc. It is solely based on an attention mechanism that captures global dependencies between input tokens, dispensing with recurrence and convolutions entirely. The key idea of the self-attention mechanism is updating token representations based on a weighted sum of all input representations. However, recent research (Tang et al., 2018) has shown that the Transformer has surprising shortcomings in long sequence learning, exactly because of its use of self-attention. As shown in Figure  1 (a), in the task of machine translation, the performance of Transformer drops with the increase of the source sentence length, especially for long sequences. The reason is that the attention can be over-concentrated and disperse, as shown in Figure 1 (b), and only a small number of tokens are represented by attention. It may work fine for shorter sequences, but for longer sequences, it causes insufficient representation of information and brings difficulty for the model to comprehend the source information intactly. In recent work, local attention that constrains the attention to focus on only part of the sequences (Child et al., 2019; Sukhbaatar et al., 2019a ) is used to address this problem. However, it costs self-attention the ability to capture long-range dependencies and also does not demonstrate effectiveness in sequence to sequence learning tasks. To build a module with both inductive bias of local and global context modelling in sequence to sequence learning, we hybrid self-attention with convolution and present Parallel multi-scale attention called MUSE. It encodes inputs into hidden representations and then applies self-attention and depth-separable convolution transformations in parallel. The convolution compensates for the in- The left figure shows that the performance drops largely with the increase of sentence length on the De-En dataset. The right figure shows the attention map from the 3-th encoder layer. As we can see, the attention map is too dispersed to capture sufficient information. For example, "[EOS]", contributing little to word alignment, is surprisingly over attended. sufficient use of local information while the self-attention focuses on capturing the dependencies. Moreover, this parallel structure is highly extensible, i.e., new transformations can be easily introduced as new parallel branches, and is also favourable to parallel computation. The main contributions are summarized as follows: • We find that the attention mechanism alone suffers from dispersed weights and is not suitable for long sequence representation learning. The proposed method tries to address this problem and achieves much better performance on generating long sequence. • We propose a parallel multi-scale attention and explore a simple but efficient method to successfully combine convolution with self-attention all in one module. • MUSE outperforms all previous models with same training data and the comparable model size, with state-of-the-art BLEU scores on three main machine translation tasks. • The proposed method enables parallel representation learning. Experiments show that the inference speed can be increased by 31% on GPUs. Although the self-attention mechanism has been prevalent in sequence modeling, we find that attention suffers from dispersed weights especially for long sequences, resulting from the insufficient local information. To address this problem, we present Parallel Multi-scale Attention (MUSE) and MUSE-simple. MUSE-simple introduces the idea of parallel multi-scale attention into sequence to sequence learning. And MUSE fuses self-attention, convolution, and point-wise transformation together to explicitly learn global, local and token level sequence representations. Especially, we find from empirical results that the shared projection plays important part in its success, and is essential for our multiscale learning. Beyond the inspiring new state-of-the-art results on three machine translation datasets, detailed analysis and model variants also verify the effectiveness of MUSE. In future work, we would like to explore the detailed effects of shared projection on contextual representation learning. We are exited about future of parallel multi-scale attention and plan to apply this simple but effective idea to other tasks including image and speech.
This paper propose a new model  which combines multi scale information for sequence to sequence learning.
{"src": "scitldr-AIC_train", "paper_id": "SJe-3REFwr"}