Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU. There has been much recent progress in sequence modeling through recurrent neural networks (RNN; BID54 , convolutional networks (CNN; BID28 BID14 BID7 ) and self-attention models BID40 BID58 . RNNs integrate context information by updating a hidden state at every time-step, CNNs summarize a fixed size context through multiple layers, while as self-attention directly summarizes all context. Attention assigns context elements attention weights which define a weighted sum over context representations BID52 BID8 BID36 . Source-target attention summarizes information from another sequence such as in machine translation while as self-attention operates over the current sequence. Self-attention has been formulated as content-based where attention weights are computed by comparing the current time-step to all elements in the context FIG0 ). The ability to compute comparisons over such unrestricted context sizes are seen as a key characteristic of self-attention BID58 . However, the ability of self-attention to model long-range dependencies has recently come into question BID57 and the unlimited context size is computationally very challenging due to the quadratic complexity in the input length. Furthermore, in practice long sequences require the introduction of hierarchies .In this paper, we introduce lightweight convolutions which are depth-wise separable BID51 BID7 , softmax-normalized and share weights over the channel dimension. The result is a convolution with several orders of magnitude fewer weights than a standard nonseparable convolution. Different to self-attention, lightweight convolutions reuse the same weights for context elements, regardless of the current time-step.Dynamic convolutions build on lightweight convolutions by predicting a different convolution kernel at every time-step. The kernel is a function of the current time-step only as opposed to the entire context as in self-attention ( FIG0 . Dynamic convolutions are similar to locally connected layers in the sense that the weights change at every position, however, the difference is that weights are dynamically generated by the model rather than fixed after training BID30 BID56 BID6 . Our approach also bears similarity to location-based attention which does not access the context to determine attention weights, however, we do not directly take the attention weights from the previous time-step into account BID8 BID36 . BID49 reduce complexity by performing attention within blocks of the input sequence and BID48 BID50 perform more fine-grained attention over each feature. BID47 and BID17 use input-dependent filters for text classification tasks.Our experiments show that lightweight convolutions perform competitively to strong self-attention results and that dynamic convolutions can perform even better. On WMT English-German translation dynamic convolutions achieve a new state of the art of 29.7 BLEU, on WMT English-French they match the best reported result in the literature, and on IWSLT German-English dynamic convolutions outperform self-attention by 0.8 BLEU. Dynamic convolutions achieve 20% faster runtime than a highly-optimized self-attention baseline. For language modeling on the Billion word benchmark dynamic convolutions perform as well as or better than self-attention and on CNN-DailyMail abstractive document summarization we outperform a strong self-attention model. We presented lightweight convolutions which perform competitively to the best reported results in the literature despite their simplicity. They have a very small parameter footprint and the kernel does not change over time-steps. This demonstrates that self-attention is not critical to achieve good accuracy on the language tasks we considered.Dynamic convolutions build on lightweight convolutions by predicting a different kernel at every time-step, similar to the attention weights computed by self-attention. The dynamic weights are a function of the current time-step only rather than the entire context.Our experiments show that lightweight convolutions can outperform a strong self-attention baseline on WMT'17 Chinese-English translation, IWSLT'14 German-English translation and CNNDailyMail summarization. Dynamic convolutions improve further and achieve a new state of the art on the test set of WMT'14 English-German. Both lightweight convolution and dynamic convolution are 20% faster at runtime than self-attention. On Billion word language modeling we achieve comparable results to self-attention.We are excited about the future of dynamic convolutions and plan to apply them to other tasks such as question answering and computer vision where inputs are even larger than the tasks we considered in this paper.
Dynamic lightweight convolutions are competitive to self-attention on language tasks.
{"src": "scitldr-AIC_train", "paper_id": "SkVhlh09tX"}