Simple CNN and LSTM networks treat all the words that are present in a sentence equally. But this is not appropriate for most NLP tasks, where some words carry more information than others specific to the task. We need neural models that can automatically learn which words are more important than others. These kinds of networks are called attention networks. An attention network includes one or more attention layers where each layer contains trainable parameters. We learn the weights of these parameters during the training process. An attention layer assigns normalized attention scores to all the words, where some words get higher scores, and some get lower scores. In this way, important words contribute more to the current prediction than the others. This idea leads to better performance on the task. [1]} used such attention networks for neural machine translation. [2]} showed that simple feed-forward neural networks with attention achieve performance similar to LSTM networks.
