One key difference between our approach and <cite>Vaswani et al. (2017)</cite> 's is that we only perform input-input attention with self-attention, as we do not have sequences as output but a text classification.