Recently, transformer networks have been shown to perform well for neural machine translation <cite>[11]</cite> and many other NLP tasks [12] . A Transformer layer distinguishes itself from a regular recurrent network by entirely relying on a key-value "self"-attention mechanism for learning relationships between distant concepts, rather than relying on recurrent connections and memory cells to preserve information, as in LSTMs, that can fade over time steps. Transformer layers can be seen as bagof-concept layers because they don't preserve location information in the weighted sum self-attention operation.