However, on tasks like Question Answering, in all the existing well-performing models, RNNs are employed in a bidirectional way, or a self-attention mechanism is employed [11,<cite> 2,</cite> 4, 8] . This means these models need to processes the whole input sequence to compute the final answer. This is a reasonable approach if the input sequence is as short as a sentence, but it becomes less effective and efficient as the length of the input sequence increases.