We performed a sanity check by removing the positional encoding when evaluating a <cite>transformer model</cite> trained with positional encoding, and obtained results which are only around 0.1% worse absolute.