This is rather surprising since our models do not use any attention mechanism. A closer look into these models reveal that there seems to be an architectural difference:<cite> Vinyals et al. (2015a)</cite> use 3-layer LSTM with 256 cells and 512-dimensional embeddings; whereas our models use 4-layer LSTM with 1000 cells and 1000-dimensional embeddings.