Indeed, in <cite>[2]</cite> , the final vectorial representation of a piece of text is defined by the sum of the embeddings Ω with the position encoding E, which would require d model = d input . However, we introduce a layer that processes embeddings and encodings separately before summing them up.