The embedding layer converts the input sequence of discrete tokens to a sequence of continuous embedding vectors.
As illustrated in Figure REF , for a given token, its embedding vector is constructed by summing the corresponding token, segment, and position embeddings.
The Transformer in its vanilla form [1]} uses relative position embeddings while BERT uses absolute position embeddings, and DeBERTa, a recent variant to BERT, uses disentangled position embeddings [2]}.
<FIGURE><FIGURE>