This component extracts information from a question of length \(I\) , \(Q=\lbrace q_1,...,q_I\rbrace \) , and a passage of length \(J\) , \(P=\lbrace p_1,...,p_J\rbrace \) , at the word level and normalizes for lexical variants. It typically maps each word to a dense vector using a pre-trained word embedding model, such as word2vec [1]} or GloVe [2]}, such that semantically similar words are mapped to the vectors that are close to each other in neural space.
Word embedding can be enhanced by concatenating each word embedding vector with other linguistic embeddings such as those derived from Part-Of-Speech (POS) tags, and named entities etc.
To tackle the out-of-vocabulary problem, character-level word representations or sub-word representations are used, such as CharCNN [3]},
FastText [4]},
Byte-Pair Encoding (BPE) [5]} and
WordPiece [6]}.
Given \(Q\)  and \(P\) , the word embeddings for the tokens in \(Q\)  is a matrix \(\mathbf {E}^Q \in \mathbb {R}^{d \times I}\)  and
tokens in \(P\)  is \(\mathbf {E}^P \in \mathbb {R}^{d \times J}\) , where \(d\)  is the dimension of the word embedding vectors.
