The pre-trained LM first converts the input sequence into a sequence of token embeddings independent
of their context then applies 12 or more Transformer layers [1]} on top.
The self-attention mechanism in the Transformer layers convert the word embeddings into a sequence of contextualized embeddings.
These vector representations depend not only on the tokens themselves (e.g., “1797”) but also their context (e.g., “Albany”).
As such, we can extract the representations of the separator tokens (i.e., “<s>”) to be the contextualized column embeddings to capture the meaning of each column that takes the table context into account.
