BERT.
BERT is a deep bidirectional transformer architecture pre-trained on large quantities of text [1]}, which has recently achieved state-of-the-art results on ad-hoc retrieval [2]}, [3]}. BERT encodes two meta-tokens, namely [SEP] and [CLS], and uses text segment embeddings to simultaneously encode multiple text segments. [SEP] and [CLS] are used for token separation and making judgments about the text pairs, respectively. In the original pre-training task [CLS] is used for determining whether the two sentences are sequential, but it can be fine-tuned for other tasks. In our experiments we adopted BERT for classification, and to this end, a linear combination layer is added on top of the classifier [CLS] token [2]}.
