BERT [1]} is a very effective pre-trained model which can be fine tuned for various tasks. BERT learns a masked language model in which a few words of a piece of text are masked and the model learns how to predict them. It is bidirectional and uses attention [2]}, which includes interactions of a word with other words following it or preceding it. By adding task specific loss functions and learning on a comparatively smaller set of data, the model can be fine tuned. For example, the final layer of the model could be a Softmax layer which can do text classification or tagging. Or it could possibly be a linear layer.
