BERT <cite>(Devlin et al., 2019)</cite> improved over previous transformer models and recurrent networks by allowing the system to learn from input text in a bidirectional way, rather than only from left-to-right or the other way around.