To that end, researchers have investigated the linguistic knowledge that these models learn by analyzing BERT (Goldberg, 2018; Lin et al., 2019) directly or training probing classifiers on the contextualized embeddings or attention heads of BERT (Tenney et al., 2019b,a;<cite> Hewitt and Manning, 2019)</cite> .