Overall, the results of both analysis methods suggest that, although some attention heads of BERT capture specific dependency relation types, they do not reflect the full extent of the significant amount of syntactic knowledge BERT and RoBERTa are known to learn as shown in previous syntactic probing work (Tenney et al., 2019b;<cite> Hewitt and Manning, 2019)</cite> .