Indeed, Tran et al. (2018) finds that transformerbased models perform worse than LSTM models on the <cite>Linzen et al. (2016)</cite> agreement prediction dataset. In contrast, (Tang et al., 2018) find that self-attention performs on par with LSTM for syntax sensitive dependencies in the context of machine-translation, and performance on syntactic tasks is correlated with the number of attention heads in multi-head attention. I adapt the evaluation protocol and stimuli of <cite>Linzen et al. (2016)</cite> , Gulordava et al. (2018) and Marvin and Linzen (2018) to the bidirectional setting required by BERT, and evaluate the pretrained BERT models (both the LARGE and the BASE models).