I adapt the evaluation protocol and stimuli of Linzen et al. (2016) , <cite>Gulordava et al. (2018)</cite> and Marvin and Linzen (2018) to the bidirectional setting required by BERT, and evaluate the pretrained BERT models (both the LARGE and the BASE models).