Assessing the syntactic abilities of monolingual neural LMs trained without explicit supervision has been the focus of several recent studies: Linzen et al. (2016) analyzed the performance of LSTM LMs at an English subject-verb agreement task, while <cite>Gulordava et al. (2018)</cite> extended the analysis to various long-range agreement patterns in different languages.