Our approach is informed by previous linguistically-motivated probing studies such at those discussed in [1]} ([1]}) and [3]} ([3]}). Though [3]} ([3]}) discusses important findings on psycholinguistic probing experiments of BERT, we find
[7]} ([7]}) particularly useful for our study due to its clear explanations of linguistic probing experiment setup. In their study, Ettinger et al. present methods for constructing linguistically-targeted datasets (including example sentences, as we use) and classification tasks for probing word embeddings for semantic knowledge. As one of our tasks also seeks to probe for semantic knowledge, we were able to use this setup as a rough guideline. In addition, the authors are careful to create linguistic data with sufficient diversity as not to give potentially helpful cues to their classifier which are not related to the knowledge they wish to probe for. We thus carefully create our task data in a similar manner by limiting the distribution of our data (described more later) and forbidding duplicates.
