To check whether synonymy, ungrounded symbols and paraphrases also impact natural language examples, we perform simple modifications of our original process. To control for synonymy, we replace every word by the first lemma of its first synset in WordNet [1]}, if any such lemma can be found. To control for ungrounded symbols, we remove stop-words from our definitions; as this results in definitions that greatly differ from the parser's training examples, we do not compute ted-based mfc scores in this case.
To control for paraphrases, we redo the selection process, and this time randomly sample \(4~123\)  items out of the total \(20~109\)  definitions, without the constraint of having only one definition per definiendum: hence these samples only contain on average \(2~507\)  distinct definienda (standard deviation: \(\pm 14.53\) ). We duly note that different definitions correspond to distinct senses of the definiendum and thus are not strictly speaking paraphrases. However, Arora18LinearWordsenses show that the embedding of a polysemous word corresponds to a weighted sum of underlying meaning embeddings; thus a word embedding ought to be matched to the entire set of definitions for the corresponding token.
In any event, we expect these methods to indicate a consistent trend in mfc scores, despite their crudeness.
