In recent years, transformer-based language models such as BERT [1]}, XLM [2]}, and DistilBERT [3]} have achieved unprecedented results on a wide array of natural language understanding tasks, even when such models are trained on other tasks (i.e. transfer learning). In light of this success, there has been increased interest in investigating what particular information transformer-based language models encode in their word embeddings during pretraining that allows them to perform well in transfer learning experiments. Put into the context of this paper, we may ask: do such models gain certain linguistic/compositional understanding from pretraining? Attempts to assess such phenomena are commonly referred to as probing experiments. In this paper, we introduce a novel probing task using targeted datasets and classification tasks aimed at evaluating models’ compositional reasoning capabilities in relation to numbersFor our purposes, numbers are spelled out, i.e. written out as words such as “ninety”. in multiple natural languages.
