Word distribution in natural language is characterized by an exponential distribution with a long tail of rare words [1]}.
Uniform sampling of words in SMLMT puts a disproportionately high weight on the long tail, leading to inefficient use of the training corpora since the low frequency words occur in only a small proportion of the sentences.
On the other hand, simple frequency-based sampling can be highly skewed towards a handful of high frequency words.
We thus propose to simply
sample words in proportion to their log-frequency instead.
