XLM-RoBERTa and RuBERT start to over-fit after 3 epochs, see learningCurveLJ for the LJ corpus, the behaviour for FTD is similar. Since the third epoch, RuBERT continues improving on the training subset, while its accuracy on the test does not change. It looks like RuBERT is slightly more vulnerable to over-fitting beyond 3-4 epochs on the genre classification task. The authors of [1]} claim that for their data and BERT-like models, the optimal number of epochs equals 4 on their text classification tasks. Since [1]} do not use RuBERT or XLM-RoBERTa, the novelty of our study is also in the fact that we show optimality of this number of epochs for other BERT-like models.
