@inproceedings{zmitrovich-etal-2024-family,
    title = "A Family of Pretrained Transformer Language Models for {R}ussian",
    author = "Zmitrovich, Dmitry  and
      Abramov, Aleksandr  and
      Kalmykov, Andrey  and
      Kadulin, Vitaly  and
      Tikhonova, Maria  and
      Taktasheva, Ekaterina  and
      Astafurov, Danil  and
      Baushenko, Mark  and
      Snegirev, Artem  and
      Shavrina, Tatiana  and
      Markov, Sergei S.  and
      Mikhailov, Vladislav  and
      Fenogenova, Alena",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.45",
    pages = "507--524",
    abstract = "Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.",
}

Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.
Zmitrovich, Dmitry Abramov, Aleks r Kalmykov, Andrey Kadulin, Vitaly Tikhonova, Maria Taktasheva, Ekaterina Astafurov, Danil Baushenko, Mark Snegirev, Artem Shavrina, Tatiana Markov, Sergei S. Mikhailov, Vladislav Fenogenova, Alena
A Family of Pretrained Transformer Language Models for Russian