BERT-base

Pretrained bidirectional encoder for russian language. The model was trained using standard MLM objective on large text corpora including open social data. See Training Details section for more information.

⚠️ This model contains only the encoder part without any pretrained head.

Developed by: deepvk
Model type: BERT
Languages: Mostly russian and small fraction of other languages
License: Apache 2.0

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("deepvk/bert-base-uncased")
model = AutoModel.from_pretrained("deepvk/bert-base-uncased")

text = "Привет, мир!"

inputs = tokenizer(text, return_tensors='pt')
predictions = model(**inputs)

Training Details

The model was trained using the NVIDIA source code. See the pretraining documentation for details.

Training Data

250 GB of filtered texts in total. A mix of the following data: Wikipedia, Books and Social corpus.

Architecture details

Argument	Value
Encoder layers	12
Encoder attention heads	12
Encoder embed dim	768
Encoder ffn embed dim	3,072
Activation function	GeLU
Attention dropout	0.1
Dropout	0.1
Max positions	512
Vocab size	36000
Tokenizer type	BertTokenizer

Evaluation

We evaluated the model on Russian Super Glue dev set. The best result in each task is marked in bold. All models have the same size except the distilled version of DeBERTa.

Model	RCB	PARus	MuSeRC	TERRa	RUSSE	RWSD	DaNetQA	Score
vk-deberta-distill	0.433	0.56	0.625	0.59	0.943	0.569	0.726	0.635
vk-roberta-base	0.46	0.56	0.679	0.769	0.960	0.569	0.658	0.665
vk-deberta-base	0.450	0.61	0.722	0.704	0.948	0.578	0.76	0.682
vk-bert-base	0.467	0.57	0.587	0.704	0.953	0.583	0.737	0.657
sber-bert-base	0.491	0.61	0.663	0.769	0.962	0.574	0.678	0.678

deepvk
/

bert-base-uncased