Our model is illustrated in Figure REF .
We base our model on pretrained BERT models.
BERT is a contextual language model with multiple transformer layers and hundreds of millions trainable parameters trained on a massive English corpora with the masked language modeling objective.
Several variations of pretrained weights were released.
We use BERT-base cased and uncased for English as well as Albert [1]}, a smaller version of BERT.
BERT also has a multilingual version, mBERT that supports Hungarian along with 100 other languages.
We use mBERT and the recently released Hungarian-only BERT, huBERT [2]} for Hungarian.
These models all apply wordpiece tokenization with their own predefined WordPiece vocabulary.
They then generate continuous representations for every wordpiece.
Our model adds a two-layer multilayer perceptron on top of these representation with 1568 hidden dimension, ReLU activation and an output layer, and finally a softmax layer that produces a distribution over the labels.
We also apply dropout with a probability of 0.2 before and after the first linear layer. Similarly to [3]}, we apply a sliding window over the input data, generate multiple predictions for each token and then aggregate the probabilities for each position by taking the label-wise mean and thus output the most probable label. The process is illustrated in Figure REF  and REF .
<FIGURE><FIGURE><FIGURE>