BERT models are trained on subword (wordpiece) tokens. We created a wordpiece vocabulary using bert-vocab-builder toolhttps://github.com/kwonmha/bert-vocab-builder, which is built upon tensor2tensor library [1]}. We did not process the whole corpora in creating the wordpiece vocabulary, but only a smaller subset. To balance the language representation in vocabulary, we used samples from each language. The sizes of corpora subsets are shown in Table REF . The created wordpiece vocabularies contain 74,986 tokens for FinEst and 49,601 tokens for CroSloEngual model.
