Inspired by RoBERTa [1]} and BERT, CamemBERT is a multi-layer bidirectional Transformer as described by Martin et al. in [2]}.
CamemBERT is trained on the Masked Language Modeling (MLM) task which consists of replacing some tokens by either the token \(<\) MASK\(>\)  or a random token and asking the model to correct the tokens. The network uses a cross-entropy loss. The input consists of a mix of whole words and sub-words in order to take advantage of the context.
