The extra layers on top of BERT use word-based tokenization instead of sub-words, apply the factored version of self-attention proposed in <cite>Kitaev and Klein (2018)</cite> , and are randomly-initialized instead of being pre-trained.