We initialize the Transformer with CamemBERT [1]} weights for DEFT and BioBERT [2]} for GENIA unless mentioned otherwise, and the remaining parameters using the method of [3]}. Dropout [4]} is applied with a probability of 0.25 everywhere. We optimize the parameters by backpropagation with Adam [5]} without weight decay, over 40 epochs for DEFT and 10 epochs for GENIA. We use two learning steps: one for the Transformer weights, initialized at \(4\times 10^{-5}\) , and one for the rest of the model, initialized at \(9\times 10^{-3}\) . The learning rate follows a linear decay schedule with a warmup for 10% of the steps. We insert the tag embeddings in the Transformers at layer \(L_{tag} = 6\)  for BERT with 12 layers and 19 for BERT with 24 layers. On an Nvidia K80 GPU graphics card, learning on 100 documents takes about 20 minutes.
