The resulting vector is concatenated with BERT output of the [CLS] token for final multilabel prediction. We also combine these approaches with previously mentioned BERT intask finetuning [1]}, which proves to be fruitful for this task as it is not too dissimilar to text classification. We additionally train a simple BERT model, which predicts technique class using only [CLS] token representation. All models were trained with linear warmup for 10% of the training steps using the Adam optimizer with learning rate of 2e-5.
