We used Udify tool [1]} to train both POS tagger and dependency parsing classifiers at the same time. We finetuned each BERT model for 80 epochs on the treebank data. We kept the tool parameters at default values, except for "warmup_steps" and "start_step" values, which we changed to equal the number of training batches in one epoch.
