Our models were implemented in PyTorch [1]} using Huggingface's Transformers [2]}. Both single-task learning models and jointly learning model were trained using the AdamW optimizer [3]} and set the batch size to 32. We set the epsilon and weight decay to default value in PyTorch, i.e. 1e-8, the soft capitalization embedding dim is 256 . The learning rate and mixture weight \(\lambda \)  were tuned in \(\lbrace \) 1e-5, 2e-5, 3e-5, 4e-5, 5e-5\(\rbrace \)  and \(\lbrace \)  0.05, 0.1, 0.15 ... 0.95\(\rbrace \)  respectively. The best learning rate and mixture weight value was 5e-5 and 0.15 respectively. We train for 10 epochs and calculate the average score of the \(F_1\) -score for capitalization prediction and punctuation prediction after each training epoch on validation set and selected version that obtained the highest average score on the validation set to apply to the test set.
