Capitalization prediction layer:
Following the common approached when fine-tuning a pre-trained language model for token classification task [1]}, the punctuation prediction layer is a linear prediction layer which is added on the top of encoder layer.
In particular, we used word vector representation \(e_i\)  that is fed into feed-forward network layer (\(FFN_{cap}\) ) followed by softmax predictor for capitalization prediction.
\(p_i = softmax(FFN_{cap}(e_i)),\) 
