Traditional methods often solve these tasks independently as two sequence labeling models.
These methods exploit probabilistic graphical models like conditional random fields (CRFs) [1]} [2]} [3]} and combine with common neural networks such as CNNs and LSTM [4]} [5]} [6]}.
For Vietnamese, recently pre-trained language models [7]} have made breakthroughs in various Vietnamese NLP tasks and applications including analyzing questions [8]}, punctuation prediction [9]}, sentiment analysis [10]}, sequence tagging [11]}.
In this paper, we proposed a new joint learning model
that based on pre-trained transformers language models and extend with a capitalization prediction layer to produce the soft capitalization features to combine with contextual features for the punctuation prediction layer.
Our contributions are summarized as follows:
