In this paper, we presented the public dataset for Vietnamese capitalization and punctuation recovery. ViCapPunc dataset can be utilized to develop a text normalization module for ASR systems focus on medical domain which used to record the conversation between doctors and COVID-19 patient. In addition, we also proposed an effective model, namely JointCapPunc, for jointly learning capitalization and punctuation recovery. In particular, JointCapPunc based on transformers based pre-trained language models to learn essential words in the sentences. Soft capitalization embedding combined with word representation can produce useful features for punctuation prediction task. Therefor our model are able to capture the relationship between the two tasks. We conduct experiments and error analysis on our dataset and the results show that jointly learning model is more effective than baseline models (ie., single training approach, joint model Monica et al [1]}). However our model is based on transformers pre-trained language models that has a giant parameters scale, which might face difficulties in ASR pipeline due to the increase of latency.
For future work, we plan to optimize our model to smaller size for faster and lighter footprint.
