README.md

Punctuator for Simplified Chinese

The model is fine-tuned based on DistilBertForTokenClassification for adding punctuations to plain text (simplified Chinese). The model is fine-tuned based on distilled model bert-base-chinese.

Usage

from transformers import DistilBertForTokenClassification, DistilBertTokenizerFast

model = DistilBertForTokenClassification.from_pretrained("Qishuai/distilbert_punctuator_zh")
tokenizer = DistilBertTokenizerFast.from_pretrained("Qishuai/distilbert_punctuator_zh")

Model Overview

Training data

Combination of following three dataset:

• News articles of People's Daily 2014. Reference

Model Performance

• Validation with MSRA training dataset. Reference
• Metrics Report:

  	precision	recall	f1-score	support
  C_COMMA	0.67	0.59	0.63	91566
  C_DUNHAO	0.50	0.37	0.42	21013
  C_EXLAMATIONMARK	0.23	0.06	0.09	399
  C_PERIOD	0.84	0.99	0.91	44258
  C_QUESTIONMARK	0.00	1.00	0.00	0
  micro avg	0.71	0.67	0.69	157236
  macro avg	0.45	0.60	0.41	157236
  weighted avg	0.69	0.67	0.68	157236