
No-pretrain: the scoring model is trained on the human-labeled data directly. No pretraining process is applied;

Real-pretrain: we calculate GOP score of each word in task-related unlabeled data, and use it to pretrain word-level scorer, which is then fine-tuned with the labeled data in training set shown in Table REF . This pretrain and fine-tune strategy achieved the highest PCC for sentence scoring in the recent study [1]}, hence we apply it for word-level scoring and treat it as a strong baseline;

Mixup-pretrain: The pretraining is first preformed on the task-related unlabeled data and the artificial data generated by mixup. The model is then fine-tuned with the labeled data in training set.

