Pre-finetuning on an intermediate task
Recently, finetuning a pre-trained model on a downstream task has become the norm [1]}, [2]}. However, it cannot be guaranteed that the model finetuned with a small dataset compared to its size will be sufficiently well-adjusted for the target downstream task [3]}. Pre-finetuning is a technique to train the model on a task before the target task [4]}. This can help the model learn the data patterns or reduce the tuning time so that it converges quickly to better fit the target task. According to [5]}, [4]}, the more closely the intermediate task is related to the target task, the better the effect of pre-finetuning. And inference tasks involving the reasoning process show a remarkable improvement in the target task performance. We adopt this method to train a pre-trained language model through two stages for hate speech detection. As the intermediate task, we propose MRP, which guides the model to infer the human rationale of each token based on surrounding tokens.
