Recent pretrained language models extend from millions to billions of parameters. Thus the need to fine-tune an extremely large pretrained model with a limited training corpus arises in various downstream tasks.
In this paper, we propose a straightforward yet effective fine-tuning technique,Child-Tuning, which updates a subset of parameters (called child network)  of large pretrained models via strategically masking out the gradients of the non-child network during thebackwardprocess.
Experiments on various downstream tasks in GLUE benchmark show thatChild-Tuningconsistently outperforms the vanilla fine-tuning byaverage score among four different pretrained models, and surpasses the prior fine-tuning techniques bypoints.
Furthermore, empirical results on domain transfer and task transfer show thatChild-Tuningcan obtain better generalization performance by large margins.