@inproceedings{thirukovalluru-etal-2024-sequence,
    title = "Sequence Reducible Holdout Loss for Language Model Pretraining",
    author = "Thirukovalluru, Raghuveer  and
      Monath, Nicholas  and
      Dhingra, Bhuwan  and
      Wiseman, Sam",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1281",
    pages = "14705--14716",
    abstract = "Data selection techniques, which adaptively select datapoints inside the training loop, have demonstrated empirical benefits in reducing the number of gradient steps to train neural models. However, these techniques have so far largely been applied to classification. In this work, we study their applicability to language model pretraining, a highly time-intensive task. We propose a simple modification to an existing data selection technique (reducible hold-out loss training) in order to adapt it to the sequence losses typical in language modeling. We experiment on both autoregressive and masked language modelling, and show that applying data selection to pretraining offers notable benefits including a 4.3{\%} reduction in total number of steps, a 21.5{\%} steps reduction in average, to an intermediate target perplexity, over the course of pretraining an autoregressive language model. Further, data selection trained language models demonstrate significantly better generalization ability on out of domain datasets - 7.9{\%} reduction in total number of steps and 23.2{\%} average steps reduction to an intermediate target perplexity.",
}

Data selection techniques, which adaptively select datapoints inside the training loop, have demonstrated empirical benefits in reducing the number of gradient steps to train neural models. However, these techniques have so far largely been applied to classification. In this work, we study their applicability to language model pretraining, a highly time-intensive task. We propose a simple modification to an existing data selection technique (reducible hold-out loss training) in order to adapt it to the sequence losses typical in language modeling. We experiment on both autoregressive and masked language modelling, and show that applying data selection to pretraining offers notable benefits including a 4.3{\%} reduction in total number of steps, a 21.5{\%} steps reduction in average, to an intermediate target perplexity, over the course of pretraining an autoregressive language model. Further, data selection trained language models demonstrate significantly better generalization ability on out of domain datasets - 7.9{\%} reduction in total number of steps and 23.2{\%} average steps reduction to an intermediate target perplexity.
Thirukovalluru, Raghuveer Monath, Nicholas Dhingra, Bhuwan Wiseman, Sam
Sequence Reducible Holdout Loss for Language Model Pretraining