@inproceedings{filevich-etal-2024-language,
    title = "A Language Model Trained on Uruguayan {S}panish News Text",
    author = "Filevich, Juan Pablo  and
      Marco, Gonzalo  and
      Castro, Santiago  and
      Chiruzzo, Luis  and
      Ros{\'a}, Aiala",
    editor = "Gaspari, Federico  and
      Moorkens, Joss  and
      Aldabe, Itziar  and
      Farwell, Aritz  and
      Altuna, Begona  and
      Piperidis, Stelios  and
      Rehm, Georg  and
      Rigau, German",
    booktitle = "Proceedings of the Second International Workshop Towards Digital Language Equality (TDLE): Focusing on Sustainability @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.tdle-1.5",
    pages = "53--60",
    abstract = "This paper presents a language model trained from scratch exclusively on a brand new corpus consisting of about 6 GiB of Uruguayan newspaper text. We trained the model for 30 days on a single Nvidia P100 using the RoBERTa-base architecture but with considerably fewer parameters than other standard RoBERTa models. We evaluated the model on two NLP tasks and found that it outperforms BETO, the widely used Spanish BERT pre-trained model. We also compared our model on the masked-word prediction task with two popular multilingual BERT-based models, Multilingual BERT and XLM-RoBERTa, obtaining outstanding results on sentences from the Uruguayan press domain. Our experiments show that training a language model on a domain-specific corpus can significantly improve performance even when the model is smaller and was trained with significantly less data than more standard pre-trained models.",
}

This paper presents a language model trained from scratch exclusively on a brand new corpus consisting of about 6 GiB of Uruguayan newspaper text. We trained the model for 30 days on a single Nvidia P100 using the RoBERTa-base architecture but with considerably fewer parameters than other standard RoBERTa models. We evaluated the model on two NLP tasks and found that it outperforms BETO, the widely used Spanish BERT pre-trained model. We also compared our model on the masked-word prediction task with two popular multilingual BERT-based models, Multilingual BERT and XLM-RoBERTa, obtaining outstanding results on sentences from the Uruguayan press domain. Our experiments show that training a language model on a domain-specific corpus can significantly improve performance even when the model is smaller and was trained with significantly less data than more standard pre-trained models.
Filevich, Juan Pablo Marco, Gonzalo Castro, Santiago Chiruzzo, Luis Ros{\'a}, Aiala
A Language Model Trained on Uruguayan Spanish News Text