@inproceedings{akkurt-etal-2024-evaluating,
    title = "Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models",
    author = {Akkurt, Furkan  and
      Gungor, Onur  and
      Mar{\c{s}}an, B{\"u}{\c{s}}ra  and
      Gungor, Tunga  and
      Ozturk Basaran, Balkiz  and
      {\"O}zg{\"u}r, Arzucan  and
      Uskudarli, Susan},
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.577",
    pages = "6504--6514",
    abstract = "Pretrained language models and large language models are increasingly used to assist in a great variety of natural language tasks. In this work, we explore their use in evaluating the quality of alternative corpus annotation schemes. For this purpose, we analyze two alternative annotations of the Turkish BOUN treebank, versions 2.8 and 2.11, in the Universal Dependencies framework using large language models. Using a suitable prompt generated using treebank annotations, large language models are used to recover the surface forms of sentences. Based on the idea that the large language models capture the characteristics of the languages, we expect that the better annotation scheme would yield the sentences with higher success. The experiments conducted on a subset of the treebank show that the new annotation scheme (2.11) results in a successful recovery percentage of about 2 points higher. All the code developed for this work is available at https://github.com/boun-tabi/eval-ud .",
}

Pretrained language models and large language models are increasingly used to assist in a great variety of natural language tasks. In this work, we explore their use in evaluating the quality of alternative corpus annotation schemes. For this purpose, we analyze two alternative annotations of the Turkish BOUN treebank, versions 2.8 and 2.11, in the Universal Dependencies framework using large language models. Using a suitable prompt generated using treebank annotations, large language models are used to recover the surface forms of sentences. Based on the idea that the large language models capture the characteristics of the languages, we expect that the better annotation scheme would yield the sentences with higher success. The experiments conducted on a subset of the treebank show that the new annotation scheme (2.11) results in a successful recovery percentage of about 2 points higher. All the code developed for this work is available at https://github.com/boun-tabi/eval-ud .
Akkurt, Furkan Gungor, Onur Mar{\c{s}}an, B{\"u}{\c{s}}ra Gungor, Tunga Ozturk Basaran, Balkiz {\"O}zg{\"u}r, Arzucan Uskudarli, Susan
Evaluating the Quality of a Corpus Annotation Scheme Using Pretrained Language Models