@inproceedings{arnardottir-etal-2024-beyond,
    title = "Beyond Error Categories: A Contextual Approach of Evaluating Emerging Spell and Grammar Checkers",
    author = "Arnard{\'o}ttir, {\TH}{\'o}runn  and
      Ing{\'o}lfsd{\'o}ttir, Svanhv{\'\i}t Lilja  and
      S{\'\i}monarson, Haukur Barri  and
      Einarsson, Hafsteinn  and
      Ingason, Anton Karl  and
      {\TH}orsteinsson, Vilhj{\'a}lmur",
    editor = "Melero, Maite  and
      Sakti, Sakriani  and
      Soria, Claudia",
    booktitle = "Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.sigul-1.6",
    pages = "45--52",
    abstract = "Automatic spell and grammar checking can be done using various system architectures, and large language models have recently been used to solve the task with promising results. Here we describe a new method of creating test data to measure the performance of spell and grammar checkers, including large language models. Three types of test data represent different approaches to evaluation, from basic error detection to error correction with natural language explanations of the corrections made and error severity scores, which is the main novelty of this approach. These additions are especially useful when evaluating large language models. We present a spell and grammar checking test set for Icelandic in which the described approach is applied. The data consists of whole texts instead of discrete sentences, which facilitates evaluating context awareness of models. The resulting test set can be used to compare different spell and grammar checkers and is published under permissive licenses.",
}

Automatic spell and grammar checking can be done using various system architectures, and large language models have recently been used to solve the task with promising results. Here we describe a new method of creating test data to measure the performance of spell and grammar checkers, including large language models. Three types of test data represent different approaches to evaluation, from basic error detection to error correction with natural language explanations of the corrections made and error severity scores, which is the main novelty of this approach. These additions are especially useful when evaluating large language models. We present a spell and grammar checking test set for Icelandic in which the described approach is applied. The data consists of whole texts instead of discrete sentences, which facilitates evaluating context awareness of models. The resulting test set can be used to compare different spell and grammar checkers and is published under permissive licenses.
Arnard{\'o}ttir, {\TH}{\'o}runn Ing{\'o}lfsd{\'o}ttir, Svanhv{\'\i}t Lilja S{\'\i}monarson, Haukur Barri Einarsson, Hafsteinn Ingason, Anton Karl {\TH}orsteinsson, Vilhj{\'a}lmur
Beyond Error Categories: A Contextual Approach of Evaluating Emerging Spell and Grammar Checkers