@inproceedings{mubarak-etal-2024-halwasa,
    title = "Halwasa: Quantify and Analyze Hallucinations in Large Language Models: {A}rabic as a Case Study",
    author = "Mubarak, Hamdy  and
      Al-Khalifa, Hend  and
      Alkhalefah, Khaloud Suliman",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.705",
    pages = "8008--8015",
    abstract = "Large Language Models (LLMs) have shown superb abilities to generate texts that are indistinguishable from human-generated texts in many cases. However, sometimes they generate false, incorrect, or misleading content, which is often described as {``}hallucinations{''}. Quantifying and analyzing hallucination in LLMs can increase their reliability and usage. While hallucination is being actively studied for English and other languages, and different benchmarking datsets have been created, this area is not studied at all for Arabic. In our paper, we create the first Arabic dataset that contains 10K of generated sentences by LLMs and annotate it for factuality and correctness. We provide detailed analysis of the dataset to analyze factual and linguistic errors. We found that 25{\%} of the generated sentences are factually incorrect. We share the dataset with the research community.",
}

Large Language Models (LLMs) have shown superb abilities to generate texts that are indistinguishable from human-generated texts in many cases. However, sometimes they generate false, incorrect, or misleading content, which is often described as {``}hallucinations{''}. Quantifying and analyzing hallucination in LLMs can increase their reliability and usage. While hallucination is being actively studied for English and other languages, and different benchmarking datsets have been created, this area is not studied at all for Arabic. In our paper, we create the first Arabic dataset that contains 10K of generated sentences by LLMs and annotate it for factuality and correctness. We provide detailed analysis of the dataset to analyze factual and linguistic errors. We found that 25{\%} of the generated sentences are factually incorrect. We share the dataset with the research community.
Mubarak, Hamdy Al-Khalifa, Hend Alkhalefah, Khaloud Suliman
Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study