@inproceedings{ferreira-etal-2024-towards,
    title = "Towards Automated Evaluation of Knowledge Encoded in Large Language Models",
    author = "Ferreira, Bruno Carlos Lu{\'\i}s  and
      Silva, Catarina  and
      Gon{\c{c}}alo Oliveira, Hugo",
    editor = "S{\'e}rasset, Gilles  and
      Oliveira, Hugo Gon{\c{c}}alo  and
      Oleskeviciene, Giedre Valunaite",
    booktitle = "Proceedings of the Workshop on Deep Learning and Linked Data (DLnLD) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.dlnld-1.7",
    pages = "76--85",
    abstract = "Large Language Models (LLMs) have a significant user base and are gaining increasing interest and impact across various domains. Given their expanding influence, it is crucial to implement appropriate guardrails or controls to ensure ethical and responsible use. In this paper, we propose to automate the evaluation of the knowledge stored in LLMs. This is achieved by generating datasets tailored for this specific purpose, in any selected domain. Our approach consists of four major steps: (i) extraction of relevant entities; (ii) gathering of domain properties; (iii) dataset generation; and (iv) model evaluation. In order to materialize this vision, tools and resources were experimented for entity linking, knowledge acquisition, classification and prompt generation, yielding valuable insights and lessons. The generation of datasets for domain specific model evaluation has successfully proved that the approach can be a future tool for evaluating and moving LLMs {``}black-boxes{''} to human-interpretable knowledge bases.",
}

Large Language Models (LLMs) have a significant user base and are gaining increasing interest and impact across various domains. Given their expanding influence, it is crucial to implement appropriate guardrails or controls to ensure ethical and responsible use. In this paper, we propose to automate the evaluation of the knowledge stored in LLMs. This is achieved by generating datasets tailored for this specific purpose, in any selected domain. Our approach consists of four major steps: (i) extraction of relevant entities; (ii) gathering of domain properties; (iii) dataset generation; and (iv) model evaluation. In order to materialize this vision, tools and resources were experimented for entity linking, knowledge acquisition, classification and prompt generation, yielding valuable insights and lessons. The generation of datasets for domain specific model evaluation has successfully proved that the approach can be a future tool for evaluating and moving LLMs {``}black-boxes{''} to human-interpretable knowledge bases.
Ferreira, Bruno Carlos Lu{\'\i}s Silva, Catarina Gon{\c{c}}alo Oliveira, Hugo
Towards Automated Evaluation of Knowledge Encoded in Large Language Models