@inproceedings{oses-grijalba-etal-2024-question,
    title = "Question Answering over Tabular Data with {D}ata{B}ench: A Large-Scale Empirical Evaluation of {LLM}s",
    author = "Os{\'e}s Grijalba, Jorge  and
      Ure{\~n}a-L{\'o}pez, L. Alfonso  and
      Mart{\'\i}nez C{\'a}mara, Eugenio  and
      Camacho-Collados, Jose",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1179",
    pages = "13471--13488",
    abstract = "Large Language Models (LLMs) are showing emerging abilities, and one of the latest recognized ones deals with their ability to reason and answer questions from tabular data. Although there are some available datasets to assess question answering systems on tabular data, they are not large and diverse enough to properly assess the capabilities of LLMs. To this end, we propose DataBench, a benchmark composed of 65 real-world datasets over several domains, including 20 human-generated questions per dataset, totaling 1300 questions and answers overall. Using this benchmark, we perform a large-scale empirical comparison of several open and closed source models, including both code-generating and in-context learning models. The results highlight the current gap between open-source and closed-source models, with all types of model having room for improvement even in simple boolean questions or involving a single column.",
}

Large Language Models (LLMs) are showing emerging abilities, and one of the latest recognized ones deals with their ability to reason and answer questions from tabular data. Although there are some available datasets to assess question answering systems on tabular data, they are not large and diverse enough to properly assess the capabilities of LLMs. To this end, we propose DataBench, a benchmark composed of 65 real-world datasets over several domains, including 20 human-generated questions per dataset, totaling 1300 questions and answers overall. Using this benchmark, we perform a large-scale empirical comparison of several open and closed source models, including both code-generating and in-context learning models. The results highlight the current gap between open-source and closed-source models, with all types of model having room for improvement even in simple boolean questions or involving a single column.
Os{\'e}s Grijalba, Jorge Ure{\~n}a-L{\'o}pez, L. Alfonso Mart{\'\i}nez C{\'a}mara, Eugenio Camacho-Collados, Jose
Question Answering over Tabular Data with DataBench: A Large-Scale Empirical Evaluation of LLMs