@inproceedings{chen-etal-2024-factors,
    title = "What Factors Influence {LLM}s{'} Judgments? A Case Study on Question Answering",
    author = "Chen, Lei  and
      Li, Bobo  and
      Zheng, Li  and
      Wang, Haining  and
      Meng, Zixiang  and
      Shi, Runfeng  and
      Fei, Hao  and
      Zhou, Jun  and
      Li, Fei  and
      Teng, Chong  and
      Ji, Donghong",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1519",
    pages = "17473--17485",
    abstract = "Large Language Models (LLMs) are now being considered as judges of high efficiency to evaluate the quality of answers generated by candidate models. However, their judgments may be influenced by complex scenarios and inherent biases, raising concerns about their reliability. This study aims to bridge this gap by introducing four unexplored factors and examining the performance of LLMs as judges, namely answer quantity, inducing statements, judging strategy, and judging style. Additionally, we introduce a new dimension of question difficulty to provide a more comprehensive understanding of LLMs{'} judgments across varying question intricacies. We employ ChatGPT, GPT-4, Gemini, and Claude-2 as judges and conduct experiments on Vicuna Benchmark and MT-bench. Our study reveals that LLMs{'} judging abilities are susceptible to the influence of these four factors, and analyzing from the newly proposed dimension of question difficulty is highly necessary. We also provide valuable insights into optimizing LLMs{'} performance as judges, enhancing their reliability and adaptability across diverse evaluation scenarios.",
}

Large Language Models (LLMs) are now being considered as judges of high efficiency to evaluate the quality of answers generated by candidate models. However, their judgments may be influenced by complex scenarios and inherent biases, raising concerns about their reliability. This study aims to bridge this gap by introducing four unexplored factors and examining the performance of LLMs as judges, namely answer quantity, inducing statements, judging strategy, and judging style. Additionally, we introduce a new dimension of question difficulty to provide a more comprehensive understanding of LLMs{'} judgments across varying question intricacies. We employ ChatGPT, GPT-4, Gemini, and Claude-2 as judges and conduct experiments on Vicuna Benchmark and MT-bench. Our study reveals that LLMs{'} judging abilities are susceptible to the influence of these four factors, and analyzing from the newly proposed dimension of question difficulty is highly necessary. We also provide valuable insights into optimizing LLMs{'} performance as judges, enhancing their reliability and adaptability across diverse evaluation scenarios.
Chen, Lei Li, Bobo Zheng, Li Wang, Haining Meng, Zixiang Shi, Runfeng Fei, Hao Zhou, Jun Li, Fei Teng, Chong Ji, Donghong
What Factors Influence LLMs' Judgments? A Case Study on Question Answering