Papers
arxiv:2406.12624

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges

Published on Jun 18
ยท Submitted by singh96aman on Jun 24
#2 Paper of the day

Abstract

Offering a promising solution to the scalability challenges associated with human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models (LLMs). However, there are still many open questions about the strengths and weaknesses of this paradigm, and what potential biases it may hold. In this paper, we present a comprehensive study of the performance of various LLMs acting as judges. We leverage TriviaQA as a benchmark for assessing objective knowledge reasoning of LLMs and evaluate them alongside human annotations which we found to have a high inter-annotator agreement. Our study includes 9 judge models and 9 exam taker models -- both base and instruction-tuned. We assess the judge model's alignment across different model sizes, families, and judge prompts. Among other results, our research rediscovers the importance of using Cohen's kappa as a metric of alignment as opposed to simple percent agreement, showing that judges with high percent agreement can still assign vastly different scores. We find that both Llama-3 70B and GPT-4 Turbo have an excellent alignment with humans, but in terms of ranking exam taker models, they are outperformed by both JudgeLM-7B and the lexical judge Contains, which have up to 34 points lower human alignment. Through error analysis and various other studies, including the effects of instruction length and leniency bias, we hope to provide valuable lessons for using LLMs as judges in the future.

Community

Paper author Paper submitter

๐‚๐š๐ง ๐‹๐‹๐Œ๐ฌ ๐ฌ๐ž๐ซ๐ฏ๐ž ๐š๐ฌ ๐ซ๐ž๐ฅ๐ข๐š๐›๐ฅ๐ž ๐ฃ๐ฎ๐๐ ๐ž๐ฌ โš–๏ธ?
We aim to identify the right metrics for evaluating Judge LLMs and understand their sensitivities to prompt guidelines, engineering, and specificity. Key findings -

๐ŸŒŸ ๐—ง๐—ผ๐—ฝ ๐—ฃ๐—ฒ๐—ฟ๐—ณ๐—ผ๐—ฟ๐—บ๐—ฒ๐—ฟ๐˜€: Only ๐—š๐—ฃ๐—ง-๐Ÿฐ and ๐—Ÿ๐—Ÿ๐—ฎ๐—บ๐—ฎ-๐Ÿฏ ๐Ÿณ๐Ÿฌ๐—• shine among 9 judge models. However, they still fall short of inter-human annotator agreement.
๐Ÿ“Š ๐—˜๐˜ƒ๐—ฎ๐—น๐˜‚๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐— ๐—ฒ๐˜๐—ฟ๐—ถ๐—ฐ: Scores assigned by judges with 80%+ percent alignment with humans can be 20 points apart! Cohen's kappa is a superior metric.
โš–๏ธ ๐—ฅ๐—ฎ๐—ป๐—ธ๐—ถ๐—ป๐—ด ๐˜ƒ๐˜€ ๐˜€๐—ฐ๐—ผ๐—ฟ๐—ถ๐—ป๐—ด: Most aligned in scores != most discriminative, in some cases, judge models with low alignment such as Contains (lexical match), and JudgeLM-7B outperform better models in terms of ๐‘Ÿ๐‘Ž๐‘›๐‘˜๐‘–๐‘›๐‘” models, because their biases are more systematic.
๐Ÿงฉ ๐—Ÿ๐—ฒ๐—ป๐—ถ๐—ฒ๐—ป๐—ฐ๐˜†: Judge LLMs tend to be more lenient than strict.
๐ŸŽญ ๐—ฉ๐˜‚๐—น๐—ป๐—ฒ๐—ฟ๐—ฎ๐—ฏ๐—ถ๐—น๐—ถ๐˜๐˜†: Judge LLMs can be easily tricked by controlled responses like "Yes," "Sure," and "I don't know."
๐ŸŽฏ ๐—–๐—ผ๐—ป๐˜๐—ฟ๐—ผ๐—น๐—น๐—ฎ๐—ฏ๐—ถ๐—น๐—ถ๐˜๐˜†: It's not easy to steer large models while smaller models get confused by adding too much detail.

So,Judge LLM more weak๏ผŒthe student LLM more strong ?

ยท
This comment has been hidden

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2406.12624 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2406.12624 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2406.12624 in a Space README.md to link it from this page.

Collections including this paper 11