Post
2046
๐๐๐ฑ๐ด๐ถ๐ป๐ด ๐๐ต๐ฒ ๐๐๐ฑ๐ด๐ฒ๐: ๐๐๐ฎ๐น๐๐ฎ๐๐ถ๐ป๐ด ๐๐น๐ถ๐ด๐ป๐บ๐ฒ๐ป๐ ๐ฎ๐ป๐ฑ ๐ฉ๐๐น๐ป๐ฒ๐ฟ๐ฎ๐ฏ๐ถ๐น๐ถ๐๐ถ๐ฒ๐ ๐ถ๐ป ๐๐๐ ๐-๐ฎ๐-๐๐๐ฑ๐ด๐ฒ๐
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges (2406.12624)
๐๐๐ง ๐๐๐๐ฌ ๐ฌ๐๐ซ๐ฏ๐ ๐๐ฌ ๐ซ๐๐ฅ๐ข๐๐๐ฅ๐ ๐ฃ๐ฎ๐๐ ๐๐ฌ โ๏ธ?
We aim to identify the right metrics for evaluating Judge LLMs and understand their sensitivities to prompt guidelines, engineering, and specificity. With this paper, we want to raise caution โ ๏ธ to blindly using LLMs as human proxy.
Blog - https://huggingface.co/blog/singh96aman/judgingthejudges
Arxiv - https://arxiv.org/abs/2406.12624
Tweet - https://x.com/iamsingh96aman/status/1804148173008703509
@singh96aman @kartik727 @Srinik-1 @sankaranv @dieuwkehupkes
Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges (2406.12624)
๐๐๐ง ๐๐๐๐ฌ ๐ฌ๐๐ซ๐ฏ๐ ๐๐ฌ ๐ซ๐๐ฅ๐ข๐๐๐ฅ๐ ๐ฃ๐ฎ๐๐ ๐๐ฌ โ๏ธ?
We aim to identify the right metrics for evaluating Judge LLMs and understand their sensitivities to prompt guidelines, engineering, and specificity. With this paper, we want to raise caution โ ๏ธ to blindly using LLMs as human proxy.
Blog - https://huggingface.co/blog/singh96aman/judgingthejudges
Arxiv - https://arxiv.org/abs/2406.12624
Tweet - https://x.com/iamsingh96aman/status/1804148173008703509
@singh96aman @kartik727 @Srinik-1 @sankaranv @dieuwkehupkes