Spaces:

AtlaAI

/

judge-arena

like 46

Running

What about these models:

• microsoft ( Phi-3-medium-4k-instruct 14B )
• Alibaba ( Qwen 2.5 32B, 14B )
• Upstage ( solar-pro-preview-instruct 22B)
• Mistral ( Mistral-Large-Instruct-2407 123B )

(as reference for which models to choose) other than the some common benchmarks

• open_llm_leaderboard
• eqbench

here's one [benchmark] that is related to judging:

• judgemark

But how are the judging scores extracted?, - by number, words or something else? (see https://arxiv.org/abs/2305.14975)

Gemini models.

https://www.flow-ai.com/judge? I believe it fits the criteria and seems like an interesting smoller competitor based on their pitch in the release blo

https://www.flow-ai.com/judge? I believe it fits the criteria and seems like an interesting smoller competitor based on their pitch in the release blo

hey! great initiative :) Would love to see a small model like Flow-Judge-v0.1 here! Happy to support with the integration if needed.

What about these models:

• microsoft ( Phi-3-medium-4k-instruct 14B )
• Alibaba ( Qwen 2.5 32B, 14B )
• Upstage ( solar-pro-preview-instruct 22B)
• Mistral ( Mistral-Large-Instruct-2407 123B )

(as reference for which models to choose) other than the some common benchmarks

• open_llm_leaderboard
• eqbench

here's one [benchmark] that is related to judging:

• judgemark

But how are the judging scores extracted?, - by number, words or something else? (see https://arxiv.org/abs/2305.14975)

Good shouts! I'm curious to see how those Qwen models would do given that the 2.5 7B is doing pretty well. And those benchmarks are very interesting, evaluating writing quality is a seriously tough task...

The judge score and critique are extracted from a JSON output {"feedback": "", "result": } similar to the Lynx paper

https://www.flow-ai.com/judge? I believe it fits the criteria and seems like an interesting smoller competitor based on their pitch in the release blo

hey! great initiative :) Would love to see a small model like Flow-Judge-v0.1 here! Happy to support with the integration if needed.

👀 will add flow judge in our next update, I'm super excited to see how a dedicated 3.8B model does

Add Command-r and Command-r+, both old and new. They were the least positively biased in my experience.

What a great work! we are looking forward such an arena for Judge models!

How about add compassjudger series (https://github.com/open-compass/CompassJudger),
which reached top performance on
RewardBench(https://huggingface.co/spaces/allenai/reward-bench),
JudgerBench(https://huggingface.co/spaces/opencompass/judgerbench_leaderboard),
JudgeBench(https://huggingface.co/spaces/ScalerLab/JudgeBench) between generative models.
And also can be applied to many subjective evaluation datasets as judge model. For example in ArenaHard: https://github.com/lmarena/arena-hard-auto/issues/49