How to request performance evaluation with 'llm-perf-backend'?

#27
by lifelongeeek - opened

I am planning to submit LLM models to this leaderboard for testing various compression and quantization frameworks.

I saw the description "Hardware/Backend/Optimization performance requests should be made in the llm-perf-backend repository and will be added to the πŸ€— LLM-Perf Leaderboard πŸ‹οΈ automatically."

Which script(s) do I need to run for performance evaluation requests in 'llm-perf-backend'? With a quick tour of scripts, maybe the benchmark script and push-dataset script seems enough. However, I want to confirm the community's opinion in advance.

Hugging Face Optimum org

Hi @LLMNewbie, we will soon refactor the LLM Perf Leaderboard to benchmark a list of canonical LLMs, instead of scrapping a list of unique architectures from the Open LLM Leaderboard. We can discuss model submissions then but generally a quantized version of a finetuned version of a pretrained model will have the same hardware performance as the the quantized pretrained model.

@IlyasMoutawwakil Thanks for the info. Looking forward to submit & test the models on this ldbd. It's hard to find leaderboards that explore the efficiency of LLM. :(

Sign up or log in to comment