![](https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/xMBly9PUMphrFVMxLX4kq.png)
models to evaluate
collecting models I want to evaluate on shadereval-task2: https://github.com/bigcode-project/bigcode-evaluation-harness/pull/173 at fp16!!
Text Generation • Updated • 385 • 32Note currently #1 with error rate of 0.353
deepseek-ai/deepseek-coder-1.3b-base
Text Generation • Updated • 18.2k • 60Note # previous #1 error rate 0.38
stabilityai/stable-code-3b
Text Generation • Updated • 2.74k • 623bigcode/starcoder2-7b
Text Generation • Updated • 12.5k • 145bigcode/starcoder2-3b
Text Generation • Updated • 83.2k • • 123
Vipitis/santacoder-finetuned-Shadertoys-fine
Text Generation • Updated • 6 • 1Note has noteable difference between fp16 and fp32, will need to run bf16 likely contaminated
google/gemma-7b
Text Generation • Updated • 119k • • 2.97kgoogle/codegemma-2b
Text Generation • Updated • 1.96k • 61
Vipitis/santacoder-finetuned-Shadertoys
Text Generation • Updated • 5 • 2Note likely contaminated
Deci/DeciCoder-1b
Text Generation • Updated • 7.34k • 245Note current result is not fully correct, need to rerun model - however I don't know a transformers version that runs without errors
google/gemma-2b
Text Generation • Updated • 287k • • 793
Salesforce/codegen2-1B_P
Text Generation • Updated • 2.72k • 36Note needs rerun with incomplete_generation tag
Vipitis/santacoder-finetuned-the-stack-glsl
Text Generation • Updated • 2 • 3microsoft/phi-1_5
Text Generation • Updated • 85.9k • 1.3kmicrosoft/phi-1
Text Generation • Updated • 12.2k • 199
microsoft/phi-2
Text Generation • Updated • 391k • • 3.2kNote performs the worst with error rate of 0.79
Sleeping2🚀ShaderMatch
Note this space holds the evaluation metric that is used. It also has a usually up to date leaderboard. check for updates: https://huggingface.co/spaces/Vipitis/shadermatch/blob/main/result_preview.png
THUDM/codegeex2-6b
Updated • 192 • 250deepseek-ai/deepseek-coder-5.7bmqa-base
Text Generation • Updated • 243 • 7deepseek-ai/deepseek-coder-6.7b-base
Text Generation • Updated • 14.5k • 75codellama/CodeLlama-7b-hf
Text Generation • Updated • 70.7k • • 310bigcode/gpt_bigcode-santacoder
Text Generation • Updated • 26.4k • 25bigcode/starcoderbase
Text Generation • Updated • 1.32k • 387google/codegemma-7b
Text Generation • Updated • 2.84k • 138aiXcoder/aixcoder-7b-base
Text Generation • Updated • 29 • 44Qwen/CodeQwen1.5-7B
Text Generation • Updated • 3.18k • 79ibm-granite/granite-3b-code-base
Text Generation • Updated • 5.39k • 32mistralai/Codestral-22B-v0.1
Text Generation • Updated • 21.6k • 1.01kdeepseek-ai/DeepSeek-Coder-V2-Lite-Base
Text Generation • Updated • 7.97k • 31Salesforce/codet5p-2b
Text2Text Generation • Updated • 2.88k • 33facebook/llm-compiler-7b
Text Generation • Updated • 566 • 103