Edit model card

Qwen2.5-Coder-1.5B-Instruct-GGUF

Original Model

Qwen/Qwen2.5-Coder-1.5B-Instruct

Run with LlamaEdge

  • LlamaEdge version: v0.14.3

  • Prompt template

    • Prompt type: chatml

    • Prompt string

      <|im_start|>system
      {system_message}<|im_end|>
      <|im_start|>user
      {prompt}<|im_end|>
      <|im_start|>assistant
      
  • Context size: 128000

  • Run as LlamaEdge service

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen2.5-Coder-1.5B-Instruct-Q5_K_M.gguf \
      llama-api-server.wasm \
      --model-name Qwen2.5-Coder-1.5B-Instruct \
      --prompt-template chatml \
      --ctx-size 128000
    
  • Run as LlamaEdge command app

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen2.5-Coder-1.5B-Instruct-Q5_K_M.gguf \
      llama-chat.wasm \
      --prompt-template chatml \
      --ctx-size 128000
    

Quantized GGUF Models

Name Quant method Bits Size Use case
Qwen2.5-Coder-1.5B-Instruct-Q2_K.gguf Q2_K 2 676 MB smallest, significant quality loss - not recommended for most purposes
Qwen2.5-Coder-1.5B-Instruct-Q3_K_L.gguf Q3_K_L 3 880 MB small, substantial quality loss
Qwen2.5-Coder-1.5B-Instruct-Q3_K_M.gguf Q3_K_M 3 824 MB very small, high quality loss
Qwen2.5-Coder-1.5B-Instruct-Q3_K_S.gguf Q3_K_S 3 761 MB very small, high quality loss
Qwen2.5-Coder-1.5B-Instruct-Q4_0.gguf Q4_0 4 935 MB legacy; small, very high quality loss - prefer using Q3_K_M
Qwen2.5-Coder-1.5B-Instruct-Q4_K_M.gguf Q4_K_M 4 986 MB medium, balanced quality - recommended
Qwen2.5-Coder-1.5B-Instruct-Q4_K_S.gguf Q4_K_S 4 940 MB small, greater quality loss
Qwen2.5-Coder-1.5B-Instruct-Q5_0.gguf Q5_0 5 1.10 GB legacy; medium, balanced quality - prefer using Q4_K_M
Qwen2.5-Coder-1.5B-Instruct-Q5_K_M.gguf Q5_K_M 5 1.13 GB large, very low quality loss - recommended
Qwen2.5-Coder-1.5B-Instruct-Q5_K_S.gguf Q5_K_S 5 1.10 GB large, low quality loss - recommended
Qwen2.5-Coder-1.5B-Instruct-Q6_K.gguf Q6_K 6 1.27 GB very large, extremely low quality loss
Qwen2.5-Coder-1.5B-Instruct-Q8_0.gguf Q8_0 8 1.36 GB very large, extremely low quality loss - not recommended
Qwen2.5-Coder-1.5B-Instruct-f16.gguf f16 16 3.09 GB

Quantized with llama.cpp b3751

Downloads last month
547
GGUF
Model size
1.54B params
Architecture
qwen2

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for second-state/Qwen2.5-Coder-1.5B-Instruct-GGUF

Base model

Qwen/Qwen2.5-1.5B
Quantized
(32)
this model

Collection including second-state/Qwen2.5-Coder-1.5B-Instruct-GGUF